Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Jul 17.
Published in final edited form as: Neuron. 2019 May 29;103(2):266–276.e4. doi: 10.1016/j.neuron.2019.04.038

Songbird ventral pallidum sends diverse performance error signals to dopaminergic midbrain

Ruidong Chen 1, Pavel A Puzerey 1, Andrea C Roeser 1, Tori E Riccelli 1, Archana Podury 1, Kamal Maher 1, Alexander Farhang 1, Jesse H Goldberg 1,2,*
PMCID: PMC6639146  NIHMSID: NIHMS1528321  PMID: 31153647

SUMMARY

Motor skills improve with practice, requiring outcomes to be evaluated against ever-changing performance benchmarks. Yet it remains unclear how performance error signals are computed. Here we show that the songbird ventral pallidum (VP) is required for song learning and sends diverse song timing and performance error signals to ventral tegmental area (VTA). Viral tracing revealed inputs to VP from auditory and vocal motor thalamus, auditory and vocal motor cortex, and VTA. Our findings show that VP circuits, commonly associated with hedonic functions, signal performance error during motor sequence learning.

Keywords: reinforcement learning, birdsong, basal ganglia, dopamine, actor-critic, skill learning, ventral tegmental area, performance error, prediction error, vocal learning

eTOC blurb

Songbird dopamine neurons encode error signals important for learning. Chen et al. discover that the ventral pallidum is required for song learning, encodes syllable timing, and sends error and error predection signals to dopaminergic midbrain.

INTRODUCTION

When practicing a piano concerto, you could evaluate your performance relative to a fixed template, such as an auditory memory of Glenn Gould playing Chopin’s Prelude No. 4. Yet reinforcement learning (RL) theory suggests improved learning if you instead learn from prediction errors, in which a note is reinforced only if it sounds better (closer to the template) than predicted based on your past performance (Sutton and Barto, 1998). Reinforcement learning proceeds via incremental improvement in performance quality, requiring performance to be evaluated against benchmarks that change with practice (Schmidt et al., 2018; Thelen, 1995). Neural mechanisms of performance benchmarking and evaluation remain poorly understood.

Songbirds provide a tractable model system to study performance evaluation. First, songbirds have a specialized circuit, the ‘song system’, that includes a projection from VTA dopamine (DA) neurons to the striatopallidal nucleus Area X (Gale et al., 2008; Reiner et al., 2004). Second, zebra finches gradually learn to imitate a sequence of song notes, or syllables, acquired from a single tutor, suggesting they have a ‘fixed template’ they aspire to learn (Marler, 1997). Yet consistent with RL theory, song syllables are not evaluated exclusively against this fixed template but are additionally evaluated against syllable-specific performance benchmarks that change with practice. Specifically, Area X projecting VTA (VTAX) DA neurons recorded during singing are phasically suppressed by distorted auditory feedback (DAF) during singing (Gadagkar et al., 2016). DAF, though not generally aversive (Murdoch et al., 2018), induces a perceived vocal error on distorted renditions such that undistorted renditions are reinforced (Ali et al., 2013; Andalman and Fee, 2009; Hamaguchi et al., 2014; Keller and Hahnloser, 2009; Tumer and Brainard, 2007). Over day timescales, DAF also reduces the predicted quality (i.e. proximity to template) of DAF-targeted syllables. When a reliably distorted target syllable is left undistorted, VTAX neurons exhibit phasic bursts at the precise moment of the song when DAF is predicted to occur but does not occur; and the magnitude of this burst depends on past error probability (Gadagkar et al., 2016). Thus VTAX neurons signal errors in predicted song quality, i.e. the difference between how good (close to the template) a syllable sounded and how good it was predicted to be based on recent practice.

To signal performance prediction error, songbird DA neurons must compute the difference between actual ‘just heard’ error and the error that was predicted at that specific time-step of the song. The roles of upstream projections to VTA in these computations remain unclear. One projection to VTA comes from a high-order auditory cortical area that forms a ‘cup’ around the robust vocal motor nucleus of the arcopallium (RAcup) (Bottjer et al., 2000; Kelley and Nottebohm, 1979; Mello et al., 1998; Vates et al., 1996). RAcup, which is located in the ventral intermediate arcopallium (AIV), is necessary for song learning and sends error signals to VTA (Gale et al., 2008; Mandelblat-Cerf et al., 2014), but it remains unknown how AIV influences VTA firing.

The second major forebrain input to VTA in songbirds comes from a ventral pallidal (VP) region outside the classic song system, ventral and medial to Area X (Gale and Perkel, 2010; Gale et al., 2008). Yet it remains unknown if VP is important for song learning and what, if any, singing-related signals it exhibits.

Here we combine lesion, electrophysiology, and viral tracing studies to demonstrate for the first time that songbird VP is required for learning, exhibits performance error signals, and receives previously unknown inputs from nuclei of the ‘classic’ song system. More generally, our results directly implicate VP in learning a purely motor sequence task like birdsong.

RESULTS

Auditory cortical stimulation causes diverse responses in VTAX neurons

To identify circuits important for performance evaluation, we injected retrograde tracer into the Area X-projecting part of VTA, where performance error encoding DA neurons important for song learning reside (Gadagkar et al., 2016; Hisey et al., 2018; Hoffmann et al., 2016; Xiao et al., 2018). Consistent with past work, retrogradely labeled neurons were observed in AIV (Bottjer et al., 2000; Gale et al., 2008; Kelley and Nottebohm, 1979; Mandelblat-Cerf et al., 2014; Mello et al., 1998; Vates et al., 1996) (Figure S1A–B). Previous studies showed that during singing, DAF causes activation of VTA-projecting AIV neurons and, at a slightly longer latency, pauses in VTAX neurons (Gadagkar et al., 2016; Mandelblat-Cerf et al., 2014).

To test if AIV stimulation affects VTAX firing, we recorded VTA neurons while electrically stimulating AIV in anesthetized birds (n = 7 birds, Figure 1A). AIV stimulation induced phasic rate changes in all wide spiking, antidromically identified VTAX neurons (n=8 antidromic neurons, spike width 0.38±0.03ms, Figure S2K, L) (Schultz and Romo, 1987). Responses included suppression followed by activation (n=3, Figure S2B–D, latency 75+/−19ms), activations (n=3, Figure S2F, H–I, latency 112+/−54ms), and suppressions (n=2, Figure 1F and S2E, J, latency 15–35ms). In putative VTA interneurons with thin spikes (n=13, spike width 0.26±0.02ms, STAR Methods), AIV stimulation caused low latency activations (n=7, latency 29±12ms) or suppressions (n=5, latency 20±7.8ms). When we recorded simultaneously from a VTAX neuron and a thin spiking interneuron at the end of the same electrode, we found that AIV stimulation activated the interneuron, which in turn could suppress the VTAX neuron (Figure 1BF).

Figure 1. Auditory cortical and ventral pallidal stimulation drive diverse changes in VTAX neuron firing. See also Figure S1 and S2.

Figure 1.

Open in a new tab

(A) Schematic of experimental strategy for recording VTAX neuronal response to AIV and VP stimulation. (B to G) Experiment conducted on simultaneously recorded wide-spiking VTAX neuron and thin-spiking VTA interneuron. (B) Antidromic identification (left), and AIV stimulation (right). Arrows, VTAX antidromic spiking (black) and collisions (blue). Asterisks: interneuron spikes. Filled triangles: stimulation artifacts. Scale bars: 0.1mV (vertical), 2ms (horizontal). (C) Left, overlay of raw trace from 10 VTAX spikes (red). Middle, 10 VTA interneuron spikes (black). Right, amplitude and duration of all recorded VTAX (red) and interneuron (black) spikes. (D) Cross-correlogram of spontaneous firing between the two units. Horizontal bars, significant response (p<0.05, z-test) (STAR Methods). (E) Raster plots (top) and rate histograms (bottom) of interneuron (black) and VTAX neuron (red), aligned to AIV stimulation. Horizontal bars indicate significant response (p<0.05, z-test) (STAR Methods). (F) Same as (E) for VP stimulation. (G) Summary of VTAX response types (n=8 neurons with AIV stimulation, n=5 neurons with VP stimulation).

Together, these findings suggest that VTA contains complex local circuitry, including one that implements feedforward inhibition to invert excitatory signals from AIV, consistent with the idea that performance error-induced activations in AIV can drive pauses in VTAX firing during singing. Notably, the AIV-VTA projection resembles cortical projections to VTA in mammals that also can target local GABAergic interneurons and inhibit dopaminergic firing (Beier et al., 2015; Carr and Sesack, 1999; Creed et al., 2014; Moreines et al., 2017; Patton et al., 2013).

Ventral pallidal stimulation causes diverse responses in VTAX neurons

Following tracer injection into the Area X projecting part of VTA, retrogradely labeled neurons were also observed in a ventromedial basal ganglia region termed VP in previous studies (Figure S1C) (Gale and Perkel, 2010; Gale et al., 2008; Reiner et al., 2004). To test if VP can influence VTAX activity, we electrically stimulated VP while recording antidromically identified VTAX neurons in anesthetized birds (n=4 birds, Figure 1A). Response to VP stimulation included suppression followed by activation (n=3/5 neurons tested, Figure 1G, S2NP), suppression (n=1/5, Figure S2Q), and activation (n=1/5, Figure S2R). Responses to VP stimulation had lower latency than AIV (VP: 15+/−2.7 ms, AIV: 22.8+/−7.8 ms, p<0.01, WRS test, Figure S2L). The observation that VP can influence VTAX activity in complex ways is consistent with diverse VTA-projecting cell types in VP (Person et al., 2008), as well as the presence of feedforward inhibition in VTA that can invert incoming signals.

Ventral pallidal lesions impair song learning

To test if VP is important for song learning, we conducted lesion experiments in juvenile birds. To specifically lesion the part of VP that is part of the Area X - VP - VTA - Area X loop previously hypothesized to play a role in learning (Gale and Perkel, 2010), we carried out electrophysiologically-guided excitotoxic lesions of VP in juvenile birds and evaluated their adult songs (STAR Methods). During lesion surgery, stimulation electrodes were implanted into Area X and the boundaries of orthodromic stimulation-evoked responses were mapped with recording electrodes in VP. This mapping specified the locations of excitotoxin injections (STAR Methods). VP lesions significantly impaired song learning (Song imitation score, WRS test, p=0.014, n=6 lesion birds and 7 controls, STAR Methods) (Figure 2).

Figure 2. VP lesions impair song learning.

Figure 2.

Open in a new tab

(A) Lesions were confirmed in neuronal nuclear stained (anti-NeuN) slices as extensive cell death. Scale bars, 200 microns. (B) Tutor song (top), adult song of sham lesioned (middle), and VP lesioned (bottom) siblings. Each spectrogram is 2 seconds long. (C) Same as (B) for another pair. (D) Adult song of vBG lesioned birds had lower similarity to their tutor compared to controls (rank-sum test between all controls and all lesioned birds, p=0.014. n=7 sham lesion, n=6 VP lesion).

Firing patterns of VP neurons in singing birds

To test how VP may guide learning, we recorded VP neurons in singing birds. With some exceptions detailed below, neurons could not easily be categorized into distinct cell classes because they exhibited a continuum of mean firing rates, spike widths, and discharge patterns (firing rate in singing, 43±3.3 Hz, range 0.38–310 Hz; spike width, 0.25±0.008 ms, range 0.05–0.59 ms; CVISI in singing: 1.10±0.03, range: 0.4–2.9; n=162 neurons; Figure S3A–G). This heterogeneity is consistent with the diverse striatal and pallidal cell types intermingled inside songbird VP (Person et al., 2008).

Notably, neuronal discharge heterogeneity is also observed in mammalian VP, leading several studies to categorize neurons on the basis of their responses to primary rewards and cues that predict them (Ahrens et al., 2016; Ito and Doya, 2009; Ottenheimer et al., 2018; Richard et al., 2016; Richard et al., 2018; Smith et al., 2009; Tachibana and Hikosaka, 2012; Tindell et al., 2004). Below we will proceed similarly by categorizing neurons on the basis of their responses to song error and syllable timing.

VP neurons exhibit performance and prediction error signals during singing

To specifically test the role of VP neurons in learning, we used syllable - targeted distorted auditory feedback (DAF) to control perceived error during our recordings (Andalman and Fee, 2009; Tumer and Brainard, 2007) (STAR Methods). Beginning days prior to recordings, a specific ‘target’ song syllable was either distorted with DAF or, on randomly interleaved renditions, left undistorted altogether (distortion rate at target syllable 48.0±1.4%, mean±s.e.m., n=39 birds). Days of pre-training with syllable-targeted DAF reduces the predicted quality of target syllables such that undistorted renditions are signaled as better-than-predicted by VTAX DA neurons (Gadagkar et al., 2016).

To test if VP neurons exhibited online error responses, we compared the activity between randomly interleaved renditions of distorted and undistorted songs. We defined an error response as a significant difference of firing in distorted and undistorted renditions. Surprisingly, although there are no known inputs to VP that carry auditory information in awake singing birds, significant error responses were observed in 31/128 VP neurons tested (Figure 3) (assessed by WRS test on number of spikes in the 125 ms window following DAF onset time (Keller and Hahnloser, 2009), 34/162 neurons were not tested due to low number of trials, STAR Methods). Neurons were either activated (n=16) or suppressed (n=15) by DAF during singing (quantified as z-scored rate difference between undistorted and distorted renditions, Methods). Biphasic responses were observed in 6/31 error responsive neurons: initial suppressions (or activations) were followed by significant activations (or suppressions) (Figure 3D, STAR Methods).

Figure 3. VP neurons exhibit performance and prediction error signals during singing. See also Figure S3, S4, and S5.

Figure 3.

Open in a new tab

(A) Spectrogram (top), spike discharge (middle) and instantaneous firing rate (bottom) of a prediction error encoding VP neuron (DAF, red shading; undistorted targets, blue lines). (B) Expanded view of neuron from (A). Top to bottom: spectrograms, spiking activity during undistorted and distorted trials, corresponding spike raster plots and rate histograms (all aligned to target onset). Horizontal bars in histograms indicate significant different responses to distorted and undistorted renditions, and lack of horizontal bars indicate no significant different response was detected (error response, p < 0.05, WRS test) (STAR Methods). (C and D) Additional examples of error-activated and error-suppressed neurons, same format as (B). (E) Each row plots the z-scored difference between distorted and undistorted target-aligned rate histograms. Error-activated neurons (top, n=16), error-suppressed neurons (middle, n=15), and non-error responsive neurons (bottom, n=97) are independently sorted by maximal z-score. Scale bar for spiking activity in (A-D) is 0.15mV.

A subset of error responsive neurons exhibited a significant maximum or minimum of firing rate following undistorted but not distorted target times (n=10/31 error responsive neurons, p<0.05, bootstrapping, STAR Methods). We term these ‘prediction error’ responses because they cannot be explained by the external DAF sound and occur only following better-than-predicted song outcomes (Gadagkar et al., 2016). Prediction error responses could be rate peaks (n=8 error suppressed neurons, Figure 3B) or nadirs (n=2 error activated neurons, Figure 5D).

Figure 5. VP sends diverse error- and prediction-related signals to VTA. See also Figure S3.

Figure 5.

Open in a new tab

(A) Stimulation and recording electrodes were chronically implanted into VTA and VP, respectively, for antidromic identification of VTA-projecting VP neurons (VPVTA). (B) Antidromic (black) and collision (red) testing of the neuron shown in (C). Scale bars: 1ms (horizontal) and 0.1mV (vertical). (C to K) Song-locked firing patterns of 9 VPVTA neurons, plotted as in Figure 3B, reveal diverse responses including prediction errors (C and D), pre-target burst (E), error-induced activation (F-H), and pre-target pauses (F to J). Y scale bar for spiking activity is 0.15 mV.

Overall, the latencies and durations of error responses were similar to those observed in downstream VTAX DA neurons (latency: 55.0±3.8 ms; duration: 76.4±7.0 ms; n=31 cells, Figure S4A–B) (STAR Methods).

To test if error responses were performance-related (defined as error responses that only occur during singing), and not nonspecific auditory responses, we played back distorted and undistorted renditions of bird’s own song (BOS) to non-singing birds in a subset of recorded neurons. Auditory error responses were rarely observed following passive playback of distorted versus undistorted BOS to awake, non-singing birds (p>0.05 in 33/35 neurons tested, including 14/16 error responsive neurons, WRS test) (Figure S5A–D). To test if error responses were attributable to different movement patterns following distortions, we used custom head-mounted accelerometers to quantify movement during recordings. Movement did not differ between distorted and undistorted renditions (Figure S5E), confirming that error responses were not attributable to body movement (Gadagkar et al., 2016). Together these data demonstrate that VP neurons can exhibit error signals specifically during singing, consistent with performance error.

VP neurons exhibit temporally precise song-locked activity during singing

Many VP neurons also exhibited activity patterns temporally aligned to song. To quantify the precision of song locked discharge, we calculated the pairwise inter-motif correlation coefficients (IMCC) of instantaneous firing rates for all neurons recorded for 10 or more motifs (Goldberg et al., 2010; Goldberg and Fee, 2010; Kao et al., 2008; Olveczky et al., 2005) (STAR Methods). A neuron that reliably discharges at the same time-steps across song renditions will have an IMCC=1; a neuron with random discharge unrelated to singing will have an IMCC=0. Most VP neurons exhibited significant song-locked discharge with a precision that varied among the population (p<0.05 in 96/162 neurons tested; IMCC=0.13±0.016, Figure 4A). Most neurons did not show time-locked responses to BOS playback (p>0.05 in 33/35 neurons tested, including 21/23 song-locked neurons, STAR Methods).

Figure 4. VP neurons exhibit temporally precise song-locked activity during singing. See also Figure S3.

Figure 4.

Open in a new tab

(A) Top: sparseness of song-locked VP neurons plotted against the intermotif correlation coefficient (IMCC). Dashed lines: IMCC=0.3 (vertical) and sparseness=0.5 (horizontal). Bottom: Maximum error score (absolute z-scored difference between distorted and undistorted trials) plotted against IMCC. Dashed lines: IMCC=0.3 and error score=1.96. (B to F) Top to bottom: spectrograms, spiking activity during undistorted and distorted trials, corresponding spike raster plots and rate histograms for a VP neuron with sparse, temporally precise discharge (B), one with time-locked bursts that tile the song (C), one with time-locked bursts and error response (D), one with significant rate maximum immediately before target time (E), and one with significant rate minimum at target time (F). Horizontal bars on top of rate histograms indicate significant difference between distorted and undistorted firing (p<0.05, WRS test). (G) A scatter plot of the mean firing rate during singing plotted against the ratio between mean firing rates during non-singing and singing periods identified VP neurons gated by singing state. (H-I) Top to bottom: spectrograms, spiking activity, and corresponding instantaneous firing rate (IFR) for a song-on neuron which fired at high rate during singing, but silent outside song (H) and a song-off neuron which abruptly stopped firing during singing (I). Scale bar for spiking activity in (B-F, H, I) is 0.15 mV.

Three cell ‘types’ were distinguished by extremely precise song-locked firing (IMCC>0.3) (Figure 4, n=10). A first type exhibited ultra-sparse discharge aligned to specific song syllables (n=2, sparseness index>0.5, STAR Methods) (Figure 4A–B). These sparse neurons’ discharge resembled striatal medium spiny neurons (MSNs) previously recorded in Area X (Goldberg and Fee, 2010; Woolley et al., 2014) and support the previous finding that striatal and pallidal cell types can be intermingled in songbird VP (Person et al., 2008). A second type exhibited stereotyped, rhythmic firing patterns visible as high frequency bursts aligned to specific song time-steps with millisecond precision (Figure 4A and 4C, n=6). In contrast to the first two types which did not exhibit error responses, a third type exhibited error responses as well as time-locked response to BOS playback, consistent with a neuron that receives strong auditory input (n=2, IMCC>0.3 during BOS playback; Figure 4D, S5C–D).

Most song-locked neurons also exhibited significant rate modulations at various time-steps of the song (n=93/96 with significant rate maximum or minimum, STAR Methods) (Figure 4E–F). Finally, yet other neurons were distinguished by dramatic increase or decrease in firing rate at the transition between non-singing and singing states (Figure 4G–I, n=12, rate difference >85%). All of these diverse cell types were spatially intermingled (Figure S3H).

VP sends diverse error- and prediction-related signals to VTA

To test which VP signals are sent to VTA, we used antidromic and collision testing methods to identify VTA-projecting VP (VPVTA, n=10, Figure 5A–B) and putative non VTA-projecting (VPOther, n=92) neurons (n=60 not tested). Like the VP population more generally, VPVTA neurons exhibited a range of firing rates and discharge patterns (Figure S3). However, VPVTA neurons were significantly more likely to exhibit error responses compared to non-projectors (n=5/10 VPVTA neurons, 16/92 VPOther neurons. p<0.05, WRS test, STAR Methods).

VPVTA neurons were significantly more likely to exhibit a minimum in firing rate immediately prior to the target time in the song (n=5/10 VPVTA neurons, 12/92 VPOther neurons, p<0.01, WRS test, STAR Methods). These pre-target pauses are consistent with a predicted quality signal. One VPVTA neuron exhibited a robust pre-target burst, but such pre-target activations were not enriched in the VPVTA neurons relative to VPOther neurons (p>0.05, WRS test, STAR Methods). Together these findings show that VP sends diverse error- and prediction-related signals to VTA.

Viral tracing identifies inputs to VP from dopaminergic, vocal motor, and auditory regions

To test what inputs to VP could account for this diversity of singing and error-related firing, we combined retrograde and anterograde viral tracing strategies. Abundant fiber tracts course through VP, complicating the interpretation of results obtained with dextran and cholera toxin (CTB) tracers, which can be taken up by fibers of passage. We thus used a sparse GFP-expressing retrograde virus, self-complementary AAV9 (scAAV9-CBh-GFP) that is taken up by axon terminals (Xiao et al., 2018) (STAR Methods). Following injection of viral tracer into VP, retrogradely labeled neurons were observed in several singing-related cortical, thalamic and midbrain structures including: (1) RA, a vocal motor cortex-like nucleus known to send precise motor command signals to brainstem motor neurons (Leonardo and Fee, 2005; Sober et al., 2008; Yu and Margoliash, 1996) (Figure S6A–B, retrograde labelling observed in 4/11 hemispheres); (2) Uva, a motor thalamic nucleus known to send precise song timing information to HVC (5/19 hemispheres, Figure S6E–G) (Danish et al., 2017; Hamaguchi et al., 2016); (3) DLM, the Area X-recipient thalamic nucleus known to send premotor signals to cortical nucleus LMAN (3/12 hemispheres, Figure S6M–O) (Goldberg and Fee, 2012); (4) AIV, an auditory cortical area known to send error signals to VTA (4/11 hemispheres, Figure S6I–K) (Mandelblat-Cerf et al., 2014); (5) Ovoidalis, the primary auditory thalamus (2/11 hemispheres, Figure S6M,N,P) (Lei and Mooney, 2010; Vates et al., 1996); and (6) VTA (9/21 hemispheres, Figure S6Q–S).

We note that using this retrograde viral tracing strategy we only observed sparsely labeled cells in all identified input regions to VP (Figure S9). This may indicate sparse connections between input regions and VP, but could also be due to sparse uptake by axon terminals and/or sparse viral expression (STAR Methods). Indeed, in all VP injected birds we observed sparse input from the main known input to VP, Area X (Gale et al., 2008) (not shown). We additionally used dual tracer strategies and further confirmed that anterogradely labeled RA axons co-mingled with VTA-projecting VP neurons (n=1/2 hemispheres, Figure S6C–D); retrograde tracers in VP and VTA could co-label AIV neurons (3/5 cells co-labeled in 3/3 hemispheres, Figure S6I–L); retrograde tracers in VP and Area X can co-label VTA neurons (n=4/6 cells co-labeled in 2/2 hemispheres, Figure S6Q–T); and that retrograde tracers in VP and HVC can co-label Uva neurons (3/9 cells co-labeled in 2/3 hemispheres, Figure S6E–H). Together, these data show that VP receives inputs from RA, DLM, Ovoidalis, HVC-projecting Uva neurons, Area X-projecting VTA neurons, and VTA-projecting AIV neurons.

DISCUSSION

By combining lesions, viral tracing, and electrophysiology we discovered that songbird ventral pallidum is required for song learning, receives information about song syllable timing and error, and sends diverse performance prediction and error related signals to VTA. These findings demonstrate that ventral pallidal circuits can play an essential role in performance evaluation during a purely motor sequence learning task like birdsong.

Despite the importance of error signals for motor sequence learning, the identification of online error signals remained elusive in singing birds (Achiro et al., 2017; Derégnaucourt et al., 2004; Ganguli and Hahnloser, 2011). HVC and LMAN do not exhibit responses to DAF during singing (Hamaguchi et al., 2014; Kozhevnikov and Fee, 2007; Leonardo, 2004; Vallentin and Long, 2015), ruling against hypothesized roles of the classic song system in online evaluation of auditory feedback (Doupe and Konishi, 1991; Nottebohm et al., 1990; Troyer and Doupe, 2000). Recent studies instead support the idea that auditory cortical areas extract error signals and send them to VTA, which in turn sends dopaminergic prediction error signals to Area X (Fee and Goldberg, 2011; Woolley, 2019). Specifically, early stages of the auditory cortical hierarchy exhibit singing-related auditory responses that include DAF-driven modulations (Keller and Hahnloser, 2009), and AIV, a higher order auditory cortical region, contains VTA-projecting neurons that are not simply activated by singing and instead are specifically activated by DAF-induced errors (Mandelblat-Cerf et al., 2014). VTA then sends performance error signals to Area X that can modify future performance (Gadagkar et al., 2016; Hisey et al., 2018; Xiao et al., 2018). Our finding that both AIV and VP can modulate VTAX firing supports the idea that song evaluation signals can reach the classic song system through the VTA projection to Area X (Gale and Perkel, 2010).

VP is classically viewed as an output of the limbic system involved in seeking primary reinforcers such as food, drugs, or courtship (McAlonan et al., 1993; Mogenson et al., 1980; Smith et al., 2009). VP lesions in mammals cause anhedonia, reduce drug- and food-seeking, and impair reward-based place preference, implicating VP with both motivational ‘wanting’ and hedonic ‘liking’. Consistent with this idea, VP neuronal activity is modulated by reward omissions, rewards, the cues that predict them, and their hedonic value (Ahrens et al., 2016; Ito and Doya, 2009; Ottenheimer et al., 2018; Richard et al., 2016; Richard et al., 2018; Tachibana and Hikosaka, 2012; Tindell et al., 2004).

Computational models of basal ganglia (BG) dependent reinforcement learning may provide insight into how VP’s established hedonic functions relate to performance evaluation in singing birds. In classic actor-critic (AC) models the BG has two functional subdivisions: a ventral ‘critic’ with outputs to VTA and a dorsal ‘actor’ with outputs to the motor system (Figure 6A) (Daw et al., 2006; Joel et al., 2002; Takahashi et al., 2008). Both subdivisions receive dopaminergic reward prediction error signals and implement DA-modulated plasticity to weigh cortical inputs (which encode ‘state’, Figure 6A) according to their reward value. DA modulated plasticity in the critic therefore computes predicted state value, for example a cue-dependent reward prediction. Predicted state-value coding by VP, manifest as VP responses to conditioned stimuli in Pavlovian tasks (Ahrens et al., 2016; Ito and Doya, 2009; Richard et al., 2018; Tindell et al., 2004), can provide VTA with ‘prediction’ information necessary to compute reward prediction error (Tian et al., 2016). DA prediction error signals project back to the critic (to update predicted state-value) and also to an ‘actor.’ The actor also implements DA-modulated plasticity on ‘state’-encoding inputs. But because the actor has topographically organized outputs to the motor system, this plasticity links a state representation to a reward-maximizing action (Daw et al., 2006; Joel et al., 2002; Takahashi et al., 2008), manifest as action-value coding in dorsal BG structures (Samejima and Doya, 2007).

Figure 6. An actor-critic circuit motif for computing performance prediction error. See also Figure S6.

Figure 6.

Open in a new tab

(A) Actor-critic circuit motif in mammalian BG adapted from (Daw et al., 2006; Joel et al., 2002; Takahashi et al., 2008). (B) Anatomy and signaling in songbird VP reveals a similar motif.

Our findings suggest that songbird VP may implement some functions analogous to the critic in the classic AC architecture, including the computation of predicted state value (which in birdsong is analogous to predicted syllable quality). Specifically, thalamic (Uva) or cortical (RA) inputs to VP could provide state representations in the form of ‘time-step’ in song that could explain the observed VP timing responses (Figure 4B–C) (Danish et al., 2017; Hamaguchi et al., 2016; Leonardo and Fee, 2005; Sober et al., 2008; Yu and Margoliash, 1996). Next, DA inputs to VP from VTAX neurons could enable DA-modulated plasticity to weigh Uva inputs according to past error. For example, consider a three syllable song a-b-c. If syllable b is reliably distorted, then DA pauses (driven by DAF) would be coincident with those Uva inputs active at syllable b. Then DA-modulated plasticity of Uva inputs would re-weigh these synapses, resulting in a representation in VP of error-weighted timing or, equivalently, predicted error (Figure 6B). With an eligibility trace (Sutton and Barto, 1998; Yagishita et al., 2014), this process would explain the pre-target pauses observed in VP that are enriched in VPVTA neurons (Figure 5, F–K). Finally, AIV neurons are DAF-responsive during singing and could explain error responses in VP (Figure 3) (Mandelblat-Cerf et al., 2014). Altogether, VP thus contains information necessary to signal the difference between predicted and actual error, manifest in the observed prediction error responses (Figure 3, A and B) that can be sent to VTA (Figure 5C and D). A key prediction of this model is the existence of DA-modulated plasticity of Uva or RA inputs to VP.

In biologically inspired AC models, the DA error signal reaches both ventral and dorsal BG domains (Daw et al., 2006; Joel et al., 2002; Takahashi et al., 2008). Notably, VP-projecting VTA neurons also project to Area X, which is located more dorsolaterally and which has topographically organized outputs to the song motor system (Johnson et al., 1995; Luo et al., 2001). DA modulated plasticity in Area X (Ding and Perkel, 2004) reinforces the way a target syllable is produced (Hisey et al., 2018; Hoffmann et al., 2016; Xiao et al., 2018), much like manipulation of striatal DA in mammals can reinforce place preference or action selection (Tsai et al., 2009; Wise and Schwartz, 1981). Thus we propose that Area X has anatomical and functional similarities to the ‘actor’ in classic AC architecture (Charlesworth et al., 2012; Fee and Goldberg, 2011) (Figure 6B), forming the counterpart to the ventral critic.

The AC model helps unify mechanisms of reward seeking and performance evaluation into a common framework, but it fails to explain the diversity of VP signals and likely oversimplifies processes underlying the construction of the dopaminergic error signal. First, VP stimulation could cause either activation or suppression of VTAX firing (Figure 1E–F and S2N–R), which could be due to a mixture of glutamatergic and GABAergic VPVTA neurons and/or differential engagement with local inhibition in VTA that can invert afferent signals (Yang et al., 2018). Second, VPVTA neurons could be activated by distorted renditions, could be activated or suppressed by undistorted renditions, and could exhibit pre-target bursts or pauses that may encode predicted song error or predicted quality, respectively (Figure 5). Thus VP sends virtually every conceivable error-related signal to VTA. While it is possible to linearly combine these VP signals in specific ways to construct the known VTAX signal, it remains unclear how these mixed inputs are transformed by the VTA microcircuit into a remarkably homogeneous dopaminergic prediction error signal in the VTAX pathway. Notably, mixed responses to reward and reward-predicting cues are observed in VP (and other) inputs to mammalian VTA. even though VTA output is a similarly homogenous reward prediction error signal (Hong and Hikosaka, 2014; Tian et al., 2016). How relatively homogenous DA error signals are constructed from mixed inputs is similarly elusive in birds and mammals, and may involve complex, cell-type specific engagement with VTA microcircuitry (Yang et al., 2018). Future recordings of VTAX neurons in lesioned animals could clarify how DA pauses and bursts depend on specifically on inputs from AIV and VP (Takahashi et al., 2016; Takahashi et al., 2011; Tian and Uchida, 2015).

In summary, we report that the ventral pallidum, a limbic structure associated with reward seeking in mammals, is necessary for birdsong learning, receives information from VTA and auditory and vocal motor areas, and sends diverse performance and prediction error signals to VTA.

STAR★Methods

CONTACT FOR REAGENTS AND RESOURCE SHARING

Further information and requests for resources and reagents should be directed to and will be fulfilled by the Lead Contact, Jesse H. Goldberg (jessehgoldberg@gmail.com)

EXPERIMENTAL MODEL AND SUBJECT DETAILS Subjects

Subjects were 91 male zebra finches (at least 39 days post hatch, dph). Animal care and experiments were carried out in accordance with NIH guidelines and were approved by the Cornell Institutional Animal Care and Use Committee.

METHOD DETAILS

Surgery and histology

All surgeries were performed with isoflurane anesthetization. For functional mapping experiments (8 birds, 90 dph or older, Figure. 1), bipolar stimulation electrodes were implanted into AIV and Area X (Gadagkar et al., 2016; Mandelblat-Cerf et al., 2014). In 4/8 birds, additional stimulation electrodes were implanted into VP. AIV coordinate was determined by its anterior and ventral position to RA, and Area X coordinate was +5.6A, +1.5L relative to lambda and 2.65 ventral relative to pial surface, at a head angle of 20 degrees. VP coordinate was +4.9A, +1.3L relative to lambda and 3.9 ventral to pial surface at a head angle of 20 degrees. Recordings were made in VTA using a carbon fiber electrode (1 MOhm, Kation Scientific). VTA was identified by anatomical landmarks. Specifically, the boundaries of DLM and Ovoidalis were determined by spontaneous firing and auditory responses. Recordings were then made at the same AP position, +0.6L relative to lambda and 6.5 ventral relative to pial surface, at a head angle of 55 degrees. This AP position corresponds to the anterior part of VTA enriched in Area X projecting neurons (VTAX) (Gadagkar et al., 2016; Mandelblat-Cerf et al., 2014). VTAX neurons were further confirmed by antidromic response and collision testing. At the end of the experiment, small electrolytic lesions (30 uA for 60s) were made at each stimulation site. Location of the stimulating electrodes was verified histologically.

For VP lesion (13 birds, 39–52 dph), a bipolar stimulation electrode was implanted into Area X and the boundaries of VP was electrophysiologically mapped by finding units suppressed by Area X stimulation. 115nl of 2% N-methyl-DLaspartic acid (NMA; Sigma, St Louis, MO) or saline (for control birds) was injected into VP bilaterally. Lesions were histologically confirmed by labeling neurons with anti-NeuN (full bilateral VP lesions in 6 birds).

For awake-behaving electrophysiology (39 birds, 87–355 dph), custom made microdrives carrying an accelerometer (Analog Devices AD22301), linear actuator (Faulhaber 0206 series micromotor) and homemade electrode arrays (5 electrodes, 3–5 MOhms, microprobes.com) were implanted into VP by coordinates (4.4–5.4A, 1.1–1.5L, 3.5V, head angle 20 degrees). In 20/39 birds, a bipolar stimulation electrode was implanted into VTA using anatomical landmarks as described above. After each experiment, small electrolytic lesions (30 ȝA for 60 s) were made with one of the recording electrodes. Brains were then fixed, cut into 100 ȝm thick sagittal sections for histological confirmation of stimulation electrode tracks and reference lesions.

For VP tracing experiments (31 birds, 90 dph or older), 40nl of self-complementary adeno-associated virus carrying green fluorescent protein (scAAV9-CBh-GFP, UNC vector core) was injected into VP in two coordinates (4.6/4.9A, 1.3L, 4V). Upstream neurons retrogradely infected and expressing GFP could be observed in RA, AIV, Uva, Ov, DLM, and VTA (each input was checked in a subset of birds as indicated in the main text). To determine if VP shares common inputs with HVC, in addition to scAAV9 in VP, fluorescently labeled cholera toxin subunit B (CTB, Molecular Probes) was injected into HVC in 3 birds. To determine if VP share common inputs with Area X, CTB was injected into Area X for 2 birds. To test if RA axons co-mingle with VTA projecting VP neurons, CTB was injected into VTA and anterograde HSV-mCherry (MGH viral core) was injected into RA.

Recording VTA responses to stimulation of AIV/VP

Neurons were classified as Area X-projecting (VTAX) based on antidromic stimulation and collision testing (200 μs pulses, 100–300 μA). Spike duration was determined by the interval between onset and offset time of spikes (Figure 1C, S2K). VTA neurons that did not respond to Area X stimulation were classified as putative interneurons. We cannot rule out the possibility that a subset of these neurons project to the basal ganglia outside the field of influence of stimulation. A burst of AIV (or VP) stimulation consisting three 200μs pulses with 10ms inter-pulse-interval was delivered every 1.5–2 s, with 300μA current amplitude. Putative interneurons were also tested for response to AIV stimulation. VTAX neurons were found in an anterior part of VTA, intermingled with non-projecting local neurons (Gadagkar et al., 2016; Mandelblat-Cerf et al., 2014).

All VTAX neurons and those putative interneurons with rate influenced by AIV stimulation were further analyzed. To determine if VTA neurons respond to AIV (VP) stimulation, spiking activity within ±1 second relative to stimulation onset was binned in a moving window of 30ms (for VTAX neurons) or 10ms (for VTA interneurons) with a step size of 5ms. Each bin after stimulation onset was tested against all the bins in the previous 1 second (the prior) using a z-test. Windows where at least 2 consecutive bins with p<0.05 were considered significant. The response onset and offset were required to bracket lowest (for phasic decrease) or highest (for phasic increase) firing rate after stimulation onset. Response was quantified by normalized firing rate in the first significant window using the 1 sec before stimulation onset as baseline (Figure S2L).

To determine if the simultaneously recorded putative interneuron (PIN) and VTAX neuron were correlated, we constructed rate histogram of VTAX neuron spiking events aligned to spontaneous spiking events of PIN with preceding inter-spike interval (ISI) > 10ms, and assessed significance of rate changes of VTAX neuron using z-test (Figure 1D).

Song imitation score

Song learning in VP lesioned and control birds was assessed by song similarity between pupil (at 90 dph) and their tutors. We computed imitation scores using an automated procedure based on Sound Analysis Pro (SAP) algorithm (Mandelblat-Cerf and Fee, 2014; Tchernichovski et al., 2000). Briefly, the tutor motif was segmented into syllables by hand. Syllables in the pupil song were determined by finding the section of pupil song with highest SAP similarity to each tutor syllable. Additionally, a sequencing score was computed as the similarity of the next syllable in tutor song and the next section in the pupil song. Imitation score was the product of song similarity and sequence similarity (Mandelblat-Cerf et al., 2014).

Syllable-targeted distorted auditory feedback

Postoperative birds with microdrive implant were placed in a sound isolation chamber and given at least a day to habituate to distorted auditory feedback (DAF) as described previously (Gadagkar et al., 2016). Briefly, ongoing singing was analyzed by Labview software to target specific syllables, and two speakers inside the chamber played a 50ms DAF sound on top of bird’s singing on 50% of randomly selected target renditions. DAF was either a broadband sound band passed at 1.5–8 kHz, the same spectral range of zebra finch song, or a segment of one of the bird’s own non-target syllables displaced in time.

Passive playback of the bird’s own song

For passive playback of the bird’s own song (BOS), we played back randomly interleaved renditions of the undistorted and distorted motifs of the bird’s own song during awake, non-singing periods. The loudness of playback was adjusted to match the average peak loudness of zebra finch song (Gadagkar et al., 2016; Mandelblat-Cerf et al., 2014).

Analysis of neural activity

Neural signals were band-passed filtered (0.25–15 kHz) in homemade analog circuits and acquired at 40 kHz using custom Matlab software. Single units were identified as VTA-projecting by antidromic identification and antidromic collision testing (Figure 5A–B). Spike sorting was performed offline using custom Matlab software. Instantaneous firing rates (IFR) were defined at each time point as the inverse of the enclosing ISI. Firing rate histograms were constructed with 10 ms bins and smoothed with a 3-bin moving average, except for Figure 4C and 4D, where the histograms had 4ms and 5ms bins. All data were acquired during undirected song, except for the neuron in Figure 5K, which was recorded during female-directed song.

Performance error response

To identify performance-error related neurons, we assessed the difference in firing rate between distorted and undistorted singing renditions as previously described (Keller and Hahnloser, 2009; Mandelblat-Cerf et al., 2014). Neurons with less than 10 trials of either distorted or undistorted renditions of the target syllable were excluded from this analysis (n=34/162 excluded). Briefly, we performed a WRS test on the number of spikes in distorted vs. undistorted renditions in 30 ms windows. Windows were shifted in 5 ms steps and considered significant when at least 4 consecutive windows had p<0.05. Error-related neurons were classified as error-activated if the firing rate is higher in distorted trials in window of significance, and error suppressed if the firing rate is higher in undistorted trials.

To visualize error response, we calculated z-score of the difference between distorted and undistorted rate histograms (Figure 3E). We defined ‘error score’ for each neuron to be the maximum of absolute z-scored difference in the 125ms after target onset (Figure 4A).

To identify prediction-error related neurons, we quantified phasic rate changes following undistorted target time. 1000 surrogate rate histograms were generated by randomly time-shifting each trial of undistorted target aligned data over the duration of the motif. Response was considered significant when firing rate dropped below 5th percentile or exceeded 95th percentile of the surrogate data. Neurons with significant rate peak or nadir in the window of significance for error response were identified as prediction error neurons.

To test if error responses were attributable to purely auditory responses to a different sound, we performed the same analysis for distorted and undistorted renditions during passive playback of bird’s own song (BOS) playback in 16/31 error neurons and 19/97 non-error neurons. Only two neurons exhibited an error response during passive playback. One such neuron exhibited similar song-locked firing during both singing and listening (Figure S5C). One other error responsive neuron also had song time-locked response in BOS playback, although the part of playback that contained target syllable was consistently masked by calls (Figure S5D).

We compared the latency and duration of error response to those of VTAX neurons from a previous dataset (Gadagkar et al., 2016). Latency and duration were defined by the onset and onset-offset interval of significant windows in WRS test as described above.

To test if VPVTA neurons were more likely to exhibit error response, we assigned a value of 1 (if error responsive) or 0 (if not error responsive) to each neuron tested for VTA antidromic stimulation. VPVTA neurons were tested against the group of VP neurons tested but not antidromic using WRS test (p=0.05).

Song timing related activity

Sparseness index was used to identify putative song-related MSNs. This distinguishes MSNs from other striatal cell types in the dorsal basal ganglia nucleus Area X (Goldberg and Fee, 2010). For each neuron, we calculated rate histograms aligned to syllable onset for all syllables. Then we normalized these histograms over all syllables to generate a probability density function pi over N bins. An entropy-based sparseness index was computed as follows:

Sparseness Index=1+∑i=1Npilog(pi)log(N)

Intermotif pairwise correlation coefficient (IMCC) was used to identify neurons that had highly time-locked firing to song motifs (timing neurons), as previously described (Goldberg et al., 2010; Goldberg and Fee, 2010; Kao et al., 2008; Olveczky et al., 2005; Woolley et al., 2014). Motif aligned IFR was time warped to the median duration of all motifs, mean-subtracted, and smoothed with a Gaussian kernel of 20ms SD, resulting in ri for each motif. IMCC was defined as the mean value of all pairwise CC between ri as follows:

IMCC=1Npairs∑j>iNpairsCCij
CCij=ri⋅rjri2rj2

To assess the significance of IMCC values, we computed new IMCC for each neuron by adding random, circular time shifts to each spiketrain. This was repeated 1000 times. IMCC was considered significant when the real value was greater than the 95th percentile of the shuffled data.

To quantify significant song-locked rate modulations, we compared the highest rate peak and lowest nadir in target-aligned rate histogram to 1000 surrogate rate histograms generated by randomly time-shifting spike trains. Rate peaks exceeding the 95th percentile of surrogate rate maximum and rate nadirs below the 5th percentile of surrogate rate minimum were considered significant.

To test if VPvta neurons were more likely to exhibit rate maxima/minima immediately prior to target time, we assigned a value of 1 (if a significant rate maximum/minimum was present in the 100ms before target time) or 0 (if significant peak/nadir was not present) for each neuron tested for VTA antidromic stimulation. VPvta neurons were tested against the group of VP neurons tested but not antidromic using WRS test (p=0.05).

Quantification of movement

An accelerometer (Analog Devices AD22301) was mounted on microdrives to quantify gross body movements as described previously (Gadagkar et al., 2016). Briefly, movement onsets and offsets were determined by threshold crossings of the band-passed, rectified accelerometer signal. To test if error responses could be explained by a difference in movement rate following DAF, for each bird we calculated onset times of movements relative to song target time. Then we performed a WRS test (p=0.05) on the number of movement onsets in distorted vs. undistorted renditions in 30 ms windows. Windows were shifted in 5 ms steps and considered significant when at least 4 consecutive windows had p<0.05.

Imaging

Imaging data was acquired with a Leica DM4000 B microscope and a Zeiss LSM 710 Confocal microscope. Image processing was done with ImageJ.

Supplementary Material

2

Key Resources Table

REAGENT or RESOURCE SOURCE IDENTIFIER
Antibodies
Anti-Tyrosine Hydroxylase antibody Millipore Cat#AB152
Anti-NeuN antibody, Alexa Fluor 488 conjugated Millipore Cat#MAB377X
clone A60
Bacterial and Virus Strains
Self-complementary AAV9-CBh-GFP Gene Therapy Center Vector Core, University of North Carolina at Chapel Hill N/A
Short Term HSV-mCherry Gene Delivery Technology Core, Massachusetts General Hospital Cat#RN3
Chemicals, Peptides, and Recombinant Proteins
N-Methyl-DL-aspartic acid Sigma Cat#M2137
Experimental Models: Organisms/Strains
Zebra Finch (Taeniopygia guttata) Magnolia Bird Farm, Anaheim CA N/A
Software and Algorithms
MATLAB MathWorks https://www.mathworks.com/products/matlab.html
ImageJ NIH https://imagej.nih.gov/ij/
Sound Analysis Pro Tchernichovski, O., Nottebohm, F., Ho, C.E., Pesaran, B., and Mitra, P.P. (2000). A procedure for an automated measurement of song similarity. Anim Behav 59, 1167–1176. N/A
Similarity Index Mandelblat-Cerf, Y., and Fee, M.S. (2014). An automated procedure for evaluating song imitation. PLoS One 9, e96484. N/A
Open in a new tab

Highlights.

  • Songbird ventral pallidum (VP) is required for song learning.

  • Vp neurons encode song performance error and precise song timing.

  • Vp sends error and error predection signals to dopaminergic midbrain.

  • Vp receives input from auditory areas and the vocal motor song system

ACKNOWLEDGEMENTS

We thank Todd Roberts for providing virus, Joe Fetcho, Teja Bollu, and Josh Dudman for comments, and Vikram Gadagkar, Aaron Andalman, and Dmitriy Aronov for comments and analysis code. Funding to JHG was provided by the NIH (R01NS094667), the Pew Charitable Trust, and the Klingenstein Neuroscience Foundations. Funding to PAP was provided by the NIH (F32NS098634). Imaging data was acquired in the Cornell BRC-Imaging Facility using the shared, NIH-funded (S10RR025502) Zeiss LSM 710 Confocal.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

DECLARATIONS OF INTERESTS

The authors declare no competing interests.

REFERENCES

  1. Achiro JM, Shen J, and Bottjer SW (2017). Neural activity in cortico-basal ganglia circuits of juvenile songbirds encodes performance during goal-directed learning. Elife 6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Ahrens AM, Meyer PJ, Ferguson LM, Robinson TE, and Aldridge JW (2016). Neural activity in the ventral pallidum encodes variation in the incentive value of a reward cue. Journal of Neuroscience 36, 7957–7970. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Ali F, Otchy TM, Pehlevan C, Fantana AL, Burak Y, and Olveczky BP (2013). The basal ganglia is necessary for learning spectral, but not temporal, features of birdsong. Neuron 80, 494–506. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Andalman AS, and Fee MS (2009). A basal ganglia-forebrain circuit in the songbird biases motor output to avoid vocal errors. Proc Natl Acad Sci U S A 106, 12518–12523. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Beier KT, Steinberg EE, DeLoach KE, Xie S, Miyamichi K, Schwarz L, Gao XJ, Kremer EJ, Malenka RC, and Luo L (2015). Circuit Architecture of VTA Dopamine Neurons Revealed by Systematic Input-Output Mapping. Cell 162, 622–634. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Bottjer SW, Brady JD, and Cribbs B (2000). Connections of a motor cortical region in zebra finches: relation to pathways for vocal learning. J Comp Neurol 420, 244–260. [PubMed] [Google Scholar]
  7. Carr DB, and Sesack SR (1999). Terminals from the rat prefrontal cortex synapse on mesoaccumbens VTA neurons. Ann N Y Acad Sci 877, 676–678. [DOI] [PubMed] [Google Scholar]
  8. Charlesworth JD, Warren TL, and Brainard MS (2012). Covert skill learning in a cortical-basal ganglia circuit. Nature 486, 251–255. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Creed MC, Ntamati NR, and Tan KR (2014). VTA GABA neurons modulate specific learning behaviors through the control of dopamine and cholinergic systems. Frontiers in behavioral neuroscience 8, 8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Danish HH, Aronov D, and Fee MS (2017). Rhythmic syllable-related activity in a songbird motor thalamic nucleus necessary for learned vocalizations. PloS one 12, e0169568. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Daw ND, Niv Y, and Dayan P (2006). Actions, policies, values and the basal ganglia. Recent breakthroughs in basal ganglia research 10, 1214–1221. [Google Scholar]
  12. Derégnaucourt S, Mitra P, Fehér O, Maul K, Lints T, and Tchernichovski O (2004). Song development: In search of the error Ǧ signal. Annals of the New York Academy of Sciences 1016, 364–376. [DOI] [PubMed] [Google Scholar]
  13. Ding L, and Perkel DJ (2004). Long-term potentiation in an avian basal ganglia nucleus essential for vocal learning. J Neurosci 24, 488–494. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Doupe AJ, and Konishi M (1991). Song-selective auditory circuits in the vocal control system of the zebra finch. Proc Natl Acad Sci U S A 88, 11339–11343. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Fee MS, and Goldberg JH (2011). A hypothesis for basal ganglia-dependent reinforcement learning in the songbird. Neuroscience 198, 152–170. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Gadagkar V, Puzerey PA, Chen R, Baird-Daniel E, Farhang AR, and Goldberg JH (2016). Dopamine neurons encode performance error in singing birds. Science 354, 1278–1282. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Gale SD, and Perkel DJ (2010). A basal ganglia pathway drives selective auditory responses in songbird dopaminergic neurons via disinhibition. The Journal of neuroscience 30, 1027–1037. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Gale SD, Person AL, and Perkel DJ (2008). A novel basal ganglia pathway forms a loop linking a vocal learning circuit with its dopaminergic input. J Comp Neurol 508, 824–839. [DOI] [PubMed] [Google Scholar]
  19. Ganguli S, and Hahnloser R (2011). Bird song learning without reinforcement: the Hebbian self-organization of sensorimotor circuits In CoSyne (Salt Lake City, UT: ). [Google Scholar]
  20. Goldberg JH, Adler A, Bergman H, and Fee MS (2010). Singing-related neural activity distinguishes two putative pallidal cell types in the songbird basal ganglia: comparison to the primate internal and external pallidal segments. J Neurosci 30, 7088–7098. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Goldberg JH, and Fee MS (2010). Singing-related neural activity distinguishes four classes of putative striatal neurons in the songbird basal ganglia. Journal of Neurophysiology 103, 2002–2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Goldberg JH, and Fee MS (2012). A cortical motor nucleus drives the basal ganglia-recipient thalamus in singing birds. Nature Neuroscience 15, 620–627. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Hamaguchi K, Tanaka M, and Mooney R (2016). A distributed recurrent network contributes to temporally precise vocalizations. Neuron 91, 680–693. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Hamaguchi K, Tschida KA, Yoon I, Donald BR, and Mooney R (2014). Auditory synapses to song premotor neurons are gated off during vocalization in zebra finches. Elife 3, e01833. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Hisey E, Kearney MG, and Mooney R (2018). A common neural circuit mechanism for internally guided and externally reinforced forms of motor learning. Nature neuroscience, 1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Hoffmann LA, Saravanan V, Wood AN, He L, and Sober SJ (2016). Dopaminergic Contributions to Vocal Learning. J Neurosci 36, 2176–2189. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Hong S, and Hikosaka O (2014). Pedunculopontine tegmental nucleus neurons provide reward, sensorimotor, and alerting signals to midbrain dopamine neurons. Neuroscience 282, 139–155. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Ito M, and Doya K (2009). Validation of decision-making models and analysis of decision variables in the rat basal ganglia. Journal of Neuroscience 29, 9861–9874. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Joel D, Niv Y, and Ruppin E (2002). Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw 15, 535–547. [DOI] [PubMed] [Google Scholar]
  30. Johnson F, Sablan MM, and Bottjer SW (1995). Topographic organization of a forebrain pathway involved with vocal learning in zebra finches. J Comp Neurol 358, 260–278. [DOI] [PubMed] [Google Scholar]
  31. Kao MH, Wright BD, and Doupe AJ (2008). Neurons in a forebrain nucleus required for vocal plasticity rapidly switch between precise firing and variable bursting depending on social context. J Neurosci 28, 13232–13247. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Keller GB, and Hahnloser RH (2009). Neural processing of auditory feedback during vocal practice in a songbird. Nature 457, 187–190. [DOI] [PubMed] [Google Scholar]
  33. Kelley DB, and Nottebohm F (1979). Projections of a telencephalic auditory nucleus-field L-in the canary. J Comp Neurol 183, 455–469. [DOI] [PubMed] [Google Scholar]
  34. Kozhevnikov AA, and Fee MS (2007). Singing-related activity of identified HVC neurons in the zebra finch. J Neurophysiol 97, 4271–4283. [DOI] [PubMed] [Google Scholar]
  35. Lei H, and Mooney R (2010). Manipulation of a central auditory representation shapes learned vocal output. Neuron 65, 122–134. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Leonardo A (2004). Experimental test of the birdsong error-correction model. Proc Natl Acad Sci U S A 101, 16935–16940. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Leonardo A, and Fee MS (2005). Ensemble coding of vocal control in birdsong. J Neurosci 25, 652–661. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Luo M, Ding L, and Perkel DJ (2001). An avian basal ganglia pathway essential for vocal learning forms a closed topographic loop. J Neurosci 21, 6836–6845. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Mandelblat-Cerf Y, and Fee MS (2014). An automated procedure for evaluating song imitation. PLoS One 9, e96484. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Mandelblat-Cerf Y, Las L, Denisenko N, and Fee MS (2014). A role for descending auditory cortical projections in songbird vocal learning. Elife 3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Marler P (1997). Three models of song learning: evidence from behavior. J Neurobiol 33, 501–516. [PubMed] [Google Scholar]
  42. McAlonan GM, Robbins TW, and Everitt BJ (1993). Effects of medial dorsal thalamic and ventral pallidal lesions on the acquisition of a conditioned place preference: further evidence for the involvement of the ventral striatopallidal system in reward-related processes. Neuroscience 52, 605–620. [DOI] [PubMed] [Google Scholar]
  43. Mello CV, Vates GE, Okuhata S, and Nottebohm F (1998). Descending auditory pathways in the adult male zebra finch (Taeniopygia guttata). J Comp Neurol 395, 137–160. [PubMed] [Google Scholar]
  44. Mogenson GJ, Jones DL, and Yim CY (1980). From motivation to action: functional interface between the limbic system and the motor system. Progress in neurobiology 14, 69–97. [DOI] [PubMed] [Google Scholar]
  45. Moreines JL, Owrutsky ZL, and Grace AA (2017). Involvement of Infralimbic Prefrontal Cortex but not Lateral Habenula in Dopamine Attenuation After Chronic Mild Stress. Neuropsychopharmacology 42, 904–913. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Murdoch D, Chen R, and Goldberg JH (2018). Place preference and vocal learning rely on distinct reinforcers in songbirds. Sci Rep 8, 6766. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Nottebohm F, Alvarez-Buylla A, Cynx J, Kirn J, Ling CY, Nottebohm M, Suter R, Tolles A, and Williams H (1990). Song learning in birds: the relation between perception and production. Philos Trans R Soc Lond B Biol Sci 329, 115–124. [DOI] [PubMed] [Google Scholar]
  48. Olveczky BP, Andalman AS, and Fee MS (2005). Vocal experimentation in the juvenile songbird requires a basal ganglia circuit. PLoS Biol 3, e153. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Ottenheimer D, Richard JM, and Janak PH (2018). Ventral pallidum encodes relative reward value earlier and more robustly than nucleus accumbens. Nat Commun 9, 4350. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Patton MH, Bizup BT, and Grace AA (2013). The infralimbic cortex bidirectionally modulates mesolimbic dopamine neuron activity via distinct neural pathways. J Neurosci 33, 16865–16873. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Person AL, Gale SD, Farries MA, and Perkel DJ (2008). Organization of the songbird basal ganglia, including area X. J Comp Neurol 508, 840–866. [DOI] [PubMed] [Google Scholar]
  52. Reiner A, Perkel DJ, Mello CV, and Jarvis ED (2004). Songbirds and the revised avian brain nomenclature. Ann N Y Acad Sci 1016, 77–108. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Richard JM, Ambroggi F, Janak PH, and Fields HL (2016). Ventral Pallidum Neurons Encode Incentive Value and Promote Cue-Elicited Instrumental Actions. Neuron 90, 1165–1173. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Richard JM, Stout N, Acs D, and Janak PH (2018). Ventral pallidal encoding of reward-seeking behavior depends on the underlying associative structure. eLife 7, e33107. [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Samejima K, and Doya K (2007). Multiple representations of belief states and action values in corticobasal ganglia loops. Ann N Y Acad Sci 1104, 213–228. [DOI] [PubMed] [Google Scholar]
  56. Schmidt RA, Lee T, Winstein C, Wulf G, and Zelaznik H (2018). Motor Control and Learning, 6E (Human kinetics).
  57. Schultz W, and Romo R (1987). Responses of nigrostriatal dopamine neurons to high-intensity somatosensory stimulation in the anesthetized monkey. J Neurophysiol 57, 201–217. [DOI] [PubMed] [Google Scholar]
  58. Smith KS, Tindell AJ, Aldridge JW, and Berridge KC (2009). Ventral pallidum roles in reward and motivation. Behav Brain Res 196, 155–167. [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. Sober SJ, Wohlgemuth MJ, and Brainard MS (2008). Central contributions to acoustic variation in birdsong. J Neurosci 28, 10370–10379. [DOI] [PMC free article] [PubMed] [Google Scholar]
  60. Sutton RS, and Barto AG (1998). Reinforcement learning: an introduction (Cambridge, MA: MIT Press; ). [Google Scholar]
  61. Tachibana Y, and Hikosaka O (2012). The primate ventral pallidum encodes expected reward value and regulates motor action. Neuron 76, 826–837. [DOI] [PMC free article] [PubMed] [Google Scholar]
  62. Takahashi Y, Schoenbaum G, and Niv Y (2008). Silencing the critics: understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an actor/critic model. Front Neurosci 2, 86–99. [DOI] [PMC free article] [PubMed] [Google Scholar]
  63. Takahashi YK, Langdon AJ, Niv Y, and Schoenbaum G (2016). Temporal Specificity of Reward Prediction Errors Signaled by Putative Dopamine Neurons in Rat VTA Depends on Ventral Striatum. Neuron 91, 182–193. [DOI] [PMC free article] [PubMed] [Google Scholar]
  64. Takahashi YK, Roesch MR, Wilson RC, Toreson K, O’Donnell P, Niv Y, and Schoenbaum G (2011). Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat Neurosci 14, 1590–1597. [DOI] [PMC free article] [PubMed] [Google Scholar]
  65. Tchernichovski O, Nottebohm F, Ho CE, Pesaran B, and Mitra PP (2000). A procedure for an automated measurement of song similarity. Anim Behav 59, 1167–1176. [DOI] [PubMed] [Google Scholar]
  66. Thelen E (1995). Motor development. A new synthesis. Am Psychol 50, 79–95. [DOI] [PubMed] [Google Scholar]
  67. Tian J, Huang R, Cohen JY, Osakada F, Kobak D, Machens CK, Callaway EM, Uchida N, and Watabe-Uchida M (2016). Distributed and Mixed Information in Monosynaptic Inputs to Dopamine Neurons. Neuron 91, 1374–1389. [DOI] [PMC free article] [PubMed] [Google Scholar]
  68. Tian J, and Uchida N (2015). Habenula Lesions Reveal that Multiple Mechanisms Underlie Dopamine Prediction Errors. Neuron 87, 1304–1316. [DOI] [PMC free article] [PubMed] [Google Scholar]
  69. Tindell AJ, Berridge KC, and Aldridge JW (2004). Ventral pallidal representation of pavlovian cues and reward: population and rate codes. J Neurosci 24, 1058–1069. [DOI] [PMC free article] [PubMed] [Google Scholar]
  70. Troyer TW, and Doupe AJ (2000). An associational model of birdsong sensorimotor learning I. Efference copy and the learning of song syllables. J Neurophysiol 84, 1204–1223. [DOI] [PubMed] [Google Scholar]
  71. Tsai HC, Zhang F, Adamantidis A, Stuber GD, Bonci A, de Lecea L, and Deisseroth K (2009). Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 324, 1080–1084. [DOI] [PMC free article] [PubMed] [Google Scholar]
  72. Tumer EC, and Brainard MS (2007). Performance variability enables adaptive plasticity of ‘crystallized’ adult birdsong. Nature 450, 1240–1244. [DOI] [PubMed] [Google Scholar]
  73. Vallentin D, and Long MA (2015). Motor Origin of Precise Synaptic Inputs onto Forebrain Neurons Driving a Skilled Behavior. J Neurosci 35, 299–307. [DOI] [PMC free article] [PubMed] [Google Scholar]
  74. Vates GE, Broome BM, Mello CV, and Nottebohm F (1996). Auditory pathways of caudal telencephalon and their relation to the song system of adult male zebra finches. J Comp Neurol 366, 613–642. [DOI] [PubMed] [Google Scholar]
  75. Wise RA, and Schwartz HV (1981). Pimozide attenuates acquisition of lever-pressing for food in rats. Pharmacol Biochem Behav 15, 655–656. [DOI] [PubMed] [Google Scholar]
  76. Woolley SC (2019). Dopaminergic regulation of vocal-motor plasticity and performance. Current opinion in neurobiology 54, 127–133. [DOI] [PubMed] [Google Scholar]
  77. Woolley SC, Rajan R, Joshua M, and Doupe AJ (2014). Emergence of context-dependent variability across a basal ganglia network. Neuron 82, 208–223. [DOI] [PMC free article] [PubMed] [Google Scholar]
  78. Xiao L, Chattree G, Oscos FG, Cao M, Wanat MJ, and Roberts TF (2018). A Basal Ganglia Circuit Sufficient to Guide Birdsong Learning. Neuron 98, 208–221 e205. [DOI] [PMC free article] [PubMed] [Google Scholar]
  79. Yagishita S, Hayashi-Takagi A, Ellis-Davies GC, Urakubo H, Ishii S, and Kasai H (2014). A critical time window for dopamine actions on the structural plasticity of dendritic spines. Science 345, 1616–1620. [DOI] [PMC free article] [PubMed] [Google Scholar]
  80. Yang H, de Jong JW, Tak Y, Peck J, Bateup HS, and Lammel S (2018). Nucleus Accumbens Subnuclei Regulate Motivated Behavior via Direct Inhibition and Disinhibition of VTA Dopamine Subpopulations. Neuron 97, 434–449 e434. [DOI] [PMC free article] [PubMed] [Google Scholar]
  81. Yu AC, and Margoliash D (1996). Temporal hierarchical control of singing in birds. Science 273, 1871–1875. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

2
NIHMS1528321-supplement-2.pdf (2.6MB, pdf)

RESOURCES