Discrete evaluative and premotor circuits enable vocal learning in songbirds

Summary

Virtuosic motor performance requires the ability to evaluate and modify individual gestures within a complex motor sequence. Where and how the evaluative and premotor circuits operate within the brain to enable such temporally precise learning are poorly understood. Songbirds can learn to modify individual syllables within their complex vocal sequences, providing a system for elucidating the underlying evaluative and premotor circuits. We combined behavioral and optogenetic methods to identify two afferents to the ventral tegmental area (VTA) that serve evaluative roles in syllable-specific learning and to establish that downstream cortico-basal ganglia circuits serve a learning role that is only premotor. Further, song performance-contingent optogenetic stimulation of either VTA afferent was sufficient to drive syllable-specific learning, and these learning effects were of opposite valence. Finally, functional, anatomical, and molecular studies support the idea that these evaluative afferents bi-directionally modulate VTA dopamine neurons to enable temporally precise vocal learning.

eTOC Blurb

Kearney et al. used behavioral and optogenetic methods in singing birds to distinguish neural pathways that evaluate song performance from downstream premotor circuits that are guided by these evaluations to learn new vocal behaviors.

Introduction

Remarkable skills such as fluent speech or virtuosic musicianship involve rapid sequences of more elementary movements, or gestures, controlled with millisecond (ms) precision. Learning such rapid and precise sequences requires extensive practice and sensory feedback-dependent performance evaluation (Jabusch et al., 2009, Bryan and Harter, 1899, Ericsson et al., 1993). Furthermore, practice and evaluation can be used to modify individual gestures within a previously learned sequence, ultimately generating new behaviors. How the brain enables such temporally precise learning is a mystery but must depend on both premotor and evaluative computations. Resolving whether the brain circuits mediating these computations are largely segregated or more distributed and possibly overlapping presents a daunting experimental challenge, especially for rapid behavioral sequences where motor learning can occur with millisecond precision, the premotor and sensory feedback signals are separated by only tens of milliseconds, and where the central neurons important to behavioral learning often display both premotor and sensory properties (Hessler and Doupe, 1999, Prather et al., 2008, McCasland, 1987, Dichter et al., 2018).

Male zebra finches use auditory feedback to learn and maintain song motifs comprising a rapid and precise sequence of 2–7 syllables, each lasting 50–150 ms (Figure 1A) (Nordeen and Nordeen, 1992, Price, 1979, Immelman, 1969). Moreover, an adult male can learn to modify the fundamental frequency (pitch) of a single syllable within this motif in response to syllable-triggered noise bursts, a process referred to as pitch learning (Tumer and Brainard, 2007). This highly tractable form of learning facilitates exploration of the brain mechanisms underlying temporally precise modification of a single gesture within a longer sequence (Andalman and Fee, 2009, Charlesworth et al., 2012, Warren et al., 2011) (Figure1A–E).

Figure 1: Pitch-contingent auditory feedback negatively reinforces syllable pitch.

(A) Spectrogram of zebra finch song highlighting three sequential syllables from a motif. A target syllable (black note), showing the related premotor window (orange) and auditory feedback window (purple). (B) Experimental design for pitch learning, here targeting lower pitch syllable variants (After Tumer and Brainard 2007). (C) Left, pitch of the first and last 50 renditions of a target syllable on a single day at baseline (black). Right, the first and last 50 renditions of a target syllable on a single day where white noise (WN) was delivered on low pitch renditions of the syllable (red). (D) Mean increase in pitch of target syllables over one day of WN learning (n = 15 syllables, n = 15 birds, p=0.00017, paired t-test). (E) Schematic of song learning. (F) Sagittal drawing of zebra finch brain emphasizing some of the nuclei involved in song learning and production.

The male finch’s brain contains a well delineated network for singing and song learning, including an anterior forebrain pathway with clear homologies to mammalian cortico-basal ganglia (CBG) circuitry (Brainard and Doupe, 2013, Reiner et al., 2004). The output of this CBG circuit, the lateral magnocellular nucleus of the anterior nidopallium (LMAN), plays a central role in song learning (Andalman and Fee, 2009, Warren et al., 2011, Bottjer et al., 1984, Brainard and Doupe, 2000). Whether this role is evaluative or premotor, or some combination of both, has engendered substantial debate. The finding that LMAN neurons can respond selectively to auditory presentation of the bird’s own song, as well as the tutor song from which it was copied, raised the possibility that LMAN evaluates song performance (Solis and Doupe, 1997, Solis et al., 2000). However, LMAN neurons display singing-related activity that is insensitive to auditory feedback perturbation, suggestive of a purely premotor function (Leonardo, 2004). Consistent with this idea, LMAN activity is a source of acute song variability and, following exposure to pitch-contingent noise, biases the pitch of the target syllable to escape noise playback (Andalman and Fee, 2009, Warren et al., 2011, Kao et al., 2005). Consequently, LMAN’s role may be premotor rather than evaluative, but the inactivation methods used to explore LMAN’s role in pitch learning are too slow to make this distinction and causal tests of a premotor learning role are lacking.

Further, if LMAN serves solely a premotor role in pitch learning, then which circuits enable performance evaluation? The songbird CBG includes projections from dopamine neurons in the ventral tegmental area (VTA) and the substantia nigra pars compacta (SNc) to a song-specialized BG region (Area X) that is essential to juvenile song copying and adult pitch learning (Figure 1G,H) (Hisey et al., 2018, Hoffmann et al., 2016, Person et al., 2008, Lewis et al., 1981, Scharff and Nottebohm, 1991, Sohrabji et al., 1990, Xiao et al., 2018). Furthermore, in adult finches chronically exposed to syllable-triggered noise, the firing rates of VTA and SNc neurons that project to Area X (referred to collectively as VTA_X neurons) decrease on renditions that trigger noise and increase when noise is withheld, demonstrating that these neurons learn to bi-directionally encode reward prediction error (Gadagkar et al., 2016). This variation in VTA_X neuron activity is causally linked to vocal learning, as pitch-contingent optogenetic excitation or inhibition of VTA_X terminals is sufficient to drive syllable-specific pitch learning in adult finches (Hisey et al., 2018, Xiao et al., 2018). Together, these studies advance a model in which VTA_X neurons transmit a bi-directional reinforcement signal that guides vocal learning.

A crucial issue is how evaluative information reaches the VTA to compute a reinforcement signal. The songbird VTA receives major inputs from the ventral intermediate arcopallium (Aiv) and the ventral pallidum (VP) (Gale et al., 2008, Person et al., 2008), providing points of entry to address this issue. Intriguingly, Aiv receives input from auditory cortical regions and the firing rates of VTA-projecting Aiv (Aiv_VTA) neurons increase when the bird sings a syllable that triggers a noise burst, but do not change when noise is withheld (Mandelblat-Cerf et al., 2014). These findings suggest that Aiv_VTA neurons evaluate auditory feedback to detect vocal errors and could suppress VTA_X neuron activity in response to syllable-triggered noise, while also raising the question of what drives increased VTA firing on renditions when noise is withheld. The VP also receives inputs from auditory cortical regions but, unlike Aiv, also receives direct input from Area X, and thus may receive both auditory and premotor signals related to song performance. Moreover, the VP in mammals is implicated in hedonic pleasure and reward, suggesting that songbird VP_VTA neurons could encode positive evaluations of song performance (Richard et al., 2016, Smith et al., 2009, Tindell et al., 2004, Panagis et al., 1995, Itoga et al., 2016, Ottenheimer et al., 2018). In fact, a recent study of VP neurons reveals that a small number selectively respond to syllable renditions accompanied by normal feedback (Chen et al., 2019). These observations support a model in which both Aiv_VTA and VP_VTA neurons process singing-related auditory feedback but provide different types of evaluative information that enable the VTA to compute a bi-directional reinforcement signal. Despite the appeal of this model, whether feedback-related variation in the activity of either of these projections is necessary for vocal learning remains untested, and whether pitch-contingent variation in Aiv_VTA and VP_VTA activity can drive either similar or opposing effects on learning remains unknown. Here we combined closed-loop behavioral and optogenetic manipulations to distinguish evaluative from premotor circuits important to pitch learning. Then we used functional, anatomical, and molecular methods to determine how evaluative circuits bidirectionally modulate VTA activity to enable temporally precise learning.

Results

Pitch-contingent noise negatively reinforces syllable pitch

We employed online syllable detection methods to deliver a brief (50 ms) burst of white noise (WN) contingent upon rendition to rendition changes in the frequency (pitch) of a target syllable, (Figure 1B) (Tumer and Brainard, 2007). Delivering WN on low pitch syllable renditions over the course of a day drove birds to significantly increase the pitch of the targeted syllable (Figure 1C, D; n = 15 syllables, n = 15 birds, p=0.00017, paired t-test). Confirming that this method is sufficient to drive syllable-specific pitch learning provided us with a means for systematically probing the neural circuit mechanisms that underlie such temporally precise learning.

Aiv_VTA and VP_VTA convey evaluative information important for pitch learning

We reasoned that if Aiv_VTA and VP_VTA neurons play a role in evaluating song performance, then imposing stereotyped patterns of activity on their synapses in the VTA should interfere with their ability to encode differences in performance quality across syllable renditions and thus impair pitch learning. Moreover, this “interference” with pitch learning should be restricted in time to when auditory feedback is being evaluated, rather than earlier (premotor) time windows prior to syllable onset. To test this idea, we optogenetically interfered with Aiv_VTA terminal activity either during the auditory feedback or premotor window of a syllable targeted for learning via pitch-contingent noise (Figure 2A–D).

Figure 2: Aiv_VTA and VP_VTA convey evaluative information important for vocal learning.

(A) Syllable sequence highlighting premotor and auditory feedback “evaluative” windows associated with syllable B. (B) Low pitch syllable renditions trigger disruptive WN feedback while high pitch renditions “escape” WN. (C) Schematic of premotor optogenetic “interference” experiment. As in Figure 1, pitch-contingent WN is delivered to syllable D. Independently, the preceding syllable C is detected on every rendition and triggers the laser at a brief fixed delay to deliver optogenetic stimulation during the premotor window for 100% of syllable D renditions. (D) Schematic of auditory feedback optogenetic “interference” experiment. The same approach is used as in the premotor jamming experiment, but now the delay for triggering the laser is set to coincide with the auditory feedback window associated with syllable D. (E) Schematic of Aiv_VTA experiments. (F) Change in target syllable pitch across a single day without laser stimulation or WN (baseline (B), black), WN on low pitch variants (No Laser + WN, red), WN with feedback laser (Feedback Laser + WN, purple), and WN with premotor laser (Premotor Laser + WN, orange). (G) Bar graph of percent change in pitch of target syllables following one day in each of the four conditions (No Laser + WN, n=5, p=0.0489, Feedback Laser + WN, n=5, p = 0.2374, Premotor Laser + WN, n=5, p=0.0235, all paired t tests). (H) Schematic VP_VTA experiments. (I) Same as in (F) except for VP_VTA. (J) Same as in (G) except for VP_VTA, (No Laser + WN, n=4, p=0.0308, Feedback Laser + WN, n=4, p=0.4812, Premotor Laser + WN, n =4, p = 0.0077, all paired t tests).

We injected an adeno-associated virus (AAV) containing a Channelrhodopsin (ChR2) gene and a fluorescent reporter (AAV2/1.CAG-ChR2.mCherry, n = 4, or AAV2/9.CAG-NRXN.ChR2.YFP, n = 1) bilaterally into Aiv of young adult male zebra finches, waited to achieve functional ChR2 expression in Aiv_VTA terminals, and implanted optical fibers bilaterally over the VTA and SNc (referred throughout simply as the VTA) (Figure 2E; n = 5 animals; mean age at viral injection: 121.6 ± 24.0 dph; mean interval between viral injections in Aiv and fiber implantation in VTA: 131.2 ± 22.1 days; mean age at implantation: 252.8 ± 15.2 dph). We then applied pitch-contingent noise to a target syllable in the bird’s motif to establish control learning values for each bird (Figure 2F, G). After allowing the pitch of the target syllable to recover to baseline, we again exposed the bird to pitch-contingent noise while optogenetically stimulating Aiv_VTA terminals on all renditions of the target syllable (see Methods for absolute hit rates, laser wavelength = 473 nm, power = 5–15 mW). Optogenetic stimulation of Aiv_VTA terminals was precisely timed to target the time window when auditory feedback related to that syllable is processed in the brain (mean onset of stimulation was ~10 ms after the mean noise onset and extended for the duration of the noise burst (50 ms); see Methods for a discussion of auditory latencies in the songbird’s brain). Pairing syllable-triggered noise with Aiv_VTA terminal stimulation strongly reduced the amount of noise-evoked pitch learning, such that pitch distributions measured after a day of pairing noise and optogenetic stimulation were not significantly different from baseline values (Figure 2F, G, n = 5 syllables, n = 5 birds, p = 0.0146 paired t test feedback laser + noise vs noise alone; p = 0.2374 feedback laser + noise vs baseline). In contrast, normal levels of pitch learning occurred when the timing of Aiv_VTA terminal stimulation was shifted earlier in time, to a 50 ms window just prior to syllable onset, when premotor signals for a given syllable are detectable within various forebrain song nuclei (Figure 2F, G, n = 5 syllables, n = 5 birds, p = 0.7211 paired t test premotor laser + noise vs noise alone; p = 0.0235 premotor laser + noise vs baseline; see Methods for a discussion of vocal premotor latencies in singing birds). Moreover, Aiv_VTA terminal stimulation in ChR2-expressing birds during either the premotor or evaluative window exerted no acute effects on the spectral features of the target syllable (Figure S1). Therefore, optogenetically stimulating Aiv_VTA terminals on all renditions of a target syllable interferes with noise-evoked pitch learning, but only if this interference occurs during evaluative rather than premotor time windows associated with the target syllable.

We used a similar approach to determine whether variation in VP_VTA terminal activity encodes evaluative information important to pitch learning (Figure 2H; n = 4 animals injected bilaterally in VP with AAV2/1.CAG-ChR2.mCherry; mean age at injection: 103.5 ± 27.3 days; mean age at fiber implantation: 435.3 ± 25.1 dph). After establishing control pitch learning values (Figure 2I, J), we exposed the bird to pitch-contingent noise while optogenetically stimulating VP_VTA terminals on all detected renditions of the target syllable during the auditory feedback or premotor window related to that syllable (see Methods). Pairing pitch-contingent noise with syllable-triggered VP_VTA terminal stimulation during the auditory feedback window strongly reduced the amount of pitch learning, and pitch values measured after one day of such pairing did not differ from baseline values (Figure 2I, J, n = 4 syllables, n = 4 birds, p = 0.0329 paired t test feedback laser + noise vs noise alone; p = 0.4812 feedback laser + noise vs baseline). In contrast, normal pitch learning occurred when VP_VTA terminal stimulation was applied during the premotor window (Figure 4I, J n = 4 syllables, n = 4 birds, p = 0.9041 paired t test premotor laser + noise vs noise alone; p = 0.0077 premotor laser + noise vs baseline). Therefore, optogenetically stimulating VP_VTA terminals on all target syllable renditions interferes with noise-evoked pitch learning, but only when stimulation occurs during the auditory feedback window rather than the premotor window associated with the target syllable. Finally, VP_VTA terminal stimulation in ChR2-expressing birds exerted no acute effects on the spectral features of the target syllable (Figure S2). Together, these findings support the idea that Aiv_VTA and VP_VTA neurons encode a feedback-dependent evaluation of song performance and transmit this evaluation to the VTA to drive syllable-specific learning.

Figure 4: Pitch-contingent stimulation of Aiv_VTA terminals negatively reinforces target syllable pitch.

(A) Viral targeting and stimulation strategy. (B) Top, ChR2 (red) expression in Aiv. Bottom, Aiv terminals in VTA (TH+ cells in green). Scale bars: 50 μm. (C) Top, schematic of experimental design, scale bars: 100 ms, 2 kHz. Left, pitch of all “catch” trials of the target syllable at baseline and on each day of laser stimulation (B, baseline; L1, first day light stimulation, L2, second day light stimulation, etc., renditions below threshold in blue). Right, z-scored pitch distribution of a target syllable before and after stimulation on low pitch renditions. (D) Left, baseline z-scored pitch distributions from all target syllables, each distribution represents one target syllable. Middle, z-scored pitch distribution from target syllables after stimulation on high pitch renditions. Right, as in middle but for stimulation on low pitch renditions. (E) auROC for all target syllables on B1 or L4 compared to B2 (n=6 syllables, 4 birds). (F) Change in frequency (hertz) for two baseline days and four days of VP_VTA stimulation for syllables targeted on low pitches (n = 3, blue, bold line is average across syllables) and high pitch renditions (n = 4, red, bold line is average across syllables). (G) Mean percent change in pitch on B1 and L4, relative to B2, in experimental birds (p=0.0029 paired t test). Right, same but in control birds (n=4 syllables, n=4 birds, p=0.4805, paired t test, green: GFP, gray: uninjected).

The output of the BG pathway serves a premotor role important to pitch learning

A remaining issue is whether other parts of the song system also contribute to performance evaluation, or instead serve solely as premotor components that are guided by the outcome of this evaluation. In songbirds, the lateral magnocellular nucleus of the anterior nidopallium (LMAN) is the sole cortical output of the cortical-BG loop, extending axons to the song motor nucleus RA as well as axon collateral projections into Area X (Figure 3A) (Nixdorf-Bergweiler et al., 1995, Vates and Nottebohm, 1995). These divergent anatomical projections along with the mixed sensorimotor properties of LMAN neurons have made a causal distinction between LMAN’s role in performance and evaluation difficult.

Figure 3: The cortical output of the BG pathway serves a premotor role important to pitch learning.

(A) Viral targeting and stimulation strategy. (B) Histological confirmation of ChR2 expression (red) in LMAN, scale bar, 50 μm (green anti-Calbindin). (C) High power image of ChR2 expression (red) in LMAN, scale bar 10 μm. (D) 5 trials of single unit activity recorded in LMAN in an isoflurane-anesthetized finch, in response to 20 ms laser pulses (blue triangles). (E) Top, latency to first spike relative to onset of laser stimulation. Bottom, inter-spike interval histogram of LMAN activity evoked by laser (blue) and spontaneous activity (red). (F) Change in target syllable pitch across a single day without laser stimulation or WN (baseline (B), black), WN on low pitch variants (No Laser + WN, red), WN with feedback laser (Feedback Laser + WN, purple), and WN with premotor laser (Premotor Laser + WN, orange). (G) Bar graph of percent change in pitch of target syllables following one day in each of the four conditions (No Laser + WN, n=5, p=0.0353, Feedback Laser + WN, n=5, p=0.0019, Premotor Laser + WN, n=5, p=0.6114, all paired t tests).

To make this distinction, we injected AAV2/1.CAG-ChR2.mCherry bilaterally in the LMAN of young adult male zebra finches and implanted optical fibers bilaterally over LMAN (Figure 3A–D; n = 5 animals; mean age at injection: 86 ± 31.3 days; mean age at implantation: 181.4 ± 42.9 dph). Single unit extracellular recordings in anesthetized birds revealed that optogenetic stimulation of LMAN neurons drove highly reliable, high frequency (> 400 Hz) action potential bursts, similar to firing rates recorded in singing birds (Figure 3D, E) (Kao et al., 2005). After establishing control noise-evoked pitch learning values, we exposed the bird to pitch-contingent noise while optogenetically stimulating LMAN cells on all detected renditions of the target syllable during the auditory feedback or premotor window associated with that syllable. Pitch learning proceeded normally when optogenetic stimulation was applied in the auditory feedback window; in contrast, there was no learning when stimulation was applied during the premotor window (Figure 3F, G no laser vs premotor laser p = 0.0276, paired t test, no laser vs feedback laser, p = 0.9366, paired t test). We noted a slight reduction in the coefficient of variability in syllable pitch on stimulated trials, consistent with LMAN’s premotor role in sculpting motor output (Figure S3). In summary, optogenetically stimulating LMAN neurons on all renditions of a target syllable reduced noise-evoked pitch learning, but only if this interference occurred during premotor rather than auditory feedback time windows associated with the target syllable. Therefore, LMAN’s role in pitch learning is premotor and not evaluative, in direct contrast to the evaluative roles played by Aiv_VTA and VP_VTA neurons.

Pitch-contingent stimulation of Aiv_VTA terminals negatively reinforces syllable pitch

We reasoned that if Aiv provides evaluative signals to the VTA necessary to pitch learning, then artificially elevating Aiv_VTA terminal activity in a pitch-dependent manner should drive pitch learning in the absence of noise. To test this idea, we injected AAV2/1.CAG-ChR2.mCherry bilaterally into Aiv of young adult male zebra finches and implanted optical fibers bilaterally over the VTA (Figure 4A, B; Figure S5A,C; n = 4 birds; mean age at injection = 77 ± 7.4 dph; mean age at implantation: 205.3 ± 9.5 dph). Following recovery from fiber implantation (mean =17.5 ± 4.1 days), we optogenetically stimulated Aiv_VTA terminals when the target syllable pitch fell above (or below) a specified threshold (Figure 4C; pulse duration, 50 ms, pulse wavelength, λ = 473 nm; threshold was set to deliver stimulation to either the upper (n = 4 syllables) or lower (n = 2 syllables) ~65% of the syllable’s pitch distribution, measured during baseline recordings). Pitch-contingent stimulation of Aiv terminals applied over several days was sufficient to drive significant changes in the pitch of the targeted syllable, as measured on unstimulated “catch” renditions (Figure 4C–G; Figure S6A; n = 6 syllables, n = 4 birds; p = 0.000032 paired t-test).

Similar to pitch learning induced by noise, the pitch of the target syllable shifted away from the frequency region receiving optogenetic stimulation, and the change in pitch was temporally restricted to the target syllable (Figure 4D, E, F; Figure S6C, E; n = 6 syllables from n = 4 birds, n = 4 syllables before target p = 0.7592 one sample t test, n = 13 syllables after target syllable p = 0.5768, one sample t test). Moreover, pitch-contingent stimulation of Aiv_VTA terminals exerted no acute effects on spectral features of the target syllable and did not alter the amount of singing (Figure S5E, G and S1; n = 6 syllables, p = 0.786, paired t test). The rate of pitch learning evoked by Aiv_VTA terminal stimulation was similar to previously reported values for pitch-contingent stimulation of VTA_X terminals, both of which are slightly less than half the rates evoked by pitch-contingent noise (Hisey et al., 2018). Lastly, the pitch of target syllables did not change following several days of pitch-contingent laser illumination of the VTA in adult finches injected with AAV-GFP in Aiv or without any viral injection (Figure 4G; n = 2 syllables from n = 2 birds injected in Aiv with AAV2/9.CMV-GFP and n = 2 syllables from n = 2 uninjected birds, p = 0.1476, paired t test). Therefore, pitch-contingent optogenetic stimulation of Aiv_VTA terminals is sufficient to drive syllable-specific learning in adult zebra finches, at least in part by negatively reinforcing syllable variants associated with elevated Aiv_VTA terminal activity.

Pitch-contingent activation of VP_VTA terminals positively reinforces syllable pitch

The optogenetic interference experiments we conducted indicate that Aiv and VP both provide evaluative signals to the VTA, but do not resolve whether they serve similar or distinct roles in pitch learning. To resolve this issue, we injected AAV2/1.CAG-ChR2.mCherry bilaterally into VP of young adult male zebra finches to functionally express ChR2 in VP_VTA terminals (Figure S5B, D), then implanted optical fibers bilaterally over the VTA (Figure 5A, B; n = 6 animals; mean age at injection = 80.8 ± 5.3 dph; mean age at implantation: 266.5 ± 9.5 dph). Following recovery from fiber implantation (mean = 17.5 ± 4.1 days), we optogenetically stimulated VP_VTA terminals when the pitch of a target syllable fell either above (or below) a specified threshold (as Figure 3C; pulse duration, 50 ms, pulse wavelength, λ = 473 nm; threshold was set to deliver stimulation to either the upper (n = 4 syllables) or lower (n = 3 syllables) ~65% of the syllable distribution).

Figure 5: Pitch-contingent activation of VP_VTA terminals positively reinforces target syllable pitch.

(A) Viral targeting and stimulation strategy. (B) Top, ChR2 (red) expression in VP. Bottom, VP terminals in VTA (TH+ cells, green). Scale bars: 50 μm. (C) Top, schematic of experimental design. Left, pitch of the target syllable at baseline and on each day of laser stimulation (convention as in Figure 4). Right, z-scored pitch distribution of a target syllable before and after stimulation on low pitch renditions. (D) Left, baseline z-scored pitch distributions from all target syllables. Each histogram represents one syllable. Middle, z-scored pitch distributions from target syllables after stimulation on high pitch variants. Right, as in middle but for stimulation on low pitch variants. (E) auROC for all target syllables on B1 or L4 compared to B2 (n=7 syllables, 6 birds). (F) Change in frequency (hertz) for two baseline days and four days of VP_VTA stimulation for syllables targeted on low pitches (n=3, blue, bold line is average across syllables) and high pitch renditions (n=4, red, bold line is average across syllables). (G) Mean percent change in pitch on B1 and L4, relative to B2, in experimental birds (n=7 syllables, n=6 birds, p=0.0006 paired t test). Right, same but in control birds (n=4 syllables, n=4 birds, p=0.2017, paired t test, green: GFP, gray: uninjected).

Pitch-contingent stimulation of VP_VTA terminals applied over several days drove significant changes to the pitch of the targeted syllable (Figure 5C–G; n = 7 syllables, n = 6 birds, p = 0.000044 paired t-test). However, the pitch of the target syllable shifted towards the frequency region paired with VP_VTA terminal stimulation, exactly opposite to the effects of Aiv_VTA terminal stimulation or pitch-contingent noise (Figure 5C–F). Additionally, the pitch changes evoked by VP_VTA terminal stimulation were similar in absolute magnitude to the changes elicited by Aiv_VTA terminal stimulation (compare Figure 5F, G to Figure 4F, G) and were temporally restricted to the target syllables (Figure S6D, F, n = 9 syllables before target p = 0.4652 one sample t test, n = 9 syllables after target syllable p = 0.3992 one sample t test). Furthermore, VP_VTA terminal stimulation in ChR2-expressing birds exerted no acute effects on the spectral features of the target syllable and chronic stimulation did not alter the amount that the bird sang (Figure S5F, H and S2; n = 7 syllables, p = 0.2145, paired t test). Finally, pitch-contingent illumination of either GFP-expressing VP_VTA terminals or of the VTA in uninjected animals did not alter the pitch of the target syllable (Figure 5G; n = 4 syllables, n = 4 birds, p = 0.4993). In summary, pitch-contingent optogenetic stimulation of VP_VTA terminals positively reinforces syllable variants paired with this stimulation, opposite in valence to the negatively reinforcing effects of pitch-contingent stimulation of Aiv_VTA terminals (compare Figure 5F, G to Figure 4F, G).

Aiv_VTA and VP_VTA terminals drive opposing effects on VTA neurons

A remaining unresolved issue is the circuit mechanisms by which Aiv_VTA and VP_VTA terminals drive their complementary effects on vocal learning. In fact, VTA_X terminals can provide bidirectional reinforcement signals for vocal learning, depending on whether they are activated or suppressed (Hisey et al., 2018, Xiao et al., 2018). Therefore, a parsimonious explanation for the opposing behavioral effects of pitch-contingent Aiv_VTA and VP_VTA terminal stimulation is that VTA_X neurons are suppressed by Aiv_VTA terminals and excited by VP_VTA terminals. To test this idea, we examined how optogenetic stimulation of either Aiv_VTA or VP_VTA terminals affected the spontaneous firing rates of different types of VTA neurons (Figure 6A, B). More specifically, electrophysiological recordings in the zebra finch have established that TH+ VTA neurons (i.e., dopamine-releasing neurons), including VTA_X neurons, fire spontaneously at low rates (<15 Hz) and display relatively broad action potentials (“thick spiking” neurons; Figure 6C) (Gale and Perkel, 2006, Gale and Perkel, 2010). In contrast, VTA interneurons fire spontaneously at more variable and often higher rates with narrower action potentials (“thin-spiking” neurons; Figure 6D) (Gale and Perkel, 2006, Gale and Perkel, 2010).

Figure 6: Optogenetic stimulation of Aiv_VTA and VP_VTA terminals drive bi-directional and opposing effects on VTA neurons.

(A) Schematic showing optogenetic activation of Aiv_VTA terminals while recording extracellularly from VTA neurons. (B) Same as (A), but for VP_VTA terminals. (C) Left, example recording of fast firing ‘thin-spiking’ unit in VTA (pink). Right, spike waveforms, mean waveform in bold and single spikes in gray. (D) Same as (C) but for a slow firing “thick-spiking” unit in VTA (E) Top, raster plot of thin-spiking single unit in VTA. Below, PSTH of activity showing increased firing with Aiv terminal activation (laser, cyan). Right, spike waveform, mean and single spikes shown. Bottom, raster plot of thick-spiking single unit in VTA and below, PSTH of activity showing decreased firing with Aiv terminal activation. Right, spike waveform, mean and single spikes shown. (F) Same as E but for VP_VTA terminals (G) Summary data from n=20 neurons from n=14 birds recorded in the VTA while stimulating Aiv terminals. Each unit that significantly modulated with laser stimulation plotted by spike width and firing rate (+, increased activity with laser; − decreased activity with laser, red region containing putative ‘thin-spiking’ interneurons and pink region containing putative ‘thick-spiking’ dopamine neurons). (H) Same as G but for VP_VTA experiments, data from n=12 neurons from n=8 birds recorded in the VTA while activating VP.

We used these electrophysiological criteria to distinguish how optogenetic stimulation of Aiv_VTA or VP_VTA terminals affected the spontaneous firing rates of putative TH+ neurons and interneurons in the VTA of isoflurane-anesthetized adult male zebra finches. In one set of animals (n = 16), we expressed ChR2 in Aiv_VTA terminals by injecting AAV2/1.CAG-ChR2.mCherry (n = 12) or AAV 2/9.CAG-ChR2.NRXN.YFP (n = 4) bilaterally into Aiv of young adult male zebra finches (mean age at injections = 89 ± 8.4 dph; mean interval between injections and recordings = 199.3 ± 46.4 days). We then used optrode methods to record from single units in the VTA, and briefly stimulated Aiv terminals at regular intervals with a laser (Figure 6E; pulse widths = 50–100ms, interpulse intervals = 50–500 ms, wavelength = 473 nm, power = 5–15 mW; units were sorted offline, see Methods) (Dufour and De Koninck, 2015). Optogenetic stimulation of Aiv_VTA terminals increased the spontaneous firing rates of a majority (11/14) of putative interneurons (Figure 6E, G; p = 0.0287 one-sided binomial test). In contrast, optogenetic stimulation of Aiv_VTA terminals suppressed a majority (9/11) of putative TH+ VTA neurons (Figure 6E, G; p = 0.0327 one-sided binomial test). In another set of animals (n = 9), we expressed ChR2 in VP_VTA terminals by injecting AAV2/1.CAG-ChR2.mCherry (n = 6) or AAV 2/9.CAG-ChR2.NRXN.YFP (n = 3) bilaterally into VP of young adult male zebra finches (mean age at injection = 69.9 ± 5.8 dph at time of injection; mean interval between injections and recordings = 323.1 ± 61.0 days). We then used optrodes to record from different neurons in the VTA while optogenetically activating VP_VTA terminals (Figure 6F). In contrast to the effects of Aiv_VTA terminal stimulation, optogenetic stimulation of VP_VTA terminals suppressed the spontaneous action potential activity of a majority (8/9) putative interneurons while driving excitatory responses in all (5/5) putative TH+ VTA neurons (Figure 6F, H; p = 0.0195 and p = 0.0313 one-sided binomial test).

Neurotransmitter phenotypes and synaptic structure of VTA afferents

Prior studies in mammals indicate that the VTA and SNc receive highly convergent input from excitatory (glutamate-releasing) and inhibitory (GABA-releasing) neurons distributed throughout the forebrain and brainstem (Watabe-Uchida et al., 2012, Eshel et al., 2015, Beier et al., 2015, Faget et al., 2016, Geisler et al., 2007). To begin to explore the neurotransmitter profiles of Aiv and VP neurons that provide input to the songbird VTA, we retrogradely labeled Aiv_VTA and VP_VTA neurons by injecting cholera toxin subunit B (CTB) into the VTA (Figure 7A, B). We then performed fluorescence in situ hybridizations for the vesicular glutamate transporter 2 (VGLUT2), which is expressed selectively in glutamatergic neurons, and the vesicular GABA transporter (VGAT), a selective marker of GABAergic neurons (Figure 7A, B; Figure S7A). Confocal imaging of fixed tissue sections revealed that that almost all Aiv_VTA neurons expressed VGLUT2 but not VGAT, whereas almost all VP_VTA neurons expressed VGAT but not VLGUT2 (Figure 7A, B). These findings indicate that Aiv_VTA neurons provide a predominantly glutamatergic input to the VTA, whereas VP_VTA neurons are predominantly GABAergic.

Figure 7: Neurotransmitter phenotypes and synaptic structure of VTA afferents.

(A) Upper left, schematic of injection of CTB into VTA to label Aiv_VTA neurons. Upper right, Aiv_VTA projecting neurons (magenta), VGLUT2 (green), and VGAT (red), scale bar 50 μm. Bottom left, higher magnification views, all scale bars 10 μm. Bottom right, pie chart of percentage of Aiv_VTA cells positive for VGLUT2, VGAT, both, and neither (gray). (B) Upper left, schematic of injection of CTB into VTA to label VP_VTA neurons. Upper right, VP_VTA projecting neurons (teal), VGLUT2 (green), and VGAT (red), scale bar 50 μm. Bottom left, higher magnification views, all scale bars 10 μm. Bottom right, pie chart of percentage of VP_VTA cells positive for VGLUT2, VGAT, both, and neither. (C) Left, viral injection of GFP into Aiv to express in Aiv_VTA terminals. Top middle and right, PV+ (red) and TH+ neurons (pink) respectively with Aiv terminals (green) in VTA, scale bar 50 μm. Bottom middle and right, higher magnification view showing Aiv terminals in apposition to PV+ (middle, red) and TH+ (right, pink) cell bodies, scale bar 10 μm. (D) Same as (C) but for VP_VTA GFP terminals. (E) Top, bar graph of appositions from Aiv_VTA GFP terminals on PV+ (red) and TH+ (pink) cells in the VTA. Bottom, same as (E) but for VP_VTA GFP terminals onto VTA neurons. (F) Upper Left, schematic of injection of CTB into VTA and VP to retrogradely label Aiv_VTA neurons and Aiv_VP neurons. Bottom left, pie chart of overlap of Aiv_VP neurons with Aiv_VTA neurons. Top right, Aiv_VTA (red), Aiv_VP (green), and co-localized (yellow) neurons, scale bar 200 μm, sagittal view with the song nucleus RA outlined. Bottom right, higher magnification view, scale bar 50 μm.

A related issue is the anatomical organization of Aiv_VTA and VP_VTA axon terminals relative to GABA- and DA-releasing neurons in the VTA. To address this issue, we injected AAV-GFP into the Aiv or VP, resulting in GFP expression Aiv_VTA and VP_VTA axon terminals, and then immunostained brain sections containing the VTA for Parvalbumin (PV), a marker of GABA-releasing VTA neurons, and for Tyrosine Hydroxylase (TH), a synthetic enzyme expressed in DA-releasing cells (n = 6 hemispheres from 3 adult male zebra finches for Aiv_VTA terminals; n = 4 hemispheres from 2 adult male zebra finches for VP_VTA terminals). This approach revealed that Aiv_VTA axons in the VTA were studded with varicosities characteristic of en passant synapses, which could be found in close apposition to PV+ cell bodies and also some TH+ cell bodies (Figure 7C). Using a similar approach, we found that VP axons form a robust investment of GFP-labeled terminals encircling PV+ but not TH+ cell bodies in the VTA (Figure 7D), a feature characteristic of inhibitory calyceal synapses that pallidal neurons in Area X make with their postsynaptic targets in the thalamus (Luo and Perkel, 1999a, Luo and Perkel, 1999b). Quantification revealed a significant bias in the close appositions of Aiv_VTA and VP_VTA terminals onto PV+ neurons relative to their close appositions to TH+ cells (Figure 7E; Aiv_VTA: n = 6 hemispheres, n = 3 birds, p = 0.0040 paired t-test; VP_VTA: n = 5 hemispheres, n = 3 birds, p = 0.000029, see Methods). Along with the electrophysiological recordings made in the VTA, these anatomical findings advance a circuit model in which inhibitory interneurons in the VTA invert the sign of excitatory inputs from Aiv and inhibitory inputs from VP to bi-directionally modulate the activity of VTA dopamine neurons (Figure 8 and Figure S8).

Figure 8: Circuit enabling vocal learning in the songbird.

(A) Diagram of zebra finch brain with brain nuclei involved in vocal learning. (B) Schematic of circuit for vocal learning, informed by the present study.

Finally, because Aiv provides input to VP as well as the VTA, a remaining issue is the extent to which Aiv and VP provide independent sources of evaluative information to the VTA. For example, one possibility is that all Aiv_VTA neurons also project to the VP, with the consequence that VP could simply invert the sign of the error detection signal transmitted by Aiv_VTA neurons. Therefore, we used dual retrograde tracer methods to characterize the Aiv neurons that project to the VTA and the VP (Figure 7F). These tracing experiments revealed that Aiv_VTA neurons and Aiv_VP neurons are largely non-overlapping (Figure 7F), consistent with the idea that Aiv and VP can transmit independent signals to the VTA.

Discussion

The current study establishes that evaluative and premotor functions for pitch learning are mediated by discrete and anatomically distinct circuit nodes within the songbird’s brain. A combination of behavioral, functional, and anatomical studies supports a model in which evaluative signals from Aiv and VP are routed through a local inhibitory network in the VTA to enable dopaminergic output neurons to encode performance quality. The outcome of this evaluation is then transmitted to downstream cortical-basal ganglia circuitry that serves a premotor function to implement changes to vocal production (Figure 8).

To our knowledge, the combination of noise-evoked pitch learning and temporally precise optogenetic stimulation of specific circuit nodes provides the first causal evidence showing how evaluative and premotor components that enable vocal learning are arrayed along the VTA – CBG axis in the songbird’s brain. Specifically, we found that noise-evoked pitch learning requires normal patterns of activity in Aiv and VP inputs to the VTA during an evaluative window immediately following the utterance of a target syllable, but not during a premotor window merely 50 milliseconds earlier, just prior to syllable onset. This temporally precise requirement for normal patterns of activity in these two pathways coincident with the processing of syllable-related auditory feedback advances these two pathways as primary components of the evaluator. This observation extends prior correlative studies showing that Aiv_VTA neurons and upstream auditory nuclei can detect vocal errors, by showing that their activity, as well as that of VP_VTA neurons, is causally linked to vocal learning specifically during performance evaluation (Mandelblat-Cerf et al., 2014, Keller and Hahnloser, 2009, Chen et al., 2019). Taken together, these findings show that pitch learning depends on auditory feedback-dependent variation in the activity of these two pathways across renditions of the target syllable. Indeed, here we found that minimizing rendition-by-rendition variation in feedback-related activity in either pathway by optogenetic “interference” strongly reduces noise-evoked pitch learning. Conversely, pitch-contingent optogenetic stimulation of either pathway applied during the auditory feedback window can effectively drive pitch learning, but in opposite directions, with Aiv stimulation negatively reinforcing pitch and VP stimulation positively reinforcing pitch. These findings underscore that the VTA integrates multiple sources of evaluative information about vocal performance to generate a bidirectional reinforcement signal for vocal learning.

The current findings also causally link variation in LMAN premotor activity and pitch learning, while excluding an evaluative role for this nucleus, as proposed in some song learning models (Achiro et al., 2017, Doya and Sejnowski, 1995, Doupe and Konishi, 1991). Various lines of evidence indicate that LMAN provides premotor signals that can contribute to vocal learning by providing acute vocal variability that enables motor exploration and also by biasing this variability adaptively to drive the pitch of the target syllable away from the frequency region targeted by noise (Olveczky et al., 2005, Kao et al., 2005, Andalman and Fee, 2009, Warren et al., 2011, Charlesworth et al., 2012). Notably, pharmacologically silencing LMAN synaptic activity in the song motor nucleus RA reduces song variability but fails to prevent noise-evoked pitch learning, whereas inactivating LMAN cell bodies reduces acute song variability and also prevents pitch learning (Charlesworth et al., 2012, Andalman and Fee, 2009, Warren et al., 2011, Kao et al., 2005). This dissociation, along with the demonstrated necessity of dopamine signaling in Area X in pitch learning (Hisey et al., 2018), suggests that pitch learning involves the integration within the basal ganglia of activity transmitted by LMAN axon collaterals and dopamine from the VTA. Indeed, an emerging idea is that LMAN contributes to pitch learning by sending premotor signals to RA that drive song variability and transmitting a copy of this variability signal to the basal ganglia (Fee and Goldberg, 2011, Charlesworth et al., 2012). In this model, LMAN activity patterns that are associated with successful syllable renditions (i.e., those that escape noise) are selectively reinforced in the basal ganglia by dopamine signals from the VTA, which over tens or hundreds of renditions manifests as an adaptive motor bias signal. The finding that optogenetically interfering with variations in LMAN activity during the premotor but not auditory feedback window blocks pitch learning firmly anchors LMAN in the premotor circuitry, in distinction to the evaluative circuitry comprising the VTA and its afferents, including Aiv and VP.

A major research goal is to understand how the VTA integrates information to enable the computations that ultimately reinforce behavior (Watabe-Uchida et al., 2017, Cohen et al., 2012). Recent studies have established that dopamine release from VTA terminals in Area X is necessary and sufficient to drive pitch learning in adult zebra finches (Hisey et al., 2018, Xiao et al., 2018, Hoffmann et al., 2016). The current study extends these findings by showing that two major inputs to the VTA – Aiv and VP – can also drive pitch learning. Moreover, as VTA_X neurons in zebra finches can encode both negative and positive reinforcement signals (Gadagkar et al., 2016), our finding that Aiv and VP can negatively and positively reinforce target syllable pitch, respectively, provides a circuit mechanism by which the VTA can generate bidirectional reinforcement signals. Such opponent mechanisms are of increasing significance in studies of afferents to the mammalian VTA that are involved in valence processing, motivation, reward, and reinforcement learning (Tye, 2018, Faget et al., 2018, Yang et al., 2018, Lammel et al., 2012). Although our understanding of the functional organization of the songbird VTA remains relatively primitive, we found that a relatively simple input-output organization could account for much of the role of VTA inputs in enabling pitch learning. Specifically, a glutamatergic projection from Aiv to the VTA excites GABAergic interneurons and drives feedforward inhibition that suppresses dopamine-releasing cells. Conversely, a GABAergic projection from VP to the VTA suppresses the activity of GABAergic neurons, resulting in the disinhibition of dopamine-releasing cells. This afferent organization in the songbird VTA is reminiscent of the feedforward inhibitory and disinhibitory circuits described in the mammalian VTA that can negatively or positively reinforce behavior (Nieh et al., 2016, Faget et al., 2018, Yang et al., 2018). In the songbird, this combination of feedforward inhibition and disinhibition of DA-releasing VTA neurons could provide a relatively straightforward circuit mechanism for encoding the quality of vocal performance and adaptively reinforcing syllable pitch.

Beyond localizing evaluative and premotor circuitry necessary for pitch learning, our study also provides insight into how Aiv and VP enable performance evaluation and behavioral reinforcement. For instance, our findings suggest that neither Aiv_VTA or VP_VTA acting alone can likely account for pitch learning, because absolute rates of pitch learning elicited by pitch-contingent optogenetic stimulation of Aiv_VTA or VP_VTA terminals (current study) were approximately half that induced by exposure to pitch-contingent noise (Hisey et al., 2018). One interpretation of these differences is that, during noise-induced pitch learning, those syllable renditions that trigger noise activate Aiv to suppress VTA activity, whereas renditions that escape noise engage VP to elevate VTA activity, thus enabling the animal to learn from every trial regardless of whether performance was “good” or “bad.” In fact, recordings made in singing birds randomly exposed to syllable-triggered noise reveal such bidirectional activity in VTA_X neurons, consistent with a mechanism involving their positive and negative modulation by VP and Aiv, respectively (Gadagkar et al., 2016).

The use of closed-loop optogenetic methods provides temporal and spatial precision but is unlikely to recapitulate the more heterogeneous patterns of endogenous activity in Aiv and VP that occur during pitch learning. Thus, while synergistic activity in these two pathways is likely necessary for full learning rates in response to pitch-contingent noise, the current experiments cannot determine whether Aiv and VP act in a simple all-or-none fashion, with Aiv signaling VTA on renditions that trigger noise and VP signaling VTA when noise is withheld, or instead operate in a parallel and graded manner. Moreover, a limitation inherent to pitch-contingent reinforcement with noise or optogenetic stimulation of VTA circuitry is that performance evaluation is strictly binary (“hit” or “escape”) and based on a single feature (pitch). In contrast, adults that maintain stable song with auditory feedback likely evaluate many different acoustic features of a syllable in parallel. This complicated learning task undoubtedly requires more complex evaluative signals than needed for noise-driven pitch learning. Furthermore, the current study imposed patterns of activity on Aiv and VP, rather than monitoring their activity during learning, and thus cannot distinguish whether Aiv_VTA or VP_VTA neurons only encode negative or positive reward, or instead compute expectation or reward prediction errors, both of which figure prominently in temporal difference learning models (Sutton and Barto, 1998, Watabe-Uchida et al., 2017). Thus, an important goal of future studies will be to longitudinally monitor Aiv_VTA or VP_VTA neurons during natural learning to determine whether they operate in parallel, provide graded evaluative signals, simply encode reward, or also transmit more complex signals to the VTA, such as expectation or prediction error.

The VTA and SNc are remarkably well conserved, with origins prior to the avian-mammalian branch point 300 million years ago (Reiner, 2010, Reiner et al., 2004). Therefore, a further elucidation of the functional organization of evaluative afferents to the songbird VTA is likely to provide broader insights into the computational principles employed by the vertebrate brain to support the learning of complex behaviors. Particularly, Aiv and VP in songbirds are likely homologues of frontal cortical and ventral pallidal structures in mammals (Reiner et al., 2004, Dugas-Ford et al., 2012, Reiner et al., 1998, Puelles et al., 2000). Similar to their mammalian homologues, Aiv and VP are distinguished in part by non-overlapping afferents: Aiv receives input primarily from higher levels of the auditory cortex; in contrast, VP receives input from the VTA, as well as a subset of Aiv neurons partly distinct from those that project to the VTA (current observations as well as (Mandelblat-Cerf et al., 2014, Gale et al., 2008)). Thus, although Aiv and VP presumably have access to information about singing-related auditory feedback, the VP also has access to reinforcement signals from the VTA. This dichotomy is reminiscent of “fixed” versus “adaptive” critics that are fundamental to temporal difference learning models (Barto, 1995, Sutton and Barto, 1998, Takahashi et al., 2008). A noteworthy feature of the “adaptive” critic is that the same reinforcement signal that helps to train the “actor,” which learns what actions to take, can also update the evaluative “critic,” allowing it to better evaluate the actor’s performance. This adaptive quality may be especially important to behaviors that change dramatically with learning, as when juvenile songbirds copy the song of a tutor (Eales, 1985, Tchernichovski et al., 2001, Deregnaucourt et al., 2005, Konishi, 1965). Because the VTA is essential to song copying as well as pitch learning (Hoffmann et al., 2016, Hisey et al., 2018, Xiao et al., 2018), further analysis of how Aiv and VP participate in song copying should prove fruitful for understanding how the VTA enables complex imitative learning in the absence of external reinforcement. Indeed, our study suggests how an ancient, conserved circuit that likely evolved to reinforce relatively simple behaviors can enable virtuosic vocal performance.

STAR * Methods

LEAD CONTACT AND MATERIALS AVAILABILITY

This study did not generate new unique reagents. Further information and requests for resources should be directed to and will be fulfilled by the Lead Contact, Richard Mooney (mooney@neuro.duke.edu)

EXPERIMENTAL MODEL AND SUBJECT DETAILS

Male zebra finches (61–531 dph) were obtained from the Mooney lab breeding colony within the Duke University Medical Center animal facility. Experimental procedures were conducted in accordance with the National Institutes of Health guidelines and were reviewed by the Duke University Medical Center Animal Care and Use Committee. Viral vectors were acquired from University of Pennsylvania Vector Core, University of North Carolina - Chapel Hill Vector Core, AddGene, or custom made in the laboratory.

METHODS DETAILS

Pitch contingent noise experiments

All birds in pitch contingent noise experiments were subsequently used for optogenetic “jamming” experiments described below. Birds were recorded and a template to detect the frequency (i.e., pitch) of a tonal syllable was made in a custom software program (EvTAF, Tumer and Brainard, 2007). Supplementary scripts to generate and evaluate template performance were made in Matlab. Once the bird sang, we created a template that detected no less than 75% of the renditions of the targeted syllable with no more than a 5 millisecond jitter in detection onset. Then we set a threshold at to target renditions sung lower than the 70^th percentile of the target syllable’s pitch distribution with a 50 millisecond burst of white noise (WN). The bird’s pitch for the target syllable was measured in the morning and early evening and changes in the pitch were quantified by comparing the mean and standard deviation of the first 50 syllables sung in the morning and last 50 syllables sung in the evening.

Optogenetic interference experiments

Young adult male zebra finches (61–195 dph) from the colony were screened for producing multiple syllables with clear tonal components and for the amount of song produced in a day. They were then either injected in Aiv, VP, or LMAN depending on brain area under investigation. One of the birds used in Aiv_VTA jamming experiments was previously used for the pitch-contingent optogenetic experiments described below. For Aiv and VP interference experiments, birds were bilaterally injected in the Aiv or VP using stereotactic coordinates (Aiv coordinates: head angle 35 degrees, −0.6 mm rostral, 2.8 lateral, and 2.6 ventral and VP coordinates: head angle 35 degrees, 2.4 mm rostral, 1.3 mm lateral, 4.5 mm ventral) with a virus containing a channelrhodopsin construct (2/1.AAV-CAG-ChR2-mCherry or 2/9.AAV-CAG-ChR2-YFP-neurexin). After waiting a minimum of 3 months for viral expression to express, birds were anesthetized and placed in a stereotaxic apparatus and craniotomies were made over VTA bilaterally. A subset of the birds used for these experiments were tested for terminal field optogenetic responses in VTA with a tungsten electrode, impendence: 500 kOhm to 1.5 MOhm, (MicroProbes Inc.) coupled to a fiberoptic cable (ThorLabs, 200um diameter core) through which 20–100 millisecond pulses of light were delivered while neural activity was simultaneously recorded (Differential A-C Amplifier 1700, A-M Systems). All birds were then implanted bilaterally over VTA with fiberoptic ferrules (Kientec FZI-LC-230, 200 um core) at coordinates: 37-degree head angle, 1.65 mm rostral; 1.0 mm lateral, 6.1 mm ventral). Craniotomies were then sealed with Kwik-Sil (World Precision Instruments) and ferrules were secured in place with Metabond (Parkell) and then reinforced with a final layer of VetBond (3M). After birds recovered from anesthesia under a heat lamp, fiberoptic cables (ThorLabs, 200 um core, 0.37 NA) were connected to the newly implanted ferrules with ferrule sleeves (Precision Fiber Products). The other end of the fiberoptic cables were attached to a two-channel optical commutator (FRJ 1×2i FC-2FC, Doric) or a one channel optical commutator (FRJ 1×1 FC-FC, Doric). The commutator was then connected by an optical patch cord (ThorLabs) to a DPSS laser (Shanghai Lasers or IkeCool). Laser power at the output of the ferrules was adjusted to be in the range of 5–15 mW, tested with a laser power meter (Spectra Physics). We let the bird recover from surgery in a sound box where their vocalizations were monitored. For LMAN, we first found LMAN by stereotaxic coordinates (head angle: 50 degrees, 4.9 mm rostral, 1.85 mm lateral, 2.0 mm ventral) and tested if we were in the proper site by with a single microelectrode (Carbostar-1, Kation Scientific), recording and listening to ongoing spontaneous neural activity to identify the characteristic tonic with intermittent bursting activity that characterize this brain region in the anesthetized state. Once the location of LMAN was confirmed we injected at this site a virus containing a channelrhodopsin construct (2/9.AAV-CAG-ChR2-mCherry). After waiting a minimum of 1 month for viral expression, birds were anesthetized and placed in a stereotaxic apparatus and craniotomies were made over LMAN bilaterally. A subset of the birds used for these experiments were tested for local optogenetic responses in LMAN with a tungsten electrode, impendence: 500 kOhm to 1.5 MOhm, (MicroProbes Inc.) coupled to a fiberoptic cable (ThorLabs, 200um diameter core) through which 20–100 millisecond pulses of light were delivered while neural activity was simultaneously recorded (Differential A-C Amplifier 1700, A-M Systems). All birds were then implanted bilaterally over LMAN with fiberoptic ferrules (Kientec FZI-LC-230, 200 um core) above LMAN using the stereotaxic and recording strategy described above.

Once the birds recovered we first identified two syllables in the song for targeting. We created a template and measured the baseline variation in the later target syllable’s pitch and determined a threshold such that pitch variants falling below the threshold triggered a brief (50 ms) white noise burst (70 dB) through a nearby speaker to the bird whenever the program template detected that the pitch of the targeted syllable was below this threshold; over hours, this manipulation results in an adaptive shift in the pitch of the target syllable (threshold set at the 70th percentile of the pitch distribution). An earlier syllable was then identified, an independent template to detect this syllable was created. When detected, a brief fixed delay was applied to drive optogenetic stimulation on all detected renditions of the pre targeted syllable at a specified time relative to the syllable targeted for noise. This allowed the precise placement of temporally restricted circuit disruption. Both templates were designed to detect no less than 75% of the renditions of the targeted syllable with no more than a 5 millisecond jitter in detection onset. A 50 millisecond jamming laser pulse was used to blanket a window in one of two periods: either in a time window where premotor production or auditory feedback evaluation are occurring. The laser was delivered on 100% of detected trials, absolute hit rates for premotor stimulation in an example bird was 91.8% and same bird for auditory feedback jamming was 92% (Figure S4). These windows were determined with reference to the literature on known auditory (Troyer and Doupe, 2000a, Troyer and Doupe, 2000b, Margoliash and Fortune, 1992, Sakata and Brainard, 2008, Lei and Mooney, 2010) and premotor (Kao et al., 2005, Giret et al., 2014) latencies described in the songbird. The mean onset of auditory feedback optogenetic stimulation was targeted to be 10 ms after the mean noise onset; where the known auditory latency in the higher motor structure HVC is at a minimum 15 ms. Further, Aiv neurons have been shown to fire in response to noise with latencies predominantly between 10–20 ms and VP neurons have been shown to respond to noise within tens of milliseconds (Mandelblat-Cerf et al., 2014, Gale and Perkel, 2010). The mean onset of premotor optogenetic stimulation was targeted to be about 50 ms before the start of the targeted syllable. This is motivated by investigation of premotor latencies for LMAN which begin around 25–35 ms (Giret et al., 2014, Kao et al., 2005). We examined baseline changes in song over a day and three learning conditions, in all learning conditions the later syllable targeting was consistent with the threshold set based on the 70th percentile of syllables from the preceding day. These three conditions include: learning with no laser stimulation, learning with optogenetic interference during a premotor period, and lastly learning with optogenetic jamming during an auditory feedback period. The bird’s pitch for the targeted syllable was measured in the morning and early evening and changes in the pitch were quantified by comparing the mean and standard deviation of the first 50 syllables sung in the morning and last 50 syllables sung in the evening. After experiments were finished, histology was performed in Aiv and VP birds to visualize the VTA with alternate sections stained against mCherry (Abcam) or GFP (Abcam) and TH (Milipore, Invitrogen) and for LMAN birds with antibodies against mCherry and Calbindin which labels thalamic terminals in LMAN(Pinaud et al., 2007).

Pitch contingent closed loop optogenetic experiments

Young adult male birds (62–95 dph) were bilaterally injected in the Aiv or VP using stereotactic coordinates (Aiv coordinates: head angle 35 degrees, −0.6 mm rostral, 2.8 lateral, and 2.6 ventral and VP coordinates: head angle 35 degrees, 2.4 mm rostral, 1.3 mm lateral, 4.5 mm ventral) with a virus containing a channelrhodopsin construct (2/1.AAV-CAG-ChR2-mCherry or 2/9.AAV-CAG-ChR2-YFP-neurexin). After waiting a minimum of 3 months for viral expression to express, birds were anesthetized and placed in a stereotaxic apparatus and craniotomies were made over VTA bilaterally. A subset of the birds used for these experiments were tested for terminal field optogenetic responses in VTA with a tungsten electrode, impendence: 500 kOhm to 1.5 MOhm, (MicroProbes Inc.) coupled to a fiberoptic cable (ThorLabs, 200um diameter core) through which 20–100 millisecond pulses of light were delivered while neural activity was simultaneously recorded (Differential A-C Amplifier 1700, A-M Systems). All birds were then implanted bilaterally over VTA with fiberoptic ferrules (Kientec FZI-LC-230, 200 um core) at coordinates: 37-degree head angle, 1.65 mm rostral; 1.0 mm lateral, 6.1 mm ventral). Craniotomies were then sealed with Kwik-Sil (World Precision Instruments) and ferrules were secured in place with Metabond (Parkell) and then reinforced with a final layer of VetBond (3M). After birds recovered from anesthesia under a heat lamp, fiberoptic cables (ThorLabs, 200 um core, 0.37 NA) were connected to the newly implanted ferrules with ferrule sleeves (Precision Fiber Products). The other end of the fiberoptic cables were attached to a two-channel optical commutator (FRJ 1×2i FC-2FC, Doric) or a one channel optical commutator (FRJ 1×1 FC-FC, Doric). The commutator was then connected by an optical patch cord (ThorLabs) to a DPSS laser (Shanghai Lasers or IkeCool). Laser power at the output of the ferrules was adjusted to be in the range of 5–15 mW, tested with a laser power meter (Spectra Physics). We let the bird recover from surgery in a sound box where their vocalizations were monitored. After the birds recovered from surgery and began singing readily at a stable rate, their songs were recorded and a template to detect the frequency (i.e., pitch) of a tonal syllable was made in a custom software program (EvTAF, Tumer and Brainard, 2007). Supplementary scripts to generate and evaluate template performance were made in Matlab. First, we recorded two days of “baseline” song production with the laser off. From these songs, we created a template that detected no less than 75% of the renditions of the targeted syllable with no more than a 5 millisecond jitter in detection onset. After the second baseline day, a threshold at the upper (or lower) 60–70^th percentile of the target syllable’s pitch distribution was set and a 50 millisecond pulse of blue light (473 nm, 5–15 mW emitted at each ferrule) was delivered to VTA whenever the program detected that the pitch of the targeted syllable was below (or above) this threshold. To assess the change in the song independent of any possible acute effects of laser stimulation, a random 5% of all trials were “catch” trials in which regardless of the syllable pitch, no stimulation was triggered. The pitch of the targeted syllable on “catch” trials was measured every day for the next four days. Light stimulation was then ended at the end of the fourth day of pitch contingent laser stimulation and song in the absence of stimulation was recorded for up to four days. After all experiments were finished, histology was performed to visualize the VTA with alternate sections stained against mCherry (Abcam) or GFP (Abcam) and TH (Milipore, Invitrogen).

Functional Electrophysiology

Young adult male zebra finches (42–183 dph) were injected with virus into Aiv and VP as described above. Custom built optrodes were fabricated by cutting fire polished borosilicate glass to length (Sutter Instruments O.D.: 1.5 mm, I.D.: 0.86 mm, 10 cm length) and inserting a tungsten microelectrode (impedance: 0.1 – 1.5 MOhms) (Microprobes) and stripped optical fiber (ThorLabs, 200 um core, 0.37 NA) on one end with a FC/PC connector on the other end were inserted adjacent with the tip of the electrode ~0.5 mm below the optical fiber. These were then glued together with Loctite (Henkel) and Metabond (Parkell). Songbirds were then acutely anesthetized with isoflurane and a small craniotomy was made 6 mm laterally where a Ag/Cl ground pellet (A-M Systems) soldered to a connector pin (A-M Systems) was implanted with Kwik-Sil (WPI) over craniotomy and stabilized with Metabond. Then the Y sinus was uncovered and a craniotomy over VTA was performed (coordinates: head angle - 37 degrees, 1.65 mm rostral, 0.5 mm lateral). Next, the optrode was lowered into the VTA and units were amplified (A-M Systems Model 1700) and recorded with a custom written Labview (National Instruments) program that also allowed delivery of custom pulse trains (pulse widths = 50–100ms, interpulse intervals = 50–500 ms) which drove a ChR2 laser (BL473T3–100, Shanghai Lasers, wavelength = 473 nm). Power was calibrated to deliver between 5–15 mW in the VTA. Electrode signals were filtered (300 to 10,000 Hz) and action potentials from individual neurons were sorted offline based on visualization of the action potential waveform and principal component analysis of the waveform using custom Matlab software (PostHawk, Credit: David Schneider). Peristimulus time histograms (PSTHs) ware produced and laser stimulation evoked action potential responses were examined for each neuron. Significant modulations of activity by optogenetic stimulation were defined by a p value less than 0.05 in a paired t test of spikes per trial during laser stimulation and an equal time interval immediately preceding the onset of stimulation. Neurons demonstrating statistically significant modulations were then further processed and their baseline firing rate was calculated as the total number of spikes in all recorded trials divided by time recorded. Further, the spike width was characterized by identifying the first peak of the spike waveform and measuring the distance from the time the spike took to reach 10% of the height of this first peak to when the mean spike waveform (+/− 0.5 SDs) was no longer significantly different from zero for at least 500 microseconds.

Fluorescence in situ hybridization

cDNA fragments used for synthesizing the in situ hybridization probes were commercially synthesized (GenScript). Sequences were designed using NCBI cDNA database as a reference and listed in Supplementary Table 1. In vitro transcription was performed to synthesize the antisense probes using T7 RNA polymerase (Roche). Young adult male zebra finches were head-fixed on stereotaxic apparatus and Cholera Toxin Subunit B Alexa Fluor 647 (Thermofisher, C34778) was injected bilaterally into VTA (coordinates measured from y-sinus: head angle 37°, 1.65 mm anterior; 0.5, 1, and 1.5 mm lateral; 6.2 mm ventral). One week after CTB647 injection, fresh frozen brains were prepared and consecutive sagittal sections (12μm thickness) were collected with a Leica CM 1850 cryostat and mounted on slides (FisherBand). Slides were dried and stored at −80°C before use. For two-color fluorescence in situ hybridization, digoxigenin (DIG)-labeled VGLUT2 and Fluorescein isothiocyanate (FITC)-labeled VGAT probes were used. VGLUT2-DIG probes was diluted 1:500 and VGAT-FITC probes were diluted 1:2000 into hybridization buffer (300 mM NaCl, 20 mM Tris-HCl [pH 8.0], 5mM EDTA, 10 mM Na₂HPO₄l [pH 7.2], 10% dextran sulfate, 1× Denhardt’s solution, 0.5mg/ml tRNA, 0.2mg/ml HS-DNA, 50% deionized formamide). Hybridization was performed at 65 °C for 12–18h. Slides were washed in lab made 5X Saline-Sodium Citrate (SSC) once at 68 °C, once in 0.2X SSC at 68 °C, and once in 0.2X SSC at room temperature. VGLUT2-DIG probes were detected with anti-DIG-AP (Sigma #11093274910) followed by FastRed treatment (Sigma# F4648). VGAT-FITC probes were detected with anti-FITC-POD (Sigma# 11426346910), followed by lab made FITC-TSA.

All the sections with retrogradely labeled Aiv-VTA and VP-VTA neurons were imaged on a Zeiss 710 confocal microscope and saved for further analysis. To quantify Aiv-VTA cells, one image was taken for each brain section containing Aiv-VTA labeled cells. As VP retrogradely labeled cells were sparser, multiple non-overlapping images were taken for each brain section containing VP-VTA cells to increase the sample size. CTB647 labeled Aiv_VTA or VP_VTA neurons were semi-automatically quantified using Image J multi-point tool (ImageJ). Each retrogradely cell was determined to be in one of four categories: VGAT positive, VGLUT2 positive, VGAT and VGLUT2 double positive, and double negative.

Immunostaining

Birds were deeply anesthetized with intramuscular injection of 20μl Euthasol (Virbac), and transcardially perfused with 0.025 M phosphate-buffered saline (PBS) followed by 4% paraformaldehyde (PFA). Brains were removed, post-fixed in 4% PFA at 4 °C overnight and moved to cryoprotective 30% sucrose PFA solution for two days. Brainstem and thalamus including the midbrain was separated from the forebrain. Frozen consecutive coronal sections of midbrain or sagittal sections of forebrain (thickness of 40μm) were collected with a sledge microtome (Reichert). For immunostaining, free-floating sections were washed 5 minutes three times in 0.3% Triton X-100 in PBS (PBST), blocked with 10% Blocking One (Nacalai Tesque) in PBST for 1 hour, and incubated with primary antibodies in PBST at 4 °C overnight. Sections were then washed for five minutes three times in PBST and incubated with secondary antibodies at room temperature for 4 hours, followed by three 5 minute washes in PBST. Sections were coverslipped with Fluoromount-G (SouthernBiotech), and then imaged with a confocal microscope (Zeiss 710). For identification of mCherry and/or TH signals in Figure 2 and 3, the combination of antibodies was used as following: rabbit anti-mCherry (1:500, abcam Ab167453) and goat anti-rabbit AlexaFluro594 (1:500, Invitrogen A11012), mouse anti-TH (1:500, Invitrogen MA1–24654) and goat anti-mouse AF488 (1:500, Invitrogen A11001). For identification of TH, PV, and GFP in Figure 7, the following combination of antibodies was used: rabbit anti-PV (1:500, abcam Ab11427) and goat anti-rabbit AF594 (1:500, Invitrogen A11037), chicken anti-GFP (1:500, abcam Ab13970) and donkey anti-chicken AF488 (1:500, Jackson ImmunoResearch 703–545-155), mouse anti-TH (1:500, Invitrogen MA1–24654) and goat anti-mouse AF405 (1:200, Invitrogen A31556).

Apposition Quantification

Coronal sections of the VTA of adult male zebra finches with viral injection into either Aiv (n = 3 birds) or VP (n = 3 birds) were collected and antibody staining against TH and PV were performed as described earlier. The boundary region of VTA/SNc was identified by zone of TH positive cells in the midbrain medial to from midline to 1500 μm laterally. Within this target area, z-stack images were taken at with 0.5 μm interval for 20 μm total stack. The middle 10 μm were stacked to one image by maximum projection. First, TH+ and PV+ neurons were quantified in a semiautomated way (ImageJ, multi-point tool). Next, for each of these cells, cells with significant overlap of GFP-labeled terminals at the boundary of the target cell body were counted.

Aiv_VTA, Aiv_VP quantification

Two different color CTB tracers were injected in VTA and VP in adult male zebra finches (n = 2 birds). A region of interest was identified by the presence of Aiv_VTA cells found ventral and anterior to RA, arcopallial neurons within this selected region were then considered for further analysis. The separate channels of tracer labeled Aiv_VTA and Aiv_VP were split and analyzed individually. The total number of Aiv_VTA and Aiv_VP neurons were quantified automatically (ImageJ, Analyze Particles). Masks were created containing the cells selected and then merged for colocalization quantification. The colocalized Aiv_VTA and Aiv_VP cells were quantified in a semiautomated way (ImageJ, multi-point tool). Graphical pie chart was made in Matlab.

QUANTIFICATION AND STATISTICAL ANALYSIS

Data are presented as mean ± standard error of the mean (s.e.m.) unless otherwise noted. Error bars in all figures indicate the standard error of the mean. Groups with fewer than 8 samples were not large enough to detect normality, but parametric tests were still used in order to detect differences in small samples. P values were calculated from two-tailed t-tests between only two groups and listed in the figure legends. For groups of more than two, ANOVAs were performed first before t-tests were performed. P values of 0.05 or below were considered significant. Star values as: *p < 0.05; **p < 0.01; ***p < 0.001. The experimenters were not blinded to allocation of subjects and allocation of subjects was not randomized. Automatic detection and calculation of syllable frequencies allowed the experimenter to be blind to conditions before and after viral expression.

All data were analyzed with Python (Python Software Foundation, https://www.python.org/) and Matlab (MATLAB, The MathWorks, Inc., Natick, Massachusetts, United States. http://www.mathworks.com/) software.

Analysis of song data

Change in auROC

To evaluate shifts in syllable rendition pitch distributions after closed loop learning perturbations, change in the area under the Receiver Operator Characteristics (ΔauROC) of the pitch of the syllable targeted for learning was determined for channelrhodopsin pitch learning experiments. First, the pitch of the entire tonal component of the syllable was measured on “catch” syllables on the day before the manipulation began and then again on the last day of the manipulation. Then, all the pitch values produced on “catch” trials on of these days were each z-scored, $z=\frac{x-\mu }{\sigma }$. Then, the ΔauROC was calculated as the integral of the proportion of baseline pitches correctly considered baseline pitches and the proportion of experimental pitches incorrectly considered baseline pitches. Bootstrapping of 10,000 random permutations was employed to create an estimate of the mean.

Percent change in pitch

The percent change in pitch of the targeted syllable with learning was calculated by measuring the pitch of the entire small stable component of the target syllable on the first 50 syllables produced on the learning day with the same procedure was performed on the last 50 syllables produced that learning day. The pitch of the last 50 syllables was subtracted from the baseline pitch, divided by the baseline pitch, and then multiplied by 100 to calculate the percent change in pitch to quantify learning.

Coefficient of variation

The pitch of a small stable component of the syllable was measured and the mean pitch of these syllable renditions was then divided by the standard deviation of the same syllable components, $CV=\frac{\sigma }{\mu }$.

DATA AND CODE AVAILABILITY

The datasets and code supporting the current study have not been deposited in a public repository because of size, but are available from the corresponding author on request.

ADDITIONAL RESOURCES

None.

KEY RESOURCES TABLE.

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Antibodies
mouse anti-GFP	Invitrogen	A-11122
mouse anti-TH	Invitrogen	MA1–24654
rabbit anti-mCherry	Abcam	AB167453
rabbit anti-PV	Abcam	AB11427
rabbit anti-TH	Millipore	AB152
chicken anti-GFP	Abcam	AB13970
donkey anti-chicken	Jackson ImmunoResearch	703–545–155
goat anti-rabbit	Invitrogen	A31556, A11037
goat anti-mouse	Invitrogen	A11001
Bacterial and Virus Strains
AAV1-CAG-ChR2(H134R)-mCherry	Penn Vector Core, This manuscript	Addgene 100054-AAV1
AAV9-CAG-NRXN-ChR2-YFP	Penn Vector Core, In House
scAAV9-CMV-GFP	UNC Vector Core
AAV9-CAG-ChR2(H134R)-mCherry	Penn Vector Core
Chemicals, Peptides, and Recombinant Proteins
Cholera Toxin Subunit B 647	Invitrogen	Cat# 34778
Dextran, Alexa Fluor 488	Invitrogen	Cat# D22910
C&B Metabond	Parkell	S380
Low Toxicity Silicone Adhesive	World Precision Instruments	KWIK-SIL
Fluoromount-G	Southern Biotech	Cat# 0100–01
Blocking One Histo	Nacalai Tesque	06349–64
Experimental Models: Organisms/Strains
Zebra Finches (Taeniopygia guttata)
Recombinant DNA
Zebra Finch VGLUT2 cDNA	GenScript
Zebra Finch VGAT cDNA	GenScript
Software and Algorithms
MATLAB	Mathworks	https://www.mathworks.com/
Python	Python Software Foundation	https://www.python.org/
Labview	National Instruments	http://www.ni.com/enus/shop/labview.html
Custom-written Labview software (EvTaf)	Michael Brainard Lab
ImageJ	NIH	https://imagej.nih.gov/ij/index.html
Other
Fiber Optic Fiber	Thor Labs	Cat# TS1843490
Fiber Optic Cannula	Kientec Systems	Cat# FSS-LC-330
Fiber Mating Split Sleeves	Precision Fiber Products	SM-C5125S
Heat Curing Epoxy	Precision Fiber Products	PFP-353ND
Optical Patch Cord	Thor Labs	Cat# M72L05
Branching Fiberoptic Patch Cord	Doric	BFP(2)FCM-2×(CL)
Tungsten Microelectrodes	Microprobes	Part# WE3PT30.1B10
Electrode Pellet	A-M Systems	Cat# 550010
Differential Amplifier Model 1700	A-M Systems	Cat# 692000
473 nm Laser	Shanghai Lasers	BL473T3–100
473 nm Laser	IkeCool	IKE-PS-500
Laser Power Meter	Spectra Physics	407A
Two Channel Optical Commutator	Doric	FRJ 1×2 FC-2FC
One Channel Optical Commutator	Doric	FRJ 1×1 FC-FC
Isolated Pulse Stimulator Model 2100	A-M Systems	Cat# 720000
Digital Acquisition Board	National Instruments	BNC-2090

Highlights.

• Two inputs onto songbird midbrain dopamine neurons evaluate song performance.

• Activating these evaluative inputs drives song learning effects of opposite valence.

• Physiology and anatomy reveal midbrain circuitry mediating these opposing effects.

• Downstream cortical neurons serve a premotor role to enable song learning.