Building a state space for song learning

Abstract

The songbird system has shed light on how the brain produces precisely timed behavioral sequences, and how the brain implements reinforcement learning (RL). RL is a powerful strategy for learning what action to produce in each state, but requires a unique representation of the states involved in the task. Songbird RL circuitry is thought to operate using a representation of each moment within song syllables, consistent with the sparse sequential bursting of neurons in premotor cortical nucleus HVC. However, such sparse sequences are not present in very young birds, which sing highly variable syllables of random lengths. Here, we review and expand upon a model for how the songbird brain could construct latent sequences to support RL, in light of new data elucidating connections between HVC and auditory cortical areas. We hypothesize that learning occurs via four distinct plasticity processes: 1) formation of ‘tutor memory’ sequences in auditory areas; 2) formation of appropriately-timed latent HVC sequences, seeded by inputs from auditory areas spontaneously replaying the tutor song; 3) strengthening, during spontaneous replay, of connections from HVC to auditory neurons of corresponding timing in the ‘tutor memory’ sequence, aligning auditory and motor representations for subsequent song evaluation; and 4) strengthening of connections from premotor neurons to motor output neurons that produce the desired sounds, via well-described song RL circuitry.

Learning complex sequential behaviors

Some of the brain’s most fascinating and expressive functions, like music, athletic performance, speech, and thought itself, are learned sequential behaviors that require thousands of repetitions of trial and error learning and observation of others. However, a fundamental challenge is learning how to structure these behaviors into appropriate chunks that can be acquired by trial and error learning. ‘Chunking’ has been highlighted as a central mechanism for learning action sequences in a variety of systems [1–4]. For example, the brain of a young musician does not know a priori the number or durations of melodies it will need to generate. The brain of a young athlete does not know how many maneuvers it will need to practice and eventually perfect. How does the brain flexibly construct motor programs that have the correct temporal state representations that then support trial-and-error learning to achieve complex behavioral goals?

Avian song learning, which shares crucial behavioral, circuit-level and genetic mechanisms with human speech [5–7,8^•,9] (Figure 1a), has provided a rich system for understanding how complex sequential behaviors are produced by the brain, including how they are learned through observation and practice. All songbirds, such as the widely studied zebra finch, learn to imitate the song of a conspecific tutor, typically the father. Juvenile zebra finches start off babbling subsong, then introduce a stereotyped ‘protosyllable’ of ~100 ms duration. New syllables emerge through the differentiation of this protosyllable into multiple syllable types, until the song crystallizes into an adult song composed of 3–7 distinct syllables [10^••]. After several weeks of practice, zebra finches can produce a precise moment-to-moment imitation of their tutor’s song.

Figure 1. Reinforcement learning of sequential behavior, implemented by the songbird brain.

(a) Schematic showing several homologous structures in the songbird (left) and human (right) brains. Unlike the layered mammalian cortex, the songbird ‘cortex’ is organized in pallial fields [95^•], which share molecular markers with mammalian cortical layers [96]. Careful analyses of cell types, projection patterns, and gene expression has led to the view that the songbird brain has homologs to all major parts of the mammalian brain, including cortex, thalamus, basal ganglia, and dopaminergic VTA [8^•,97,98–101]. Within these distinct brain regions lie cell groups (or nuclei) whose primary function is song. (b) The song nuclei connect to each other, forming several pathways that underlie song production and learning: a descending ‘cortical’ motor pathway consisting of premotor and motor output nuclei (HVC and RA respectively) [76,102–104]; a learning pathway consisting of a basal-ganglia-thalamo-cortical loop (Area X, DLM and LMAN respectively)[105–108]; and an auditory pathway that stores a memory of the tutor song [50,51], interacts directly with song areas [60,61], and also interacts indirectly through midbrain reward centers (VTA)[22,23^••,109]. These pathways are thought to implement distinct functions of reinforcement learning [11,12,13^••,14]:

Timing/Motor Pathway Each individual projection neuron in adult HVC bursts at a particular moment in the song, always occurring at the same moment in the song to submillisecond precision [29,30]. Different neurons burst at different times in the song, so that collectively, the population of projection neurons provides a sequence of timestamps that cover the entire song [31^••,32^•]. In this model, HVC drives different ensembles of downstream neurons in RA and the vocal motor nucleus at different moments in the song.

Reinforcement Learning Pathway Song variability is largely driven by nucleus LMAN, a premotor cortical region that projects to the motor output nucleus RA [15^••,16–18]. During learning, song variations generated by LMAN become biased toward successful song variations [25^••,26]. This bias represents the gradient of song performance in motor space, which could be used to shape motor circuitry through learning of HVC→RA synapses [13^••,27]. It has been hypothesized that this gradient is computed using three signals that converge in the song-related basal ganglia, Area X [13^••]. Specifically, local ensembles of medium spiny neurons in Area X receive: an efference copy of LMAN activity that generates song variations, a timing signal from HVC, and an evaluation signal from VTA. These signals allow Area X to determine which song variations, at which times, have led to improved performance, and thus to bias LMAN to produce the same variations at the same time in future song renditions, through a topographically organized BG-thalamo-cortical feedback loop to LMAN [107,110^•].

Evaluation pathway through VTA. Song is evaluated by listening and comparing to a template memory of the tutor song. The evaluation pathway starts in higher-order auditory cortical areas, which contain neurons selective for song errors [21,22]. The output of this evaluation pathway is VTA. Neurons in VTA projecting to Area X convey a song performance prediction error signal used to guide learning [23^••,109].

Song acquisition through reinforcement learning (RL)

The RL framework underlies a predominant view of song learning [11,12,13^••,14]. In recent incarnations of this view, premotor and motor nuclei HVC (proper name) and RA (robust nucleus of the arcopallium) generate a song ‘policy’ — what vocal outputs to produce when. A variability-generating circuit LMAN (lateral magnocellular nucleus of the anterior nidopallium) serves as an ‘actor’ that injects variability into RA to produce variable song outputs [15^••,16–20]. A pathway from auditory circuits through dopaminergic VTA (ventral tegmental area) to a basal ganglia circuit may serve as an ‘evaluator’ that detects which vocal variations successfully matched a memory of the tutor song [21,22,23^••]. Finally, the vocal basal ganglia circuit, which is necessary to learn changes in song acoustics [24], improves song policy by biasing the variability-generating circuit to produce successful variations more often [25^••,26]. This bias drives plasticity in the song motor pathway to consolidate an improved song policy [25^••,26,27]. See Figure 1b, and [13^••] for more extensive discussion of an RL model.

Note that song is a sequential behavior in which errors early in the sequence do not propagate to the rest of the sequence. In contrast with behaviors like navigating a maze or playing a game, where actions early in the sequence can have a profound effect on potential outcomes later in the sequence, one flubbed note need not spoil the whole song. This type of RL, described by Sutton and Barto as ‘contextual bandit’ RL, is simpler than what they term ‘full’ RL, because actions only affect reward, but not future states [28]. Thus, learning can occur independently at each state in time, using a time-dependent reward signal to associate each state with the appropriate action. This form of RL requires as input a representation of the context on which associated actions or emissions are learned.

Consistent with this view, songbird RL models take as input a sequence of contexts represented by a unique timestamp for each moment in the song [12,13^••,14]. This representation is encoded in nucleus HVC of the song motor system. In adult birds, each individual premotor HVC neuron bursts at a single moment in the song. The bursts of each neuron are only 6 ms in duration occurring at the same moment in the song with submillisecond precision. Because different neurons burst at different times [29,30], the population of neurons collectively provides a continuous sequence of bursts that spans the entire song [31^••,32^•]. Each burst in the sequence activates a different ensemble of downstream neurons to generate the appropriate vocal output at that moment. Due to the sparseness of HVC coding, each syllable of the song is generated by a unique sequence of HVC bursts, which behavioral measurements of variability suggest are initiated at the offset of the previous syllable [33^•]. See Box 1 for discussion of the neural mechanisms underlying sparse sequential bursting in HVC. Finally, HVC transmits sparse sequences both to the song motor pathway and to the song-related basal ganglia, which uses this sequential state space representation of song timing to perform RL [13^••,14].

Box 1. Mechanisms underlying HVC sequences.

HVC plays a key role in song timing [76]. Lesions to HVC eliminate all consistently timed vocal gestures, and leave birds babbling subsong [77]. Further evidence that HVC controls stereotyped song timing is that localized cooling of HVC slows song syllables by roughly 3% per degree C of cooling [45,78]. In contrast, cooling the downstream area (RA) does not slow the song beyond what would be expected from the small residual temperature change in HVC (which is several millimeters away) suggesting that the dynamics governing song timing are in HVC, and not RA [78] nor in a loop involving RA (but see [79]).

A prevalent model of how HVC produces precisely timed sequences is the synaptic chain model [30,47,48^••,80,81], which hypothesizes that sequential activity in HVC neurons is due to direct synaptic connections between neurons that burst at successive moments in the song. Preliminary analysis of a small number of burst times seemed to suggest that HVC bursts only occur at certain special moments in the song [82], which would be inconsistent with a synaptic chain model. More recently, analysis of large datasets of HVC neurons has revealed that the network generates a sequence of bursts that spans the entire song with nearly uniform density [31^••,32^•]. Temporal patterning in HVC does not appear to be strongly spatially organized; neurons that burst at similar times appear to be scattered throughout HVC [83]. However, projections to HVC exhibit non-uniform topology, and lesion experiments suggest that medial HVC may play a disproportionate role in controlling transitions between syllables [84,85,86^•].

Inhibitory interneurons within HVC may play a role in governing the timing of HVC projection neurons [87]. In the simplest case, inhibitory feedback may stabilize the propagation of activity through feedforward excitatory chains [30,47,48^••,81]. In addition, interneurons may play a more precise role in patterning projection neuron burst times [88,89]. More recently, the interaction between excitatory and inhibitory neurons in HVC has been investigated using connectomic approaches [90^•], revealing patterns of connectivity between projectionneurons and interneurons consistent with synaptically connected chains of excitatory neurons embedded in a local inhibitory network.

HVC receives inputs from other brain areas that may influence song timing and structure. For example, inputs from cortical area Nucleus Interface (NIf) appear to affect syllable ordering and higher-order song structure [91]. Inputs to HVC from NIf and the thalamic nucleus Uvaformis (Uva) exhibit a peak immediately prior to syllable onsets and a pronounced minimum during gaps between syllables [92^•,93], consistent with a special role of these areas in syllable initiation and timing. However, cooling studies reveal that HVC also plays a role in syllable syntax [94], and inputs to HVC from Uva may also play some role in within-syllable song timing [79].

Here we come to the crux of our problem: Like our young musician, prior to hearing their tutor, juvenile songbirds do not know how many syllables their songs will contain, making it unlikely that HVC could form, prior to tutoring, sequences for each syllable in the song to be acquired. Importantly, recent work suggests that such sequences may not already exist in the earliest stages of song development. This work also suggests that the sequential representation of time underlying RL emerges gradually during song acquisition, perhaps through an unsupervised Hebbian learning process [34^••].

Generating appropriate latent representations on which learning may efficiently operate is an area of active research in the machine learning and learning theory communities. In particular, RL algorithms are known to suffer the ‘curse of dimensionality’, and work poorly with inefficient high-dimensional representations of the state space [35]. However, RL algorithms have achieved impressive human-level performance on a variety of tasks when based on efficient state representations obtained from a separate learning process involving deep learning and artificial neural networks [36,37]. More generally, several recent advances in machine learning involve using interacting networks that each learn via separate processes [38,39]. Furthermore, the use of deep networks, historically difficult to train, experienced a major breakthrough [40] with the application of pre-training using unsupervised processes to achieve generalizable representations [41]. How latent structure learning could be implemented is a key research direction at the interface of biological and machine learning theory [42^•,43^•]. Thus, to the extent that the brain employs RL, it also must model latent structure in the world to build representations that support RL [44].

Learning sequences for song timing

The songbird is an excellent model system to examine how the brain builds a latent state space representation — namely, sequences in HVC. Recent findings suggest that these sequences emerge gradually throughout vocal development. At the earliest stage, only half of HVC projection neurons are reliably locked to song, with bursts clustered near the onsets of subsong syllables [34^••], and lesions of HVC do not affect the babbling subsong [45]. Maturation past subsong requires HVC, and is defined by the emergence of a consistently timed vocal gesture, called a protosyllable [10^••,46]. During the emergence of the protosyllable, HVC appears to grow a protosequence in which bursts span the entire syllable duration [34^••].

How does a single protosequence transform into a different distinct sequence for each syllable in the adult song? At the level of the behavior, it has been observed that protosyllables can gradually differentiate into two syllable types [10^••]. Neural recordings in HVC during this process suggest that early protosequences gradually split to form multiple distinct syllable sequences. Sequence splitting is evidenced by the observation that, while some neurons burst selectively for one or the other emerging syllables, many neurons are active during both. Furthermore, the number of shared neurons decreases significantly at later stages of development. This splitting process repeats as birds differentiate enough new syllable sequences to compose their adult songs, at which point most neurons are syllable-specific, with very few shared neurons [34^••].

Model of the growth and splitting of motor sequences during song development

The activity of HVC at different stages of vocal development is consistent with a model (Figure 2) in which an initially random network assembles into synaptic chains via simple learning mechanisms [34^••]: spike-timing-dependent plasticity (STDP), recurrent inhibition, and synaptic competition — mechanisms previously hypothesized to play a role in HVC sequence formation [47,48^••]. In our model, these synaptic mechanisms transform an initially random network into a feedforward protosyllable chain, under the influence of rhythmic external inputs to a small population of ‘seed’ neurons.

Figure 2. Model of the growth and splitting of neural sequences in HVC.

Schematic of a recurrent network model of HVC development. See [34^••] for more detail, including supplemental code. Neurons are drawn as circles, sorted by when they fire relative to syllable onset. The network is shown at four stages of development — ‘subsong’: before any learning, when weights and input timing are random; ‘protosyllable’: Learning a single protosyllable chain using STDP and synaptic competition under the influence of rhythmic seed inputs; ‘chain splitting (early)’: splitting the protosyllable chain using STDP and increased synaptic competition. Seed neurons are divided into two groups and activated alternately; ‘chain splitting (late)’: at the end of learning. This model results in patterns of activity in HVC similar those observed during development [34^••].

Splitting of the protosyllable chain into multiple daughter chains occurs when the seed neuron inputs are split into multiple groups and activated separately. Synaptic pruning is encouraged by synaptic competition and increased inhibition. Since daughter chains split from a common protosyllable chain, this enables reuse of learning for gestures that are common to all syllables. For example, a critical feature that emerges during the formation of a protosyllable is the coordination of respiration with vocalizations [46]. Such coordination would then be automatically inherited by any daughter syllables that arise from splitting of the protosyllable chain.

A crucial aspect of this model of sequence formation and splitting is the rhythmic patterning of external inputs to ‘seed’ neurons. The ‘seed’ neuron inputs effectively tutor the model network to learn sequences of the proper number and duration. However, the origin of these external inputs is unspecified in our original model. New data elucidating connections between HVC and auditory cortex [49^••] allows us to expand our hypothesis to include a potential role for auditory cortex in seeding HVC sequences.

Does the auditory system shape motor sequences to reflect a tutor memory?

At a computational level, song imitation can be viewed as learning a generative model of the tutor song. Exposure to a tutor song could imprint the desired number of appropriately sized syllable chunks in the auditory system [50,51], which through interaction with HVC could then create an appropriate latent representation of song timing in the form of HVC sequences. More specifically, we propose that auditory cortex may directly influence sequence formation in HVC by appropriately activating seed neurons during development. Consistent with this view, tutor exposure produces rapid overnight changes in the song motor system, including dramatic alterations of song features [10^••], spontaneous activity [52], and spine growth and stabilization [53]. Furthermore, auditory inputs to HVC are gated off during singing itself, consistent with a role for these inputs other than online auditory feedback [54,55].

Four processes for song learning: a hypothesis

Several models have been proposed for how plasticity in the songbird brain gives rise to aspects of song learning [11,12,13^••,14,34^••,47,48^••,56^•,57,58]. We present a new hypothesis that incorporates ideas from these models, but resolves several gaps, particularly in light of new data showing direct links between the nucleus HVC and higher auditory cortical areas [49^••]. We propose that song learning may be implemented by four processes: 1) formation of synaptically connected chains in auditory cortex encoding a memory of the tutor song; 2) replay of ‘tutor memory’ sequences to seed the formation of chains in HVC of the appropriate number and duration; 3) formation, through synchronized replay of HVC and ‘tutor memory’ sequences, of connections from HVC neurons to auditory neurons, supporting subsequent song evaluation; and 4) refinement of connections, via RL, from HVC to downstream motor output neurons that produce the desired sounds (Figure 3).

Figure 3. Hypothesis for the role of auditory-motor interactions in song learning.

(a) Diagram showing interactions between auditory cortex and HVC that could facilitate the emergence and splitting of sequences in HVC, as well as song evaluation. In this model, the auditory memory for each tutor syllable initiates the formation of a different sequence in HVC, thus allowing the number and duration of sequences in HVC to match the tutor song. At this early stage of song learning, the RL pathway (through LMAN) drives subsong babbling; dashed lines emphasize that the role of RL circuitry is unclear at these earliest stages. In later stages, the diagrammed interactions between HVC and auditory cortex would facilitate readout of the auditory memory for song evaluation during singing. (b) Illustration of four hypothesized plasticity processes involved in learning one tutor syllable. Connections strengthened in each process are colored red. Process 1: exposure to a tutor song forms ‘tutor memory’ sequences in auditory cortex (by Hebbian mechanisms described in [58]). Process 2: replay of ‘tutor memory’ sequences seeds the formation of HVC sequences (by Hebbian mechanisms described in [34^••]). Process 3: concurrent replay of ‘tutor memory’ and HVC sequences allows Hebbian strengthening of connections from HVC to auditory neurons active at the same time in the sequence. This automatically aligns sequentially active HVC neurons [49^••] to auditory neurons selective for the desired sound at each time. Coordinated inputs to auditory cortex from HVC and auditory afferents generate a match-to-target song evaluation signal. Process 4: This evaluation signal is used by RL circuitry to strengthen HVC to RA connections that produce sounds that match the tutor song. (c) Detail of how connections from HVC to auditory cortex, learned in Process 3, could create a match-to-target signal. Specifically, each auditory neuron acts as a coincidence detector and only spikes if it simultaneously receives timing input from HVC and the correct auditory input from earlier auditory areas. The population of auditory neurons thus produces a continuous, temporally precise evaluation signal informing the song performance prediction error signal observed in VTA [23^••].

Process 1:

A ‘tutor memory’ is formed in auditory cortex by simple Hebbian learning rules, as outlined by Hahnloser and Ganguli [58]. Specifically, connections between neurons selective for consecutive features of the tutor song would strengthen after repeated exposure to the song. In higher order cortical area CM, neurons exhibit sparse and background-invariant coding of song features [59], a representation ideal for forming sequence generating connections, for example, a synaptic chain. Sufficient strengthening of such connections would enable ‘tutor memory’ sequences to replay autonomously. Next, we envision that such replay would drive HVC, at a rhythm set by the tutor song, via projections from auditory cortical areas [60,61]. Activation of HVC at the tutor song rhythm would be facilitated by stronger activity at syllable onsets, consistent with the observed firing preference of many auditory neurons [62]. Each syllable in the tutor song would be represented by a different sequence in auditory cortex, each of which could facilitate the formation of a distinct chain in HVC in process 2.

Process 2:

Activation of auditory sequences drives the formation of syllable chains in HVC, out of an initially random network, via Hebbian STDP ([34^••], and Figure 2). Such activation could also facilitate the splitting of existing HVC sequences, and could happen either during tutor exposure or during reactivation of auditory sequences in singing or even sleep. In support of the latter idea, sleep replay in the song motor system is believed to be important for song learning [63–65], is driven by the auditory system [66,67], and increases dramatically following exposure to tutor song [52]. A notable consequence of our model is that, after the auditory sequences form temporally aligned HVC chains, auditory exposure to a tutor-like song would sequentially drive HVC neurons. After vocal learning is complete, exposure to the birds’ own song would lead to sequential activation of HVC neurons at times corresponding to their activity during singing, similar to the observed ‘mirror neuron’ responses in HVC [54,68]. Such responses would arise in our model without requiring any learning of auditory-to-motor connections — random connectivity is sufficient. This stands in contrast with models that generate mirror neuron activity by learning auditory-motor connections to create an ‘inverse model’ [56^•,57,58].

Process 3:

Learning of the projection from HVC to auditory cortex connects HVC neurons active at a particular time to auditory neurons selective for the desired sound at that time. It has recently been shown that there exists a specific population of neurons in HVC that projects to auditory cortex, is important for vocal learning, and is sequentially active during singing [49^••]. It has been proposed that HVC inputs to the auditory system could play a role in temporally aligning the readout of the tutor memory with auditory feedback during song performance [49^••]. Process 3 provides a simple biologically plausible mechanism for creating such alignment (Figure 3c). More specifically, coordinated activation of HVC sequences and auditory sequences (process 2) would allow simple Hebbian plasticity to link HVC neurons to auditory neurons selective for the desired acoustic features.

After this HVC-to-auditory mapping is learned, individual auditory neurons would receive coincident input from auditory afferents and from HVC whenever the bird sings the correct sound at the correct time. If these neurons acted as coincidence detectors, then as a population they would provide a dynamic and temporally precise measure of how well the song matches the tutor memory. Notably, error-related signals have been observed in CM [21], and at each stage in a pathway that connects CM to the basal ganglia RL circuitry [22], via a projection from the dopaminergic midbrain area VTA [23^••]. A key open question is how the performance evaluation signal generated by our model might be transformed into a performance prediction error signal of the type reported in songbird VTA.

Process 4: Learning connections from HVC to downstream nucleus RA to generate the desired sound at each time in the HVC sequence. This learning proceeds in two stages: first, computation of a bias in vocal variability that drives the motor system up a local gradient of song performance, and second, Hebbian learning at HVC-to-RA synapses to integrate the local gradient over time, thus consolidating long-term changes in the song motor pathway [13^••,14]. Recent modeling work [27] suggests that efficient learning at HVC-to-RA synapses requires a matching of the biased activity in LMAN with the form of the local learning rule in RA [69,70].

Discussion

Several functions have been proposed for auditory-HVC interactions other than the role we have hypothesized. Prather et al. argued that HVC may provide a motor-based prediction of auditory feedback [68], based on the observation that HVC sequences in adult birds are reactivated during exposure to the bird’s own song, and building on previous ideas that HVC may be involved in computing an internal prediction or ‘efference copy’ of auditory feedback [71]. In addition, disruption of HVC during tutor exposure impairs song imitation [72], leading to the suggestion that HVC may actually encode auditory memories. Other hypotheses are that such interactions play a role in constructing a sensorimotor ‘inverse model’ [56^•,57,58,73,74], or inferring latent structure in songs of conspecifics to aid recognition [75]. These views are not mutually exclusive with our proposed hypothesis.

An emerging principle is that motor sequences for complex learned behaviors could be shaped by behavioral targets represented in sensory areas, and that this shaping may involve direct synaptic interactions between motor and sensory circuits, independent of reinforcement learning mechanisms. Such direct shaping could solve two fundamental challenges in learning complex sequential behaviors. First, through observing a tutor, sensory areas could specify the number and durations of behavioral chunks that the motor system should perform. Second, motor sequences built this way would, by construction, be aligned with corresponding sensory representations, allowing for a temporally specific readout of performance errors. With these challenges met, the brain could then efficiently deploy simple associative RL algorithms to refine behavior through trial and error.