Neural encoding of sensory and behavioral complexity in the auditory cortex

Abstract

Converging evidence now supports the idea that auditory cortex is an important step for the emergence of auditory percepts. Recent studies have extended the list of complex, nonlinear sound features coded by cortical neurons. Moreover, we are beginning to uncover general properties of cortical representations, such as invariance and discreteness, which reflect the structure of auditory perception. Complexity, however, emerges not only through nonlinear shaping of auditory information into perceptual bricks. Behavioral context and task-related information strongly influence cortical encoding of sounds via ascending neuromodulation and descending top-down frontal control. These effects appear to be mediated through local inhibitory networks. Thus, auditory cortex can be seen as a hub linking structured sensory representations with behavioral variables.

Introduction

Auditory perception is an essential, complex process through which humans and other animals use pressure waves to interpret and interact with the environment. The cochlea initially decomposes pressure waves into their frequency spectrum and transduces these signals into neural impulses [1]. This tonotopic organization propagates throughout the auditory system. At the level of perception, however, audition is not merely designed for precise frequency encoding [2], but rather interprets complex acoustic motifs into distinct auditory objects reflecting our experience of the acoustic environment. Thus, one may struggle to identify the absolute pitch of a tone but most people accustomed to western music will recognize if the tone is coming from a piano [3]. How does the auditory system turn the initial spectral decomposition of sounds into the timbre of instruments, the words of a phrase, or the siren of a police car? In vision science, Gestalt theories [4,5] argue that perception is based on a skeleton of complex representations which does not correspond to a ‘one-to-one copy’ of the input signals but rather to archetypal building blocks that are innate or experience-dependent and are used to construct perception [2,6]. This idea likely extends to audition [7–9]. A major challenge for understanding auditory perception is to experimentally isolate and mechanistically explain the elements of such complex auditory representations. These sensory percepts, moreover, are bound to be sensitive to the current internal state and near-term goals of the subject while it explores or interacts with the environment. Increasing evidence now shows that, in auditory cortex, an elaborate vocabulary of auditory representations comes along with information about behavioral context. The purpose of this review is to highlight recent discoveries about these two levels of complexity in auditory cortex and discuss their functional roles and potential interactions.

Structure of auditory cortex representations and their link to perception

Whether auditory cortex represents simple acoustic features or building blocks for ‘auditory objects’ is a longstanding question [7,10–13]. The existence of frequency tuned neurons and their tonotopic mapping in primary auditory subfields [14,15] suggests a degree of similarity to cochlear representations. Yet this might be the only similarity. The precision of the tonotopic map weakens in secondary areas [15]. Moreover, frequency selectivity is, in general, more broadly tuned in auditory cortex than in subcortical areas [16,17], and response time constants appear to be longer [7] so that fine temporal details present in the input are lost or converted into rate codes [18^•]. This readily suggests that cortex integrates over a wider range of information than subcortical structures. But most importantly, multiple studies have shown that selectivity of auditory cortex neurons greatly extend pure tone frequency coding [13,19–22]. Recently, two-photon calcium imaging and electrophysiology in rodents demonstrate that loosely mapped ensembles of cortical neurons in primary and secondary auditory cortex code for the direction of frequency variations [23], interaural differences [24^•] as well as the frequency [25], amplitude and direction [26^••] of intensity variations (Figure 1). These temporal features are crucial for recognizing particular classes of sounds. For example, recognition of musical instruments greatly depends on the steepness of tone intensity rise and decay [3] which is coded in primary auditory cortex, even in mice [26^••]. Frequency modulations are important components of vocalization in most species [27–29]. For highly vocal animals, the structure of frequency harmonics represent important cues, as in human language [12]. Interestingly, a recent study showed that the core auditory cortex of primates includes cells which specifically detect patterns of frequency harmonics [30^••] (Figure 1).

Figure 1.

Structuring of auditory information via nonlinear computations in the auditory cortex. (a) Throughout the auditory system raw cochlear inputs are structured into biologically relevant percepts (auditory objects). This transformation requires a complex ensemble of non-linear computations, which we here schematize as a multilayer network linking simple nonlinearities (e.g. spike threshold) with an elaborate connection graph that includes feedforward, feedback and lateral connections. Interestingly, appropriately trained multilayer networks (Deep Learning) were recently demonstrated to boost performance on artificial perceptual tasks such as speech recognition. (b) Beyond simple computations such as spatially organized preference for particular frequency ranges (left), the auditory cortex displays a number of non-linear computations. These leads to the emergence of neurons sensitive to specific features, as sketched in the middle column, including sound onsets and offsets of particular amplitude [26^••], amplitude modulation (AM) frequencies [25], or even harmonicity (at least in primates) [30^••]. Also, generic response properties, such as invariance to modification of basic acoustic parameters as sketched in the top right graph [31] and discrete coordinated population response switches (bottom right) (data from [33]), indicate that these nonlinear computations endow cortical representations with some of the expected properties of auditory object representations.

The presence of complex features in auditory cortex, however, does not prove that it codes for auditory objects. To address this issue, two major and somewhat orthogonal properties of object-like representations must be observed: invariance and discreteness. Invariance refers to the stability of representations with respect to small changes in acoustic parameters. Discreteness refers to the categorical and rapid switching of representations between objects as separate entities. Two recent studies in rats have shown that auditory cortex neurons respond to vocalizations or water sounds with a certain degree of robustness against various acoustic modifications [31,32]. Interestingly, invariance for vocalization was tested both in primary and non-primary auditory cortex and was found to be more pronounced in non-primary areas [31] (Figure 1), suggesting that invariance properties progressively emerge along the cortical hierarchy, correlating with the weakening of the tonotopic map [15]. As for discreteness, a two-photon calcium imaging study has shown that local ensemble of neurons in the mouse auditory cortex respond in a step-wise manner to gradual changes in sound mixtures [33] (Figure 1). The object-like representation in cortex predicted how mice categorized diverse sounds during a behavioral task [33]. Thus, many of the ingredients necessary to build object-like or categorical representations are present in auditory cortex and evidence exists that cortical representations are close to perceptual space.

Are object-like representations hard-wired into the cortex or are they experience-dependent, dynamically updating based on the ecological needs of a particular animal? In rodents, for example, ultrasonic pup vocalizations elicit robust responses even in low frequency areas [34], suggesting the possibility for an object-like representation. Future studies will need to address this on the axes described above, invariance and discreteness, and determine whether these representations are innate or experience-dependent. Moreover, a comparative approach that looks across the phylogenetic tree in mammals might yield further insights into the types of objects encoded by the auditory cortex, including speech in humans.

Computations and circuits underlying auditory cortex representations

Computational models of sound encoding must account for the richness, diversity and complexity of auditory cortex representations. One influential model, linear spectro-temporal receptive fields (STRFs) [35,36] attempts to explain the activity of auditory cortex neurons as generic weighted sums of spectral characteristics over multiple time steps (i.e. the neurons computations are approximated by a linear filter). Combined with a simple threshold non-linearity, this model can capture many frequency selectivity characteristics, including selectivity to the direction of frequency variations [37], and can even be used, with significant success, to reconstruct played sound spectrograms based on cortical activity [38]. But while they successfully account of the raw input space, STRFs struggle to capture many of the transformations of the acoustic inputs that are required to structure auditory information into percepts. In general, STRFs can only predict a fraction of cortical neurons’ responses [39,40]. The linearity hypothesis prevents them from capturing selectivity to amplitude modulations [26^••,41], as well as the invariance [31,32] and discreteness [33] properties of cortical representations. A generic solution to these limitations might be to consider that the auditory system implements several layers of linear (STRF-like) computations interleaved with nonlinearities (Figure 1) [26^••,42], as in artificial deep networks [43]. The primate visual system has functional analogies with deep networks trained to recognize visual objects [44] and ongoing work is beginning to address this in the auditory system [45]. Thus, deep networks might represent new tools, complementary to biologically constrained modeling [46], to identify the actual features encoded by auditory cortex neurons, particularly where linear filter methods and their weakly non-linear extensions [42,47,48] are not successful.

A major goal will be to link the emerging computational models, identified by deep networks or other approaches, to the underlying neural implementation and circuit mechanisms. A promising avenue is to combine observation of auditory cortical representations with manipulation of local and long-range circuit elements with optogenetics. Recent work demonstrates that interneurons influence representation of frequency in auditory cortex [49,50]. Currently, these studies are in their infancy with the tools now emerging to obtain more precise control of neural circuitry. Going forward, robust circuit dissection will require exploration of more complex sensory stimuli, cell-type specific genetic strategies [51], patterned and activity-dependent optogenetics [52–54], and incorporation of large-scale network modeling. This largely remains to be done, not only for interneuronal circuits but also for important elements of biological circuits such as intracortical layer connections and interareal feedback projections.

Modulations by behavior and context

Another current limitation of the essentially ‘sensory-centric’ framework proposed by existing encoding models is the absence of account for the context in which auditory perception occurs. However, this is an essential parameter in auditory cortex computations. David Hubel and colleagues demonstrated in the 1950s that the auditory cortex responded to non-sensory features when they found putative excitatory neurons that fired only when a cat would ‘pay attention’ [55]. They remarked, however, that ‘attention is an elusive variable that no one has yet been able to quantify’ [55]. This simple statement continues to resonate fifty years later. Yet beginning with pioneering studies done in the ferret [56], several experiments have chipped away at this notion and identified several emerging cortical computations related to behavioral context.

First, during locomotion, in which sounds have no apparent valence, multiple circuits work in concert to achieve robust suppression of auditory cortical output during locomotion [57^•,58,59^•]. Top-down projections from the motor cortex feed back into the auditory cortex, activate local inhibitory circuits, and suppress excitatory output [57^•,58]. In parallel, ascending cholinergic projections from the basal forebrain innervate multiple inhibitory neurons to suppress excitatory output [60]. These signals likely supplement a reduction of the feedforward thalamic drive also induced by locomotion [60]. This shows that even a simple neural computation, such as blanket suppression when auditory information is less important to the animal, can be accomplished by the coordinated actions of distributed brain regions projecting in the auditory cortex.

Second, in contrast to these suppressive effects, cognitively demanding tasks appear to both suppress and facilitate auditory cortical responses [61^••,62,63^•,64]. Anticipatory top-down inputs from the pre-frontal cortex prepares auditory cortex to receive incoming sensory information based on behavioral conditions [63^•]. Moreover, rats given the opportunity to voluntarily initiate a given trial exhibit similar anticipatory patterns of activity in the auditory cortex [62] and a reduction of baseline activity with trial initiation is also seen in gerbils [65]. These signals likely augment ascending cholinergic neuromodulation which alters the output of the auditory cortex during engagement in a cognitively demanding task [61^••]. Interestingly, cholinergic modulation works to simultaneously suppress some excitatory neurons (via direct inhibition through parvalbumin and somatotastin-positive interneurons) and facilitate others (via disinhibition through VIP+ interneurons) (Figure 2). Thus, inhibitory networks in auditory cortex are not only enabling stimulus-related sensory computations [50] but also mediate the complex action of multiple long-range modulatory inputs [61^••].

Figure 2.

Two computations that govern behavioral context in auditory cortex. During locomotion (left), evidence suggests that top-down inputs from the motor cortex (black solid line) drive the activity of PV+ interneurons which suppress the activity of pyramidal neurons. These same inputs may also activate somatostatin neuron (black dashed line, hypothetical). At the same time, ascending cholinergic neuromodulation (red lines) works on the same cell types but potentially in non-overlapping neurons. During task engagement (right), evidence suggests that all three major interneuron subtypes are co-activated by ascending cholinergic neuromodulation (red lines). Top-down inputs from the frontal cortex are known to modulate PV+ interneurons (black solid line) while potentially also operating on other interneuron subtypes (black dashed line, hypothetical).

These data suggest that all contexts are not created equal and, rather, that the auditory cortex integrates the specific features of the surrounding context. These effects may be distinct from and complementary to long-term neural plasticity, including changes in receptive fields and tonotopic maps, which have been observed in associative and perceptual learning [66–69]. Why and how does the auditory cortex treat locomotion differently than cognitively demanding tasks? These two different contexts likely reflect the importance of experience-dependent changes in circuitry. One possibility is that locomotion-related suppression reflects the hard-wiring of feedforward, top-down, and ascending projections onto inhibitory neurons that directly suppress excitatory output. Since sounds hold little to no value in these contexts, suppression dominates as the critical computation in auditory cortex. Some contexts, in contrast, may require the processing of behaviorally relevant sounds. In these tasks, the facilitation of a subnetwork of excitatory neurons may be crucial for faithful execution of a given task.

Broad suppression and selective facilitation may therefore be a prudent coding scheme in these cases to minimize metabolically expensive spikes [70] while maximizing information transfer. Population analysis further suggests that choice-related information transiently exists in the auditory cortex such that rate increases may be predictive of behavioral choice [64]. Moreover, these effects extend to more ethological behaviors such as motherhood. Oxytocin appears to work on auditory cortex through disinhibitory mechanisms to increase the sensitivity of auditory cortex to ultrasonic distress calls of pups [71,72] likely via multiple interneuron subtypes [73]. Taken together, the auditory cortex exhibits remarkable context-dependent responses, allowing for simple computations such as blanket suppression during locomotion and more complex computations such as bidirectional modulation during cognitively demanding tasks. In both cases, however, neurons in the auditory cortex negotiate a barrage of inputs including feedforward sensory drive, ascending neuromodulation and descending frontal control. These signals likely converge in the auditory cortex to prime downstream circuits for action.

Conclusions/perspectives

There is now broad evidence that auditory cortex combines high level representations of sounds with different types of behavioral modulations. Future work will need to systematically investigate the related algorithms, computations and neural circuit implementations. First, at the algorithmic level, some questions are unexplored. For example pioneering work has shown that sound guided navigation leads to plasticity of sound amplitude representations [74]. However it remains unclear which operations allow the brain to guide navigation with sounds and whether this involves processing of contextual inputs by auditory cortex. Second, at the computational level, it is not yet clear what are the circuit principles allowing auditory objects to gain behavioral relevance? One possibility is that there exist hard-wired computations in auditory cortex fostering the detection of behaviorally relevant stimuli. This could concern conspecific communication (e.g. mother-pup interactions in rodents, vocal interactions in primates) and/or vocalization-specific features (e.g. frequency modulations, harmonics) which could thereby be boosted against other sensory features. Similarly, sounds that gain behavioral relevance through experience may be detected by similar, but learnt computations operating above the blanket suppression induced by a non-specific active state (i.e. locomotion) to enhance them as particular auditory objects. Third, at the circuit level, what are the local and long-range connections that enable the interplay between sensory and behavioral complexity? Addressing these questions will require probing the activity of precisely identified functional cell types or neural ensembles during parametrically controlled behavioral contexts using more complex auditory objects and representations. Emerging technologies including large-scale optical and electrophysiological recordings, cell-type specific targeting, and longitudinal observation of the same neurons promise to aid in this endeavor. Thus, exciting times lie ahead, in which we may soon get a handle on the sensory and behavioral complexity of audition.