Quantitative models of auditory cortical processing

Abstract

To generate insight from experimental data, it is critical to understand the inter-relationships between individual data points and place them in context within a structured framework. Quantitative modeling can provide the scaffolding for such an endeavor. Our main objective in this review is to provide a primer on the range of quantitative tools available to experimental auditory neuroscientists. Quantitative modeling is advantageous because it can provide a compact summary of observed data, make underlying assumptions explicit, and generate predictions for future experiments. Quantitative models may be developed to characterize or fit observed data, to test theories of how a task may be solved by neural circuits, to determine how observed biophysical details might contribute to measured activity patterns, or to predict how an experimental manipulation would affect neural activity. In complexity, quantitative models can range from those that are highly biophysically realistic and that include detailed simulations at the level of individual synapses, to those that use abstract and simplified neuron models to simulate entire networks. Here, we survey the landscape of recently developed models of auditory cortical processing, highlighting a small selection of models to demonstrate how they help generate insight into the mechanisms of auditory processing. We discuss examples ranging from models that use details of synaptic properties to explain the temporal pattern of cortical responses to those that use modern deep neural networks to gain insight into human fMRI data. We conclude by discussing a biologically realistic and interpretable model that our laboratory has developed to explore aspects of vocalization categorization in the auditory pathway.

1. Introduction

A critical requirement for extracting insight from experimental data is to place individual experimental observations and their inter-relationships in context within a well-defined framework. In this review, we describe how quantitative modeling can provide such a framework for gaining an understanding of auditory cortical processing from an ever-increasing mountain of electrophysiological, imaging, and behavioral data. The field of quantitative modeling of the auditory system spans the entirety of the auditory pathway and a wide range of auditory processes, a vast scope that cannot be covered in a single review. Therefore, in this review, we focus on quantitative models of auditory cortical processing at the level of single neurons and neural circuits that have been developed in the past decade or so. Our main objective is to highlight the insights about auditory cortical processing that have been generated by a small sample of quantitative models of different aspects of cortical activity at different levels of abstraction. In addition to informing experimental neuroscientists about contemporary computational frameworks and tools that are available to understand auditory cortical activity, our hope is to also encourage the widespread adoption of quantitative models across different levels of analysis so that these may be synthesized in the future to obtain an integrated understanding of auditory cortical function.

1.1. Purpose of quantitative models

There are three main reasons for developing quantitative models of neural processing. First, as stated above, by making the relationships between experimental observations explicit, quantitative models can provide the broader scaffolding needed to gain insight from experimental observations. Second, quantitative models can provide a simple or compact explanation for observed experimental phenomena, thereby summarizing large or complex data sets. In doing so, underlying assumptions about the system and the relative importance of various observed variables are also made explicit. Third, once a validated model that can explain existing data is developed, it can serve as a hypothesis generator for future experiments. For example, perturbations to the system can be made in-silico which can lead to novel predictions that can be verified experimentally. Most importantly, a validated model can be easily shared across the community, which can in turn enable its use in other experiments and contexts. But these models cannot be ‘one size fits all’; as we detail below, different types of models, with different motivations and levels of abstraction may be required to explain the available data or the observed phenomenon. In this review, we address a diverse set of problems that have been addressed using quantitative models, including understanding temporal-to-rate transformations in auditory cortex, the biophysical bases of cortical states, explaining the effects of behavioral state of auditory cortical activity, estimating the receptive fields of auditory cortical neurons, evaluating the impact of network activity on auditory neurons, and suggesting biologically feasible circuits to perform auditory categorization (summarized in Table 1).

Table 1:

Schema for model classification and illustrative examples of quantitative models at different scales in auditory cortex.

Model Type	Scale	Example purposes	References
Mechanistic	Single neuron	E/I dynamics for ACtx ‘up’ and ‘down’ states	DeWeese & Zador, 2006
		E/I dynamics for temporal-to-rate code transformation in A1	Bendor, 2015 Gao and Wehr, 2015
	Circuit	Interval timing detection	Buonomano & Merzenich, 1995 Mottanis et al., 2018
		Determine E/I operational regime of ACtx circuits	Kato et al., 2017
Descriptive	Single neuron	STRF estimation/response prediction with input transformations	Gill et al., 2006 David et al., 2009
		STRF estimation/response prediction with contextual modulations	Ahrens et al., 2008a Williamson et al., 2016
	Network/population	STRF estimation with network interactions	Harper et al., 2016 Rahman et al., 2019
		STRF estimation with convolutional neural networks	Keshishian et al., 2020 Pennington & David, 2022
Normative	Single neuron	Contrast gain control by rescaling output nonlinearity	Rabinowitz et al., 2011 Mesgarani et al., 2014
		Categorization in noise using adaptive mechanisms	Parida et al., 2022
	Multi-layer	Generative model for sound representation using mid-level features	Mlynarski & McDermott, 2018
		Categorization of complex sounds using informative and non-redundant features	Liu et al., 2019

Abbreviations: A1, primary auditory cortex; ACtx, auditory cortex.

1.2. Types of quantitative models

One influential framework for understanding information processing systems was suggested by Marr (Marr, 1982), who proposed three essential ‘levels’ of inquiry. These are 1) the computational goal of the system, 2) the algorithms that may be used to achieve this goal, and 3) how these algorithms may be implemented in hardware, i.e., in the neural circuit. This overarching philosophical framework suggests several levels at which models may be developed. For example, divisive normalization by population activity to preserve dynamic range (Carandini and Heeger, 2012) is an algorithmic-level model, whereas implementing this normalization using specific connectivity patterns between excitatory and inhibitory neurons (Seybold et al., 2015) is an implementation-level model.

From a practical standpoint in terms of actual model implementation, quantitative models can be categorized into three broad classes (Wang et al., 2020). First, mechanistic models incorporate known anatomical and/or biophysical properties to build neuron models to explain experimental observations. For example, detailed auditory nerve models (Bruce et al., 2018; Zilany et al., 2014) that include cochlear nonlinearities, rate adaptation, and realistic discharge rate generators, can emulate recorded auditory nerve responses as well as explain some behavioral observations (Saremi et al., 2016). A compartmental model that includes both passive filtering in dendrites and a biologically realistic distribution of potassium channels can successfully recapitulate the sub-millisecond precision required to compute inter-aural time differences in the medial superior olive (Mathews et al., 2010). Second, descriptive models, which likely form the most prevalent model class, can be used to quantitatively characterize experimental data. For example, simple linear fitting essentially tests the model that the dependent variable varies linearly with the independent variable. Spectrotemporal receptive field (STRF) models of cortical neurons (Sharpee, 2013) attempt to explain a neuron’s responses to a given stimulus as a linear combination of the neuron’s responses to simple sound features (e.g., frequencies). Finally, normative models aim to explain why a certain computation may be functionally necessary at a particular stage of processing. For example, by starting with the principle that the auditory system needs to represent information efficiently at the input stage, and by recognizing that the system is likely biased to optimally represent some classes of natural sounds such as speech, Smith and Lewicki, 2006 were able to obtain tuning functions for theoretical input neurons that closely matched the tuning functions of recorded auditory nerve fibers. Some models of contrast gain control start with the principle that auditory neurons need to maintain their dynamic range for encoding stimuli in a variety of listening conditions (Rabinowitz et al., 2011; Willmore et al., 2014) and use dynamic adjustments of the input-output functions of model neurons to arrive at experimentally testable predictions of how firing rates of neurons would be affected in different conditions if this principle was indeed in operation. While some parallels between this practical model classification scheme and the more philosophical framework of Marr are evident (e.g., mechanistic model – implementation level, normative model – algorithmic level), for clarity, in this review we will classify models from the practical standpoint (i.e., as mechanistic, descriptive, or normative models; see Table 1 and Figure 1).

Figure 1: Schematic of model types and scales.

(A) Mechanistic models use biophysical details to explain experimentally observed phenomena. Examples include single-neuron models to explain temporal-to-rate firing rate transformations (left; based on Bendor, 2015), or circuit models to explain interval selectivity of cortical responses (right; adapted from Motanis et al., 2018). (B) Descriptive models are used to quantitatively characterize experimental data. These include single-neuron encoding models such as LN models (left) as well as multi-layer encoding models (right; adapted from Keshishian et al., 2020). (C) Normative models aim to explain why a computation may be necessary at a give stage of processing. For example, one proposed reason for why selectivity for some complex features is observed in auditory cortex is that detecting a set of non-redundant features is optimal for categorizing sounds (based on Liu et al., 2019). Details of all models are discussed in the main text.

Models at each of these levels can also explain data at different scales with corresponding degrees of abstraction. For example, a mechanistic model may focus on single neuron responses, including details such as the distribution of channels along the dendrite (Migliore and Shepherd, 2002) or the time course of synaptic depression (e.g., Abbott et al., 1997), or focus on the dynamics of neural circuits with some simplifying assumptions about biophysical details (e.g., Kuchibhotla et al., 2017). Alternatively, a descriptive model may focus on stimulus encoding by single neurons, or model multiple auditory processing stages consisting of thousands of neurons (for example, modern deep neural network [DNN] models) built using simple (more abstract) integrate-and-fire units. In some cases, such as the first mechanistic example, the motivation for developing the model may be to test the contributions of fine biophysical details to neuronal function. But for the large-scale descriptive example, the details of channel distribution may be irrelevant to the question at hand, and/or modeling that level of detail may be computationally intractable. Thus, choosing the level of abstraction that is appropriate for the scale of the model is a key consideration for gaining insight into the question at hand using a quantitative model. In this review, we will discuss quantitative models at different scales, from single neurons to multi-layered neural networks (see Table 1).

1.3. Quantitative models in the ascending auditory system

Like other sensory systems, the auditory system shows increasing processing complexity along the hierarchy, i.e., neurons in earlier stages respond to simple sensory features and neurons in later stages respond to more complex features and experience greater modulations by internal state and behavioral context (Fritz et al., 2003; Montes-Lourido et al., 2021; Saderi et al., 2021; Sharpee et al., 2011). Much more anatomical and electrophysiological data is available at the earlier stages of the auditory pathway. The combination of relatively simple neural tuning and abundance of experimental data has led to the development of a wide range of models of early auditory processing across levels of analysis. An exhaustive review of these studies is beyond the scope of this review focusing on models of cortical processing. However, because these models may be used as biologically realistic ‘front-ends’ to models of higher auditory processing stages, in this section, we will briefly highlight a few of these models.

1.3.1. Auditory nerve models

At a mechanistic level, detailed models of auditory nerve tuning matched to experimental observations in humans and animals have been developed. These models include some cochlear nonlinearities, properties of the ribbon synapses, and capture the heterogeneity of nerve fiber types (Bruce et al., 2018; Zilany et al., 2014). These mechanistic details are necessary to capture some aspects of auditory nerve responses that cannot be explained by simpler models. For example, the addition of synaptic depression (using power-law dynamics) at the ribbon synapse was necessary to capture the temporal dynamics of auditory nerve fiber responses (Zilany et al., 2009). These detailed auditory nerve models also attempt to recreate neural coding aspects such as synchrony capture, two-tone suppression, and level-dependent frequency tuning. With appropriate readout strategies, these responses have been used to predict human performances, such as frequency, level, and amplitude modulation discrimination (Colburn et al., 2003; Dau et al., 1998a, 1998b; Heinz et al., 2001). The output of these mechanistic models can be used as biophysically realistic front-ends to other models as well (Carney et al., 2015; Liu et al., 2019; Saddler et al., 2021).

At a descriptive level, auditory nerve fibers show spectrally well-defined excitatory tuning with limited suppressive sidebands (due to mechanical properties of the cochlea) and simple low-pass modulation tuning (Joris and Yin, 1998; Liberman, 1978). These tuning features are well-captured by models using systems identification approaches to estimate receptive fields (Boer and Jongh, 1998; Carney and Yin, 1988; Eggermont et al., 1983). In these models, auditory nerve fibers are conceptualized as filters with specific impulse responses that confer the fiber with its frequency tuning and temporal response properties. As a further abstraction, the shapes of these auditory nerve filters can be captured as a sinusoidal carrier wave modulated by a gamma function envelope, termed a ‘gammatone’ filter (Patterson et al., 1987). In an engineering sense, auditory nerve responses can thus be approximated as a gammatone filter bank, which is a common front-end used in other auditory processing models (for example, Rahman et al., 2020). Thus, in addition to explaining some auditory nerve response properties, these descriptive quantitative models of the auditory periphery have proved invaluable in the development of other auditory processing models.

At a normative level, some elegant studies have asked why auditory nerve fibers show ‘gammatone’ tuning in the first place. Starting with the assumption that the goal of peripheral auditory encoding was the efficient representation of sounds, Smith and Lewicki, 2006 used a set of wavelet-like filters and a matching-pursuit algorithm to reconstruct incoming acoustic inputs with the least residual errors. Critically, the shapes of the filters and the times of the spikes were derived by training this model using natural stimuli such as speech sounds. Surprisingly, the shapes of the filters derived using this purely theoretical approach were starkly similar to the filters observed in electrophysiological experiments. This model thus provided critical insight into the underlying encoding strategies used by the auditory system underlying the aforementioned experimental observations.

1.3.2. Brainstem models

In contrast to the auditory nerve, neurons in the brainstem (including the inferior colliculus) show heterogenous spectral (narrow and broad) and modulation tuning (e.g., multiple peaks or band-reject) and more complex interactions between inhibitory and excitatory inputs (Krishna and Semple, 2000; Langner and Schreiner, 1988). Specialized circuits in the medial and lateral superior olive show specific tuning for spatial location (Caird and Klinke, 1983; Grothe, 2003; Kandler and Gillespie, 2005). At a mechanistic level, recent models have demonstrated how the inclusion of anatomical details such as the distribution of potassium channels along dendrites of medial superior olive neurons is necessary for explaining coincidence detection with sub-millisecond precision (Mathews et al., 2010). Models of the cochlear nucleus and inferior colliculus have been used to explain neural selectivity for second-order stimulus features such as modulation rate (Dicke et al., 2007; Nelson and Carney, 2004). At a descriptive level, tuning for modulation rates has been implemented as modulation filtering – for example, as the additive response of a narrowly tuned excitatory subunit and widely tuned inhibitory subunit tuned to the same carrier frequency (Nelson and Carney, 2004). With various readout strategies, these modulation filter bank-based models have been used to predict speech intelligibility in degraded listening conditions (Jørgensen and Dau, 2011; Scheidiger et al., 2018). Systems identification approaches have also been widely used to characterize neurons in the inferior colliculus and thalamus, with these models revealing the tuning of these neurons to more complex stimulus features (Andoni et al., 2007; Escabí and Schreiner, 2002; Miller et al., 2002). The auditory brainstem is the locus of one of the best known normative models – the Jeffress model for the computation of azimuthal location from binaural inputs using delay lines and coincidence detectors (Jeffress, 1948).

In summary, the auditory hierarchy (and that of other sensory systems) can be conceptualized as being composed of early encoding and later decoding stages. Pre-thalamic models may largely be characterized as encoding models, with their explicit aim being to explain empirically observed neural response properties or behavioral results that are closely tied to low-level stimulus representations. While detailed mechanistic models may be necessary to explain neural response properties at this level, abstractions or engineering constructs (such as filter banks) are also available that can be used as front-ends to post-thalamic decoding models. We acknowledge that this brief discussion has barely scratched the surface of the body of research on encoding models of the ascending auditory pathway. Our objective in this review is to highlight recent quantitative models of the auditory cortex, which use the features extracted from these pre-thalamic representations to accomplish specific behavioral goals.

2. Mechanistic models in the auditory cortex

2.1. Single-neuron models

Few quantitative models of the auditory cortex operate at a mechanistic level or incorporate biophysical details. These models are generally focused on explaining specific features of single auditory cortical neuron responses from in-vivo recordings. For example, one observation, particularly in anesthetized animals, is that the membrane potential of neurons is typically far from the spiking threshold (‘down’ state), with infrequent threshold crossings occurring due to large depolarizations (‘up’ states) (DeWeese and Zador, 2006; Harris et al., 2011; Hromádka et al., 2013). Conceptually, this distribution of membrane potentials is not consistent with many uncorrelated inputs; rather, it suggests that during ‘up’ states, the neuron is bombarded by a volley of spikes from a highly correlated presynaptic population. Comparing three variants of a simple model could establish the basic conditions that are necessary for such large depolarizations (DeWeese and Zador, 2006). In these variants, the membrane potential was modeled as ‘random walk’ processes with 1) purely excitatory inputs with a homogenous response rate following Poisson statistics, 2) excitatory and inhibitory inputs with homogenous firing rates, or 3) excitatory and inhibitory inputs with input firing rates varying at different time scales. This model showed that the observed data could only be matched by the third model, and only when the time scale of firing rate variations was rapid (consistent with highly correlated firing of the presynaptic population). This model illustrates how a quantitative model can both provide rigor to a conceptual explanation, but also reveal essential underlying assumptions (in this case, about the necessity of both excitation and inhibition to recapitulate the experimental observation).

Another phenomenon that has been explored using mechanistic models is the transformation of neural representations of amplitude-modulated stimuli from a temporal- to a rate-based representation by a population of cortical neurons. Two broad response types have been reported in recordings from the primary auditory cortex of unanesthetized animals (Bendor and Wang, 2007; Gao and Wehr, 2015; Lemus et al., 2009; Liang et al., 2002; Lu et al., 2001; Lu and Wang, 2004). Similar to sub-cortical responses, the spiking responses of one cortical response type (the ‘synchronized’ population) faithfully follow slow fluctuations of stimuli, but do not respond when the stimulus fluctuates too rapidly. The second cortical response type (‘non-synchronized’ population) does not respond to slow fluctuations but increases its firing rate in a manner that is proportional to the rate of stimulus fluctuations for rapid fluctuations. Passive filtering by membrane properties alone cannot account for this transformation, as the membrane potential responses of cortical neurons (already filtered by membrane properties) can synchronize to high stimulus repetition rates (Gao and Wehr, 2015). Rather, a model that included push-pull amplification of membrane potential responses by inhibition that was slightly delayed with respect to excitation was critical in producing both the observed strength of synchronization for slow stimulus rates (Bendor, 2015; Fig. 1A, left), as well as in producing high asynchronous firing rates for fast stimulus rates (Gao and Wehr, 2015). A further extension to the Bendor, 2015 model demonstrated that incorporating short-term synaptic depression was necessary to explain additional features of cortical neuron responses (Lee et al., 2020). While the specific neuron types that provide the inhibition required by these models are unidentified, and the role of other intrinsic properties such as bursting (Liu and Wang, 2022) remain unexplored, these studies show how quantitative models at this level can help identify putative mechanisms and precise parameter ranges (for example, E-I delay) that are required to explain observed neural responses.

2.2. Circuit models

Models at the mechanistic level can also be used to gain insight into how some critical computations can be implemented by simple neural circuits. For example, a key question in auditory processing is that of timing. Detecting precise temporal intervals in the range of tens to hundreds of milliseconds is important for the accurate recognition of complex sounds that contain spectrotemporal information over such wide timescales, such as speech (Aasland and Baum, 2003; Tallal, 1994), vocalizations (Montes-Lourido et al., 2021; Sadagopan and Wang, 2009), and music (Janata and Grafton, 2003). But how neural circuits can achieve selectivity for precise temporal intervals is yet to be understood. To address this question, Buonomano and Merzenich, 1995 built a simple model with integrate-and-fire units but incorporating known synaptic properties of cortical neurons such as paired-pulse facilitation (a form of short-term plasticity). The model consisted of two layers of excitatory and inhibitory neurons, approximating the microstructure of a cortical column (layers 4 and 3). The task was to discriminate the timing between two pulses delivered at the inputs. Delayed inhibition with respect to excitation, slow kinetics of the inhibitory post-synaptic potential, and the kinetics of paired-pulse facilitation altered the state of the circuit after the first pulse and caused neurons in the second layer (layer 3) to respond differentially to the second pulse. The pattern of activity in second layer neurons could be used to decode the time interval between the two pulses with high accuracy. Critically, although the model was trained to discriminate the timing between only two pulses using a handful of inter-pulse intervals, the model generalized to more complex trains of pulses without further adjustments. In a follow-up study, Buonomano, 2000 demonstrated that a simple disynaptic circuit with the above properties was sufficient to achieve order- and interval-selectivity but required precise parameter tuning. However, a large circuit with greater heterogeneity could achieve generalization without the need for such tuning. Thus, these simple circuit models demonstrated how cortical neurons can achieve selectivity for order and interval in an ethologically relevant timescale (tens to hundreds of milliseconds) using experimentally observed mechanisms such as short-term plasticity (Motanis et al., 2018; Fig 1A, right) and without the need for using components that may not be biologically realistic, such as temporal delay lines.

In a similar vein, simple circuit models incorporating the unique dynamics of short-term plasticity in different cortical neuron populations, such as parvalbumin (PV) and somatostatin (SOM) expressing inhibitory neurons, have been shown to recapitulate the varied temporal profiles of excitatory cortical neuron responses to pulse trains (Phillips et al., 2017a, 2017b). Using a spiking model (rather than a firing rate-based model) that captured these differential synaptic dynamics, excitation-inhibition delays, and the relative balance of inhibition from these two inhibitory neuron types was a critical factor in the success of this approach in reconstructing cortical response types (Seay et al., 2020). But importantly, the model could also generate experimentally testable predictions regarding the spike latencies of excitatory neurons and how inactivating specific subpopulations of inhibitory neurons would alter excitatory neuron responses. During sequential tone presentation, the model predicted that inactivation of PV interneurons (but not SOM interneurons) during the first tone would decrease the firing rate of excitatory neurons to the second tone. This prediction was verified by performing optogenetic inactivation of the PV population during in-vivo recordings, further validating the model. This example illustrates how quantitative models can help discern between multiple mechanisms that might underlie an observed phenomenon.

Other circuit models have been used to explain or predict the effects of the targeted activation or inactivation of cortical cell types on the activity of excitatory neurons. For example, using optogenetic manipulations targeted to PV interneurons, Aizenberg et al., 2015 demonstrated that photoactivation of PV neurons led to increased acuity of frequency discrimination, whereas photo-suppression led to impaired performance and led to degraded specificity of learning. To understand these experimental observations, the authors built a mutually coupled excitatory–inhibitory firing rate model. The model also included inhibitory-excitatory synapse non-linearities, particularly PV-generated synaptic depression. With only these simple considerations, the model could replicate experimental observations, generating insight into possible circuit wiring motifs that could explain these experimental observations. Seybold et al., 2015 also used a simple circuit model to explain the diverse effects of experimentally manipulating inhibition. To capture the effects of optogenetically manipulating PV and SOM neurons on excitatory neurons, they modeled a simple excitatory-inhibitory circuit regulating the firing thresholds and the excitatory and inhibitory conductances. The central insight the model provided was that changes to the firing threshold of neurons, varying the strength of suppression onto it and the interaction of these changes with the network could “mask” divisive scaling as subtractive and vice-versa, providing a cautionary tale for experimentalists. Similarly, Phillips and Hasenstaub, 2016 showed using a simple circuit model how relatively small changes to parameters such as spontaneous rate and the strength of light manipulation could result in vastly different outcomes when modeling the effect of photoactivation or inactivation. These examples illustrate how in addition to helping gain a deeper understanding of existing data, quantitative models can also guide experimental design.

Larger-scale network models incorporating more biological details have also provided insight into how activity in the cortex is maintained in a stable state and how external as well as internal factors can alter cortical state. For example, Kuchibhotla et al., 2017 used a network model consisting of four major cortical cell types – excitatory, PV, SOM, and vasointestinal peptide (VIP) expressing interneurons – to demonstrate that neuromodulation of all three major types of inhibitory neurons played a role in active behavior, and that driving either only inhibition or disinhibition could not reproduce the effects of active engagement seen on the auditory cortical neurons. Larger scale recurrent network models and state space models have been used to gain a deeper understanding of neural dynamics observed using large-scale electrophysiological or optical neural recordings. One of the most powerful models of recurrent network dynamics to emerge in recent years across sensory domains is the inhibition-stabilized network (ISN). In a network where excitatory recurrent connections are strong enough to result in instability (runaway excitation) in the absence of any inhibition (as is the case in the cortex), increasing external excitatory input to inhibitory neurons can paradoxically lead to decreased activity in inhibitory neurons (Latham and Nirenberg, 2004; Tsodyks et al., 1997). Experimental support that the cortex operates in such an inhibition-stabilized regime was provided in the visual cortex by Ozeki et al., 2009, who used this framework to explain the phenomenon of surround suppression in primary visual cortex neurons. Using in-vivo intracellular recordings, Ozeki et al., 2009 demonstrated a transient increase in inhibitory conductance followed by a sustained decrease when inhibitory receptive field regions were excited, as predicted by an ISN model. In fact, only an ISN architecture could explain these experimental results. In the auditory cortex, Kato et al., 2017 used the analogous phenomenon of lateral inhibition and in-vivo intracellular recordings to show that the auditory cortex may also operate in an inhibition-stabilized regime. More recently, Aponte et al., 2021 used an auditory cortex model with ISN architecture to explain direction selectivity of A1 neurons to frequency-modulated sweeps. Operation of the model in the inhibition-stabilized regime was crucial in ensuring that the cortical circuitry amplified thalamic inputs in one sweep direction, whereas such amplification was prevented by leading inhibition in the other sweep direction. Bondanelli et al., 2021 used a state-space model to gain new insights into OFF responses in the auditory system. They showed that a network model including recurrent dynamics outperformed standard single-neuron models in explaining OFF responses. Not only did the model exhibit transient OFF responses similar to those observed in experiments, but could also make predictions on the low-dimensional nature of such OFF responses that were verified experimentally. These models illustrate how larger-scale models can be used in conjunction with large data sets generated from electrophysiological and imaging experiments to gain a deeper understanding of the network dynamics that surround and shape single neuron responses.

3. Descriptive or encoding models

3.1. Single neuron models

While the mechanistic examples in section 2 attempt to explain the responses of cortical neurons in terms of the interaction between excitatory and inhibitory inputs and intrinsic properties, validating mechanistic models typically requires access to membrane potential recordings that are technically difficult to acquire. Extracellular recordings of spiking activity are by far the most common type of auditory cortical response data available at the single neuron level. This has spurred the development of several powerful approaches to characterize how stimuli are encoded by cortical neurons, i.e., to characterize the spectrotemporal receptive fields (STRFs) of these neurons. Early studies employed reverse correlation approaches using white noise acoustic stimuli to determine the average stimulus waveform that evoked a spike in auditory nerve fibers (Boer, 1975; de Boer, 1991, 1969). These studies laid the foundation for STRF models that characterized the input-output relationships of peripheral auditory neurons (Aertsen and Johannesma, 1980; 1981a; 1981b). STRF models have since undergone several adaptations to explain the activity of neurons across the auditory hierarchy including the cochlear nucleus (Bandyopadhyay et al., 2007; Tan et al., 2015), auditory thalamus (Miller et al., 2002), and the auditory cortex (Christianson et al., 2008; Klein et al., 2000; Kowalski et al., 1996a, 1996b; Miller et al., 2002; Schnupp et al., 2001; Sharpee, 2013). However, the ability to predict neural responses to new stimuli (prediction accuracy) of early linear STRF models was not high (Sahani and Linden, 2003), likely because auditory cortical responses show a range of strong nonlinearities. A previous review has extensively described the development of STRF models of auditory cortical responses (Sharpee, 2013). In this section of the review, we will focus on more recent descriptive models that, in addition to modeling linear receptive field components, also include some nonlinearities that are apparent in cortical data.

The spiking threshold as well as power law relationship between the membrane potential and firing rate are strong nonlinearities that shape cortical responses. Linear-nonlinear (LN) models for STRF estimation have been developed to account for these effects. Like linear models, LN models also convolve a linear filter with the stimulus spectrogram, but then impose a static non-linearity on the filter output to obtain time-varying firing rate estimates (Atencio et al., 2008). The static nonlinearity often takes the form of a rectified-linear or power law function. Low degrees of ‘match’ between the STRF and the stimulus evoke no spiking output, whereas a high degree of match would evoke correspondingly (linear) or disproportionately (power law) high outputs. A further extension to the LN model framework is the LNP (Linear-Nonlinear-Poisson) model, which uses the time-varying firing rate output of the LN model as the instantaneous firing rate of a non-homogenous Poisson process to generate spike outputs (Moskovitz et al., 2018; Paninski et al., 2007; Schwartz et al., 2006; Williamson et al., 2015). In addition to achieving higher prediction accuracy compared to purely linear models, the LN model is powerful in that for the same inputs and linear STRFs, minor adjustments to the slope or exponent of the static nonlinearity can result in sizeable changes to the output firing rate. This property has proved critical for modeling contrast gain control. For example, several studies have shown that varying the nonlinearity in a contrast-dependent manner can better capture the responses of cortical neurons in noisy listening conditions (Cooke et al., 2018; Rabinowitz et al., 2012, 2011; Willmore et al., 2014). Athough LN models exhibit better predictive power than purely linear models, the performance of LN models in predicting responses to natural stimuli is still modest and fails to capture some important aspects of auditory coding (David et al., 2007; Machens et al., 2004).

Some of these issues may arise because of sub-optimal inputs to the STRF models. Typically, STRF models have used the stimulus spectrogram as a front-end to the model, but as discussed earlier, given the availability of detailed models of the auditory periphery, a more biophysically realistic front-end could be applied. Moreover, because the auditory cortex is located several synapses away from the auditory periphery, it is reasonable to expect that cortical inputs have been substantially transformed by the subcortical hierarchy. A recent study showed, however, that relatively simple front-ends to STRFs could perform as well as biophysically realistic front-ends (Rahman et al., 2020), suggesting alternate causes for the low predictive power of STRFs. For example, since it is known that inferior colliculus (IC) neurons also experience contrast gain control and adapt their firing rates to stimulus statistics (Dean et al., 2008), this IC adaptation could be used to transform the inputs to cortical linear filters. Such an ‘IC adaptation’ model indeed resulted in an improvement in prediction accuracy (Willmore et al., 2016). Other studies have applied various transformations to the input to the linear filter to take into account biological observations (Gill et al., 2006). For example, the relationship between sound intensity and neural response may be nonmonotonic (Brugge and Merzenich, 1973; Pfingst and O’Connor, 1981; Polley et al., 2007, 2004; Sadagopan and Wang, 2008). Modeling level-dependence as an input nonlinearity improved the performance of STRF models (Ahrens et al., 2008b). Similarly, transforming the inputs to represent stimulus surprise, i.e., whether a stimulus component was louder or quieter than expected given stimulus history and context, improved prediction accuracy in the avian forebrain (Gill et al., 2008). Other input transformations include accounting for the strongly depressing nature of the synaptic inputs to the cortex (Wehr and Zador, 2005). Including strongly depressing synapses at the inputs improved prediction accuracy by differentially affecting the STRF for stimulus classes with different temporal dynamics (David et al., 2009). Incorporating synaptic depression with multiple timescales similarly improved prediction accuracy (David and Shamma, 2013). More broadly, applying short-term plasticity dynamics to all or a subset of cortical inputs in this framework, resulted in varying levels of benefit (Espejo et al., 2019). Recent results have shown that models incorporating short-term plasticity mechanisms at the inputs are complementary to the output nonlinearity-based gain control mechanisms described above (Pennington and David, 2020). More generally, the local context in which a sound occurs (of which the above mechanisms are a subset) can powerfully modulate the activity of cortical neurons (e.g., (Sadagopan and Wang, 2010, 2009)). Modeling the local context as a gain field applied to the inputs improved prediction accuracy (Ahrens et al., 2008a; Williamson et al., 2016). But even with the addition of these nonlinearities, STRF predictions often systematically deviate from observed firing rates, likely because the fundamental linear model does not consider nonlinear spectrotemporal interactions that are known to shape cortical responses (Machens et al., 2004; Sadagopan and Wang, 2009). Furthermore, the underlying assumption of linearity requires stimuli to exhibit Gaussian statistics along relevant dimensions. But because natural stimuli are not Gaussian and consist of strongly correlated time-frequency components, STRFs estimated using reverse correlation from these stimuli are often strongly dependent on stimulus characteristics. This results in poor cross-predictivity, i.e., STRFs derived using one stimulus set are quite poor at predicting responses to another stimulus set (David et al., 2009; Laudanski et al., 2012; Theunissen et al., 2000; Woolley et al., 2005).

To account for non-Gaussian stimuli and some potential nonlinear interactions, alternative methods of STRF estimation have been developed. Sharpee et al., 2004 took an approach based on estimating the most important stimulus dimensions for generating neuronal responses in an information-theoretic sense (also see Sharpee, 2013). While the first most informative dimension typically corresponded to the spike-triggered average, i.e., the STRF estimated using a linear estimator, the second most informative dimension captured unique response characteristics such as non-monotonicity that were not captured by the first component (Atencio et al., 2008). Thus, the two receptive field components were synergistic, and provided higher prediction accuracy compared to only using the first component, i.e., the LN-like component. Extending this approach (but in the visual cortex), Rowekamp and Sharpee, 2011 determined maximally informative subspaces that incorporated several receptive field components that synergistically provided high levels of information about the stimulus. This has also been observed in higher auditory cortical regions of the avian forebrain, where individual neurons respond to multiple distinct features of bird song and are their receptive fields are composed of numerous components (Kozlov and Gentner, 2016). The observation of several synergistic receptive field components suggests that perhaps the inputs to cortical neurons may themselves be modeled as individual LN units. In this vein, Harper et al., 2016 developed a network receptive field (NRF) model in which inputs to the cortical neuron to be modeled were themselves modeled as 20 LN units. This model outperformed traditional LN models but showed longer latencies and integration times than seen in auditory cortical neurons. Extending the NRF model further, Rahman et al., 2019, built in exponentially decaying memory for the input LN units of the model and modeled each unit’s firing using a dynamic equation. Capturing the dynamic integration of recent stimulus history by multiple input units in this manner indeed improved prediction accuracy further.

A second approach to ameliorate the effect of stimulus correlations in estimating receptive fields uses the generalized linear model (GLM) framework instead of reverse correlation to estimate linear STRF and nonlinearity parameters of LN neurons. Calabrese et al. (2011) demonstrated that the GLM framework, with appropriate sparseness constraints, could generate STRFs with high prediction accuracy across stimulus classes. In their model, each neuron’s output spike train was generated from a Poisson process, whose instantaneous rate was determined by the convolution of a linear filter (STRF) with the stimulus spectrogram and an additional post-spike filter capturing the neuron’s recent spike history (Fig. 1B, left). Paralleling the results of Smith and Lewicki (2006) in the auditory nerve, the use of a sparseness constraint, i.e., assuming that the neuron was selective only to a small number of stimulus features, was critical in achieving high prediction accuracies. More generally, Stevenson et al., 2012 proposed a GLM encoding model which modeled a neuron’s firing rate in terms of 1) external variables or tuning properties of the neuron alone, 2) the activity of the other simultaneously recorded neurons, or 3) a combination of neuron tuning and the activity of other neurons. In several brain areas including A1, incorporating even a small number (10 – 30) of simultaneously observed neurons into the encoding model improved prediction accuracy considerably compared to purely tuning models. In addition to incorporating the impact of the activity of other neurons, the GLM framework also lends itself easily to incorporating other non stimulusrelated factors affecting neural responses, such as task variables, arousal, and reward (Pachitariu et al., 2015; Runyan et al., 2017; Saderi et al., 2021). As access to larger data sets become more feasible and as model fitting techniques improve, it is likely that even more complex model architectures will provide increasingly better quantitative descriptions of single neuron spiking-data (Pennington and David, 2022), essentially placing single-neuron spiking activity in the context of the spiking activity of a large circuit. The limited biological interpretability of these models, however, will present new challenges in the quest to gain insight into auditory cortical processing.

3.2. Multi-layer models

In section 3.1, we briefly mentioned models where the inputs to neurons were themselves modeled as LN units (e.g., Harper et al., 2016; Kozlov and Gentner, 2016; Rahman et al., 2019). A generalization to this approach is to model the inputs to cortical neurons as the output of a multi-layered network, approximating the biological inputs to cortical neurons that are filtered by many subcortical processing stages. Here, individual units in the input layers are often abstracted as LN models, where the linear component is an STRF that is convolved with the stimulus, and the nonlinear component is a static output nonlinearity. The large parameter space and computational demands of such models often preclude the addition of any additional complexity or biophysical realism. As a recent example, Pennington and David, 2022 explored the use of several convolutional neural network (CNN) architectures that differed in terms of the number of convolutional layers and the number of dense or fully connected layers to fit the experimentally observed spiking activity of single neurons as well as neural populations. They demonstrated that for the same number of free parameters per neuron, CNN models handily outperformed traditional LN models in explaining cortical neuron responses. In addition, the models showed a high degree of generalizability in that the model, when trained on one population of neurons, performed well in fitting the activity of another population of neurons. Although a high degree of variance of cortical neuron firing rates can be explained by these models, the trade-off is human interpretability. Whereas the interpretation of an LN model is straightforward (convolution with linear receptive field followed by thresholding), the patterns of activity in the CNN that lead to the final fit are complex and difficult to visualize. Although there is no reason to expect that cortical computations need to be human interpretable (as discussed later in this review), this is one drawback of using such multilayer models to gain intuition about underlying neural computations from experimental observations. To address interpretability, Keshishian et al., 2020 used a standard linear STRF or convolutional neural networks (CNN) to fit cortical responses to speech recorded from human auditory cortex (Fig. 1B, right). Responses to speech stimuli were taken to be the envelope of the high-gamma band of the electrocorticogram, recorded from human Heschl’s gyrus and superior temporal gyrus. As with the Pennington and David, 2022 study, this study also demonstrated that CNNs outperformed linear models in terms of prediction accuracy. But to better understand the nonlinear mapping between the stimulus and the response, the authors developed a technique, termed the dynamic STRF or DSTRF, which determines the linear function at each time point of the stimulus that is mathematically equivalent to the nonlinear transformation performed by the CNN. In this framework, neurons could be classified into those showing gain changes to the STRF, those showing delay changes to the STRF, or those showing STRF shape changes over time. The conceptualization of the CNN mapping as a dynamically varying STRF, and the description of responses in these categories helped improve the interpretability of these multi-layer models.

4. Normative models of auditory cortical computational

4.1. Single neuron models

At the single-neuron level, experimental observations have demonstrated that cortical neurons tend to maintain their dynamic activity range in a wide variety of listening environments (Rabinowitz et al., 2011; for a detailed review, see Willmore et al., 2014). Based on ideas developed in the visual cortex (Carandini and Heeger, 2012), it has been suggested that the reason for maintaining this dynamic activity range is to preserve neural sensitivity to acoustic features across listening environments (invariance). Contrast-dependent adjustments to the nonlinear stage of LN models, termed contrast gain control, has been proposed as a possible method of implementing contast invariance (Rabinowitz et al., 2011; Willmore et al., 2014). High-contrast sounds (e.g., speech in quiet) show greater variations in sound pressure levels (SPL) than do low-contrast sounds (e.g., speech in noise), consequently providing a narrower range of inputs to neurons in low-contrast situations. To preserve the sensitivity of neurons to this reduced range of inputs, one possible solution could be to adjust the slope of the nonlinearity in LN models of neurons in a contrast-dependent manner to maintain a consistent output dynamic range. In addition to rescaling of the output nonlinearity, it is also possible that other input transformations such as synaptic depression could contribute to the contrast invariance of auditory cortex responses (Mesgarani et al., 2014). While both inhibition and contrast-dependent fluctuations in membrane potentials have been proposed as mechanistic implementations of contrast gain control in the visual cortex, the precise mechanisms underlying contrast gain control in the auditory cortex have not yet been identified. For example, a recent study found that contrast gain control could occur independently of inhibitory neuron activity in the auditory cortex (Cooke et al., 2020). Our laboratory has recently proposed another normative model for why such rescaling mechanisms are important in auditory cortex from the perspective of accomplishing auditory categorization (Parida et al., 2022; see section 6 below).

4.2. Network models

Normative multi-layer models have been proposed to develop intuition about why complex receptive fields are observed in higher auditory cortical areas, and what computations such receptive fields might subserve. For example, Młynarski and McDermott, 2018 developed a two-layer generative hierarchical model, where the first layer consisted of spectrotemporal kernels (similar to STRFs) that were trained to sparsely represent natural sounds. Units in the second layer learned time-dependent combinations of first-layer spectrotemporal kernels, also under a sparseness constraint. After training with natural stimuli such as speech and environmental sounds, model units showed several similarities with experimental data. For example, first- and second-layer spectrotemporal kernels showed differences in their modulation transfer functions that were consistent with those measured from the thalamus and cortex (Miller et al., 2002). The feature selectivity of first- and second-layer units were also consistent with previous studies (Atencio et al., 2009). This increasing feature selectivity is reminiscent of that observed across auditory cortical laminae in other species (Montes-Lourido et al., 2021; Sadagopan and Wang, 2009). The fact that model units resembled experimental data thus suggested that the theoretical principles from which the model was constructed - learning dynamic second-order kernels from natural sounds – may also be the computation being performed by cortical neurons. Critically, the model also generated predictions regarding the similarity of second-layer receptive fields (due to the pooling of first-layer neurons) and opponent tuning of second-layer receptive fields that could be tested experimentally. In section 6, we discuss in detail work from our laboratory that proposes another normative model for the observation of complex receptive fields (Liu et al., 2019) that uses the detection of maximally informative features to accomplish auditory categorization tasks.

5. Goal-directed network models

Sensory information in the world is hierarchically organized. In audition, for example, speech is hierarchical at a linguistic level (sentences, phrases, words) as well as at an acoustic level (information content in timescales from tens of milliseconds to seconds). To process information at these varying levels, it can be argued that the brain as an information processing system might therefore adopt a hierarchical structure as well. The ultimate purpose of extracting information at these levels is to guide appropriate behavioral responses. For example, low-level representations of the words ‘Go’ and ‘Stop’ might show variations in pitch, duration, and formant peak locations, but these variations are inconsequential to perform the behaviors associated with these words. Goal-directed models emulate hierarchical architectures to transform early acoustic representations to higher-level representations that facilitate performance in tasks such as stimulus categorization. In this supervised approach, models are trained on a labeled dataset. The input layer typically is a dense representation of the stimulus or stimulus features, and the output layer typically reports the stimulus category. The number of layers between the input and output layers (network depth) varies greatly depending on the task, with more complex mappings requiring greater layer depth. For example, a recent DNN-based automatic speech recognition system that achieved near-human recognition accuracy for conversational speech used a 25-layer network to process stimulus spectrograms (Thomas et al., 2019). Neurons in these layers are modeled as abstract LN units. From an engineering perspective, many previous studies have used deep network architectures and large-scale data driven approaches to tackle issues of auditory object recognition such as phoneme categorization (Lee et al., 2009), extracting vocals from music (Simpson et al., 2015), audio segmentation (Sainath et al., 2007), audio-tagging (Xu et al., 2017), and speech separation (Hershey et al., 2016; Luo et al., 2018). However, only a few studies have used this framework to gain insight into auditory processing in the brain (Kell and McDermott, 2019; Richards et al., 2019). In this section, we will highlight a few such studies.

5.1. Insights into auditory processing from DNN-based models

Güçlü et al., 2016 developed one of the earliest DNN-based auditory models that quantified the similarity between the representation of music in a DNN and in the human brain (using fMRI data). Using representational similarity analyses, the authors found that anterior STG (Superior Temporal Gyrus) was similar to shallower layers of the DNN representing low level features, whereas the posterior STG represented higher level features of music and was more similar to deeper DNN layers. Kell et al., 2018 trained various DNN architectures to perform speech (syllable recognition) and music (genre recognition) tasks. While distinct networks separately trained on each task performed the best, the study showed that networks that shared many early layers of processing (low-level features) and segregated into two pathways at deeper layers could also perform well in these tasks. The activation patterns in such networks strongly resembled human fMRI voxel-activation patterns when subjects were performing the same tasks. These results suggest that earlier layers (in both artificial and biological networks) may efficiently represent stimulus statistics whereas deeper layers show flexible and/or parallel pathways to perform specific tasks. Deeper layers of DNNs also show higher correlations with central and frontal components of EEG activity that originate from higher regions of the auditory processing hierarchy (Huang et al., 2018). This transformation is consistent with sequential nonlinear feature extraction by the auditory pathway, a phenomenon that was also observed in a hierarchical spiking neural network model (Khatami and Escabi, 2020). Here, the authors constructed a six-layer densely connected network of integrate-and-fire units, and optimized connection weights for maximizing performance in a word-in-noise recognition task. The STRFs of model neurons in the trained network resembled those observed in-vivo, with temporal integration durations increasing with layer depth, mirroring the increasing integration windows along the auditory processing hierarchy. The model also captured other experimentally observed phenomena such as increasing sparsity along the hierarchy. The authors suggested that the model is indicative of a lossy transformation, where task-relevant cues are enhanced and noisy or irrelevant cues are discarded from successive neural representations along the hierarchy. However, one aspect of the data that this model did not capture was the increase in auditory receptive field complexity along the frequency axis at the higher levels of the auditory processing hierarchy, including selectivity for harmonic stacks (Feng and Wang, 2017; Norman-Haignere et al., 2013; Patterson et al., 2002; Penagos et al., 2004; Tang et al., 2017) and other complex features (Montes-Lourido et al., 2021; Sadagopan and Wang, 2009).

5.2. Limitations of DNN-based approaches

DNN models are powerful in that they can achieve near-human performance on complex auditory tasks, and can explain a high fraction of the variance of some experimental observations (such as BOLD activity in the auditory cortex). A range of available tools and software facilitate the ease of building and deploying these models, and these models are particularly well-suited for the analysis of large datasets. However, several questions remain regarding the extent to which DNNs can be used to gain insight into biological mechanisms. For example, state-of-the-art DNNs boast tens of layers, which is substantially greater than the number of processing stages in the brain. In comparison to biological systems, DNNs need vast amounts of labeled training data to train and model performance often generalizes poorly to other (even related) tasks unless the model is retrained. Although DNNs can approach or surpass human performance in specific visual and auditory tasks, they exhibit very different patterns of error compared to human subjects (Schrimpf et al., 2020; Weerts et al., 2022). Finally, it is not straightforward how biological/neural mechanisms that are known to improve neural representations in different brain layers can be implemented in these models. For example, DNNs use a single abstract cell type to accomplish the task at hand, whereas the diversity of cell types in both subcortical and cortical stages of the auditory hierarchy is well-established (Arlotta and Paşca, 2019; Osen, 1969; Peruzzi et al., 2000; Winer, 1984; Winer and Morest, 1983). In the visual system, CNNs are powerful because they use the same linear filter to tile visual space to model translation invariance, but it is unclear how this maps onto the auditory system where translation along the cochlea is a critical determinant of auditory object identity. Finally, inputs to DNN models treat short segments of spectrograms as images and do not capture long term temporal dependencies. While recent advances using recurrent architectures and attention units address some of these discrepancies between artificial and biological neural networks (Magnuson et al., 2020), it is likely that current DNN-based models of the auditory pathway approximate some of these complexities by using additional network layers, complicating their biological interpretation. Finally, from the perspective of Marr’s levels of understanding, the computational goal (categorization) and implementation (convolution) levels of DNNs are clear, but the algorithm used to achieve categorization cannot be easily stated. Thus the development of biologically interpretable models is an important avenue of research from a neuroscientific perspective.

6. A biologically realistic and interpretable model of auditory categorization

Over the past few years, our laboratory has developed a hierarchical model that is trained to solve specific categorization tasks but that emphasizes biological realism and interpretability (Liu et al., 2019; Montes-Lourido et al., 2021; Kar et al., 2022; Parida et al., 2022). We adopted a top-down approach inspired by Marr’s levels of understanding and first identified a computational goal or task for the model to solve. To maximize the insight that could be gained about cortical mechanisms from the model, it was critical to choose both stimuli and tasks that would be of sufficient complexity and behavioral relevance to engage cortical pathways. In vocal animals, one such task is the categorization of conspecific vocalizations. Vocal animals (in particular, mammals) such as marmosets (Agamaite et al., 2015) and guinea pigs (Berryman, 1976; Grimsley et al., 2012) use several categories of vocalizations to communicate internal states such as displeasure or fear, external events such as food availability, and social contexts such as affiliative or aggressive intent. Although they are not as complex as human speech, these vocalizations exhibit reasonable spectrotemporal complexity, containing features such as frequency-modulated sweeps, harmonic stacks, and temporally repeating segments. Additionally, within each species, vocalization types often overlap in spectral content (likely constrained by the biomechanics of that species’ vocal tract), which means that animals cannot determine category based on spectral content alone. But critically, these vocalizations are produced with high trial-to-trial and inter-subject variability. To produce appropriate behavioral responses, the auditory system must generalize over within-category variability and ascertain the category of a given vocalization, a complex task. Thus, vocalizations are complex stimuli that are behaviorally relevant, and categorizing them is a complex task, meeting the conditions necessary for robust cortical engagement. Experimentally as well, while simple stimuli such as pure tones do not drive robust activity in higher cortical areas (Bendor and Wang, 2008; Rauschecker and Tian, 2000), vocalizations are known to evoke robust responses in secondary and higher cortical areas (Rauschecker et al., 1995; Tian et al., 2001) as well as in prefrontal regions (Cohen et al., 2007; Gifford et al., 2005; Romanski and Goldman-Rakic, 2002). Specialized regions that preferentially process vocalizations may exist in higher auditory cortical regions of humans (Belin, 2006; Bodin et al., 2021), macaques (Perrodin et al., 2011; Petkov et al., 2008), and marmosets (Sadagopan et al., 2015). Thus, it is reasonable to posit that vocalization categorization is one computational goal of auditory cortical processing.

Having identified a computational goal, we next asked how this goal could best be achieved, i.e., we constructed a normative model to determine the algorithms the auditory system might use to solve vocalization categorization. In this endeavor, we looked at parallels in the visual system. Like vocalizations, faces are complex objects, are behaviorally relevant, and exhibit tremendous within-category variability. Cortical regions at the highest stages of the visual processing hierarchy seem to preferentially respond to faces over other objects (Freiwald and Tsao, 2010; Tsao et al., 2003; Tsao and Livingstone, 2008). Thus, despite the different sensory modalities, and while recognizing that faces and vocalizations are not comparable in stimulus space (face information is available instantaneously in two spatial dimensions whereas vocalization information is only available in one dimension over time), the apparent similarities in how faces and vocalizations are represented in the visual and auditory hierarchies led us to examine algorithms for face categorization. Of these, a categorization algorithm based on the detection of maximally informative features of intermediate complexity (Ullman et al., 2016, 2002) stood out as a particularly suitable candidate for adaptation to vocalization categorization.

In this approach, the objective was to identify a set of intermediate-sized features that (as a set) best distinguishes between the category of interest (a particular vocalization type), and all other categories. For our model (Liu et al., 2019), the first step was to use a suitable front end, such as a biophysically realistic auditory nerve model (Zilany et al., 2014), to generate biologically realistic input representations (cochleagrams) of a training set of within-category and outside-category stimuli. Next, a large number of random candidate features were generated from the within-category stimuli. Each feature corresponded to the spatiotemporal activity of a random subset of auditory nerve fibers within a random time window (a rectangular region of the cochleagram). These features may be conceptualized as the linear receptive field of a neuron that would be best driven by that feature. Then, we used a template-matching algorithm (normalized cross-correlation) to determine how well each candidate feature “matched” within-category and outside-category vocalizations in the training stimulus set. We convolved each feature with the stimulus (restricted to the bandwidth of the feature), resulting in a one-dimensional vector of normalized cross-correlation values, which may be conceptualized as the membrane potential fluctuations of the feature-sensitive neuron. The distributions of the maximum normalized cross-correlation values for the within-category and outside-category stimuli determine the usefulness of the feature in categorization (Fig. 1C). For example, a good feature would result in high maximum cross-correlation values for within-category stimuli and low cross-correlation values for outside-category stimuli, with little overlap between the two distributions. For each feature, we determined an optimal threshold for feature detection at which within- and outside-category distributions were best separated, with the feature deemed present in the stimulus if the cross-correlation value exceeded this threshold. At this optimal threshold, we also determined the merit or usefulness of the feature for categorization using mutual information, and a weight defined as the feature’s log-likelihood ratio. The threshold may be conceptualized as the spiking threshold of a feature-selective neuron – when the membrane potential (cross-correlation) exceeds this threshold, the feature-selective neuron spikes, signifying the presence of its preferred feature. Thus, in this framework, each feature-selective neuron is an LN unit with a linear receptive field (feature) and a stationary nonlinearity (optimal threshold).

The critical computation in the model was how a final set of features was chosen from this initial randomly generated set. In the initial set, because of the random nature of candidate feature generation, many features were not informative, and the best features could be redundant (i.e., encoding the same stimulus features). Thus, as with the face categorization studies (Ullman et al., 2002), we used greedy search optimization to select, from this initial random set, a most informative feature (MIF) set that maximized categorization performance while minimizing self-redundancy. Briefly, we chose the best performing feature as the first member of the MIF set. The second member of the MIF set was the feature that, together with the first member, added the most information for categorization. We sequentially chose members that added the most information for categorization in a pairwise fashion until no more information could be added or a stopping criterion was reached (e.g., added information < 0.001 bits; see Liu et al., 2019 for details). At the end of this procedure, we were left with MIF sets consisting of a handful (10 – 20) of features for categorizing each vocalization type. For any given stimulus, we convolved all MIFs with the stimulus and determined which MIFs were detected, i.e., which of the maximum normalized cross-correlation values crossed the MIFs’ respective thresholds. The total evidence for the presence of a stimulus category was then taken to be the sum of the weights (log-likelihoods) of the detected features (Fig. 1C). We demonstrated that using this set of MIFs alone was sufficient for the model to generalize over the training vocalization set, as well as to a new testing set of within- and outside-category vocalizations. The model achieved high categorization performance (>95% accuracy) on vocalization categorization tasks across several species (marmosets, guinea pigs, and macaques). Additionally, using the same algorithmic strategy, we achieved high performance on other auditory tasks (caller recognition for marmoset twitter calls). Interestingly, similar to face categorization, features of intermediate size (in terms of duration and bandwidth) and complexity were most informative for vocalization categorization. When features were too large (encompassing the entire vocalization), generalization to other within-category exemplars was impaired; when features were too small (only integrating over a few fibers for a few milliseconds), their frequency of occurrence was similar in within- and outside-category vocalizations.

Thus, in our model, the algorithmic solution for the categorization goal was to detect a set of most-informative features that were best-performing as a set and that were least self-similar. Our model can be conceptualized as a three-layer network – the MIF units form a sparse feature-detector layer that integrates inputs from a dense spectrotemporal (cochleagram) layer (Figure 1A). The MIFs are biologically interpretable because they were derived to perform a specific and well-defined task – the set is chosen to maximize performance and minimize redundancy. Individual MIFs are LN units that spatiotemporally integrate over a small number of input units. A final output layer linearly weights detected features from the feature-detector layer and reports category. Thus, the model architecture is also interpretable and biologically realistic. MIFs can be learned from tens to hundreds of examples. However, the task specificity of the MIFs – a distinct set of MIFs need to be learned to categorize each vocalization type or caller identity – also makes our algorithm quite inefficient. The conceptualization that each task requires a distinct set of features is supported by some experiments. For example, face-selective (Freiwald and Tsao, 2010; Tsao et al., 2003) and vocalization-selective (Perrodin et al., 2011; Grimsley et al., 2012) neurons exist in higher visual and auditory cortex, respectively. But the true value of the model lies in its ability to generate biologically grounded hypotheses and experimentally testable predictions, as detailed below.

First, the model predicted an increase in feature-selectivity between two layers of processing in the auditory hierarchy, i.e., between a layer that densely represents all features of the stimulus (Fig. 2A, bottom) and a layer that sparsely represents specific vocalization features (Fig. 2A, top). By recording from three stages of the auditory hierarchy – the thalamus, the thalamorecipient layers (A1 L4), and the superficial layers (A1 L2/3) - we demonstrated such an increase between A1 L4 and A1 L2/3 in guinea pigs (Montes-Lourido et al., 2021). The receptive fields of A1 L4 neurons were selective for frequency (Fig. 2B, bottom), resulting in dense responses to many vocalization categories, whereas the receptive fields of A1 L2/3 neurons were selective for more complex features (Fig. 2B, top), resulting in selective responses to one or a few vocalization categories. Single spikes of A1 L2/3 neurons were more informative about vocalization category than the spikes of A1 L4 neurons. Thus, we could map the layers of the model architecture to A1 laminae and provide support for the model at the electrophysiological level. Note that our claim is not that the precise features identified by our algorithm are the ones represented in the cortex; rather, we argue that the algorithmic principle of selecting the most contrastive and least redundant features is also implemented in cortex. Further experiments are needed to quantify the information content and redundancy of experimentally recorded feature-selective neurons and determine how they compare with model feature-selective neurons. The neural mechanisms by which feature-selectivity in A1 L2/3 can be generated, how and where in auditory cortex feature-selective responses are combined to produce a categorical output, and whether the temporal ordering detected features is important for category determination are all critical open questions.

Figure 2: Schematic of feature-based hierarchical model for auditory categorization.

(A) Model architecture. In the first model layer (i.e., the cochleagram), the spectrotemporal content of the stimulus is represented in a dense manner. Individual model excitatory neurons (gray) in this layer show classic frequency-selective receptive fields (simplified schematics of receptive fields shown as insets). In the second layer, individual model excitatory neurons are selective for stimulus features that are maximally informative for performing specific tasks such as the categorization of conspecific vocalizations. These model neurons show receptive fields of intermediate complexity (simplified schematics of feature-selective receptive fields are shown as insets). A third layer (not shown) pools detected features to output a category decision. Results from electrophysiological recordings suggest that the responses and receptive fields of actual neurons in A1 L4 and A1 2/3 are consistent with those of model neurons in the cochleagram and feature-selective layers. Putative inhibitory neurons in these layers (blue and green cells) modulate neuronal excitability. (B) Additive noise imposed by the listening environment alters the statistics of the auditory input (increase in mean and decrease in variance, histograms). In the input layer, putative inhibitory-neuron-based mechanisms help denoise the input representation by normalizing population activity (gain control). (C) Additive noise also decreases the ‘match’ between feature templates and the inputs, resulting in distributions of correlation values that are shifted to lower values (histograms). Cholinergic inputs recruited during increased listening effort may, through cortical disinhibitory circuit motifs, increase the excitability of feature-selective neurons and help separate within-category and outside-category stimuli.

Second, by replacing the categorical output stage with a trial-by-trial decision stage, the model could be used to predict behavioral responses to natural and modified vocalization stimuli in a Go/No-go task (Kar et al., 2022). We modeled the decision stage using a simple winner-take-all framework – for each category, all the features that spiked to a given stimulus were weighted by their log-likelihood ratios and summed to obtain the total evidence for the presence of each category, and the category with the highest evidence was taken to be model output category. We trained the model on categorization tasks using natural guinea pig vocalizations and tested the model with both novel natural vocalizations and vocalizations altered using various spectrotemporal manipulations. The categorization performance of the model was sensitive to some manipulations (addition of noise and frequency shifting; Fig. 2D, E) and insensitive to others (temporal stretching). We also trained guinea pigs on the same categorization tasks and tested them using the same manipulated stimuli. Guinea pigs showed similar sensitivity to the manipulations as the model (Fig. 2D, E), with the model explaining about 50% of the variance in guinea pig behavior (Kar et al., 2022). These experiments provided support for the model at the behavioral level. In addition, by examining which features the model relied on for task performance in these conditions, we could generate additional predictions regarding how features with distinct characteristics would be flexibly used in different contexts. Further experiments to test these predictions are underway. The decision stage may be conceptualized as a downstream area integrating feature-selective responses, or as a computation implemented as a mutual inhibition between feature-selective neurons in A1 L2/3 (Chettih and Harvey, 2019). Further experiments are necessary to distinguish between these possibilities.

The MIFs capture within-category variations that are random fluctuations about an archetypal feature that are caused during production. However, other variations that are imposed by the listening environment, such as the addition of noise or reverberations, do not follow this pattern. Noise results in both decreased overall cross-correlation values between the stimulus and the feature, as well as decreased separation between the within- and outside-category distributions (Fig. 1B, C). Thus, the MIF-based approach fails when no normalization is used for feature detection (Parida et al., 2022). This shortcoming could be ameliorated to an extent by training the MIFs in noisy and reverberant conditions, which is biologically realistic – it is reasonable to expect that during early development, animals acquire vocalization categories in a range of acoustic conditions. However, the benefits of training in one condition (say, noise) did not generalize to other conditions (say, reverberation). Therefore, we developed two modules based on experimental observations to improve model performance in adverse conditions. First, guided by contrast gain control studies (Rabinowitz et al., 2011; Willmore et al., 2014), we implemented normalization in the dense spectrotemporal layer (Fig. 1B). This normalization mimicked inhibitory neuron-based contrast gain control (Atallah et al., 2012; Carandini and Heeger, 1994; Wilson et al., 2012, but see Cooke et al., 2020) to maintain the overall activity level in the spectrotemporal stage across listening conditions. Shifts to the distribution of cross-correlation values in noisy conditions were lower after normalization, but the benefits of normalization varied widely across vocalization categories. Second, inspired by recent studies demonstrating how arousal may modulate cortical activity by activating specific inhibitory neuronal populations (vasointestinal peptide expressing or VIP interneurons) (Pi et al., 2013), we implemented the effects of listening effort as a top-down modulation of feature-detector excitability. Here, increased listening effort activated VIP interneurons that in turn inhibited other cortical interneurons, with the net effect being increased cortical excitability (Fig.1C). Thus, although the distributions of cross-correlation values were shifted to lower values, the thresholds of the feature detecting neurons were also proportionally decreased to values that better separated within- and outside-class distributions. This top-down approach resulted in more uniform benefits across noisy conditions, and sometimes out-performed animals in categorization-in-noise tasks (Parida et al., 2022). Interestingly, our implementation of top-down listening effort provided limited benefit in reverberant conditions, consistent with observations from human psychoacoustics (McCloy et al., 2017; Picou et al., 2016; Prodi and Visentin, 2022). Overall, these results highlight the modular nature of our model and demonstrate how experimental observations can be incorporated into the basic hierarchical architecture of the model.

7. Conclusion

In this review, we have discussed a selection of quantitative models ranging from those that use detailed biophysical data to explain experimental observations to those that use large-scale neural network approaches to solve specific tasks. These approaches are characterized by different motivations (modeling a behavioral process vs. exploring the necessity of a specific channel) and varied levels of abstraction (detailed biophysics vs. integrate-and-fire units). All of these models succeed at explaining specific experimental phenomena, but it is still not clear how these different models interact. Some examples suggest that the same underlying mechanisms could indeed contribute to different phenomena – for example, synaptic depression has been used to explain features of stimulus-specific adaptation (Lee and Sherman, 2008; Vanattou-Saïfoudine et al., 2021; Yarden and Nelken, 2017), and also to explain reverberation- and noise-resistant representation of sounds (David and Shamma, 2013; Mesgarani et al., 2014). Disinhibitory circuit motifs developed to explain the modulation of cortical activity by internal state (Letzkus et al., 2011; Pi et al., 2013) may also prove useful in auditory categorization in noise (Parida et al., 2022). A deeper exploration of such inter-relationships could lead to novel insight into auditory processing and generate novel predictions. As our ability to acquire large experimental datasets increases, and as tools to incorporate increasing levels of biological details into large-scale models become available (e.g., Olah et al., 2022), a close collaboration between modelers generating testable predictions and experimentalists testing these predictions and generating new data and constraints for modeling is indispensable. It is our hope that interest in and support for such collaborations will continue to increase in the coming years.

Figure 3: Model comparisons to electrophysiological and behavioral data.

(A) (bottom) STRFs of two model units in the model’s dense cochleagram layer (Layer 1) and PSTHs (blue lines) of their responses to a sequence of guinea pig vocalizations. Gray dashed lines separate two example vocalizations from each labeled category. These model neurons exhibit compact, frequency-selective STRFs and respond to numerous vocalization types. (top) STRFs of a wheek MIF and a whine MIF from the feature-detector layer (Layer 2) of the model. These model neurons exhibit complex STRFs and respond to only a few call types. (B) Electrophysiological data recorded from neurons in A1 of awake, passively listening guinea pigs. (bottom) Like model Layer 1 neurons, A1 L4 neurons exhibited compact and frequency-selective STRFs and responded to numerous call types. (top) Like model Layer 2 neurons, many A1 L2/3 neurons exhibited complex STRFs and responded to only a few vocalization types. (C) We trained five models with non-overlapping MIF sets (black circles and lines) and a winner-take-all decision stage, and two guinea pigs (green and purple discs) on a categorization task in a Go/No-go framework, where the objective was to categorize wheek vocalizations from other (purr, chut, and whine) vocalizations. We measured performance using the sensitivity index d’. Guinea pigs learned the task in about a month of training (note that these two guinea pigs were previously trained on a wheek vs. whine task). Both guinea pigs as well as the model generalized to novel stimuli equally well. (D) The performance of both guinea pigs as well as the model degraded with the addition of white noise. Although the model overperformed in this task, it explained about 65% of the variance in guinea pig behavior as quantified using R-squared (R²). (E) The performance of both guinea pigs as well as the model were similarly affected by pitch-shifts to the vocalization stimuli. GP, Guinea pig; MAE, median absolute error.