Efficient neural coding in auditory and speech perception

Abstract

Speech has long been recognized as “special”. Here, we suggest that one of the reasons for speech being special is that our auditory system has evolved to encode it in an efficient, optimal way. The theory of efficient neural coding argues that our perceptual systems have evolved to encode environmental stimuli in the most efficient way. Mathematically, this can be achieved if the optimally efficient codes match the statistics of the signals they represent. Experimental evidence suggests that the auditory code is optimal in this mathematical sense: statistical properties of speech closely match response properties of the cochlea, the auditory nerve, and the auditory cortex. Even more interestingly, these results may be linked to phenomena in auditory and speech perception.

The relevance of efficient neural coding for speech perception

Speech has long been recognized as “special” [1–6]. We prefer it over other sounds from birth onwards [6], and we are able to make fine-grained discriminations that allows us to convey an infinite amount of messages. The special status of speech has been studied from a variety of perspectives. Researchers of social cognition approach it as our species-specific communicative signal, and as the basis of learning and cultural transmission [7, 8]. Others have claimed that speech is special because it is the only auditory signal that we ‘feel’, in the sense of perceiving the movement of our articulators, when producing it [1]. Here, we review experimental evidence for the hypothesis that speech is special for another reason – i.e, because our auditory system has evolved to encode it in an efficient way.

Organisms need to process the environmental signals they encounter, and the efficiency with which they do so may considerably impact their survival. The ability to process environmental signals efficiently is, therefore, assumed to be an important principle shaping the evolution of the sensory systems. Specifically, the theory of efficient neural coding [9, 10] argues that our perceptual systems have evolved to encode environmental stimuli in the most efficient way. Information theory provides a mathematically precise and empirically testable framework to evaluate this theory. It defines efficient or optimal coding as one that transmits the highest fidelity information at the lowest cost, i.e. if the encoding maximally reduces the redundancy in the signal. Mathematically, this can be achieved if the optimally efficient codes match the statistics of the signals they represent [11].

In the last decades, the hypothesis that the neural code used by the perceptual systems is optimal in this mathematical sense has gained considerable empirical and theoretical support in vision [12]. More recently, experimental evidence has suggested that the auditory code may also be optimal [13–17]. Here, we link these findings to auditory perception, with special attention to speech perception. Conceiving of speech as an auditory signal that is particularly well suited to match the encoding capabilities of the auditory system may contribute to a better understanding of speech perception phenomena and the “special” nature of speech. This hypothesis is now gaining momentum [13, 14, 18–20], motivating the current Review.

The statistical structure of sounds

To test whether the mammalian auditory system codes sound in a mathematically optimal way, it is first necessary to describe the statistical structure of sounds. The space (in the mathematical sense) of all potential sounds is vast (Box 1). Within this space, natural sounds including speech comprise a compact, yet multi-dimensional subspace. Analyses of statistical regularities in natural sounds have identified several prominent features. The temporal structure of many natural environmental sounds has a self-similar property: its power spectrum scales as 1/f [21], which means that the signals exhibit correlations across multiple time scales. These spectral correlations translate into statistical dependencies across frequency and time, which can be captured with a histogram of the statistical features of sounds in the spectro-temporal domain [22] (Figure 1A). These dependencies can be encoded by a neuronal population that processes the inputs at multiple time scales with varying degrees of resolution across scales [23, 24]. Scale-invariant dependency occurs not just within the amplitude spectrum of sounds, but also across spectral bands: if we consider the spectrogram of a natural sound, we observe that the temporal fluctuations occur on a faster timescale in higher frequency bands than in lower frequency bands. As result, the temporal correlations in the spectrogram are shorter at high than at low frequencies [25, 26].

Box 1: The efficient auditory coding hypothesis.

The potential space (in the abstract, mathematical sense) of all possible sounds is vast, but environmental sounds, animal vocalizations, and speech, occupy specific subspaces. These subspaces are determined by the spectro-temporal statistics of the acoustic properties of sounds from different groups (Figure I–A). It is, however, difficult to identify the relevant dimensions in the sound space. Research in exploring the space of environmental, vocalization and speech sounds has focused on complementary approaches (Figure I–B): (i) by shaping random noise according to some statistical constraints to generate sounds from different groups, or (ii) by using recorded sounds, and applying directed perturbations to these sounds along specific dimensions within this complex space to produce distorted sounds. The first approach allows to test whether a particular statistical constraint is sufficient to define a sound category. The second approach tests whether the particular statistical constraint is required to define a sound category. We propose that throughout evolutionary history and during human development, transformations in the constraints over the perceived sounds produce an auditory code that encodes speech in an efficient fashion (Figure I–C).

Figure I:

Framework for understanding the development of speech selectivity. A. Diagram of the spectro-temporal statistical space of different types of sounds projected on a subset of dimensions, including d1, Spectro-temporal correlation; d2, frequency modulation (FM); d3, Frequency; d4, scale-invariant coefficient; dn -- other components to be identified. B. Diagram of complementary methods to identify the relevant dimension. C. Speech statistics are shapes throughout evolution and development.

Figure 1.

Spectrotemporal characteristics of different types of sounds. (A) The time-frequency histogram of a speech signal shows that speech shares characteristics with both animal vocalizations and environmental sounds [adapted from 22]; (B) The spectro-temporal characteristics of mathematically computed optimal filters also suggest that speech resembles both vocalizations and environmental sounds [adapted from 13].

Interestingly, the relation between frequency and temporal correlations drives differential perception of sounds that are generated under this statistical relationship. Varying the value of a single statistical parameter that controls the correlation within the temporal structure can yield a range of sound percepts, to which both adults and infants exhibit sensitivity [26]. More generally, controlling a small number of statistical parameters for first- and second-order distributions of the means and variance of spectro-temporal channel components of sounds can reproduce “sound textures”, yielding percepts that range from a chorus of insects to helicopter sounds [15, 27]. In contrast to environmental sounds, mammalian vocalizations, which often have a strongly harmonic structure, show peaks over their 1/f spectra, corresponding to the fundamental frequency and its harmonics [16].

The speech signal shows properties of both environmental sounds and harmonic vocalizations. Globally, speech also has a 1/f spectrum [21]. More locally, vowels have a harmonic structure, while different consonant classes show acoustic transience to different degrees, resulting in systematic variations in the local statistical structure of the speech signal. This led to conceptualizing speech as a modulated carrier signal [28, 29]. The carrier signal produced by the vocal folds is modulated slowly in amplitude and frequency as a result of the dynamic changes of the vocal tract during phonation. The amplitude modulation corresponds to the envelope of the speech signal, while the frequency modulation to its temporal fine structure. The amplitude and frequency modulations can be obtained using a Hilbert transform applied to the speech signal. Based on the modulated carrier signal view of speech, powerful analysis and synthesis algorithms have been developed, called vocoders, which can selectively manipulate the acoustic components of speech [30]. The amplitude and frequency modulation spectra of the speech signal have recently received considerable attention. It has been shown that the amplitude modulation spectrum, believed to correspond to the percept of speech rhythm, has a peak between 4–5Hz (Figure 2). This temporal modulation is found across a wide range of different languages [31, 32], with slight variations corresponding to well-established rhythmic and other prosodic differences between them [32].

Figure 2.

The amplitude modulation spectra of speech in different languages. A. The amplitude modulation spectra of speech in 10 different languages from a database of clearly articulated, well-controlled recordings show both a strong similarity with a modulation peak at around 4Hz as well as slight differences across languages in the strength and rate of modulation [adapted from 32]. B. The amplitude modulation spectra of speech in 9 different languages from large corpora with a wide range of different speech styles and registers show the same ~4Hz modulation peak, but not the smaller cross-linguistic differences [adapted from 31].

Non-redundant, optimal mathematical models of sounds

According to the efficient coding hypothesis, the brain has evolved to efficiently process and respond to stimuli that occur in nature, reducing redundancy in their neural representations [9]. This principle posits that the statistical properties of neuronal responses should match the statistical structure of natural stimuli, and should maximize the efficiency in representation [10, 33]. This is best achieved if neuronal responses constitute a sparse, non-redundant code, meaning that the code should be as parsimonious as possible, yet capture the full range of variability in the signal structure along the relevant dimensions [34].

Following these principles, recent studies have derived sparse codes for different categories of sounds, and compared them to the response properties of components of the auditory pathway. So far, to our knowledge, two mathematical approaches have been used. The first [13, 18, 19, 35] uses independent component analysis (ICA). Imposing a sparsity constraint improves the decomposition of sounds into independent components [35] – a statistical analysis that identifies the most informative dimensions in the spectro-temporal space of sounds. Optimal filter populations derived using ICA [13] for three categories of sounds, i.e. environmental sounds, animal vocalizations and speech, differ in their spectro-temporal properties, reflecting the statistics of the sounds classes (Figure 1B). Thus the optimal filters for animal sounds resemble a Fourier decomposition, in conformity with the harmonic structure of these sounds; the filters for environmental sounds approximate wavelets, reflecting the fast transients in these sounds; whereas the filters for speech are in between these two representations. Importantly, the filters derived for speech as well as for a mix of environmental sounds and animal vocalizations, but not for environmental sounds alone or vocalizations alone, very closely match auditory nerve fiber tuning properties. At the more local level, filter populations for vowels match the global properties of speech, whereas those of different consonant classes vary from Fourier-like to wavelet-like representations. Crucially, these filter populations are well aligned with the response properties of cochlear nuclei [18]. Interestingly, the filter populations for speech in different languages match well the acoustic correlates of the percept of speech rhythm [19]. Furthermore, the basis for binaural sounds reproduced sound localization networks with a small number of components, suggesting that sound localization can be carried out with reduced representation that is optimized to distribution of binaural dependencies in the natural world [36].

As a second mathematical approach, sparse coding models have been proposed. Identifying a sparse and efficient representation of sounds in terms of spikes, imposing a sparse binary code constraint on sound encoding, replicates encoding features observed in the mammalian auditory system [15]. Specifically, the spike code representation of speech approximates time-domain cochlear filter estimates, and the frequency-bandwidth dependence of auditory nerve fibers.

These non-redundant codes stand in contrast to more traditional representations of sound in terms of a waveform over different spectral bands, such as a spectrogram or cochleogram. These traditional representations require a large number of parameters to fit the sound waveform. The assumption of sparsity in acoustic signals reduces the number of parameters required to represent a sound waveform.

Sounds with naturalistic statistics are special for the mammalian auditory system

According to the efficient coding hypothesis, identifying the statistical dependencies in the structure of sounds yields insight into the structure of the neuronal code. This was tested by constructing artificial codes that were optimized according to some set of constraints to best represent natural sounds, and then compared to experimental measurements of responses of neurons in the auditory pathway. Such advanced mathematical models were, for instance, used to better understand the structure of receptive fields of auditory neurons. The assumption of sparsity responses in conjunction with analysis of a library of sounds yields spectro-temporal filters with different spectro-temporal relations that capture the diversity observed in the auditory pathway [24]. Using independent component analysis on a library of natural and speech sounds furthermore yielded a correlation between the bandwidth and center frequency of tuning, and predicted overrepresentation of the frequency of an overexposed tone. Such a relationship was identified experimentally for primary auditory cortical neurons [37]. Imposing a sparse coding constraint on natural sounds yielded single and multi-peaked frequency response units, such as found in the primate A1 [38]. Enforcing sparseness and a specific form of scaling of inputs, termed divisive normalization, in a network of neurons, reproduced the set of auditory features within the auditory processing pathway [39]. Extending this code by adding a layer of neuronal connectivity captured the non-uniform distribution of spatial tuning that was observed experimentally in mammals [40].

Natural sounds, or sounds that exhibit naturalistic statistics, evoke enhanced responses throughout the auditory system. Neurons throughout the auditory pathway represent complex sounds as a population that integrates across neurons tuned to different spectro-temporal features of sounds. These can be described through the spectro-temporal receptive fields of neurons (STRF) [41, 42]. The STRF of neurons indicates the range of frequencies and transformations in time in the stimulus amplitude that evoke a strong response and can predict the ability of neurons to represent and discriminate between complex sounds [43, 44]. STRFs are typically determined from a set of randomized sounds that obey certain statistical constraints. Modifying those statistical constraints to capture statistics of natural sounds, such as con-specific vocalizations, amplifies cortical responses [43, 45–47] (Figure 3). Furthermore, modifying the stimulus to exhibit a 1/f frequency spectrum yields tuned responses [48–50]. Indeed, such dependence is consistent with the 1/f structure of responses within the cochlea [51], and the enhanced information transmission at the level of auditory primary afferents [17] for naturalistic stimuli.

Figure 3.

Predictive model for neuronal responses to a sequence of con-specific vocalizations in awake rat A1 is improved through low-dimensional parametrization of the stimulus [adapted from 46]. A. stimulus waveform, which consisted of rat ultra-sonic-vocalizations, concatenated at the naturalistic rate of production of 10 Hz. B, E. Representation of the stimulus in: (B) a 2-dimensional space of frequency modulation and amplitude or (E) as a spectrogram. C, F. Linear filters for responses of the neuron for the two models. D, G. Instantaneous non-linearities used for the two models. H. Firing rate (black), and model prediction based on 2-parameter generalized linear model (GLNM, red) and on the full spectrogram linear-non-linear model (STRF LNM, green).

Particularly relevant to auditory communication are con-specific vocalizations. In many species, auditory cortical neurons exhibit enhanced tuning for natural vocalizations [52–56]. Vocalizations are encoded at higher information rate when their statistics are unperturbed [57]. Furthermore, predictive models for auditory processing were able to predict activity more accurately in the primary auditory cortex when the stimulus was comprised of sounds with the statistical structure of con-specific vocalizations [46]. Interestingly, the responses of cortical neurons in mammals to speech can approach estimates for perceptual speech discrimination [58], and neuronal responses to phonetic features of speech sounds can be related to their spectrotemporal tuning properties [59, 60].

Can efficient coding explain perception?

Few studies to date have directly addressed whether efficient coding principles can account for auditory percepts. Among these, one series of studies [25, 26, 61] tested how human adults, infants and newborns perceive water sounds generated by a mathematical model (Figure 4) that consisted of a population of randomly spaced gamma tone chirps from a wide range of frequencies [25]. This model generated scale-invariant sounds when the temporal structure of the chirps scaled relative to their center frequency, and variable-scale sounds when chirps in different spectral bands varied in their temporal structure relative to their center frequency. Adults rated the scale-invariant sounds generated by the model as natural, and qualitatively described them as water sounds (e.g. rain, shower, ocean etc.), whereas they rated the variable scale sounds as unnatural and qualitatively described them as noise or machine-like sounds [25], suggesting that scale-invariance is indeed a statistical property underlying the percept of naturalness in sounds. Similarly, when habituated with the same scale-invariant water sounds, 5-month-old infants readily dishabituated to the variable-scale sounds, suggesting that they formed a perceptual category for the scale-invariant sounds during habituation. If, however, they were habituated to the variable-scale sounds, they did not dishabituate when hearing the scale-invariant ones, indicating that they did not perceive the variable-scale sounds as constituting a well-formed perceptual category [26]. The perceptual advantage of scale-invariant sounds appears to be present even earlier in human development: The newborn brain also discriminates between the scale-invariant and variable-scale water sounds [61].

Figure 4.

The mathematical model of water sounds [25]. A scale-invariant or variable-scale random sound was constructed as superposition of gammatone chirps with the same time and frequency. Left: The onset timing and center frequency of each gammatone used for both scale-invariant and variable scale sounds. Right insets: two representative gammatones for either sound: Scale invariant gammatones had a constant cycle constant of decay, whereas variable scale sounds had a constant time constant of delay; as result, scale-invariant sounds differed in duration across frequencies, whereas variable scale sounds did not. In perceptual judgement experiments, adult subjects rated scale-invariant, but not variable-scale sounds as natural for a wide range of parameters.

Speech perception may also obey efficient coding principles. When speech is degraded by preserving only 6 frequency bands in a noise vocoder, adult listeners’ speech recognition performance is better if the vocoder uses mathematically derived efficient filters rather than linear or cochleotopic filters [14]. Speech perception also shows scale-invariance in time: adult [62–66], child [67, 68] and even newborn [20] listeners readily adapt to time-compressed speech in their native language as well as in rhythmically similar non-native languages. This indicates that adaptation happens at the auditory, rather than at the abstract linguistic level, confirming the auditory system’s ability to encode scale-invariance in time. It needs to be noted, however, that listeners can only adapt to speech compressed maximally to about 30% of its original duration. Beyond that, adaptation breaks down. Some researchers thus interpret adaptation to time-compressed speech not as a manifestation of scale-invariant processing [69]. Rather, it is taken as evidence in favor of the multi-time scale model of speech perception [70–72], which posits that speech is simultaneously processed at a few privileged time-scales, roughly corresponding to the linguistic units of (sub)phonemes, syllables and phrases, sustained in the brain by a hierarchy of embedded neural oscillations in the low gamma (25–35Hz), theta (4–8Hz) and delta (1–2Hz) bands. This model predicts that speech perception is not fully scale-invariant in time. Rather adaptation to compression is only possible if the rhythm of the signal remains within these privileged frequency ranges, and the lower limit on listeners’ ability to adapt to compression is seen as an indication that the theta rhythm is no longer maintained, making speech perception impossible.

Concluding remarks and Future Perspectives

The research findings discussed in this Review suggest that auditory perception may obey the principles of efficient neural coding, relying on the information theoretical notion of optimality. The existing studies demonstrate that the approaches for understanding the mathematical structure of sounds can yield predictions about neuronal encoding throughout the auditory pathway. The correspondence between neuronal responses and model predictions, conversely, is consistent with the notion that the neuronal representation of sounds is optimized for the statistical features of sounds found in nature. The auditory neural code appears to be particularly well matched to the statistical properties of speech.

The efficient neural coding approach opens up an interesting perspective on auditory and speech perception. Nevertheless, a number of issues remain unresolved (see Outstanding Questions). First, efficient coding assumes that neural representations are optimal. This stands in apparent contrast to the redundancy that is well attested in biological systems. Since both signals and the computing units (neurons) are noisy, and can be damaged, a certain amount of redundancy is necessary and even desirable to make auditory representations robust and resilient. Future models of efficient auditory coding will need to take into account the need for resilience in the face of noise or damage.

Second, the general mathematical principle of coding efficiency does not specify the aspects of the signal that need to be encoded, nor the neural structures that are involved. The theory leaves underspecified whether efficient coding principles should operate at the level of individual neurons, neuronal assemblies or even larger structures. For a better understanding of these issues, efficient coding models need to be integrated with anatomical and neurophysiological as well as acoustic and linguistic accounts.

Third, it remains open how the efficient neural coding account relates to other theories of auditory perception. As discussed above, the temporal scale-invariance prediction of the efficient coding of speech stand in (apparent) contradiction with the multiple time scale model of speech perception [70]. Whether these models may be integrated, and if so how, remains an important question for future research.