A convolutional neural-network model of human cochlear mechanics and filter tuning for real-time applications

Abstract

Auditory models are commonly used as feature extractors for automatic speech-recognition systems or as front-ends for robotics, machine-hearing and hearing-aid applications. Although auditory models can capture the biophysical and nonlinear properties of human hearing in great detail, these biophysical models are computationally expensive and cannot be used in real-time applications. We present a hybrid approach where convolutional neural networks are combined with computational neuroscience to yield a real-time end-to-end model for human cochlear mechanics, including level-dependent filter tuning (CoNNear). The CoNNear model was trained on acoustic speech material and its performance and applicability were evaluated using (unseen) sound stimuli commonly employed in cochlear mechanics research. The CoNNear model accurately simulates human cochlear frequency selectivity and its dependence on sound intensity, an essential quality for robust speech intelligibility at negative speech-to-background-noise ratios. The CoNNear architecture is based on parallel and differentiable computations and has the power to achieve real-time human performance. These unique CoNNear features will enable the next generation of human-like machine-hearing applications.

Introduction

The human cochlea is an active, nonlinear system that transforms sound-induced vibrations of the middle-ear bones to cochlear travelling waves of basilar-membrane (BM) motion [1]. Cochlear mechanics and travelling waves are responsible for hallmark features of mammalian hearing, including the level-dependent frequency selectivity [2–5] that results from a cascade of cochlear mechanical filters with centre frequencies (CFs) between 20 kHz and 40 Hz from the human cochlear base to apex [6].

Modelling cochlear mechanics has been an active field of research because computational methods can help characterise the mechanisms underlying normal or impaired hearing and thereby improve hearing diagnostics [7,8] and treatment [9,10], or inspire machine-hearing applications [11,12]. One popular approach involves representing the cochlea as a transmission line (TL) by discretising the space along the BM and describing each section using a system of ordinary differential equations that approximate the local biophysical filter characteristics (Fig. 1a; state-of-the-art model) [13–19]. Analytical TL models represent the cochlea as a cascaded system in which the response of each section depends on the responses of all previous sections. This architecture makes them computationally expensive, as the filter operations in the different sections cannot be computed in parallel. The computational complexity is even greater when nonlinearities or feedback pathways are included to faithfully approximate cochlear mechanics [15,20].

Fig 1. CoNNear Overview.

CoNNear is a fully convolutional encoder-decoder neural network with strided convolutions and skip-connections to map audio input to 201 basilar-membrane vibration outputs of different cochlear sections (N_CF) in the time-domain. CoNNear architectures with (a) and without (b) context are shown. The final CoNNear model has four encoder and decoder layers, uses context and includes a tanh activation function between the CNN layers. (c) provides an overview of the model training and evaluation procedure. Whereas reference, analytical TL-model simulations to a speech corpus were used to train the CoNNear parameters, evaluation of the model was performed using simple acoustic stimuli commonly adopted in cochlear mechanics studies.

Computational complexity is the main reason that real-time applications for hearing-aid [21], robotics [22], and automatic speech recognition applications do not adopt cochlear travelling-wave models in their preprocessing. Instead, they use computationally efficient approximations of auditory filtering that compromise on key auditory features. A common simplification implements the cochlear filters as a parallel, rather than cascaded, filterbank [23,24]. However, this parallel architecture captures neither the longitudinal coupling properties of the BM [25] nor the generation of otoacoustic emissions [26]. Another popular model, the gammatone filterbank model [27], ignores the stimulus-level dependence of cochlear filtering. Lastly, a number of models simulate the level-dependence of cochlear filtering but fail to match the performance of TL models [28]: they either simulate the longitudinal cochlear coupling locally within the individual filters of the uncoupled filterbank [29] or introduce distortion artefacts when combining an automatic-gain-control type of level-dependence with cascaded digital filters [30–32].

Thus, the computational complexity of biophysically realistic cochlear models poses a serious impediment to the development of human-like machine-hearing applications. This complexity motivated our search for an efficient model that matches the performance of state-of-the-art analytical TL models while offering real-time execution. Here, we investigate whether convolutional neural networks (CNNs) can be used for this purpose. Neural networks of this type can deliver end-to-end waveform predictions [33,34] with real-time properties [35] and are based on convolutions akin to the filtering process associated with cochlear processing.

This paper details how CNNs can best be connected and trained to approximate the computations performed by TL cochlear models [19, 36, 37], with a specific emphasis on simultaneously capturing the tuning, level-dependence, and longitudinal coupling characteristics of human cochlear processing. The proposed model (CoNNear) converts speech stimuli into corresponding BM displacements across 201 cochlear filters distributed along the length of the BM. Unlike TL models, the CoNNear architecture is based on parallel CPU computations that can be accelerated through GPU computing. Consequently, CoNNear can easily be integrated with real-time auditory applications that use deep learning. The quality of the CoNNear predictions and the generalisability of the method are evaluated on the basis of cochlear mechanical properties such as filter tuning estimates [38], nonlinear distortion characteristics [39], and spatial excitation patterns [40] obtained using sound stimuli commonly adopted in experimental studies of cochlear mechanics but not included in the training.

The CoNNear model

The CoNNear model has an auto-encoder CNN architecture and transforms a 20-kHz sampled acoustic waveform (in [Pa]) to N_CF cochlear BM displacement waveforms (in [μm]) using several CNN layers and dimension changes (Fig. 1a). The first four layers are encoder layers and use strided convolutions to halve the temporal dimension after every CNN layer. The following four, decoder, layers map the condensed representation onto L × N _CF outputs using deconvolution operations. L corresponds to the initial size of the audio input and N _CF to 201 cochlear filters with centre frequencies (CFs) between 0.1 and 12 kHz. The adopted CFs were spaced according to the Greenwood place-frequency map of the cochlea [6] and span the most sensitive frequency range of human hearing [41]. It is important to preserve the temporal alignment (or, phase) of the inputs across the architecture, because this information is essential for speech perception [42]. We used U-shaped skip connections for this purpose. Skip connections have earlier been adopted in image-toimage translation [43] and speech-enhancement applications [33,34]; they pass temporal information directly from encoder to decoder layers (Fig. 1a; dashed arrows). Aside from preserving phase information, skip connections may also improve the model’s ability to learn how best to combine the nonlinearities of several CNN layers to simulate the level-dependent properties of human cochlear processing.

Every CNN layer is comprised of a set of filterbanks followed by a nonlinear operation [44] and the CNN filter weights were trained using TL-simulated BM displacements from N_CFcochlear channels [37]. While training was conducted using a speech corpus [45] presented at 70 dB SPL, model evaluation was based on the ability to reproduce key cochlear mechanical properties using basic acoustic stimuli (e.g. clicks, pure-tones) unseen during training (Fig. 1c). During training and evaluation, the audio input was segmented into 2048-sample windows (≈ 100 ms), after which the corresponding BM displacements were simulated and concatenated over time. Because CoNNear treats each input independently, and resets its adaptation properties at the start of each simulation, this concatenation procedure could result in discontinuities near the window boundaries. To address this issue, we also evaluated an architecture that had the previous and following (256) input samples available as context (Fig. 1b). Different from the no-context architecture (Fig. 1a), a final cropping layer was added to remove the simulated context and yield the final L-sized BM displacement waveforms. Additional details on the CoNNear architecture and training procedure are given in Methods and Extended Data Fig. 1. Lastly, training CoNNear using audio inputs of fixed duration does not prevent it from handling inputs of other durations after training, thanks to its convolutional architecture. This flexibility is a clear benefit over matrix-multiplication-based neural network architectures, which can operate only on inputs of fixed-duration.

CoNNear hyperparameter tuning

Critical to this work is proper determination of the optimal CNN architecture and its hyperparameters that yield a realistic model for cochlear processing. Ideally, CoNNear should both simulate the speech training dataset with a sufficiently low L1 loss [i.e., the average mean-absolute difference between simulated BM displacements of the reference TL and CoNNear model] and also reproduce key cochlear response properties. Details on the hyperparameters that can be adjusted to define the final CoNNear architecture (number of layers, nonlinearity, context) are given in Extended Data Fig. 1. To determine the hyperparameters, we followed an iterative principled approach taking into account: (i) the L1 loss on speech material, (ii) the desired frequency- and level-dependent cochlear filter tuning characteristics, and (iii), the computational load needed for real-time execution. Afterwards, the prediction accuracy of CoNNear was evaluated on a broader set of response features. We considered a total of four metrics derived from cochlear responses evoked using simple stimuli of different frequencies and levels: filter tuning (Q_ERB), click-evoked dispersion, pure-tone excitation patterns, and the generation of cochlear distortion products. Together, these evaluation metrics form a comprehensive description of cochlear processing. Further information on the tests and their implementation is given in Methods. A benefit of our approach is that the evaluation stimuli were unseen to the model to ensure an independent evaluation procedure. Even though any fragment of the speech training material can be seen as a combination of basic acoustic elements such as impulses and pure tones of varying levels and frequencies, the cochlear mechanics stimuli were not explicitly present in the training material.

The first hyperparameter specifies the total number of encoder/decoder layers, which we set to 8 on the basis of Q_ERBsimulations. Aside from the reference experimental human Q_ERBcurve for low stimulus levels [46], Fig. 2a shows simulated Q_ERBcurves from the reference TL-model (red) overlaid with CoNNearmodel simulations. CoNNear captures the frequency-dependence of the human and reference TL-model Q_ERBfunction better as the layer depth was increased from 4 to 8. Models with 4 and 6 layers underestimated the overall Q_ERBfunction and performed worse for CFs below 1 kHz where reference ERBs were narrower, and corresponding BM impulse responses longer, than for higher CFs. Extending the number of layers beyond 8 increased the required computational resources without producing a substantial improvement in the quality of the fits. These resources are listed in Extended Data Fig. 2 for models of different layers, along with L1-loss predictions on the speech training set and a small set of evaluation metrics.

Fig 2. CoNNear hyperparameter tuning.

Cochlear filter tuning (Q_ERB) was simulated to 100-μs clicks of 40 and 70 dB peSPL for different cochlear filter centre frequencies (CF). Reference human Q_ERBestimates for low-stimulus levels [46] are shown and compared to simulations of the reference TL model and CoNNear models with (c) and without (a,b) context. (a) Increasing the number of CoNNear layers from 4 to 8, improved the Q_ERB-across-CF simulations. (b) Comparing architectures with the PReLU or tanh activation function shows that the PReLU nonlinearity failed to capture level-dependent cochlear filter tuning because the Q_ERBfunctions remained invariant to stimulus level changes. 8-layer CNN architectures were used for this simulation. (c) Adding context to the 8-layer, tanh model further improved the CoNNear predictions by showing less Q_ERBfluctuations across CF.

The second hyperparameter controls the activation function, or nonlinearity, which is placed between the CNN layers. To mimic the original shape of the outer-hair-cell input/output function [47] responsible for cochlear compression, we required that the activation function cross the x-axis. We therefore considered only the parametric rectified linear unit (PReLU) and hyperbolic-tangent (tanh) nonlinearities rather than standard activation functions such as the sigmoid and ReLU. Figure 2b depicts how the PreLU and tanh activation functions affected the simulated Q_ERBs across CFs. Whereas the PReLU activation function was unable to capture the level-dependence of cochlear filter tuning, the tanh nonlinearity reproduced both the level and frequency-dependence of human Q_ERBs. Additionally, Extended Data Fig. 3 (no context simulations) shows that the model with tanh nonlinearity reached a lower L1 loss on both training and test sets compared to the PReLU nonlinearity, while requiring a smaller number of parameters. The benefit of the tanh nonlinearity is further illustrated when comparing simulated pure-tone excitation patterns across model architectures. Figure 3 shows that the tanh nonlinearity (c) outperforms the PReLU nonlinearity (b) in capturing the compressive growth of BM displacement with stimulus level, as observed in the excitation pattern maxima (a). Although both activation functions were able to code negative input deflections, only the model with tanh nonlinearity (whose shape best resembles the cochlear input/output function) performed well. These simulations show that it is essential to consider the shape of the activation function when the reference system is nonlinear.

Fig 3. Comparing cochlear excitation patterns across model architectures.

Simulated root-mean-square (RMS) levels of BM displacement across CF for pure-tone stimuli between 10 and 90 dB SPL. From top to bottom, the stimulus frequencies were 500 Hz, 1 kHz and 2 kHz, respectively. Both the reference, TL-model, level-dependent excitation pattern shape changes and compressive growth of pattern maxima (a), were captured by the tanh architectures (c,d), but not by the PReLU architecture (b).

The final hyperparameter relates to adding context information when simulating an input of size L. As noted earlier, CNN architectures treat each input independently, which can produce discontinuities near the window boundaries. The effect of adding context [cf. architecture (b) vs (a) in Fig. 1] is best observed when simulating speech. Figure 4 shows simulated BM displacements of the reference and CoNNear models to a segment of the speech test set which was not seen during training. Panel (d) shows that providing context prevents discontinuities near the window boundaries. Context was also beneficial when simulating the reference 70-dB Q_ERBfunction in Fig. 2c. The L1-loss simulations in Extended Data Fig. 3 (tanh simulations) confirmed the overall performance improvement when providing context, and hence the final CoNNear architecture included a context window of 256 samples on either side of the 2048 sample input (Fig. 1b).

Fig 4. Effect of adding context to the CoNNear simulations.

Simulated BM displacements for a 2048-sample-long speech fragment of the TIMIT test set (i.e., unseen during training). The stimulus waveform is shown in panel (a) and panels (b)-(d) depict instantaneous BM displacement intensities (darker colours = higher intensities) of simulated TL-model outputs (b) and two CoNNear architecture outputs: without (c) and with (d) context. The left and right segments show the output of the neighbouring windows to demonstrate the discontinuity over the boundaries. The N_CF=201 output channels are labelled per channel number: channel 1 corresponds to a CF of 12 kHz and channel 201 to a CF of 100 Hz. Intensities varied between -0.5 μm (blue) and 0.5 μm (red) for all panels.

Generalisability of CoNNear

Overfitting occurs when the trained model merely memorises the training material and fails to generalise to data not present in the training set. To investigate whether the final CoNNear architecture was robust against overfitting, we tested how well the trained CoNNear model performed on unseen stimuli from (i) the same database, (ii) a different database of the same language, (iii) a different database of a different language, and (iv) a music piece.

Figure 5 shows boxplots of the L1-loss distributions of all simulated windows in the following audio material (from left to right): the TIMIT training set, the TIMIT test set using different speakers, the Dutch Matrix sentence database [49], Radioheads’ OK Computer album, and 400 randomly-drawn samples from different event categories in the AudioSet database [50]. In addition to calculated loss distributions, Fig. 5 compares instantaneous BM displacement intensities between the reference TL-model (g) and CoNNear (h) for three stimuli unseen during training: an English sentence from the Wall Street Journal Corpus [48] to show how CoNNear adapts to different recording settings (first column). A sentence from the Dutch matrix test [49] to investigate performance on a different language stimulus (middle column), and a music segment to test performance on a non-speech acoustic stimulus (right column). From an application perspective, we also tested how CoNNear handles audio input of arbitrary length. Extended Data Fig. 4 shows that CoNNear generalises well to an unseen speech stimulus of length 0.5 s (10048 samples) and a music stimulus of length 0.8 s (16384 samples), even though training was performed using shorter 2048-sample windows of fixed duration.

Fig 5. Generalisability of CoNNear to unseen input.

Top: boxplots comparing the distribution of L1 losses between TL-model and CoNNear model simulations for 2048-long windows within (a) 2310 sentences from the TIMIT training set (131760 calculated L1 losses, 16-kHz sampled audio) (b) the 550 sentences in the TIMIT test set (32834 L1 losses, 16-kHz sampled audio), (c) 100 sentences from the Dutch Matrix test (7903 L1 losses, 44.1-kHz sampled audio), (d) All songs of Radioheads OK Computer (48022 L1 losses, 16-kHz downsampled audio), (e) 400 random audio-fragments from the natural-sounds AudioSet database (53655 L1 losses, 16-kHz downsampled audio). The red line in the boxplot shows the median of the L1 errors, the black box denotes the values between the 25% (Q1) and 75% (Q3) quartiles. The blue whiskers denote the error values that fall within the 1.5 * (Q3 - Q1) range. Outliers are shown as circles. Cochlear model predictions and prediction errors for the windows associated with median and maximum L1 loss in the TIMIT test set are shown in Extended Data Fig. 5. Bottom: Along the columns, simulated BM displacements for three different 2048-long audio stimuli are shown: a recording of the English Wall Street Journal speech corpus [48], a sentence from the Dutch matrix test [49] and a music fragment taken from Radiohead - No surprises. Stimulus waveforms are depicted in panel (f) and instantaneous BM displacement intensities (darker colours = higher intensities) of the simulated TL-model and CoNNear outputs are depicted in (g-h). Panel (i) shows the intensity difference between the TL and CoNNear outputs. The N_CF=201 considered output channels are labelled per channel number: channel 1 corresponds to a CF of 12 kHz and channel 201 to a CF of 100 Hz. The same colour map was used for all figures and ranged between -0.5 μm (blue) and 0.5 μm (red).

The similar loss distributions across tested speech conditions, together with the small intensity prediction errors seen in Fig. 5(i), show that CoNNear generalises well to unfamiliar stimuli. When comparing L1-loss distributions between speech (b,c) and non-speech (d,e) audio, we observe increased mean L1-losses along with a greater performance variability, which reflects the large variety of spectral and temporal features present in music or the AudioSet database. Even though median performance is still acceptable for non-speech audio, we conclude that CoNNear performs most comparable to the original TL-model simulations for speech. CoNNear’s prediction accuracy on non-speech audio may be improved by retraining with a larger and more-diverse stimulus set than the adopted TIMIT training set. However, a clear benefit from our hybrid, neuroscience-inspired training method is that we can yield a generalisable CoNNear architecture even when using a training dataset which is much smaller than those adopted in standard data-driven NN-approaches [44]. Lastly, we note that the non-speech audio were downsampled to 16 kHz to match the frequency content of the TIMIT speech material to allow for a fair comparison between speech and non-speech audio. We can hence not guarantee that CoNNear performs well on non-speech audio with frequency content above 8 kHz.

CoNNear as a model for human cochlear signal processing

Since the primary goal of this work was to develop a neural-network based model for human cochlear mechanics, we evaluated how well the trained CoNNear model collectively performs on simulating key cochlear mechanics metrics (see Methods for a detailed description). Figure 2c shows that the final CoNNear architecture with context simulates the frequency- and level-dependence of human cochlear tuning. At the same time, CoNNear faithfully captures the shape and compression properties of pure-tone cochlear excitation patterns (Fig. 3d). However, we did observe small excitation pattern fluctuations at CFs below the stimulus frequency for the 1 and 2 kHz simulations (middle and bottom row in Fig. 3). These fluctuations had levels of ≈ 30 dB below the peak of the excitation pattern and are hence not expected to degrade the sound-driven CoNNear response to complex stimuli such as speech. This latter statement is supported by (i) the CoNNear speech simulations in Fig. 5i, which show minimally visible noise in the error patterns, and (ii) the root mean-square error (RMSE) observed between simulated TL and CoNNear model excitation patterns for a broad range of frequencies and levels (Extended Data Fig. 6c). To obtain a meaningful error estimate, we normalised the RMSE to the TL excitation pattern maximum to yield a CoNNear error percentage. The error for the final CoNNear architecture remained below 5 % for pure-tone frequencies up to 4 kHz, and stimulation levels below 80 dB. Increased errors were observed for higher stimulus levels and at CFs of 8 and 10 kHz and stemmed from the overall level and frequency content of the speech material used for training; the dominant energy in speech occurs below 5 kHz, and we presented the speech corpus at 70 dB. However, it is noteworthy that the visual difference between simulated high-level excitation patterns was mostly associated with low-level fluctuations at CFs below the stimulation frequency and not at the stimulation frequency itself (cf. the first and last columns of Fig. 3). This means that the stimulus-driven error that occurs when using broadband stimuli is likely smaller than that observed here for tonal stimuli. In summary, the error appears acceptable for processing speech-like audio. We speculate that CoNNear’s application range can be extended by retraining with stimuli at different levels and/or with greater high-frequency content.

The top panel of Fig. 6 shows the dispersion characteristics of the TL model and trained CoNNear models, illustrating the characteristic 12-ms onset delay from basal (high CF, low channel numbers) to apical (low CF, high channel numbers) cochlear sections in both models. In the cochlea, dispersion arises from the biophysical properties of the coupled BM (i.e., its spatial variation of stiffness and damping), and the CoNNear architecture captures this phenomenon. Adding context did not affect the simulation quality.

Fig 6. Cochlear dispersion and DPOAEs.

Top: Comparing cochlear dispersion properties across model architectures. Panel (a) shows the stimulus pressure, while panels (b)-(d) show instantaneous BM displacement intensities for CFs (channel numbers, CS) between 100 Hz (channel 201) and 12 kHz (channel 1). The colour scale is the same in all figure panels, and ranges between -15 μm (blue) and 15 μm (red). Bottom: Comparing simulated DPOAEs across model architectures. The frequency response of the 12-kHz CF channel (i.e., fast Fourier transform of the BM displacement waveform) was considered as a proxy for otoacoustic emissions recorded in the ear-canal. Panels (a) - (d) show simulations in response to two pure tones of f ₁of 2.0 and f ₂of 2.4 kHz for different model architectures. In humans, the most pronounced distortion product occurs at 2f ₁− f ₂(1.6 kHz). Panel (e) depicts simulated distortion-product levels (L_DP) compared between TL and the CoNNear-tanh-context model. L_DP was extracted from the frequency response of the 12-kHz CF channel at 2f ₁− f ₂. Simulations were conducted for L₂levels between 10 and 100 dB SPL and L₁=0.4 L₂+39 [51]. f₁ranged between 1 and 6 kHz, following a f₂/f₁ratio of 1.2.

Finally, we tested CoNNear’s ability to simulate a last, key feature of cochlear mechanics: distortion products, which travel in reverse along the BM to generate pressure waveforms in the ear canal (i.e., distortion-product otoacoustic emissions, DPOAEs). The bottom panel of Fig. 6 compares reference TL-model simulations with those of different CoNNear models (b)-(d). DPOAE frequencies are visible as spectral components that are not present in the stimulus (which consists of two pure tones at frequencies f ₁= 2.0 kHz and f ₂= 2.4 kHz). In humans, the strongest DP component occurs at 2f ₁− f ₂= 1.6 kHz, and as the simulations show, the level of this DP was best captured using the tanh activation function. As observed before in the excitation pattern simulations, the activation function most resembling the shape of the cochlear nonlinearity performed best when simulating responses that rely on cochlear compression. Adding context removed the high-frequency distortions that were visible in panels (b) and (c).

The quality of the DPOAE simulations across a range of stimulation levels and frequencies was further investigated in Fig. 6e. Up to L₂levels of 60 dB SPL, CoNNear matched the characteristic nonlinear growth of both TL-simulated and human DPOAE level functions well [52]. At higher stimulation levels, DPOAE levels started fluctuating in both models. Although this may reflect the chaotic character of cochlear DPs interacting with cochlear irregularities in TL models and in human hearing [26], it may simply reflect the limited training material we had available for CoNNear at these higher levels. Given that human DPOAEs are generally recorded for stimulation levels below 60 dB SPL in clinical or research contexts [53], we can conclude that CoNNear was able to capture this important epiphenomenon of hearing. The cochlear mechanics evaluations we performed in Figs. 2, 3, 6 and Extended Data Fig. 6 together demonstrate that the 8-layer, tanh, CoNNear model with context performed best on four crucial aspects of human cochlear mechanics. Despite training on a limited speech corpus presented at 70 dB SPL, CoNNear learned to simulate outputs which matched those of biophysically-realistic analytical models of human cochlear processing across level and frequency.

CoNNear as a real-time model for audio applications

In addition to its ability to simulate realistic cochlear mechanical responses, CoNNear operates in real-time. In audio applications, real-time is commonly defined as a computation duration less than 10 ms; below this limit no delay is perceived. Table 1 summarises the necessary time to compute the final CoNNear-context model for a stimulus window of 1048 samples on CPU or GPU architectures. On a CPU, the CoNNear model outperforms the TL-model by a factor of 129 and on a GPU, CoNNear is 2142 times faster. Additionally, the GPU computations show that the trained CoNNear model (8-layers, tanh, with context) has a latency of 7.27 ms, and hence reaches real-time audio processing performance.

Table 1. Model calculation speed.

Comparison of the time required to calculate a TL and CoNNear model window of 1048 samples on a CPU (Apple MacBook Air, 1.8 GHz Dual-Core processor) and a GPU (NVIDIA GTX1080). The calculation time for the first window is considered separately for the GPU computations since this window also includes the weight initialisation. For each evaluated category, the best performing architecture is highlighted with a bold font.

Model	#Param	CPU (s/window)	GPU 1st window (s)	GPU (ms/window)
PReLU/8 lay./no context	11,982,464	0.222	1.432	7.70
tanh/8 lay./no context	11,507,328	0.195	1.390	7.59
tanh/8 lay./context	11,689,984	0.236	1.257	7.27
Transmission Line	N/A	25.16	N/A	16918

Discussion

This paper details how a hybrid, deep-neural-net and analytical approach can be used to develop a real-time model of human cochlear processing (CoNNear), with performance matching that of human cochlear processing. As we demonstrate here for the first time, neither real-time performance nor biophysically realistic responses need be compromised. CoNNear combines both in a single auditory model, laying the groundwork for a new generation of human-like robotic, speech-recognition and machine hearing applications. Prior work has demonstrated the clear benefits of using biophysically realistic cochlear models as front-ends for auditory applications: e.g. for capturing cochlear compression, [7,11,28], speech enhancement at negative signal-to-noise ratio’s [12], realistic sound perception predictions [54] and for simulating the generation of human auditory brainstem responses [37, 55]. Hence, CoNNear can dramatically improve performance in application areas which, to date, have relied on computationally intensive biophysical models. Not only can CoNNear operate on running audio input with a latency below 7.5 ms, the model offers a differentiable solution which can be used in closed-loop systems for auditory feature enhancement or augmented hearing.

With the rise of neural-network (NN) based methods, computational neuroscience has seen an opportunity to map audio or auditory brain signals directly to sound perception [56–58] and to develop computationally efficient methods to compute large-scale differential-equation-based neuronal networks [59]. These developments are transformative, as they can unravel the functional role of hard-to-probe brain areas in perception and yield computationally fast neuromorphic applications. The key to these breakthroughs is the hybrid approach in which knowledge from neuroscience is combined with that of NN-architectures [60]. While the possibilities of NN approaches are numerous when large amounts of training data are available, this is rarely the case for biological systems and human-extracted data. It therefore remains challenging to develop models of biophysical systems which can generalise to a broad range of unseen conditions or stimuli.

Our work presents a solution to this problem for cochlear processing by constraining the CoNNear architecture and its hyperparameters on the basis of a state-of-the-art TL cochlear model. Our general approach consists of four steps: First, derive an analytical description of the biophysical system on the basis of available experimental data. Second, use the analytical model to generate a training data set consisting of responses to a representative range of relevant stimuli. Third, use this training data set to determine the NN-model architecture and constrain its hyperparameters. Finally, verify the ability of the model to generalise to unseen inputs. Here, we demonstrated that CoNNear predictions generalise to a diverse set of audio stimuli by faithfully predicting key cochlear mechanics features to sounds which were excluded from the training. We note that CoNNear performed best on unseen speech and cochlear mechanics stimuli with levels below 80 dB. We expect that improved generalisability to complex non-speech audio such as environmental sounds and music can be obtained when retraining CoNNear with a more diverse audio set.

Our proposed method is by no means limited to NN-based models of cochlear processing. Indeed, the method can be widely applied to other nonlinear and/or coupled biophysical models of sensory and biophysical systems. Over the years, analytical descriptions of cochlear processing have evolved based on the available experimental data from human and animal cochleae, and they will continue to improve. It is straightforward to train CoNNear to an updated/improved analytical model in step (i), as well as to include different or additional training data in (ii) to further optimise its performance.

Conclusion

We present a hybrid method which uniquely combines expert knowledge from the fields of computational auditory neuroscience and machine-learning-based audio processing to develop a CoNNear model of human cochlear processing. CoNNear presents an architecture with differentiable equations and operates in real time (<7.5 ms delay) at speeds 2000 times faster than state-of-the-art biophysically realistic models. We have high hopes that the CoNNear framework will inspire a new generation of human-like machine hearing, augmented hearing and automatic speech-recognition systems.

Methods

Extended Data Fig. 1 describes the parameters which define the CNN-based CoNNear auto-encoder architecture. The encoder layers transform audio input of size L into a condensed representation of size L/2^M× k_M, where k_M equals the number of filters in the M^th CNN layer. We used 128 filters per layer with a filter length of 64 samples. The encoder layers use strided convolutions, i.e. the filters were shifted by a time-step of two to halve the temporal dimension after every CNN layer. The decoder contains M deconvolution layers to re-obtain the original temporal dimension of the audio input (L). The first three decoder layers used 128 filters per layer and the final decoder layer had (N _CF) filters, equalling the number of cochlear sections simulated in the reference cochlear TL model.

Training CoNNear

CoNNear was trained using TL-model simulations of N _CFBM displacement waveforms in response to audio input windows of L= 2048 samples. For the context architecture (Fig. 1b), the previous and following L_l=L_r=256 input samples were also available to CoNNear. This architecture included a final cropping layer to remove the context after the last CNN decoder and yield an output size L × N _CF. Note that the CoNNear model output units are BM displacement y_BM in [μm], whereas the TL-model outputs are in [m]. This scaling was necessary to enforce a training procedure with sufficiently high digital numbers. For training purposes, and visual comparison between the TL and CoNNear outputs, the y_BM values of the TL model were multiplied by a factor of 10⁶ in all figures and analyses.

The TIMIT speech corpus [45] was used for training and contains phonetically balanced sentences with sufficient acoustic diversity. To generate the training data, 2310 sentences from the TIMIT corpus were used. They were upsampled from 16 kHz to 100 kHz to solve the TL-model accurately [36] and the root-mean square (RMS) energy of every utterance was adjusted to 70 dB sound pressure level (SPL). 550 sentences of the TIMIT dataset were omitted from the training and considered as the test set. BM displacements were simulated for 1000 cochlear sections with centre frequencies between 25 Hz and 20 kHz using a nonlinear time-domain TL model of the cochlea [37]. From the TL-model output representation (i.e., 1000 y_BM waveforms sampled at 20 kHz), outputs from 201 CFs between 100 Hz and 12 kHz, spaced according to the Greenwood map [6], were chosen to train CoNNear. Above 12 kHz, human hearing sensitivity becomes very poor [41], motivating our choice for the upper limit of considered CFs. CoNNear model parameters were optimised to minimise the mean absolute error (dubbed L1 loss) between the predicted model outputs and the reference TL model outputs. A learning rate of 0.0001 was used with Adam optimiser [61] and the entire framework was developed using the Keras machine learning library [62] with a Tensorflow [63] back-end.

Cochlear mechanics evaluation metrics

The evaluation stimuli were sampled at 20 kHz and had a duration of 102.4 ms (2048 samples) and 128 ms (2560 samples) for the CoNNear and CoNNear-context model, respectively. Stimulus levels were adjusted using the reference pressure of ρ ₀= 2 ·10⁻⁵ Pa. Only when evaluating how CoNNear generalised to longer duration continuous stimuli (Extended Data Fig. 4), or when investigating its real-time capabilities, did we deviate from this procedure. The following sections describe the four cochlear mechanics evaluation metrics we considered to evaluate the CoNNear predictions. Together, these metrics form a comprehensive description of cochlear processing.

Cochlear filter tuning

A common approach to characterise auditory or cochlear filters is by means of the equivalent-rectangular bandwidth (ERB) or Q_ERB. The ERB describes the bandwidth of a rectangular filter which passes the same total power than the filter shape estimated from behavioural or cochlear tuning curve experiments [64], and presents a standardised way to characterise the tuning of the asymmetric auditory/cochlear filter shapes. The ERB is commonly used to describe the frequency and level-dependence of human cochlear filtering [4,46,65], and Q_ERBto describe level-dependent cochlear filter characteristics from BM impulse response data [19,66]. We calculated Q_ERB as:

$$Q_{\text{ERB}}=\frac{\text{CF}}{\text{ERB}}.$$ (1)

The ERB was determined from the power spectrum of a simulated BM time-domain response to an acoustic click stimulus using the following steps [66]: (i) compute the Fast Fourier Transform of the BM displacement at the considered CF, (ii) compute the area underneath the power spectrum, and (iii) divide the area by the CF. The frequency- and level-dependence of CoNNear predicted cochlear filters were compared against TL-model predictions and experimental Q_ERBvalues reported for humans [4].

The acoustic stimuli were condensation clicks of 100-μs duration and were scaled to the desired peak-equivalent sound pressure level (dB peSPL), to yield a peak-to-peak click amplitude that matched that of a pure-tone with the same dB SPL level (L):

$$\text{click}(t)=2\sqrt{2}\cdot p_0\cdot {10}^{L/20}\cdot x(t)\text{with}x(t)=\{\begin{array}{l}1\text{for}t\le 100\mu s\hfill \\ 0\text{for}t>100\mu s\hfill \end{array}$$ (2)

Cochlear Excitation Patterns

Cochlear excitation patterns can be constructed from the RMS energy of the BM displacement or velocity at each measured CF in response to tonal stimuli of different levels. Cochlear excitation patterns show a characteristic half-octave basal-ward shift of their maxima as stimulus level increases [40]. Cochlear excitation patterns also reflect the level-dependent nonlinear compressive growth of BM-responses when stimulating the cochlea with a pure-tone of the same frequency as the CF of the cochlear measurement site [3]. Cochlear pure-tone transfer-functions and excitation patterns have in several studies been used to describe the level-dependence and tuning properties of cochlear mechanics [2,3,40]. We calculated excitation patterns for all 201 simulated BM displacement waveforms in response to pure tones of 0.5, 1 and 2 kHz frequencies and levels between 10 and 90 dB SPL using:

$$\text{tone}(t)=p_0\cdot \sqrt{2}\cdot {10}^{L/20}\cdot \sin \left(2\pi f_{\text{tone}}t\right),$$ (3)

where t corresponds to a time vector of 2048 samples, L to the desired RMS level in dB SPL, and f_tone to the stimulus frequencies. The pure-tones were multiplied with a Hanning-shaped 10-ms on- and offset ramp to ensure a gradual onset.

Cochlear Dispersion

Click stimuli can also be used to characterise the cochlear dispersion properties, as their short duration allows for an easy separation of the cochlear response from the evoking stimulus. At the same time, the broad frequency spectrum of the click excites a large portion of the BM. Cochlear dispersion stems from the longitudinal-coupling and tuning properties of BM mechanics [67] and is observed through later click response onsets for BM responses associated with more apical CFs. In humans, the cochlear dispersion delay mounts up to 10-12 ms for stimulus frequencies associated with apical processing [68]. Here, we use clicks of various sound intensities to evaluate whether CoNNear produced cochlear dispersion and BM click responses in line with predictions from the TL-model.

Distortion-product otoacoustic emissions (DPOAEs)

DPOAEs can be recorded in the ear-canal using a sensitive microphone and are evoked by two pure-tones with frequencies f ₁and f ₂and SPLs of L ₁and L ₂, respectively. For pure tones with frequency ratios between 1.1 and 1.3 [69], local nonlinear cochlear interactions generate distortion products, which can be seen in the ear-canal recordings as frequency components which were not originally present in the stimulus. Their strength and shape depends on the properties of the compressive cochlear nonlinearity associated with the electro-mechanical properties of cochlear outer-hair-cells [39], and the most prominent DPOAEs appear at frequencies of 2f ₂ − f ₁and 2f ₁ − f ₂. Even though CoNNear was not designed or trained to simulate DPs, they form an excellent evaluation metric, as realistically simulated DPOAE properties would demonstrate that CoNNear was able to capture even the epiphenomena associated with cochlear processing. As a proxy measure for ear-canal recorded DPOAEs, we considered the BM displacement at the highest simulated CF which, in the real ear, would drive the middle-ear and eardrum to yield the ear-canal pressure waveform in an OAE recording. We compared simulated DPs extracted from the fast Fourier transform of the BM displacement response to simultaneously presented pure tones of f ₁ and f ₂= 1.2 f ₁with levels according to the commonly adopted experimental scissors paradigm: L ₁= 0.4 L ₂+ 39 [51]. We considered f ₁ frequencies between 1 and 6 kHz and L ₂ levels between 10 and 100 dB SPL.

Extended Data

Extended Data Fig. 1. Overview of the CoNNear architecture parameters.

Extended Data Fig. 2. CNN layer depth comparison.

The first column details the CoNNear architecture. The next columns describe the total number of required model parameters, the required training time per epoch of 2310 TIMIT training sentences and average L1 loss across all windows of the TIMIT training set. Average L1 losses were also computed for BM displacement predictions to a number of unseen acoustic stimuli (click and 1-kHz pure tones) with levels between 0 and 90 dB SPL. Lastly, average L1 loss was also computed for the 550 sentences of the TIMIT test set. For each evaluated category, the best performing architecture is highlighted in bold font.

Extended Data Fig. 3. Activation function comparison.

The first column details the CoNNear architecture. The next columns describe the total number of required model parameters, the required training time per epoch of 2310 TIMIT training sentences and average L1 loss across all windows of the TIMIT training set. Average L1 losses were also computed for BM displacement predictions to a number of unseen acoustic stimuli (click and 1-kHz pure tones) with levels between 0 and 90 dB SPL. Lastly, average L1 loss was also computed for the 550 sentences of the TIMIT test set. For each evaluated category, the best performing architecture is highlighted in bold font.

Extended Data Fig. 4. Simulated BM displacements for a 10048-sample speech stimulus and a 16384-sample music stimulus.

The stimulus waveform is depicted in panel (a) and panels (b)-(c) depict instantaneous BM displacement intensities (darker colours = higher intensities) of the simulated TL-model (b) and CoNNear (c) outputs. The N_CF=201 considered output channels are labelled per channel number: channel 1 corresponds to a CF of 12 kHz and channel 201 to a CF of 100 Hz. The same colour scale was used for both simulations and ranged between -0.5 µm (blue) and 0.5 μm (red). The left panels show simulations to a speech stimulus from the Dutch matrix test [49] and the right panels shows simulations to a music fragment (Radiohead - No Surprises).

Extended Data Fig. 5. Comparing TL and CoNNear model predictions at the median and maximum L1 prediction error.

The figure visually compares BM displacement intensities of the BM (b) and CoNNear (c) model to audio fragments which resulted in the median and maximum L1 errors of 0.008 and 0.038 simulated for the TIMIT test set (Fig. 5). The N_CF=201 considered output channels are labelled per channel number: channel 1 corresponds to a CF of 12 kHz and channel 201 to a CF of 100 Hz. The same colour scale was used for both simulations and ranged between -0.5 μm (blue) and 0.5 μm (red).

Extended Data Fig. 6. Root mean-square error (RMSE) between simulated excitation patterns of the TL and CoNNear models reported as fraction of the TL excitation pattern maximum (cf. Fig. 3).

Using the PReLU activation function (a) leads to an overall high RMSE as this architecture failed to learn the level-dependent cochlear compression characteristics and filter shapes. The models using the tanh nonlinearity (b),(c) did learn to capture the level-dependent properties of cochlear excitation patterns, and performed with errors below 5% for the frequency ranges and stimulus levels captured by the speech training data (for CFs below 5 kHz, and stimulation levels below 90 dB SPL) The RMSE increased above 5% for all architectures when evaluating its performance on 8- and 10-kHz excitation patterns. This decreased performance results from the limited frequency content of the TIMIT training material.

Data availability

The source code of the TL-model used for training is available via 10.5281/zenodo.3717431 or github/HearingTechnology/Verhulstetal2018Model, the TIMIT speech corpus used for training can be found online [45]. Most figures in this paper can be reproduced using the CoNNear model repository.

Code availability

The code for the trained CoNNear model, including instructions of how to execute it is available from github.com/HearingTechnology/CoNNear cochlea or 10.5281/zenodo.4056552. A non-commercial, academic UGent license applies.