Frequency-based multi-band adaptive compression for hearing aid application

Abstract

Multi-band Dynamic Range (MBDR) Compression is a key part of the signal processing operation in hearing aid devices (HADs). Operating speed of the MBDR compressor plays an important role in preserving the quality and intelligibility of the output signal. Traditional fast-acting compressor preserves the audible cues in quiet speech but, in presence of surrounding noise, it can degrade the sound quality by introducing pumping and breathing effects. Alternatively, slow-acting compressor maintains the temporal cues and the listening comfort but may provide inadequate gain for soft inputs that come right after loud inputs. HADs may operate in a variable acoustic environment. Therefore, a fixed speed in compression might affect the performance of the hearing aids. In this study, we propose a frequency(FFT) based nine-band adaptive MBDR compression which uses spectral flux as a measure of the intensity change in input level to adapt the speed of the compressor in each band. Gain, threshold and compression ratio of the compressor for nine bands are adjusted based on the audiogram of the hearing impaired patient. The proposed frequency-based adaptive MBDR compression method is implemented on smartphone. The objective and subjective test results demonstrate the performance of proposed method compared to fixed compression approaches.

1. INTRODUCTION

Dynamic Range Compressor (DRC) is a variable gain amplifier where loud sounds higher than the predefined threshold level are compressed and softer sounds are provided with adequate gain in different frequency bands. This signal processing operation can be considered as loudness equalization which adjusts the amplitude ratio between highest to lowest intensities in every frequency band of a signal. DRC is commonly used in music production, sound systems, hearing aids, amplifiers, and broadcasting etc.¹ In this paper, we will focus on the importance of DRC in hearing aid application. At present hearing aids are the most common solution to sensorineural hearing loss.² A hearing-impaired (HI) person encounters difficulty in understanding or listening to sounds that are below their hearing threshold level. This reduced dynamic range of audibility is known as residual dynamic range.³ Hearing aids employ DRC to optimally utilize the residual dynamic range of HI person. DRC being responsible for mapping the wide range of input sounds to the residual dynamic range of HI person, makes it a crucial component in hearing aids. DRC is defined by parameters such as compression ratio, compression threshold, attack time, release time, gain, knee width, etc. These parameters are decided based on an individual’s listening preferences and hearing threshold levels, often accompanied by the individual’s audiogram. Setting these parameters on hearing aids is also called “hearing aid fitting”. An individual’s need and listening preference makes fitting of hearing aid a bit convoluted process.⁴

Fitted hearing aid ensures normal hearing experience and better speech recognition ability for people with hearing loss. The purpose of DRC is to provide prescribed frequency-dependent amplification to the input signal. After the amplification, DRC compress or reduce the gain of sounds that are above the preset threshold level in any frequency band because HI people may have trouble with understanding or hearing soft sounds but their perception of loud sound is still similar to a normal hearing person. DRC produces the output that is not distorted or above the discomfort loudness level but still audible to HI person. DRC being a non-linear operation, it is very important to maintain the quality and intelligibility of the input signal. Gain to be inserted or compression ratio are decided based on the input level and hearing threshold of a HI person. A compressor that divides the input level into multiple categories by having multiple threshold values is referred to as wide dynamic range compressor (WDRC).^{5, 6} Hearing aid with WDRC will have different compression ratio after each threshold level. Hearing thresholds of a HI person are also not identical across the whole frequency spectrum which suggests that gain inserted across each frequency band of input signal needs to be different. Compressor that divides the input signal into multiple bands is referred as multi-band dynamic range (MBDR) compressor.⁷ In another word, amplification gain depends on the input level and hearing threshold of a particular frequency band. These gains are decided by hearing aid fitting strategy. An audiologist uses popular fitting strategies such as Desired Sensation Level (DSL), NAL-NL2, and Half-gain rule, etc. for gain computation.^8,9

Parameters such as compression threshold, compression ratio and gain of the compressor assure the audibility and the comfort level of sound. Whereas, the attack time and release time of compressor play a vital role in maintaining the intelligibility and quality of the input signal.^10,11 When the input levels changes instantly, the compressor takes time to change the gain. Sudden and abrupt changes in the gain values can create temporal artifacts and unpleasant listening experience. In designing DRC for hearing aid, attack time and release time primarily rely on user preference and the surrounding environment where the hearing aid is used. Attack time is generally kept at a minimum (5 to 30 ms) to reduce the gain right after signal overshooting the threshold level. Release time, however, has an important role to play in the listener’s preference. The choice of shorter or longer release time decides the speed of the compressor. The fast-acting compressor has shorter release time (50 to 100 ms) and the slow-acting compressor has longer release time (800 ms to 2s). Fast-acting compression preserves the audible cues in quiet speech, but in the presence of surrounding noise, it can degrade the sound quality by introducing pumping and breathing effects.^7,12 Alternatively, slow-acting compression maintains the temporal cues and the listening comfort but may provide inadequate gain for soft inputs that come immediately after loud inputs.¹³ Therefore, a fixed speed in compression might affect the performance of the hearing aids. Some compression designs incorporate adaptive or variable release time that is release time would change based on duration, nature of input or surrounding sound environment.^14-16

In this study, we propose an FFT based multi-band dynamic range adaptive compressor. We divide the input signal into nine unequal bands in the frequency domain. Center frequencies of these nine bands are 0.25, 0.5, 1, 1.5, 2, 3, 4, 6, and 8 kHz similar to those present in the typical audiogram. The gain calculation for each band corresponding to the user’s hearing threshold and input level is carried out by the DSL-v5 strategy. However, attack and release time are adapted based on the spectral flux measurement of the input signal. Spectral flux is used as a measure of signal transient. Our specific focus will be the effect of adaptive time constants on the performance of compression. Objective and subjective evaluation for fast-acting, slow-acting, and proposed adaptive compression are done on normal hearing people to show the improvement in the quality and intelligibility of the signal. Additionally, the proposed method is also implemented on a smartphone for real-time application. Ubiquitous smartphone as a cost-effective standalone platform carrying all the signal processing functions of hearing aids makes it a viable assistive listening device.^17-19

The rest of the paper is organized as follows. Section 2 gives the background information on compression parameters and the effect of release time. The proposed method is described in Section 3. Section 4 provides a brief description of real-time implementation on a smartphone device carrying an iOS operating system. The performance comparison for different compression speeds is carried out in Section 5. Section 6 concludes the overall work in this paper.

2. BACKGROUND AND MOTIVATION

Compression is typically applied as a side-chain configuration on feed-forward topology as shown in Figure 3.¹⁴ Parameter tuning is very important in the case of DRC for hearing aids to have optimal performance. Compression parameters and their effects for hearing aid use are discussed below.

Figure 3:

Block diagram of proposed FFT based multi-band dynamic range compressor.

A. COMPRESSOR PARAMETERS

• Compression Threshold (T): It defines a threshold level above which compression or gain reduction needs to apply. T is typically kept around 65 dB SPL. There can be multiple threshold points in the WDRC compression scheme, which sets a different compression ratio after each threshold level.

• Compression Ratio (R): It determines the amount of gain reduction being applied, once the input signal crosses the threshold. For hearing aid applications, R is generally kept below 4. High values of R can result unpleasantness and unclear sound.²⁰ The output-input relation can be given by

$$output=\{\begin{array}{cc}input,\hfill & input\le T\hfill \\ \frac{input-T}{R}+T,\hfill & input>T\hfill \end{array}\phantom{\}}$$ (1)

where T is compression threshold.

• Knee Width (W): The knee is the point of threshold level where input-output relation changes its slope. This transition can be made smooth so that the effect of compression won’t be noticeable. Knee width is the spread of this smooth transition region.

• Attack Time (τ_A): Attack time is defined as the time delay between activating the compression and reaching within 3dB of its compressed final value the moment that the input signal exceeds the threshold level. Attack time is kept minimum (5 to 20ms) as we don’t want the user to listen to sounds above discomfort level for long. Figure 1a shows the impact of different attack times on the signal.

• Release Time (τ_R): Release time is defined as the time delay between deactivating the compression and reaching within 4dB of its normal value the moment that the input signal falls below the threshold level. Release time can be set between (50ms to 2s). We discuss the further effects of release time in the next sub-section. Figure 1b shows the impact of different release times on the signal.

• Makeup Gain (M): The compressor equalizes the loudness across the signal by compressing the loud sounds. Makeup gain works as a volume controller. M is set generally to increase the overall volume of the signal.

• Number of Bands: Since the hearing loss level is not identical across the frequency spectrum, it is important to provide the different amount of gains at different frequency bands. The number of bands varies from 3 to 16 in modern digital hearing aids. Typically, it is set to nine, since we know the hearing thresholds for these nine bands from the audiogram. Center frequencies of these nine bands are located at 0.25, 0.5, 1, 1.5, 2, 3, 4, 6, and 8 kHz known as octave frequencies. We will consider these nine bands in our work.

Figure 1:

Demonstration of time constants a) Attack time b) Release time. Both Figures show how the gain smoothing is applied once the signal crosses the threshold.

B. EFFECT OF COMPRESSOR SPEED

The speed of the compression has a great impact on the quality and intelligibility of the output signal.¹⁰ How quickly gain changes occurs is critically important in determining the effectiveness of the compressor and associated artifacts. The speed of the compressor is defined by the release time. Attack and Release time defines how quickly the compressor reacts once the signal crosses the threshold value. Figure 2 shows the effects of attack time and release time on an analytical signal. Attack time is usually kept minimum to suppress the signal immediately after it goes above the defined compression threshold. A compressor can be divided into two broader classes based on the value of release time “fast-acting compressor (FAC)” and “slow-acting compressor (SAC)”. Typical FAC has release time around 50 to 100ms and SAC has release time around 800ms to 2s. Many literature has discussed the advantages and disadvantages of the fast-acting and slow-acting compressors.^10,21-23

Figure 2:

(a) Analytical input signal (b) linear amplification (c) fast-acting compression (d) slow-acting compression

FAC intends to make the hearing impaired person’s perception of loudness more like that of a normal listener. A system with short recovery times (5 to 200ms) is also called “Syllabic Compressor” because the gain changes over the time are comparable with the duration of individual syllables in the speech. FAC with a high compression ratio reduces temporal cues and are problematic for people with cognitive issues.¹⁵ As a result, high compression ratios (> 3) are avoided for these systems as they are shown to have adverse effects on speech intelligibility.^23,24 Another drawback is that ambient noise is more perceptible during pauses. If the gain changes significantly during the speech itself, or during the pauses, then “breathing” and “Pumping” noises may be heard that are objectionable to the user. So during low SNR, FAC is preferred.¹⁶ Spatial cues are also reduced which are important for locating the sound sources. Another study also suggests that people with lower cognitive ability may have more difficulty with the fast-acting compression.²⁵ To reduce this problem, release time should be longer.

Fitting philosophy for SAC is to deliver comfortable loudness and naturalness for overall sound in the environment. SAC preserves the temporal structure which is not the case with FAC. One disadvantage is that loudness perception is not restored to the “normal”. In Figure 2d one can see that gain drops to a low value right after an intense sound; hearing aid goes effectively “dead” for a while. This can create inaudibility for weak sounds rapidly following the intense sounds.^21,22 However, we can see in Figure 2c that fast-acting compressor can restore this audibility. Another disadvantage linked with SAC is that when sound comes from multiple sources with different intensities, it would be difficult to understand the lower intensity sound source.

Audiologists fit the hearing aid for users at the clinic. The hearing aids that are available currently may not exactly fall clearly into these categories of fast or slow. Release time can also be set to be the in-between value of these two groups. But a fixed time constant may not be acceptable for optimal performance. Variable release time compressor employs both fast-acting and slow-acting compressors in series to vary the release time.²⁵ SNR aware compressor strategy was also proposed to select between FAC or SAC based on the SNR of the input signal.¹⁶ In our proposed method, spectral flux is used to automate the release time of the compressor. We are using spectral flux as a measure to track the transient nature of the incoming signal. The following section presents the details of the proposed method.

3. PROPOSED METHOD

Proposed frequency-based multi-band dynamic range adaptive compression is explained in this section. The input signal is considered in 20ms frame-wise manner with 50% overlap and hamming windowing. Let N be the number of samples in one frame and Fs the sampling rate. The windowed input signal is transformed into K point single-sided frequency spectrum using FFT. Figure 3 represents the block diagram of the proposed approach. Converting the linear domain into the log-scale domain yields

$$X_L(n,k)=10lo{g}_{10}\mid X(n,k){\mid }^2$$ (2)

where k denotes the frequency bin and n denotes a index frame at time instance n. Equation 2 also represents the energy of each frequency bin. We can categorize the energy of each bin into three different levels soft, moderate or loud. The K bins of the frequency domain can be grouped into nine bands with center frequencies as 0.25, 0.5, 1, 1.5, 2, 3, 4, 6, and 8 kHz. Every nine bands can be now amplified based on the user’s hearing thresholds and the input level of the respective band. To calculate gain A(k) for every nine bands, we follow the DSL-v5 hearing aid fitting procedure. Gain insertion can be given as follow:

$$E(n,k)=X_L(n,k)+A(k).$$ (3)

Bins falling in the same band will have a similar gain. After the insertion of gain, the signal passes through the compression function. The compression function makes sure that the amplified signal is within the comfortable loudness. This operation can be applied in the frequency domain as per:

$$F=\{\begin{array}{cc}E,\hfill & 2(E-T)<-W\hfill \\ E+\frac{(\frac{1}{R}-1)(E-T+\frac{W}{2}{)}^2}{2W},\hfill & 2\mid E-T\mid \le W\hfill \\ T+\frac{E-T}{R},\hfill & 2(E-T)>W,\hfill \end{array}\phantom{\}}$$ (4)

where T, R, and W are compression threshold, compression ratio, and knee width respectively. Notation of (n, k) is dropped for simplicity.

After the application of the compression function, gain smoothing is applied. Smoothing of the gain is necessary as sudden changes in the gain might create audible artifacts. In gain smoothing stage, attack time and release time comes into consideration. Gain smoothing is applied as

$$G=\{\begin{array}{cc}{\alpha }_{A}F{z}^{-1}+(1-{\alpha }_A)W_G,\hfill & W_G>F{z}^{-1}\hfill \\ {\alpha }_{R}F{z}^{-1}+(1-{\alpha }_R)W_G,\hfill & W_G\le F{z}^{-1}\hfill \end{array}\phantom{\}}$$ (5)

where, W_G can be calculated as W_G = F – X_L ; attack time and release time constants are given as ${\alpha }_A=\text{exp}\left(-\frac{K}{{\tau }_A{F}_{{}^s}}\right)$ and ${\alpha }_R=\text{exp}\left(-\frac{K}{{\tau }_R{F}_{{}^s}}\right)$ respectively. Here, τ_A and τ_R are attack time and release time respectively given in milliseconds.

Additionally, we adapt the release time based on the spectral flux. More transient the signal, the higher spectral flux and shorter the time constants needed for compression. Thus, release time is updated as

$${\tau }_R(n)=max({\tau }_{Rmin},\frac{{\tau }_{Rmin}}{S{F}_{smooth}(n{)}^{{\gamma }_R}+ϵ})$$ (6)

where τ_Rmin is the value of minimum desired release times, SF_smooth is the smoothed version of spectral flux SF(n), γ_R is constant controlling the dependence of release time constant on spectral flux, and ϵ is a small constant to prevent the denominator to be zero. The maximum value of τ_R can be limited to 2 s.

Figure 4 shows the adaption of release time based on the input signal and working of compression. SF(n) is the spectral flux at the n^th frame, given as

$$SF(n)=\frac{1}{N}\sum _k(\mid X(n,k)\mid -\mid X(n-1,k)\mid {)}^2$$ (7)

$$S{F}_{smooth}(n)=\alpha S{F}_{smooth}(n-1)+(1-\alpha )SF(n);$$ (8)

Figure 4:

Effect of adaptive compression on input signal

In cases when spectral flux changes too much, release time adaption shouldn’t be noticeable to user. Equation (8) gives the smoothed value for spectral flux SF_smooth(n). After the smoothing stage of the compression, the frequency domain output signal frame Y(n) is generated by shaping the frequency domain input signal based on a gain factor calculated from processing the side chain as shown in Figure 3. Thus,

$$Y(n,k)=X(n,k)\times {10}^{(\frac{M+G(n,k)}{20})}$$ (9)

where M is the make-up gain which controls the overall volume. The time-domain output signal frame y(n) can be obtained via inverse short-time Fourier transform (STFT) of Y(k).

4. SMARTPHONE IMPLEMENTATION

Real-time implementation of the proposed method has been done for the iPhone device operating on iOS 13 as the operating system. The bottom microphone is used to capture the incoming input signal. Real-time capturing and playback framework was developed in C++ programming on Xcode IDE. Core Audio library is used in making this real-time record and playback framework. All processing is done based on a frame-wise manner with 20 ms as an input frame and 16 kHz sampling rate. The output frame is calculated based on the proposed method as per Section 3. Output can be played back through the wireless headphone (Air-pods) or wired earphones to the HI person’s ear. Figure 5 shows the graphical user interface of the developed application. The application allows the user to input the hearing thresholds or audiogram data for each ear. Users can select the fitting strategy for gain calculation. Also, the user has the option to select the speed of the compression or set it to the specific value. Smartphone application provides the freedom of a parameter setting. An easy-to-use graphical interface allows the user to tune his/her compression parameters for the best listening experience.

Figure 5:

Screenshot of the developed smartphone application

5. EXPERIMENT AND EVALUATION

A clean speech signal was generated using the HINT database. We concatenated six sentences, selected based on intensities of the sentences as shown in Figure 4. Parameters of the compression considered are; T = 65 dB SPL, R = 2, W = 20 dB, M = 0 dB, τ_A = 10 ms. Three cases were considered for performance comparison. Fast-acting, slow-acting, and adaptive compression were considered for objective and subjective evaluation. All three types of compression were frequency-based multi-band compression with center frequencies of bands located at 0.25, 0.5, 1, 1.5, 2, 3, 4, 6, and 8 kHz. Only the release time of these methods was different. For fast-acting and slow-acting compression release time of 100ms and 800ms were considered respectively. For adaptive compression release time was varying between release times of fast and slow-acting compression and, considered to be changing as per (6). Figure 6 shows the comparison of objective measures. Perceptual evaluation of speech quality (PESQ) and hearing aid speech quality index were used as a measure of speech quality. Coherence and speech intelligibility index (CSII) and hearing aid speech perception index (HASPI) were used as a measure of speech intelligibility. Adaptive compression showed greater performance by all the measures. To subjectively evaluate these three methods, we considered a mean opinion score (MOS) of 10 normal-hearing participants suggesting score between 1 to 5, 1 being poorest and 5 being excellent quality. Figure 7a presents the MOS of the participants when compression is applied on clean speech. Figure 7b presents the MOS of the participants when machinery noise was mixed with the clean speech with 10 dB SNR and compression was applied. We can see that the fast-acting compressor performed worse in case of the presence of background noise.

Figure 6:

Objective evaluation of quality and intelligibility

Figure 7:

Subjective test results for (a) clean and (b) speech in presence of machinery noise (SNR 10dB)

6. CONCLUSION

A frequency-based multi-band dynamic range adaptive compression technique was proposed. The input signal was transformed into the frequency domain and divided into nine bands. The gain for each band was calculated based on the input intensity level and hearing threshold of a hearing-impaired person using the DSL-v5 gain fitting strategy. The effect of the parameter release time on the performance of the compressor was examined. To enhance the performance, spectral flux based adaption of release time was done. Real-time implementation of the proposed frequency-based multi-band compressor was done on a smartphone (iPhone). The developed application lets users input their hearing thresholds, set the parameters of the compression, change the speed of compression and fitting strategy according to their listening preferences. Objective and subjective evaluations showed performance improvement using the proposed adaptive compression method.