Influence of MVDR beamformer on a Speech Enhancement based Smartphone application for Hearing Aids

Abstract

This paper presents the minimum variance distortionless response (MVDR) beamformer combined with a Speech Enhancement (SE) gain function as a real-time application running on smartphones that work as an assistive device to Hearing Aids. It has been shown that beamforming techniques improve the Signal to Noise Ratio (SNR) in noisy conditions. In the proposed algorithm, MVDR beamformer is used as an SNR booster for the SE method. The proposed SE gain is based on the Log-Spectral Amplitude estimator to improve the speech quality in the presence of different background noises. Objective evaluation and intelligibility measures support the theoretical analysis and show significant improvements of the proposed method in comparison with existing methods. Subjective test results show the effectiveness of the application in real-world noisy conditions at SNR levels of −5 dB, 0 dB, and 5 dB.

I. Introduction

Researchers have developed numerous solutions for hearing impaired in the form of Hearing Aid Devices (HADs) and other hearing assistive devices. Performance of HADs and Cochlear Implants (CI) degrade in the presence of background noise, thus reducing the quality and intelligibility of speech. Speech Enhancement (SE) plays a vital role in suppressing the noise in various stationary and non-stationary environments while preserving the speech features. Already existing HADs do not have the power to handle computationally complex signal processing algorithms [1]–[4]. Lately, HADs manufacturers are using auxiliary devices as an external microphone in the form of a pen or a necklace to capture speech in real-time and transmit it to the HADs through wired or wireless connection [5]. The drawbacks of these existing devices are that they are expensive and not portable. One solution is to use a smartphone as an assistive device to HADs by capturing noisy signal using two microphones and performing complex computations. Smartphones also solve the problem of cost and portability.

In recent times, Apple iPhone and Android smartphones are coming up with sophisticated HA features such as Live Listen by Apple [6], and many third-party applications are being developed to enhance the quality and intelligibility of the speech perceived by the hearing-impaired people. In many HA applications on the smartphone, a single microphone is used to avoid audio input/ output latency. The proposed algorithm uses two microphones on the smartphone to suppress the background noise without distorting the clean speech.

Numerous studies have shown that SE is a key module in the HAD signal processing pipeline and would improve the listening comfort for the hearing impaired. Existing SE methods like Spectral Subtraction [7] and statistical model based methods proposed by Ephraim and Malah [8–9] can be implemented on a smartphone, but they do not improve the speech quality and intelligibility to a satisfactory extent. An ideal binary mask in SE can improve intelligibility [10], but precise estimation of the binary mask is challenging, especially in lower SNR conditions.

Over the last few decades, researchers have developed beamforming algorithms, which can be classified into fixed and adaptive beamformers. Fixed beamformers have static filter coefficients and signal independent spatial response [11]. The isotropic model, which is a first-order approximation of real noise fields, is commonly used in these beamformers and the noise field is not known. This limitation makes the fixed beamformer less efficient for real applications. As a solution, the beamforming filter coefficients can be changed, leading to the second class of adaptive beamformers. Among several SE techniques, adaptive beamformers are commonly used to improve the performance of the algorithm. The MVDR beamformer has wide range applications for extraction of desired speech signals in noisy environments [12]. The MVDR beamformer, known as Capon beamformer [13], dating back to 1980s, minimizes the output power of the beamformer under a single linear constraint on the response of the array towards the preferred signal. This spatial filtering process plays a critical role in extracting the signal of interest, suppressing ambient noise, and separating multiple sound sources. MVDR beamformer requires less a priori knowledge, which makes it practical for implementing it as a smartphone-based SE application for HADs.

The proposed algorithm is a combination of MVDR beamformer and Minimum mean square error Log spectral amplitude estimator (Log-MMSE) SE gain function, for suppressing noise and extracting the desired speech. This method is computationally efficient and helps in achieving minimal speech distortion for the hearing-impaired. Performance of the proposed method is compared against standard techniques of SE for speech quality and intelligibility. Subjective evaluations show promising results of the real-time application.

II. Proposed SE Gain Function

In the smartphone application, signals captured by the two microphones is composed of both clean speech and the background noise. Figure 1 shows the block diagram of the proposed method implemented on the smartphone. We consider a signal model with the first microphone as the reference point, the signal received by the n^th microphone (n = 1, 2) can be written as,

$$y_n\left(t\right)=s_n\left(t\right)+w_n\left(t\right)=s\left(t-{\tau }_n\right)+w_n\left(t\right)$$ (1)

where y_n(t), s_n(t) and w_n(t) are noisy speech, clean speech and noise signals respectively picked up by the n^th microphone at time t. Let τ₀ be the relative time delay between the two microphones given by τ₀ = δ/c with δ as the spacing between the two microphones and c being the speed of sound in air. The signals are considered to be zero mean and real, noise signal w_n(t) are assumed to be uncorrelated with s_n(t).

Fig. 1:

Block Diagram of Proposed SE method

For efficient performance, the signals are transformed to frequency domain and are re-written as,

$$\begin{gathered} Y_n\left(\omega \right)=S_n\left(\omega \right)+W_n\left(\omega \right) \\ =e^{-j\left(n-1\right)\omega {\tau }_0\cos \left({\theta }_d\right)}{S}_1\left(\omega \right)+W_n\left(\omega \right) \end{gathered}$$ (2)

where Y_n(ω), S_n(ω), W_n(ω) are the Fourier transforms of y_n(t), s_n(t), w_n(t) respectively. ω = 2πf is the angular frequency. Equation (2) can be rearranged into vector form as follows,

$$\begin{gathered} Y\left(\omega \right)\triangleq {\left[Y_1\left(\omega \right)Y_2\left(\omega \right)\right]}^T \\ =d_{{\theta }_d}\left(\omega \right)S_1\left(\omega \right)+W\left(\omega \right) \end{gathered}$$ (3)

where the superscript ^T is the transpose operator,

$$d_{{\theta }_d}\left(\omega \right)\triangleq {\left[1e^{-j\omega {\tau }_0\cos \left({\theta }_d\right)}\right]}^T$$ (4)

is the steering vector and the noisy signal W(ω), is defined similar to Y(ω). θ_d is the angle of incidence of the source at the plane of microphones. Since the signals are assumed to be uncorrelated, the correlation matrix of Y(ω) can be determined by the method explained in [11].

A. MVDR Beamformer

The goal of beamforming is to extract the desired speech signal S₁(ω), by applying a linear filter h(ω) to Y(ω). This can be shown as follows,

$$Z\left(\omega \right)=h^H\left(\omega \right)d_{{\theta }_d}\left(\omega \right)S_1\left(\omega \right)+h^H\left(\omega \right)W\left(\omega \right)$$ (5)

where Z(ω) is the output of the beamformer, h^H(ω)S₁(ω) is the filtered speech signal, and h^H(ω)W(ω) is the residual noise.

The MVDR beamformer output can be obtained by minimizing the variance on either side of (5), or the residual noise with the constraint that the signal from the desired direction is without any distortion. In this work, we consider the minimization of variance of the residual noise.

$$\underset{h\left(\omega \right)}{\min }E[{|h^H\left(\omega \right)W\left(\omega \right)|}^2]\text{subject to}{h}^H\left(\omega \right)d_{{\theta }_d}\left(\omega \right)=1$$ (6)

E[.] denotes mathematical expectation. Using a Lagrange multiplier to adjoin the constraint to the objective function, then differentiating with respect to h(ω), and equating the result to zero, (6) can be reduced to,

$$h\left(\omega \right)=\frac{{\Gamma }_W^{-1}\left(\omega \right)d_{{\theta }_d}\left(\omega \right)}{d_{{\theta }_d}^H\left(\omega \right){\Gamma }_W^{-1}\left(\omega \right)d_{{\theta }_d}\left(\omega \right)}$$ (7)

where Γ_w(ω) = Φ_w(ω)/ϕ_w1(ω) is the pseudo-coherence matrix of the noise with Φ_w(ω) = E[W(ω)W^H(ω)] and ϕ_w1(ω) = E[|W₁(ω)|²].

B. Log-Spectral Amplitude Estimator

In the Log-MMSE method, speech and noise models are considered to be statistically independent Gaussian Random Variables [14]. The aim is to minimize the mean squared error of log magnitude spectra between estimated and true speech. The input is taken to be the output of the MVDR beamformer z(n), which contains filtered speech signal s′(n), and some residual noise w′(n),

$$z\left(n\right)=s^{\prime }\left(n\right)+w^{\prime }\left(n\right)$$ (8)

The noisy k^th Discrete Fourier Transform (DFT) coefficient of y(n) for frame λ is given by,

$$Z_k\left(\lambda \right)={S^{\prime }}_k\left(\lambda \right)+W_k^{\prime }\left(\lambda \right)$$ (9)

Where S′ and W′ are the input speech and noise DFT coefficients. In polar coordinates, (9) can be written as,

$$R_k\left(\lambda \right)e^{j{\theta }_{z_k}\left(\lambda \right)}=A_k\left(\lambda \right)e^{j{\theta }_{{s^{\prime }}_k}\left(\lambda \right)}+B_k\left(\lambda \right)e^{j{\theta }_{w_k^{\prime }}\left(\lambda \right)}$$ (10)

Where R_k(λ), A_k(λ), B_k(λ) are magnitude spectra of noisy speech, input signal and noise respectively. θ_zk(λ), θ_s′k(λ), ${\theta }_{w_k^{\prime }}\left(\lambda \right)$ are the phase spectra of noisy, input speech and noise respectively. Looking at the estimator ${\widehat{A}}_k$, which minimizes the distortion measure as explained in [8], the mean-square error of the log-magnitude spectra is given by,

$$E\left\{{\left(log{A}_k-log{\widehat{A}}_k\right)}^2\right\}$$ (11)

Where, A_k is the k^th bin of magnitude spectrum, and ${\widehat{A}}_k$ is the k^th bin of estimated clean speech magnitude spectrum. The optimal log-MMSE estimator can be obtained by evaluating the conditional mean of the log A_k, that is,

$$log{\widehat{A}}_k=E\left\{log{A}_k|Z_k\left(\lambda \right)\right\}$$ (12)

Hence, the estimate of the speech magnitude is given by,

$${\widehat{A}}_k=exp\left(E\left\{log{A}_k|Z_k\left(\lambda \right)\right\}\right)$$ (13)

Solving the above expectation, the final estimate of speech magnitude spectrum according to [8] is given by,

$$\begin{gathered} {\widehat{A}}_k=\frac{{\xi }_k}{{\xi }_k+1}\exp \left\{\frac{1}{2}{\int }_{{\nu }_k}^{\infty }\frac{e^{-t}}{t}dt\right\}{R}_k \\ \triangleq G_k{R}_k \end{gathered}$$ (14)

Where ${\nu }_k=\frac{{\xi }_k}{1+\xi }{\gamma }_k$ here ${\xi }_k=\frac{{\sigma }_{{s^{\prime }}_k}^2}{{\sigma }_{w_k}^2}$ is the a priori SNR and ${\gamma }_k=\frac{R_k^2}{{\sigma }_{w_k}^2}$ is the a posteriori SNR. ${\sigma }_{W_k}^2$ is estimated using a voice activity detector (VAD) [15]. σ_s′k is the estimated instantaneous clean speech power spectral density. The optimal phase spectrum is the noisy phase spectrum itself θ_Sk = θ_Yk. The final clean speech estimate is,

$${{\widehat{S}}^{\prime }}_k=G_k{Z}_k$$ (15)

The time domain reconstruction signal ${\widehat{s}}^{\prime }(n)$ is obtained by taking inverse Fourier Transform of ${{\widehat{S}}^{\prime }}_k$.

III. Real-Time Implementation on Smartphone as an Assistive Device to HADs

In this paper, Google Pixel running Android 7.1 Nougat operating system is considered as an assistive device. Two microphones (13 cm apart) on the smartphone capture the audio signal, process the signal and transmit the enhanced signal to the HADs. The smartphone device considered has an M4/T4 HA Compatibility rating and meets the requirements set by Federal Communications Commission (FCC). Android Studio [16] is used for implementation of the SE algorithm on the smartphone. An inbuilt android audio framework was used to carry out dual microphone input/output handling. The input data is acquired at 48 KHz sampling rate and a 10ms frame, with FFT size to be 512 is considered as the input buffer. Figure 2 shows the screenshot of the proposed SE method implemented on Pixel smartphone. When the button is in “ON” mode, the microphone will record the audio signal and playback to the HADs without any processing. There is another button present on the screen to apply the developed SE algorithm to enhance the audio stream. The enhanced output signal is then played back to the HADs. Initially, when the SE algorithm is turned on, the algorithm uses approximately 3 seconds to estimate the noise variance. Hence, we assume there is no speech activity during this time. The smartphone application is computationally efficient and consumes less power.

Fig 2:

Screenshot of developed SE method

IV. Experimental Results

A. Objective Evaluation

The performance of the proposed method is evaluated by comparing with dual microphone coherence [17] and Log-MMSE [9] methods, promising results are seen. The objective evaluations are performed for 3 different noise types: machinery, multi-talker babble, and traffic noise.

The plotted results are the average over different speech signals from the HINT database. The audio files are sampled at 16 kHz, and 10 ms frames with 50% overlap are considered. Perceptual evaluation of speech quality (PESQ) [18] and short time objective intelligibility (STOI) [19] are used to measure the quality and intelligibility of the speech respectively. PESQ ranges between 0.5 and 4.5, with 4.5 being high perceptual quality. Higher the score of STOI better the speech intelligibility. Figure 3 shows the plots of PESQ and STOI versus 3 different SNR for the 3 noise types. PESQ and STOI values show substantial improvements over other methods for all three noise types considered. Objective and Intelligibility measures state the fact that the proposed SE method suppresses more noise with minimal speech distorting.

Fig.3.

Objective evaluation of speech quality and intelligibility

B. Subjective test setup and results

Subjective measures give information about the practical usability of our application in real-time. Thus, Mean Opinion Score (MOS) tests [20] was performed on 10 normal hearing subjects including 5 male and 5 female adults. They were presented with noisy speech and enhanced speech using the proposed, coherence and Log-MMSE methods at different SNR levels of −5 dB, 0 dB, and 5 dB. The audio files were played on headphones for the subjects. Each subject was instructed to rate between 1 and 5 for each audio file based on the following criteria: 5 being excellent speech quality and imperceptible level of distortion. 1 having the least quality of speech and intolerable level of distortion. This test provided a good comparison between the proposed method and other existing methods. Subjective test results in Fig. 4 illustrate the effectiveness of the proposed method in reducing the background noise, simultaneously preserving the quality and intelligibility of the speech.

Fig.4.

Comparison of Subjective results

V. Conclusion

An MVDR beamformer based dual microphone SE algorithm was developed and implemented on a smartphone as a real-time application. This method can act as an assistive device for HADs. Objective and Subjective evaluations verify that the proposed method can be used as a solution to enhance the speech in real-world noisy enviromnents.