Improved Convolutive and Under-Determined Blind Audio Source Separation with MRF Smoothing

Abstract

Convolutive and under-determined blind audio source separation from noisy recordings is a challenging problem. Several computational strategies have been proposed to address this problem. This study is concerned with several modifications to the expectation-minimization-based algorithm, which iteratively estimates the mixing and source parameters. This strategy assumes that any entry in each source spectrogram is modeled using superimposed Gaussian components, which are mutually and individually independent across frequency and time bins. In our approach, we resolve this issue by considering a locally smooth temporal and frequency structure in the power source spectrograms. Local smoothness is enforced by incorporating a Gibbs prior in the complete data likelihood function, which models the interactions between neighboring spectrogram bins using a Markov random field. Simulations using audio files derived from stereo audio source separation evaluation campaign 2008 demonstrate high efficiency with the proposed improvement.

Introduction

Blind source separation (BSS) aims to recover unknown source signals from observed mixtures with or without very limited information about their mixing process. BSS problems have been addressed in many previous studies, for example, [1–12], which were motivated by several real-world applications.

In a cocktail-party problem, microphones receive noisy mixtures of acoustic signals that propagate along multiple paths from their sources. In a real scenario, the number of audio sources may be greater than the number of microphones, audio sources may have different timbres and similar pitches, and audio signals may be only locally stationary.

A convolutive and under-determined mixing approach needs to be adopted to model this problem. There are several techniques for solving convolutive unmixing problems [13]. Some of these [14] operate in the time-domain by solving the alternative finite impulse response (FIR) inverse model using independent component analysis (ICA) methods [2]. Another method is to extract meaningful features from the time-frequency (TF) representations of mixtures. This approach seems to be more efficient than the ICA-based techniques especially when the number of microphones is lower than the number of sources. Acoustic signals are usually sparse in the TF domain, so the source signals can be separated efficiently even if they are partially overlapped and the problem is under-determined. These features can be extracted using several techniques, including TF masking [15, 16], frequency bin-wise clustering with permutation alignment (FBWC-PA) [17, 18], subspace projection [19], hidden Markov models (HMM) [20], interaural phase difference (IPD) [21], nonnegative matrix factorization (NMF) [22, 23], and nonnegative tensor factorization (NTF) [24].

Nonnegative matrix factorization [25] is a feature extraction method with many real-world applications [26]. A convolutive NMF-based unmixing model was proposed by Smaragdis [22]. Ozerov and Fevotte [23] developed the EM-NMF algorithm, which is suitable for unsupervised convolutive and possibly under-determined unmixing of audio sources using only stereo observations. Their model of the sources was based on the generalized Wiener filtering model [27–29], which assumes that each source is locally stationary and that it can be expressed in terms of superimposed amplitude-modulated Gaussian components. Thus, a power spectrogram of each source can be factorized into lower-rank nonnegative matrices, which facilitates the use of NMF for estimating the frequency and temporal profiles of each latent source component. In the TF representation, the latent components are mutually and individually independent across frequency and time bins. However, this assumption is very weak for any adjacent bins because real audio signals have locally smooth frequency and temporal structures.

Motivated by several papers on smoothness [26, 28, 30, 31] in BSS models, we attempt to further improve the EM-NMF algorithm by enforcing local smoothness both in the frequency and temporal profiles of the NMF factors. Similar to [28, 30, 32], we introduce a priori knowledge to the NMF-based model using a Bayesian framework, although our approach is based on a Gibbs prior with a Markov random field (MRF) model to describe pairwise interactions among adjacent bins in spectrograms. As demonstrated in [33], the MRF model with Green’s function, which is well known in many tomographic image reconstruction applications [34], can improve the EM-NMF algorithm. In this paper, we extend the results presented in [33] using other smoothing functions, particularly a more flexible simultaneous autoregressive (SAR) model that is more appropriate in term of hyperparameter estimation and computational complexity.

The rest of this paper is organized as follows. The next section reviews the underlying separation model. Section 3 is concerned with MRF smoothing. The optimization algorithm is described in Sect. 4. Audio source separation experiments are presented in Sect. 5. Finally, the conclusions are provided in Sect. 6.

Model

Let I microphones receive signals that can be modeled as a noisy convolutive mixture of J audio signals. The signal received by the i-th microphone () can be expressed as

1

where represents the corresponding mixing filter coefficient, is the j-th source signal (), is the additive noise, and L is the length of the mixing filter.

In the TF domain, the model (1) can be expressed as

2

where , and is the index of a frequency bin.

The noise n _ift is assumed to be stationary and spatially uncorrelated, i.e,

3

where and is a proper complex Gaussian distribution with a zero-mean and the covariance matrix .

Benaroya et al. [27] described an audio source as a superimposed amplitude-modulated Gaussian process:

4

where h _r(t) is a slowly varying amplitude parameter in the r-th component (), and is a stationary zero-mean Gaussian process with the power spectral density σ²_r (f). The TF representation of (4) leads to

5

The power spectrogram of (5) is given by |s _ft|² = ∑^R_r=1 w _fr h _rt, where w _fr = σ²_r(f). Thus, the spectrogram of the j-th source can be factorized as follows:

6

where is the number of latent components in the j-th source, and is the nonnegative orthant of the Euclidean space. The column vectors of represent the frequency profiles of the j-th source, while the row vectors of are the temporal profiles.

Févotte et al. [28] transformed the model (5) to the following form:

7

where . Thus,

8

and , where . Consequently, the model (2) can be expressed as

9

is the number of entries in the set , and is created from the columns of the matrix . For example, assuming , we have , and is the augmented mixing matrix [23] created from matrices . From (8) and (9), we have where .

To estimate the parameters , and , we formulate the following posterior:

10

from which we obtain

11

From (3) and (9), we have the joint conditional PDF for :

12

Based on (12), the log-likelihood term in (11) can be expressed as

13

where , and .

The joint conditional PDF for comes from the model (5):

14

From (14), we have the log-likelihood functional for

15

The negative log-likelihood in (15) is the Itakura-Saito (IS) divergence [35], which is particularly useful for measuring the goodness of fit between spectrograms. The IS divergence is the special case of the β-divergence when [26].

The priors and in (10) can be determined in many ways. Févotte et al. [28] proposed the determination of priors using Markov chains and the inverse Gamma distribution. In our approach, we propose to model the priors with the Gibbs distribution, which is particularly useful for enforcing local smoothness in images.

MRF Smoothing

Let us assume that prior information on the total smoothness of the estimated components W and H is modeled using the following Gibbs distributions:

16

where Z _W and Z _H are partition functions, α_W and α_H are regularization parameters, and U(P) is a total energy function, which measures the total roughness in P. The function U(P) is often formulated with respect to the MRF model, which is commonly used in image reconstruction for modeling local smoothness.

The functions U(W) and U(H) can be determined for the matrices W and H in the following way:

17

18

In the first-order interactions (nearest neighborhood), we have S _f = {f − 1, f + 1} and the weighting factor ν_fl = 1, and S _t = {t − 1, t + 1} with ν_tl = 1. In the second-order interactions, S _f = {f − 2, f − 1, f + 1, f + 2} and S _t = {t − 2, t − 1, t + 1, t + 2}. The parameters δ_W and δ_H are scaling factors, while is a potential function of ξ that can take different forms. The potential functions that can be applied to the EM-NMF algorithm are listed in Table 1.

Table 1.

Potential functions

Author(s) (name)	Functions: ψ(ξ, δ)	Reference
(Gaussian)	(ξ/δ)2
Besag (Laplacian)		[36]
Bouman and Sauer (GGMRF)	|ξ/δ|p	[37]
Geman and McClure		[38]
Geman and Reynolds		[39]
Green		[34]
Hebert and Leahy		[40]

According to Lange [41], a robust potential function in the Gibbs prior should have the following properties: nonnegative, even, 0 at ξ = 0, strictly increasing for ξ > 0, unbounded, and convex with bounded first-derivative. Of the functions listed in Table 1, Green’s function satisfies all of these properties, and consequently, it was selected for use in the tests in [33]. Unfortunately, the application of Green’s function to both matrices W and H demands the determination of two hyperparameters δ_W and δ_H, and two penalty parameters α_W and α_H. Moreover, data-driven hyperparameter estimation usually involves an approximation of the partition functions and , which is not easy in this task.

The Gaussian function , as shown in Table 1, does not have a bounded first-derivative, but its scaling parameter δ may be merged with a penalty parameter α. Consequently, only two parameters need to be determined. The MRF model with a Gaussian potential function is actually the SAR model [42–44], which is used widely in many scientific fields [44, 45] to represent the interactions among spatial data with Gaussian noise. Let be the r-th column of the matrix W, and be the r-th row of the matrix H. Random variables in the vectors and can be modeled using the following stochastic equations:

19

where and are symmetric matrices of spatial dependencies between the random variables, is an i.i.d. Gaussian noise, and I is an identity matrix with a corresponding size.

According to [45, 46], the spatial dependence matrices can be expressed as and , where γ is a constant that ensures that the matrices and are positive-definite, while and are binary symmetric band matrices indicating the neighboring entries in and , respectively. In the first-order interactions, we have z ^(W)_1,2 = z ^(W)_F,F-1 = z ^(W)_m,m-1 = z ^(W)_m,m+1 = 1 for for , and z _mf = z _tn = 0 otherwise. In the P-order interactions, each entry w _fr and h _rt has the corresponding sets of neighbors: {w _f-ν,r}, {w _f+ν,r}, {h _r,t-ν}, {h _r,t+ν} with . As a consequence, and are symmetric band matrices with P sub-diagonals and P super-diagonals, the entries of which are equal to ones, but zeros otherwise. The matrices and are positive-definite, if γ < (2P)⁻¹ for P-order interactions [45, 46]. We selected , where is a small constant, for example, .

In the SAR model, Gibbs priors (16) may be expressed as their joint multivariate Gaussian priors:

20

21

where , and is the f-th eigenvalue of the matrix . Similarly, . If and , which simplifies the hyperparameter estimation.

Algorithm

The EM algorithm [47] is applied to maximize in (11). To calculate the E-step, the log-likelihood functional (13) is transformed to the following form

22

where the correlation matrices are given by , and the cross-correlation .

Ozerov et al. [23] observed that the set {R ^(xx)_f, R ^(xs)_f, R ^(ss)_f, |c _rft|² } provides sufficient statistics for the exponential family [47], so the sources and the latent components can be estimated by computing the conditional expectations of the natural statistics. According to [23], we have the following posterior estimates:

23

24

Similarly, for the latent components, we have

25

26

The conditional expectations for the sufficient statistics are as follows:

27

28

29

Detailed derivations of the formulae (23)–(26) are presented in the ``Appendix''.

From the M-step, we have , which gives . From

we have

From , we have

30

Similarly, from , we get

31

The terms and in (30) and (31) take the following forms with respect to the potential functions:

• Gaussian (SAR model):

  32

  33
• GR function (proposed by Green [34]):

  34

  35
• HL function (proposed by Hebert and Leahy [40]):

  36

  37

Experiments

Experiments were conducted using selected sound recordings taken from the stereo audio source separation evaluation campaign (SiSEC)1 in 2007. This campaign aimed to evaluate the performance of source separation algorithms using stereo under-determined mixtures. We selected the benchmarks given in Table 2, which included speech recordings (three male voices—

male3

, and three female voices—

female3

), three nonpercussive music sources—

nodrums

, and three music sources that included drums—

wdrums

. The mixed signals were recordings that lasted 10 s, which were sampled at 16 kHz (the standard settings of recordings from the “Under-determined speech and music mixtures" datasets in the SiSEC2008). For each benchmark, the number of true sources was three (J = 3) but it only had two microphones (I = 2), that is, stereo recordings. Thus, for each case, we faced an under-determined BSS problem. All instantaneous mixtures were obtained using the same mixing matrix with positive coefficients. Synthetic convolutive mixtures were obtained for a meeting room with a 250 ms reverberation time using omnidirectional microphones with 1 m spacing.

Table 2.

Benchmarks

Instantaneous	Convolutive
male3_inst_mix	male3_synthconv_250ms_1m_mix
female3_inst_mix	female3_synthconv_250ms_1m_mix
nodrums_inst_mix	nodrums_synthconv_250ms_1m_mix
wdrums_inst_mix	wdrums_synthconv_250ms_1m_mix

The spectrograms were obtained by a short-time fourier transform (STFT) using half-overlapping sine windows. To create the spectrograms and recover the time-domain signals from STFT coefficients, we used the corresponding

stft

_

multi

and

istft

_

multi

Matlab functions from the SiSEC2008 webpage2 [48]. For instantaneous and convolutive mixtures, the window lengths were set to 1,024 and 2,048 samples, respectively.

The EM-NMF algorithm was taken from Ozerov’s homepage3, while the MRF-EM-NMF algorithm was coded and extensively tested by Ochal [49].

The proposed algorithm is based on an alternating optimization scheme, which is intrinsically non-convex, and hence, its initialization plays an important role. An incorrect initialization may result in slow convergence and early stagnation at an unfavorable local minimum of the objective function. As done in many NMF algorithms, the factors W and H are initialized with uniformly distributed random numbers, whereas the entries in the matrix A are drawn from a zero-mean complex Gaussian distribution. After W and H have been initialized, the covariance matrices and given by (8) can be computed. A noise covariance matrix is needed to update the E-step. Ozerov and Fevotte [23] tested several techniques for determining this matrix. The E-step in MRF-EM-NMF is identical to that in EM-NMF [23], and hence, all of these techniques can be used in this experiment. The initial matrix was determined based on the empirical variance of the observed power spectrograms.

The MRF-EM-NMF and EM-NMF algorithms were initialized using the same random values (given as ) and run for 1,500 iterations.

The choice of the parameters {α_W, α_H, γ_W, γ_H} used in the Gibbs distributions also affected the performance. The regularization parameters can be fixed or changed with iterations. Motivated by iterative thresholding strategies [26], we used the following strategies:

• Linear thresholding:

  
• Nonlinear thresholding:

  
• Fixed thresholding:

  

  where k is the current iteration, k _max is the maximum number of iterations, is the shape parameter, is the shift parameter, k ₁ is the threshold, and α can be equal to α_W or α_H. All of the above thresholding strategies aim to relax smoothing during the early iterations when the descent directions in the updates are sufficiently steep and to emphasize smoothing if noisy perturbations become significantly detrimental to the overall smoothness. These strategies are motivated by standard regularization rules that apply to ill-posed problems. We tested all of the thresholding strategies using instantaneous and convolutive mixtures, and we obtained the best performance with fixed thresholding using k ₁ = k _max/2.

The parameters δ_W and δ_H in the MRF models can be estimated using standard marginalization procedures or by maximizing the Type II ML estimate for (10). However, these techniques have a huge computational cost for the nonlinear potential functions in the MRF models. For practical reasons, they are not very useful for the GR or HR functions.

In this study, we tested all of the benchmarks in Table 2 and the following potential functions: the first- and second-order Gaussian, GR, and HR. For the Gaussian functions, we tested all combinations of the regularization parameters α_W and α_H from the discrete set {0.001, 0.005, 0.01, 0.05, 0.1}. For GR and HL, the regularization parameters could take only two values, {0.001, 0.01}, although the parameters δ_W and δ_H were tested with the following values: {0.1, 1, 10}. The optimal values of the smoothing parameters are summarized in Table 3.

Table 3.

Parameters of the MRF-EM-NMF algorithm for each test case shown in Fig. 1

Benchmark	Smoothing	Instantaneous mixture					Convolutive mixture
			αW	αH	δW	δH		αW	αH	δW	δH
Male	GR	12	0.01	0.01	1	1	4	0.01	0.01	0.1	10
Male	HL	12	0.001	0.001	1	10	4	0.001	0.01	1	1
Male	1-Gaussian	12	0.001	0.01	–	–	4	0.05	0.05	–	–
Male	2-Gaussian	12	0.001	0.01	–	–	4	0.05	0.01	–	–
Female	GR	12	0.01	0.01	10	10	4	0.01	0.01	1	1
Female	HL	12	0.001	0.001	1	10	4	0.001	0.001	0.1	10
Female	1-Gaussian	12	0.001	0.001	–	–	4	0.1	0.001	–	–
Female	2-Gaussian	12	0.001	0.001	–	–	4	0.05	0.005	–	–
Nodrums	GR	4	0.01	0.01	10	1	4	0.01	0.01	10	0.1
Nodrums	HL	4	0.01	0.001	1	10	4	0.01	0.01	0.1	0.1
Nodrums	1-Gaussian	4	0.001	0.01	–	–	4	0.001	0.05	–	–
Nodrums	2-Gaussian	4	0.01	0.001	–	–	4	0.005	0.01	–	–
Wdrums	GR	4	0.01	0.01	1	10	4	0.01	0.01	1	1
Wdrums	HL	4	0.01	0.001	1	10	4	0.001	0.01	1	0.1
Wdrums	1-Gaussian	4	0.001	0.001	–	–	4	0.001	0.1	–	–
Wdrums	2-Gaussian	4	0.001	0.001	–	–	4	0.005	0.1	–	–

The notations "1-Gaussian" and "2-Gaussian" represent the first- and second-order Gaussian functions, respectively

The separation results were evaluated in terms of the signal-to-distortion ratio (SDR) and the signal-to-interference ratio (SIR) [50]. Figure 1 shows the SDRs and SIRs averaged for the sources, which were estimated using the EM-NMF and MRF-EM-NMF with various smoothing functions based on instantaneous and convolutive mixing models. For each sample in Table 2 and each smoothing function, the smoothing parameters were tuned optimally for a given fixed initializer. This unsupervised learning approach evaluated the efficiency of the smoothing functions with respect to a given recording scenario. However, the smoothing parameters need to be determined with a supervised learning framework in practice. To test this option, each recording in Table 2 was divided into two 5 s excerpts during the training and testing stages. For each training excerpt, the smoothing parameters and initializer were selected to maximize the SDR performance. Testing was performed on the other excerpt with the same initializer. The results obtained during the testing stage with the instantaneous mixtures are shown in Fig. 2.

Fig. 1.

Source separation results obtained with the MRF-EM-NMF (first- and second-order Gaussian, GR, and HL functions) and EM-NMF (no smoothing) algorithms after 1,500 iterations: a mean SDR (dB) for instantaneous mixture, b mean SDR (dB) for convolutive mixture, c mean SIR (dB) for instantaneous mixture, d mean SIR (dB) for convolutive mixture. The smoothing parameters were tuned separately for each mixture in Table 2

Fig. 2.

Source separation results obtained in the testing stage with the MRF-EM-NMF (first- and second-order Gaussian, GR, and HL functions) and EM-NMF (no smoothing) algorithm after 1,500 iterations: a mean SDR (dB), b mean SIR (dB). The smoothing parameters were determined during the training stage. 5 s excerpts were used in the training and testing stages

For comparison, Table 4 shows the average SDR results produced and the running time taken when using several state-of-the-art algorithms, which were applied to the mixtures in Table 2. The generalized Gaussian prior (GGP) algorithm [51] and the statistically sparse decomposition principle (SSDP) algorithms [52] were applied to the instantaneous mixtures. The convolutive mixtures were unmixed with the IPD [21], two versions of the FBWC-PA [17, 18] algorithm, and the Convolutive NMF [22]. Note that the last method in this list was based on supervised learning, whereas the others were unsupervised learning algorithms. In this case, the first 8 s excerpts of the 10 s source recordings were used for learning, while the remainder was used for testing.

Table 4.

Mean SDR (dB) and running time (s) for sources estimated from the mixtures shown in Table 2

Benchmark	Mixture	Male	Female	Nodrums	Wdrums	Time
MRF-EM-NMF (HR)	inst	8.06	9.95	24.07	21.72	2487
MRF-EM-NMF (GR) [33]	inst	7.69	8.86	26.65	21.28	2498
EM-NMF [23]	inst	2.62	6.5	11.7	19.87	2456
GGP [51]	inst	8.4	8.57	13.9	10.3	5
SABM+SSDP [52]	inst	4.25	3.82	5.83	9.43	2
MRF-EM-NMF (HR)	conv	1.06	2.2	1.17	1.7	2760
MRF-EM-NMF (GR) [33]	conv	1.4	2.1	1.2	1.56	2762
EM-NMF [23]	conv	0.95	1.6	0.2	0.44	2720
IPD [21]	conv	1.53	1.43	2.2	−2.7	1200
FBWC-PA [17]	conv	−0.1	4.43	0.77	−2.53	40
Generalized FBWC-PA [18]	conv	5.95	7.45	1.2	−0.69	8
ConvNMF [22]	conv	−0.7	−0.47	3.85	8.13	347

The averaged elapsed time measured using Matlab 2008a for 1,500 iterations with , executed on a 64-bit Intel Quad Core CPU 3 GHz with 8 GB RAM was almost the same for the MRF-EM-NMF and EM-NMF algorithms (see Table 4).

The simulations demonstrate that MRF smoothing improved the source separation results in almost all test cases. The results confirmed that instantaneous mixtures were considerably easier to separate than convolutive ones. The MRF-EM-NMF algorithm delivered the best mean SDR performance of all the algorithms tested with instantaneous mixtures. The highest SDR values were produced with instantaneously mixed non-percussive music sources. This was justified by the smooth frequency and temporal structures of non-percussive music spectrograms. If the source spectrograms were not very smooth (as with the percussive audio recordings), MRF smoothing gave only a slight improvement (see Figs. 1, 2) in the first-order MRF interactions, and even a slight deterioration in the higher-order MRF interactions. According to Fig. 1, the HL function delivered the most promising SDR results, which were stable with a wide range of parameters. In each case with the instantaneous mixtures, the best results were produced with the same hyperparameter values, δ_W = 1 and δ_H = 10, and almost the same penalty parameter values, α_W and α_H. The SAR model also improved the results compared with the standard EM-NMF algorithm. Moreover, the SAR model was tuned using only two penalty parameters, and the partition function of the associated Gibbs prior could be derived using a closed-form expression, which might be very useful for data-driven hyperparameter estimation.

The source separation results produced with the MRF-EM-NMF algorithm for convolutive and under-determined mixtures were better than those obtained with the EM-NMF algorithm. Unfortunately, the SDR values showed that these results were still a long way from being perfect, even after 1,500 iterations, and thus, further research is needed in this field. It is likely that some additional prior information could be imposed, especially on a mixing operator, which might increase the efficiency considerably.

It should be noted that the SDR performance with both mixtures could still be improved by refining the associated parameters, especially in the MRF models, and by using more efficient initializers.

Conclusions

This study demonstrated that imposing MRF smoothing on the power spectrograms of audio sources estimated from under-determined unmixing problems may improve the quality of estimated audio sounds considerably. This was justified because any type of meaningful prior information improves the performance, especially with under-determined problems. This study addressed the application of MRF smoothing in the EM-NMF algorithm, but this type of smoothing could be applied to many other related BSS algorithms based on feature extraction from power spectrograms. Thus, the theoretical results presented in this paper may have broad practical applications. Clearly, further studies are needed to improve this technique for convolutive mixtures and to integrate regularization parameter estimation techniques in the main algorithm.

Appendix

The conditional expectations of the natural statistics can be derived from the a posteriori distributions and . Thus,

38

where ,

We can transform in (38) into the following form:

Using the Woodbury matrix identity, we have

and finally,

39

where

40

41

Thus, . From (5), it follows that , so . Since the zero-mean noise n _ft from (3) is not correlated with , we have

42

Inserting (42) and into (40) and (41), we obtain the update rules (23) and (24), respectively.

Analyzing , one can obtain , which yields the update rules (25) and (26).