Proportionate Adaptive Filtering Algorithms Derived Using an Iterative Reweighting Framework

Abstract

In this paper, based on sparsity-promoting regularization techniques from the sparse signal recovery (SSR) area, least mean square (LMS)-type sparse adaptive filtering algorithms are derived. The approach mimics the iterative reweighted ℓ₂ and ℓ₁ SSR methods that majorize the regularized objective function during the optimization process. We show that introducing the majorizers leads to the same algorithm as simply using the gradient update of the regularized objective function, as is done in existing approaches. Different from the past works, the reweighting formulation naturally leads to an affine scaling transformation (AST) strategy, which effectively introduces a diagonal weighting on the gradient, giving rise to new algorithms that demonstrate improved convergence properties. Interestingly, setting the regularization coefficient to zero in the proposed AST-based framework leads to the Sparsity-promoting LMS (SLMS) and Sparsity-promoting Normalized LMS (SNLMS) algorithms, which exploit but do not strictly enforce the sparsity of the system response if it already exists. The SLMS and SNLMS realize proportionate adaptation for convergence speedup should sparsity be present in the underlying system response. In this manner, we develop a new way for rigorously deriving a large class of proportionate algorithms, and also explain why they are useful in applications where the underlying systems admit certain sparsity, e.g., in acoustic echo and feedback cancellation.

I. Introduction

ADAPTIVE filters [1], [2], [3], [4] have been an active research area over the past few decades for their capabilities of estimating and tracking time-varying systems. In several applications, the impulse responses (IRs) of the underlying systems to be identified are often sparse or compressible (quasi-sparse), i.e., only a small percentage of the IR components have a significant magnitude while the rest are zero or small. Examples include network and acoustic echo cancellation [5], [6], [7], hands-free mobile telephony [8], acoustic feedback reduction in hearing aids [9], and underwater acoustic communications [10], to mention a few. Designing adaptive filters that can exploit the sparse structure of the underlying system response for performance improvement over the conventional approaches, e.g., the least mean square(LMS)and normalized LMS (NLMS), is of great interest and importance especially for acoustic and speech applications. In this paper, we utilize the iterative reweighted ℓ₂ and ℓ₁ algorithms that have been developed in the sparse signal recovery (SSR) area to minimize diversity measures as a starting point [11]. Incorporating an affine scaling transformation (AST) [12], [13] into the algorithm design process, a new methodology for developing a large class of adaptive filters is presented that leverage the sparse nature of the system responses.

A. Related Work

An early and influential work on identifying sparse IRs is the proportionate NLMS (PNLMS) algorithm proposed by Duttweiler [5] for acoustic echo cancellation. The main idea behind the approach is to update each filter coefficient using a step size proportional to the magnitude of the estimated coefficient, as opposed to the NLMS which assigns a uniform adaptation gain to all coefficients. Consequently, when the system is sparse, larger coefficients are adapted using relatively large steps compared to the smaller ones with PNLMS. The overall convergence can thus be sped up by focusing on adjusting the significant coefficients, rather then treating them all equally as in NLMS. Although PNLMS was developed in an intuitive way, i.e., the equations used to calculate the proportionate factors that realize step-size control were not based on any optimization criterion but were based on good heuristics, it has motivated many new proportionate variants for sparse system identification. The proportionate class of algorithms represent an important subset among sparsity-aware adaptive filters.

The recent progress on SSR has led to a number of computational algorithms, e.g., [14], [15], [16], [17], [18], [19], among others. This makes available a plethora of approaches for systematically designing sparsity-aware adaptive algorithms that are a natural complement to the SSR batch estimation techniques. As a result, different from the proportionate approaches, another class of sparse adaptive filters have been introduced by integrating sparsity-inducing regularization to accelerate the convergence of near-zero coefficients in sparse systems. This has led to several sparse adaptive filtering algorithms and even obtaining a general framework of adaptive filters that incorporate sparsity. SSR-motivated adaptive algorithms represent another important class of sparsity-aware adaptive filters. We now discuss a few works on the proportionate class followed by the SSR variants.

Several variants of the PNLMS have been proposed and [20] provides a good summary. Examples include the improved PNLMS (IPNLMS) [21], IPNLMS based on the ℓ₀ “norm” (IPNLMS-ℓ₀) [22], etc. In [23], Martin et al. utilized a natural gradient framework to deduce adaptive filters having similar features to the PNLMS that can exploit the sparse structure. Rao and Song [24] and Jin [25] proposed a framework for promoting sparsity in adaptive filters based on minimizing diversity measures. The framework is quite general and encompasses a broad range of adaptive filtering algorithms having similarity with the PNLMS algorithm. Benesty et al. [26] derived the PNLMS from a different perspective by using a basis pursuit [17] formulation. Following them, Liu and Grant [27] proposed a general framework of proportionate adaptive filters based on convex optimization and sparseness measures, which covers many traditional proportionate algorithms.

Several SSR-inspired algorithms have been introduced by integrating a sparsity-inducing regularizer into the original LMS objective function to accelerate the convergence of near-zero coefficients in sparse systems. For example, Chen et al. [28] proposed the zero-attracting LMS (ZA-LMS) derived by including the ℓ₁ norm penalty in the objective function. They also proposed the reweighted ZA-LMS (RZA-LMS) obtained by incorporating the log-sum penalty. Later, using the approximation of the ℓ₀ “norm” as a sparsity-inducing term, Gu et al. [29] proposed the ℓ₀-LMS that is capable of better estimating sparse systems. In [30], the authors utilized the p-norm-like regularization function and considered the quantitative learning of the regularizer. Another work in this area is the new reweighted ℓ₁ norm penalized LMS algorithm proposed and studied in [31] for improving the ZA-LMS and RZA-LMS.

Recently, some works have considered both proportionate adaptation and sparsity-inducing regularization together. For example, [32] presents a modified PNLMS update equation with a zero attractor as in the ZA-LMS for all the taps, derived by introducing a carefully constructed ℓ₁ norm penalty in the PNLMS objective function. Other than the ℓ₁ norm, [33], [34] apply the ℓ_p norm penalty to the PNLMS cost function and derive ℓ_p-norm-constrained proportionate algorithms for improved broadband multipath channel estimation and active noise control. [35] encompasses a number of sparsity-aware adaptive filtering algorithms that go beyond the LMS and NLMS, including proportionate and regularization-based approaches. [36], [37] provide a general framework to combine proportionate updates and sparsity-inducing regularizers. In Section III, we will derive algorithms whose update rules also consist of a proportionate term and another term due to regularization. However, our derivation follows a very different path from these previous works.

B. Contributions of the Paper

In this paper, inspired by the conceptual similarity with SSR,¹ our goal is to add to this interesting body of work on adaptive filtering and sparsity. The contributions of the paper are the following:

1. The sparsity-aware adaptive filters developed lie at the intersection of the proportionate-class and SSR-inspired adaptive algorithms and provides an interesting bridge. We start with the rigorous formulation of a regularization framework and derive sparsity-aware adaptive filtering algorithms. Specifically, based on diversity measure minimization in SSR, we adopt the reweighted ℓ₂ and ℓ₁ frameworks [11]. Using an AST methodology [12], [13] in the algorithm development, our work naturally results in a general class of proportionate algorithms. This is an unique feature of this work. The combination of AST and the reweighting frameworks contribute to the main innovation of our adaptive algorithm development framework.

2. Under the proposed framework, we introduce Sparsity-promoting LMS (SLMS) and Sparsity-promoting NLMS (SNLMS) algorithms that promote sparsity without biasing the adaptation process by adopting λ = 0, where λ is the regularization coefficient associated with the sparsity penalty. This is not possible for the class of algorithms currently in existence that utilize a sparsity-inducing regularization penalty.² The SLMS and SNLMS can be viewed as realizing proportionate adaptation like the PNLMS class of algorithms [5]. Therefore, utilizing λ = 0 in our framework paves the way for explaining why proportionate algorithms are useful in circumstances where the channels to be estimated admit certain sparsity. This provides theoretical support to existing proportionate algorithms which were mostly developed based on good heuristics rather than on optimization criteria. More importantly, unlike most of them that design the proportionate factors heuristically, our SSR-motivated framework leads to a more systematic way of designing the factors, and permits incorporation of a broad class of diversity measures that have proven to be effective for SSR in our algorithms.

3. Compared to existing derivations of proportionate-type algorithms, using the proposed framework we derive the algorithms in a more natural way in terms of incorporating sparsity using a regularization framework. For instance, in some of the existing works modified objective functions were proposed that impose sparsity on the “change” of the filter rather than on the filter itself, e.g., [24], [25], [26], [27], [32]. However, since the assumption is that the filter itself is sparse, the motivation for enforcing sparsity on the “change” rather than on the filter is not clear and at best indirect. In contrast, we work with the general mean squared error (MSE) criterion in which sparsity can be directly imposed via regularization on the filter.

4. Steady-state analysis of the proposed algorithms is conducted and simulation results are provided to demonstrate the effectiveness of the proposed algorithms compared to existing approaches. Examples with the acoustic channel response measured on a real-world hearing aid device using speech input are also presented.

A portion of this work has been previously published as a conference paper [39]. To a fuller extent, the current paper provides a general framework for incorporating flexible diversity measures into sparse adaptive filters, together with theoretical discussion and supporting simulated results.

C. Organization of the Paper

The rest of the paper is organized as follows. Section II provides background on adaptive filters. Section III derives adaptive filters that incorporate sparsity based on diversity measure minimization, by utilizing the reweighted ℓ₂ and ℓ₁ frameworks together with the AST methodology. Section IV introduces the SLMS and SNLMS that adopt λ = 0. Section V discusses the steady-state analysis. Section VI presents simulation results. Section VII concludes the paper.

D. Notation

Let ${ℝ}^M$ denote the M-dimensional real Euclidean space. ${ℝ}^{N\times M}$ denotes the set of N ×M real matrices. ${ℝ}_{+}$ denotes the set of non-negative real numbers. Superscript^T denotes the transpose of a vector or matrix. E[·] denotes the mathematical expectation. Vectors and matrices are denoted by boldface lowercase and uppercase letters, respectively. Scalars are denoted by italics. For a vector $x={\left[x_0,x_1,\dots ,x_{M-1}\right]}^T\in {ℝ}^M$, the ℓ_p norm³ (where p > 0) is defined as: $\Vert x{\Vert }_p={\left({{\sum }_{i=0}^{M-1}\left|x_i\right|}^p\right)}^{1/p}$. The ℓ₀ “norm”⁴ ‖x‖₀ is defined as the number of nonzero entries of the vector x. We use diag{x_i} to denote the M-by-M diagonal matrix whose i-th diagonal element is x_i. We use sgn(·) to denote the component-wise sign function that takes the sign of the entries of its argument. ∇_x denotes the gradient operator⁵ w.r.t. x. d denotes the differential operator. tr(X) denotes the trace of a square matrix $X\in {ℝ}^{M\times M}$. I is the identity matrix. 1 denotes the vector of all ones. 0 denotes the vector of all zeros. We use $\mathcal{N}(\cdot ,\cdot )$ to denote the normal distribution with the first and second arguments being the mean and (co)variance, respectively.

II. Background on Adaptive Filtering and SSR

We provide some preliminaries of adaptive filters in the context of system identification and present several examples of existing sparsity-aware adaptive filtering algorithms. We also discuss the iterative reweighting frameworks in SSR for developing our adaptive algorithms in later sections.

A. Adaptive Filtering for System Identification

Let h_n = [h_0,n, h₁,_n,…, h_{M − 1,n}]^T denote the adaptive filter of length M at discrete time instant n. Assume the IR of the underlying system is $h^o={\left[h_0^o,h_1^o,\dots ,h_{M-1}^o\right]}^T$, and the model for the observed or desired signal is $d_n=u_n^T{h}^o+v_n$. u_n = [u_n, u_n−1, …, u_n−M+1]^T is the vector containing the M most recent samples of the input signal u_n and v_n is an additive noise signal. The output of the adaptive filter $u_n^T{h}_n$ is subtracted from d_n to obtain the error signal $e_n=d_n-u_n^T{h}_n$ The goal in general is to sequentially update the coefficients of h_n upon the arrival of a new data pair (u_n, d_n), such that eventually h_n = h^o; i.e., to identify the unknown system.

The most classic adaptive filtering algorithms are the LMS and NLMS [1], [2], [3], which can be derived based on minimizing the MSE objective function:

$$\underset{h}{\text{min}}J(h)\triangleq \text{E}\left[e_n^2\right]=\text{E}\left[{\left(d_n-u_n^{T}h\right)}^2\right].$$ (1)

The method of steepest descent (gradient descent) for optimizing (1) suggests the following recursion for updating the filter coefficients [2]:

$$h_{n+1}=h_n-\frac{\mu }{2}{\nabla }_{h}J\left(h_n\right),$$ (2)

where μ > 0 is the step size. To develop adaptive algorithms, in practice the gradient ∇_h J(h_n) = −2E[u_ne_n] is replaced by the instantaneous estimate −2u_n e_n, i.e., the stochastic gradient [2], [3], leading to the standard LMS algorithm:

$$h_{n+1}=h_n+\mu u_n{e}_n.$$ (3)

The normalized version of (3), i.e., the NLMS algorithm, can be derived based on the principle of minimum disturbance [2]. Alternatively, it can be obtained by performing exact line search for the optimal step size for each iteration [40]. Then, practically, a scaling factor $\tilde{\mu }>0$ is introduced to exercise control over the adaptation⁶ and a small regularization constant δ > 0 is also employed to avoid division by zero [2], leading to the standard NLMS algorithm:

$$h_{n+1}=h_n+\frac{\tilde{\mu }{u}_n{e}_n}{u_n^T{u}_n+\delta }.$$ (4)

1). Sparsity-Aware Adaptive Filtering Algorithms:

When the underlying system response is sparse, a class of algorithms realizing proportionate adaptation [20] are able to take advantage of the structural sparsity. A typical update rule with proportionate adaptation is:

$$h_{n+1}=h_n+\mu P_n{u}_n{e}_n,$$ (5)

or the normalized version:

$$h_{n+1}=h_n+\frac{\tilde{\mu }{P}_n{u}_n{e}_n}{u_n^T{P}_n{u}_n+\delta },$$ (6)

Where

$$P_n=\text{diag}\left\{p_{0,n},p_{1,n},\dots ,p_{M-1,n}\right\}$$ (7)

is an M-by-M diagonal matrix assigning different weights to the step sizes for different filter taps, referred to as the proportionate matrix. It redistributes the adaptation gains among all coefficients and emphasizes the large ones in order to speed up their convergence. Typically, at the n-th iteration the diagonal entries are computed as:

$$p_{i,n}=\frac{{\gamma }_{i,n}}{{\sum }_{j=0}^{M-1}{\gamma }_{j,n}},$$ (8)

∀i = 0, 1,…, M − 1, where γ_i,n is algorithm-dependent and examples of such algorithms include the PNLMS [5], IPNLMS [21], IPNLMS-ℓ₀ [22], etc. In general, if the estimated filter coefficients h_i,n are sparse, the resulting γ_i,n (thus p_i,n) will also tend to be sparsely distributed (with positive values).

Another class of algorithms, inspired by developments in the SSR area, take sparsity into account using a regularization-based approach, e.g., [28], [29], [30], [31]. The algorithms are obtained by adding a sparsity-inducing term G(h) to the MSE objective function:

$$\underset{h}{\text{min}}{J}^G(h)\triangleq J(h)+\lambda G(h),$$ (9)

Where λ > 0 is the regularization coefficient. By simply applying (stochastic) gradient descent⁷ on (9):

$$h_{n+1}=h_n+\mu u_n{e}_n-\frac{\mu \lambda }{2}{\nabla }_{h}G\left(h_n\right),$$ (10)

various algorithms can be obtained with different sparsity-inducing functions G(·). Examples include the ZA-LMS [28], RZA-LMS [28], and ℓ₀-LMS [29], [41].

B. Iterative Reweighting Algorithms in SSR

The optimization of (9) is actually an SSR problem. The sparsity regularization term G(·) represents the general diversity measure that when minimized encourages sparsity in its argument. A separable function of the form $G(h)={\sum }_{i=0}^{M-1}g\left(h_i\right)$ is commonly used, where g(·) has the following properties [11]:

• Property 1: g(z) is symmetric, i.e., g(z) = g(−z) = g(|z|);

• Property 2: g(|z|) is monotonically increasing with |z|;

• Property 3: g(0) is finite;

• Property 4: g(z) is concave in |z| or z².

Any function that holds the above properties is a candidate for effective SSR algorithm development.

The concave nature of the regularization penalty poses challenges to the diversity measure minimization problem. The iterative reweighted ℓ₂ [15], [18] and ℓ₁ [19] methods are popular batch estimation algorithms for solving such minimization problems in SSR. By introducing a weighted ℓ₂ or ℓ₁ norm term as an upper bound for the diversity measure term in each iteration, they form and solve a convex optimization problem at each step to approach the optimal solution [11]. Specifically, instead of (9), at iteration n the reweighted ℓ₂ framework suggests solving:

$$\underset{h}{\text{min}}{J}_n^{{\mathcal{l}}_2}(h)\triangleq J(h)+\lambda {\Vert W_n^{-1}h\Vert }_2^2,$$ (11)

and the reweighted ℓ₁ framework suggests solving:

$$\underset{h}{\text{min}}{J}_n^{{\mathcal{l}}_1}(h)\triangleq J(h)+\lambda {\Vert W_n^{-1}h\Vert }_1,$$ (12)

where W_n = diag{w_i,n} is positive definite⁸ and each w_i,n is computed based on the current estimate h_i,n, depending on which framework (reweighted ℓ₂ or ℓ₁) and diversity measure (choice of G(·)) are used.

To elaborate, for using the reweighted ℓ₂ (11), the diversity measure function g(z) has to be concave in z² for Property 4; i.e., it satisfies g(z) = f(z²), where f(z) is concave for $z\in {ℝ}_{+}$. Based on [11], we have w_i,n given as:

$$w_{i,n}={\left({\frac{\text{d}f\left(z\right)}{\text{d}z}|}_{z=h_{i,n}^2}\right)}^{-\frac{1}{2}}.$$ (13)

For using the reweighted ℓ₁ (12), g(z) has to be concave in |z| for Property 4; i.e., it satisfies g(z) = f(|z|), where f(z) is concave for $z\in {ℝ}_{+}$. In this case, w_i,n is given as:

$$w_{i,n}={\left({\frac{\text{d}f\left(z\right)}{\text{d}z}|}_{z=|h_{i,n}|}\right)}^{-1},$$ (14)

To utilize the reweighting frameworks, we first choose an appropriate diversity measure G(h) and then use (13) or (14) to obtain the corresponding update form of W_n. Several examples will be presented is Section IV-B.

III. Proposed Framework for Incorporating Sparsity in Adaptive Filters

Our framework for developing sparse adaptive filters is based on (9). However, we will be deriving algorithms in a different way rather than using a simple gradient descent as is typically done in existing regularization-based adaptive filtering approaches, e.g., (10). Our novel derivation consists of two stages: i) adapting the iterative reweighting frameworks [11] popular in SSR to the adaptive filtering setting, followed by ii) the AST strategy [12], [13] from the optimization literature to obtain new adaptive filtering algorithms.

A. Reweighting Methods for Adaptive Filtering

The reweighting methods introduced in Section II-B actually belong to the more general class of majorization-minimization (MM) algorithms [42]. In each iteration n, the weighted ℓ₂ or ℓ₁ norm term majorizes G(h) at the current estimate h_n, thereby providing a surrogate function (or majorizer) $J_n^{{\mathcal{l}}_2}(h)$ or $J_n^{{\mathcal{l}}_1}(h)$ for the regularized objective function J^G(h). Sequentially minimizing the surrogate functions allows the algorithm to produce more focal estimates as optimization progresses. Hopefully when the number of iterations is large enough, the optimal solution can be well approached or even achieved [11].

In SSR, it is typical that the surrogate function is exactly minimized in each iteration n. For the purpose of developing adaptive filtering algorithms, here we consider performing only one step of gradient descent per iteration. In this sense, it corresponds to the generalized MM [43] where one does not need to minimize the majorizer but only to assure that it decreases in every iteration. Indeed, the MM viewpoint provides an interesting observation of using gradient descent for optimizing (9) and the reweighting formulations (11) and (12), as stated in the following proposition:

Proposition 3.1:

For any point h_n at which G(h) is differentiable, the gradient vector of the surrogate function $J_n^{{\mathcal{l}}_2}(h)$ or $J_n^{{\mathcal{l}}_1}(h)$ evaluated at h_n coincides with that of the regularized objective function J^G(h), i.e., ${\nabla }_h{J}_n^{{\mathcal{l}}_2}\left(h_n\right)={\nabla }_h{J}^G\left(h_n\right)$ and ${\nabla }_h{J}_n^{{\mathcal{l}}_1}\left(h_n\right)={\nabla }_h{J}^G\left(h_n\right)$.

Proof:

Since the surrogate function majorizes J^G(h) at h_n, the tangent plane (supporting hyperplane) of the majorizer coincides with that of J^G(h) at h_n. Consequently, the gradient vectors are the same at h_n.

The observation in Proposition 3.1 implies that, if the gradient descent (when using a fixed step size) is utilized for optimization,⁹ then adopting the reweighting frameworks (11) and (12) will be equivalent to directly working on (9) and lead to the existing regularization-based algorithms such as the ZA-LMS. In the following, we introduce the AST strategy naturally suggested by the reweighting frameworks, leading to new algorithms markedly different from those obtained by directly optimizing (9) with gradient descent.

B. AST-Based Adaptive Filtering Algorithms

The reweighting frameworks (11) and (12) naturally suggest the following reparameterization in terms of the (affinely) scaled variable q:

$$q\triangleq W_n^{-1}h.$$ (15)

This step can be interpreted as the AST commonly employed by the interior point approach to solving linear and nonlinear programming problems [12], where W_n is used as the scaling matrix. It is pre-calculated and treated as a given matrix at iteration n to perform a change of coordinates (variables) [44] from h to q, acting as a scaling technique in gradient descent methods [45]. In the optimization literature, AST-based methods transform the original problem into an equivalent one, favorably positioning the current point at the center of the feasible region for expediting the optimization process [13]. While we do not claim this argument is rigorous in the context of adaptive filtering, where the convergence behavior is hard to characterize due to the nonlinear nature of the update equations and the long term dependency on the data, the numerical results appear to support this observation of enjoying the benefits of AST for convergence speedup.

Now we apply (15) to reparameterize the objective functions $J_n^{{\mathcal{l}}_2}(h)$ and $J_n^{{\mathcal{l}}_1}(h)$ and perform minimization w.r.t. q, that is:

$$\underset{q}{\text{min}}{\tilde{J}}_n^{{\mathcal{l}}_2}(q)\triangleq J_n^{{\mathcal{l}}_2}\left(W_{n}q\right)=J\left(W_{n}q\right)+\lambda {\Vert q\Vert }_2^2$$ (16)

and

$$\underset{q}{\text{min}}{\tilde{J}}_n^{{\mathcal{l}}_1}(q)\triangleq J_n^{{\mathcal{l}}_1}\left(W_{n}q\right)=J\left(W_{n}q\right)+\lambda {\Vert q\Vert }_1,$$ (17)

for the reweighted ℓ₂ and ℓ₁cases, respectively. The overall update process conceptually can be summarized as follows: i) given an h compute W_n followed by q. ii) Update q using a gradient descent algorithm. iii) Use this new q to obtain the updated h. iv) Repeat Steps i)–iii) till convergence.

More formally, to proceed with gradient-based updates, following [40] we define the a posteriori AST variable at time n:

$$q_{n\mid n}\triangleq W_n^{-1}{h}_n$$ (18)

and the a priori AST variable at time n:

$$q_{n+1\mid n}\triangleq W_n^{-1}{h}_{n+1}.$$ (19)

The recursive update by using gradient descent in the q domain can be formulated as:

$$q_{n+1\mid n}=q_{n\mid n}-\frac{\mu }{2}{\nabla }_q{\tilde{J}}_n^{{\mathcal{l}}_2}\left(q_{n\mid n}\right)$$ (20)

and

$$q_{n+1\mid n}=q_{n\mid n}-\frac{\mu }{2}{\nabla }_q{\tilde{J}}_n^{{\mathcal{l}}_1}\left(q_{n\mid n}\right),$$ (21)

for optimizing (16) and (17), respectively.

Using the chain rule¹⁰ and the AST relationships (15) and (18), we can write (20) and (21) respectively as:

$$q_{n+1\mid n}=q_{n\mid n}-\frac{\mu }{2}{W}_n{\nabla }_h{J}_n^{{\mathcal{l}}_2}\left(h_n\right)$$ (22)

and

$$q_{n+1\mid n}=q_{n\mid n}-\frac{\mu }{2}{W}_n{\nabla }_h{J}_n^{{\mathcal{l}}_1}\left(h_n\right).$$ (23)

Premultiplying W_n on both sides of (22) and (23) and noting the relationships (18) and (19), we transform the q domain updates (22) and (23) back to the h domain respectively as:

$$h_{n+1}=h_n-\frac{\mu }{2}{W}_n^2{\nabla }_h{J}_n^{{\mathcal{l}}_2}\left(h_n\right)$$ (24)

and

$$h_{n+1}=h_n-\frac{\mu }{2}{W}_n^2{\nabla }_h{J}_n^{{\mathcal{l}}_1}\left(h_n\right).$$ (25)

By Proposition 3.1, we can replace ${\nabla }_h{J}_n^{{\mathcal{l}}_2}\left(h_n\right)$ and ${\nabla }_h{J}_n^{{\mathcal{l}}_1}\left(h_n\right)$ with ∇_h J^G(h_n). Thus, (24) and (25) can both be written as:

$$h_{n+1}=h_n-\frac{\mu }{2}{W}_n^2{\nabla }_h{J}^G\left(h_n\right).$$ (26)

Note that based on the aforementioned update process (i)-(iv), we can in fact directly apply (15) to reparameterize J^G(h) and obtain (26) without going through the reweighting formulation, as long as the scaling matrix W_n is specified. In this sense, the reweighing methods suggest a suitable W_n that eventually becomes a diagonal weighting matrix $W_n^2$ on the gradient ∇_hJ^G(h_n) in the update rule. Hopefully, it alters the ordinary descent direction in such a way that leads to convergence improvement. We should also emphasize that the scaling matrix W_n suggested by (24) and (25) will in general be different for a given same G(h), despite the fact that both can be written as (26).

In practice, the following update rule is suggested over (26) for avoiding instability and slow convergence issues:

$$h_{n+1}=h_n-\frac{\mu }{2}{S}_n{\nabla }_h{J}^G\left(h_n\right),$$ (27)

where

$$S_n=\frac{W_n^2}{\frac{1}{M}\text{tr}\left(W_n^2\right)},$$ (28)

referred to as the sparsity-promoting matrix, is the normalized version of $W_n^2$. As a fixed step size μ is used, performing normalization of the weighting matrix compensates for any arbitrary scaling inherent in $W_n^2$ that might cause instability (scaling too large) or slow convergence (scaling too small). Note that by (28) we always have tr(S_n) = M, same as in the ordinary gradient descent (non-AST) case which essentially has S_n = I whose trace is also M.

Finally, to obtain the adaptive algorithm, we follow the standard procedure of replacing ∇_hJ^G(h_n) = −2E[u_n e_n] + λ∇_hG(h_n) in (27) with its instantaneous estimate −2u_ne_n + λ∇_hG(h_n), leading to:

$$\boxed{h_{n+1}=h_n+\mu S_n{u}_n{e}_n-\frac{\mu \lambda }{2}{S}_n{\nabla }_{h}G\left(h_n\right).}$$ (29)

We see that there is a term with a diagonal weighting S_n on the LMS update vector u_ne_n, similar to that in proportionate algorithms (5) and (6). We also see another term weighted by λ which is due to the introduction of the regularizer, like that of (10). Therefore, the AST framework leads to a more general algorithm comprised of proportionate adaptation and sparsity-inducing regularization. We thus refer to (29) as the generalized sparse LMS algorithm.

C. Discussions

It may seem at the first glance that applying the reweighting techniques to (9) straightforwardly leads to our algorithm. We stress that it is not true. If the AST (15) was not considered, adopting the reweighting schemes would still end up with an update rule like (10) according to Proposition 3.1, rather than the proposed (29). It is also worth mentioning that there is considerable difference between the proposed algorithm (29) and existing SSR algorithms based on (11) and (12) – the conventional SSR techniques are batch estimation methods for recovering the underlying sparse representation, while the proposed algorithm is specifically tailored for the adaptive filtering scenario. That being said, as gradient descent is adopted for optimization, we actually perform a gradual update of the filter coefficients in each iteration n, rather than looking for an exact minimizer of the surrogate function as is typically pursued in SSR. This enables the algorithm to track temporal variations and environmental changes. Certainly, considering the gradient noise in real scenarios, it may post the issue of whether the algorithm is convergent. However, even the standard LMS and NLMS that are based on gradual updates, work well in many practical situations with gradient noise. In Section VI, experimental results will demonstrate that the proposed algorithm, like the LMS and NLMS, also behaves well when certain level of environmental noise is present.

Finally, the following theorem establishes the convergence of the q domain recursions (20) and (21) and their relationships to (9) to shed light on the convergence of the adaptive algorithm (29) developed based on them:

Theorem 3.1:

For the objective function J^G(h) in (9) with the general diversity measure G(h) satisfying Properties 1–4 in Section II-B,¹¹ there exists a step size sequence ${\left\{{\mu }_n\right\}}_{n=0}^{\infty }$ such that each of the update recursions (20) and (21) monotonically converges to a local minimum (or saddle point) of (9) under a wide-sense stationary (WSS) environment, i.e., u_n and d_n are jointly WSS.

Proof:

See Appendix A.

IV. Sparsity-Promoting Algorithms Adopting λ = 0

An interesting situation arises when we consider the limiting case of λ → 0⁺ for the proposed framework. By setting λ = 0 in (29), we see the λ-weighted term due to regularization vanishes, leading to a simpler equation:

$$\boxed{h_{n+1}=h_n+\mu S_n{u}_n{e}_n.}$$ (30)

The main feature of (30) is that it is able to promote sparsity of the system (through S_n) if it already exists while not strictly enforcing it (as λ = 0). This shall become clearer in later discussions. We refer to the algorithm (30) as the Sparsity-promoting LMS (SLMS).

The normalized version of (30) can also be developed by performing exact line search for the optimal step size at iteration n just like that when deriving the NLMS:

$${\mu }_n=\underset{\mu }{\text{arg}\text{min}}{\left(d_n-u_n^T\left(h_n+\mu S_n{u}_n{e}_n\right)\right)}^2=\frac{1}{u_n^T{S}_n{u}_n}.$$ (31)

Similar to the NLMS, we introduce $\tilde{\mu }>0$ to exercise control over the adaptation and δ > 0 to avoid division by zero, resulting in:

$$\boxed{h_{n+1}=h_n+\frac{\tilde{\mu }{S}_n{u}_n{e}_n}{u_n^T{S}_n{u}_n+\delta }.}$$ (32)

We refer to the algorithm (32) as the Sparsity-promoting NLMS (SNLMS).

An obvious benefit of adopting λ = 0 is that the computation for the term due to regularization is no longer needed, and we do not have to tweak this coefficient anymore (which is typically not a trivial task in practice). Still, the SLMS and SNLMS have the ability to leverage sparsity owing to the diagonal weighting S_n, which is similar to the proportionate matrix P_n in (5) and (6). Again, this is made possible due to the use of the AST (15), wherein the gradient descent update is done in the q variable rather than in the original h domain. Otherwise, we will end up with algorithms like (10) that will reduce to the ordinary LMS/NLMS when using λ = 0.

The SLMS and SNLMS can in fact be viewed as a broader class of proportionate algorithms. Actually, with certain choices of diversity measures and corresponding parameters, we can have the PNLMS (approximately) as a special case. For example, as we will see in Section IV-B, using p = 1 in (34) for W_n, the sparsity-promoting matrix S_n approximates the proportionate matrix P_n of PNLMS. Indeed, one of the main advantages of the SLMS and SNLMS is their ability to incorporate flexible diversity measures. It allows the algorithms to fit the sparsity level of the system response by optimizing corresponding sparsity control parameters in a more informed manner due to the underlying connections to SSR. Furthermore, the derivations provide theoretical support to the class of proportionate algorithms that were mostly motivated based on heuristics, explaining why they are useful in practical system identification tasks with sparse channels, e.g., in acoustic echo/feedback cancellation, from an SSR viewpoint.

A. Interpretation of λ = 0 from Optimization Perspective

We further discuss the interpretation of using λ = 0 in our framework from the optimization perspective. Recall that the AST reparameterization (15) results in the optimization problems (16) and (17). Setting λ = 0 leads both to:

$$\underset{q}{\text{min}}J\left(W_{n}q\right).$$ (33)

This actually applies a change of coordinates to the unregularized problem (1) via (15). Since W_n is invertible, the problem of finding the h that minimizes J(h) is equivalent to finding the q which minimizes J(W_nq). Therefore, the advantage of solving (33) is that the solution is guaranteed to also be a solution of (1), which is not true for (9) with λ > 0. Thus, the optimization is unbiased while promoting sparsity – it is able to take advantage of sparsity whereas avoiding any bias incurred by the introduction of the sparsity regularizer. As noted in [45], the performance of gradient-based methods is dependent on the parameterization – a new choice may substantially alter convergence characteristics. Introducing variable scalings may speed up convergence by altering the descent direction toward the optimum. In our case, solving (33) with appropriately selected W_n can expedite the adaptation procedure toward the optimum of (1).

This observation can also be illustrated by looking at (9) which indicates a trade-off between estimation quality, as reflected in the MSE objective function, and solution sparsity as controlled by λ. In the limiting case of λ → 0⁺, the objective function exerts diminishing impact on enforcing sparsity on the solution, meaning that eventually no sparse solution is favored over other possible solutions. To elaborate, with λ = 0 and under a WSS environment, all the algorithms derived from (9) minimize the MSE and converge toward the Wiener-Hopf solution. However, not surprisingly, the paths they take are different and depend on how the iterations are developed. If the Wiener-Hopf solution is sparse, then all will converge toward the same sparse solution asymptotically. Interestingly, the SLMS and SNLMS, because of their proportionate nature similar to the PNLMS-type algorithms, can take advantage of the sparsity and are capable of speeding up convergence without compromising estimation quality should sparsity be present. This observation will later be supported by experimental results in Section VI-B.

B. Example Diversity Measures and Corresponding W_n

To illustrate the flexibility of the proposed framework, we provide example algorithms instantiated with popular diversity measures that have proved effective in SSR.

Consider the p-norm-like diversity measure with g(h_i) = |h_i|^p, 0 < p ≤ 2 for the reweighted ℓ₂ framework [15], [12]. Using (13) leads to the update form of W_n:

$$w_{i,n}={\left(\frac{2}{p}{\left(|h_{i,n}|+c\right)}^{2-p}\right)}^{\frac{1}{2}}.$$ (34)

Note that we have empirically added a small regularization constant c > 0 for avoiding algorithm stagnation and instability,¹² which also ensures the positive definiteness of W_n [39]. The parameter p ∈ (0, 2] in (34) is responsible for controlling the sparsity degree, as the p-norm-like diversity measure is associated with super-Gaussian prior distributions. In general, a smaller p corresponds to a heavier-tailed distribution, encouraging stronger sparsity in the parameters. It is worth noting that using p → 1 in (34) results in a proportionate factor close to that of the PNLMS. On the other hand, letting p = 2 recovers the standard LMS/NLMS.

The p-norm-like diversity measure can also be adopted in the reweighted ℓ₁ framework if 0 < p ≤ 1. Applying (14), we obtain the update form of W_n in this case:

$$w_{i,n}=\frac{1}{p}{\left(|h_{i,n}|+c\right)}^{1-p}.$$ (35)

Again, a small constant c > 0 is added. The sparsity control parameter of (35) is now p ∈ (0, 1]. In this case, using p → 0.5 in (35) results in a proportionate factor close to that of the PNLMS, whereas letting p = 1 recovers the standard LMS/NLMS.

We can also consider the log-sum penalty with $g\left(h_i\right)=\text{log}\left(h_i^2+ϵ\right)$, ϵ > 0 for the reweighted ℓ₂ framework [18]. The function is readily amenable to the use of (13) to obtain the update form of W_n as:

$$w_{i,n}={\left(h_{i,n}^2+ϵ\right)}^{\frac{1}{2}}.$$ (36)

Or consider the log-sum penalty with g(h_i) = log(|h_i| + ϵ), ϵ > 0 for the reweighted ℓ₁framework [19]. Using (14), the update form of W_n becomes:

$$w_{i,n}=|h_{i,n}|+ϵ.$$ (37)

The sparsity control parameter is ϵ > 0 for the two log-sum penalty cases. From (36) and (37) we can see that ϵ controls how much proportionate adaptation is encouraged: as ϵ becomes smaller, the term $h_{i,n}^2$ or |h_i,n| becomes more dominant. Consequently, they exhibit a stronger proportionate adaptation characteristic. On the contrary, as ϵ becomes larger, the influence of $h_{i,n}^2$ or |h_i,n| reduces. Thus, the algorithm will approach the standard LMS/NLMS when $ϵ\gg h_{i,n}^2$ or ϵ ≫ |h_i,n|. In practice, one can start from a large ϵ and reduce it to find a suitable value.

More example functions can be found in [46], [27], including g(h_i) = arctan(|h_i|/ϵ), ϵ > 0 also suggested in [19], which works for both the reweighted ℓ₂ and ℓ₁ frameworks. Note that different diversity measures can result in different computational complexity for calculating W_n. Notably, for example, the p-norm-like function resulting in (34) or (35) might incur extra computation for calculating the quantity to the power 2 − p or 1 − p for some p value (e.g., non-integer power).

Algorithm 1 summarizes the proposed SLMS and SNLMS. A MATLAB implementation of the algorithms is available at https://github.com/chinghualee/SLMS_SNLMS.

C. Comparison to Existing Work on PNLMS-Type Algorithms

Note that in IPNLMS [21] and IPNLMS-ℓ₀ [22] there is also a parameter for fitting the sparsity degree, which was heuristically introduced to weight between proportionate and non-proportionate updates. However, this empirical parameter does not reflect the sparsity level of the underlying system directly. In our algorithms, we have the sparsity control parameters that play a similar role for fitting different sparsity levels. However, based on diversity measures in SSR, they have direct connections to the system sparsity, thereby offering a more intuitive parameter selection procedure. Our algorithms thus have the advantages of enjoying theoretical support and leveraging sparsity more straightforwardly.

In terms of algorithm derivations, PNLMS-type algorithms were mostly developed from a constrained optimization problem following the principle of minimal disturbance, e.g., [24], [25], [26], [27], [32], in which modified objective functions have been proposed that impose sparsity on the “change” of the filter rather than on the filter itself. For example, [24], [25], [32] considered enforcing sparsity on the difference between the current and updated filters; [26], [27] imposed sparsity on the so-called correctness component as defined in [26] which also represents the change in the filter coefficients. However, since the assumption is that the filter itself is sparse rather the difference between successive updates, the motivation to enforce sparsity on the “change” of the filter is less clear. Sparsity, in turn, does not seem to fit in straightforwardly under the commonly adopted constrained optimization framework. In contrast, we adopt the general MSE criterion in which filter sparsity can be directly imposed via regularization, which is more direct and makes intuitive sense.

V. Steady-State Performance Analysis

The signal model of system identification described in Section II-A is employed for performance analysis. We further assume the noise v_n is i.i.d. according to $\mathcal{N}\left(0,{\sigma }_v^2\right)$. We also introduce several other assumptions useful for simplifying analysis. Although these assumptions may seem restrictive, they make meaningful analysis possible without significant loss of insight and are also commonly adopted in the literature. We shall later see that these assumptions lead to theoretical results that are supported by experiments.

Assumption 1:

The input data vector u_n is independent of u_k for n ≠ k. Furthermore, u_n is independent of v_k for all n and k. In practice and from past experience in adaptive filters, this assumption simplifies the analysis and does lead to useful insights [2], [3], despite the fact that it does not in general hold true.

Assumption 2:

The input data vector obeys $u_n~\mathcal{N}(0,R)$ for all n. This technical assumption facilitates the analysis by taking advantage of the useful results on Gaussian random variables [4].

Assumption3:

At steady-state, the diagonal matrix W_n in the update equations can be view as a fixed matrix. As suggested in [5], [23], when the system is at steady-state and when the step size is sufficiently small, the coefficients converge in both mean and mean squared senses. Thus, replacing W_n by a fixed matrix becomes reasonable and convenient.

For convenience we shall consider the algorithm of the following form for performance analysis:

$$h_{n+1}=h_n+\mu S{u}_n{e}_n,$$ (38)

where S = diag{s_i} with s_i > 0, ∀i = 0, 1, …, M − 1.

For a fixed underlying system h^o, define the steady-state excess MSE [4]:

$$J_{\text{ex}}\triangleq \underset{n\to \infty }{\text{lim}}\text{E}\left[{\left(u_n^T\left(h^o-h_n\right)\right)}^2\right].$$ (39)

Under Assumption 1, we have the steady-state MSE:

$$J\triangleq \underset{n\to \infty }{\text{lim}}\text{E}\left[e_n^2\right]={\sigma }_v^2+J_{\text{ex}}.$$ (40)

The following theorems characterize the steady-state behavior of (38):

Theorem 5.1 (Steady-state excess MSE):

Under Assumptions 1–2 with a sufficiently small μ and assume $R={\sigma }_u^{2}I$, for the adaptive filter (38), the steady-state excess MSE is given by:

$$J_{\text{ex}}=\frac{\mu {\sum }_{i=0}^{M-1}\frac{{\sigma }_u^2{s}_i}{2-2\mu {\sigma }_u^2{s}_i}}{1-\mu {\sum }_{i=0}^{M-1}\frac{{\sigma }_u^2{s}_i}{2-2\mu {\sigma }_u^2{s}_i}}{\sigma }_v^2.$$ (41)

Proof:

See Appendix B.

Theorem 5.2 (Convergence conditions):

Under Assumptions 1–2 with a sufficiently small μ and assume $R={\sigma }_u^{2}I$, for the adaptive filter (38):

1. It converges in the mean sense if:

   $$|{\lambda }_{\text{max}}\left\{I-\mu {\sigma }_u^{2}S\right\}|<1,$$ (42)

   where λ_max {X} denotes the largest eigenvalue of a square matrix X in magnitude.
2. It converges in the mean squared sense if:

   $$0<\mu <{\left(\sum _{i=0}^{M-1}\frac{{\sigma }_u^2{s}_i}{2-2\mu {\sigma }_u^2{s}_i}\right)}^{-1}.$$ (43)

Proof:

See Appendix C.

A. Steady-State Performance of SLMS

Consider the case where Assumptions 1–3 are in position and $R={\sigma }_u^{2}I$. For analyzing the proposed SLMS (30), first we need to recognize an appropriate S with regard to Assumption 3. A useful approximation at steady-state is to replace the occurrence of h_n by the true system h^o; that is, to use $S=\frac{W^2}{\frac{1}{M}\text{tr}\left(W^2\right)}$, where W = diag{w_i} with w_i given by (13) for the reweighted ℓ₂ case, or by (14) for the reweighted ℓ₁ case, both computed based on the corresponding true coefficient $h_i^o$. Now, since tr(S) = M, the excess MSE (41) can be approximated as:

$$\begin{gathered} J_{\text{ex}}\approx \frac{\mu {\sum }_{i=0}^{M-1}\frac{{\sigma }_u^2{s}_i}{2}}{1-\mu {\sum }_{i=0}^{M-1}\frac{{\sigma }_u^2{s}_i}{2}}{\sigma }_v^2=\frac{\mu \frac{{\sigma }_u^2}{2}\text{tr}(S)}{1-\mu \frac{{\sigma }_u^2}{2}\text{tr}(S)}{\sigma }_v^2 \\ =\frac{\mu }{\frac{2}{M{\sigma }_u^2}-\mu }{\sigma }_v^2, \end{gathered}$$ (44)

where for the approximation we assume a sufficiently small step size μ such that $2\mu {\sigma }_u^2{s}_i\ll 2$, ∀_i = 0, 1, …, M − 1.

Now, for the mean squared convergence condition, although the upper bound in (43) of Theorem 5.2 contains μ itself, after some inspection it is clear that the lowest stability limit on μ occurs when S has its diagonal elements nonzero at one tap position (with a value of M) and zero at all others [5]. With such an S, it leads to:

$$0<\mu <\frac{2}{3M{\sigma }_u^2}.$$ (45)

On the other hand, the largest stability limit is associated with a proportionate matrix assigning equal gains at each position [5], i.e., S = diag{s_i} with s_i = 1, ∀_i = 0, 1, …, M − 1. With such an S we have:

$$0<\mu <\frac{2}{(2+M){\sigma }_u^2}.$$ (46)

For a large M, the largest stability limit can be approximated as $\frac{2}{M{\sigma }_u^2}=\frac{2}{\text{tr}(R)}$, which is also the stability limit of the LMS [4]. This result is not surprising since using an S that assigns uniform gains essentially becomes the LMS.

B. Steady-State Performance of SNLMS

Consider the case where Assumptions 1–3 are in position and $R={\sigma }_u^{2}I$. For analyzing the proposed SNLMS (32), first we must identify a fixed S to approximate the term $\frac{S_n}{u_n^T{S}_n{u}_n}$ (where we have ignored δ), for which an exact characterization seems difficult, if at all possible, to obtain. However, if we fix W_n = W at steady-state by Assumption 3, where W is again computed based on the true system h^o, then we have:

$$S=\frac{\frac{W^2}{\frac{1}{M}\text{tr}\left(W^2\right)}}{u_n^T\left(\frac{W^2}{\frac{1}{M}\text{tr}\left(W^2\right)}\right)u_n}=\frac{W^2}{u_n^T{W}^2{u}_n}\approx \frac{W^2}{{\sigma }_u^2\text{tr}\left(W^2\right)},$$ (47)

with the approximation $u_n^T{W}^2{u}_n\approx \text{E}\left[u_n^T{W}^2{u}_n\right]={\sigma }_u^2\text{tr}\left(W^2\right)$ utilized. A useful fact of (47) is that $\text{tr}(S)={\left({\sigma }_u^2\right)}^{-1}$. We can thus use the following approximation for (41) to express the excess MSE (41) (and replace μ by $\tilde{\mu }$):

$$\begin{gathered} J_{\text{ex}}\approx \frac{\tilde{\mu }{\sum }_{i=0}^{M-1}\frac{{\sigma }_u^2{s}_i}{2}}{1-\tilde{\mu }{\sum }_{i=0}^{M-1}\frac{{\sigma }_u^2{s}_i}{2}}{\sigma }_v^2=\frac{\tilde{\mu }{\sigma }_u^2{\sum }_{i=0}^{M-1}{s}_i}{2-\tilde{\mu }{\sigma }_u^2{\sum }_{i=0}^{M-1}{s}_i}{\sigma }_v^2 \\ =\frac{\tilde{\mu }{\sigma }_u^2\text{tr}(S)}{2-\tilde{\mu }{\sigma }_u^2\text{tr}(S)}{\sigma }_v^2=\frac{\tilde{\mu }{\sigma }_u^2{\left({\sigma }_u^2\right)}^{-1}}{2-\tilde{\mu }{\sigma }_u^2{\left({\sigma }_u^2\right)}^{-1}}{\sigma }_v^2=\frac{\tilde{\mu }}{2-\tilde{\mu }}{\sigma }_v^2, \end{gathered}$$ (48)

for $\tilde{\mu }$ sufficiently small such that $2\tilde{\mu }{\sigma }_u^2{s}_i\ll 2$, ∀_i = 0, 1, …, M − 1,

For the mean squared convergence condition, using the same argument as in the SLMS case for (45) and (46), we can obtain the lowest stability limit as:

$$0<\tilde{\mu }<\frac{2}{3}$$ (49)

and the largest stability limit as:

$$0<\tilde{\mu }<\frac{2}{1+\frac{2}{M}}.$$ (50)

For a large M, we have (50) approximately as $0<\tilde{\mu }<2$, which is the classic result of the NLMS [4].

VI. Simulation Results

The proposed algorithms are evaluated using computer simulations in MATLAB. We consider three system IRs as shown in Fig. 1 which represent different sparsity levels: quasi-sparse, sparse, and dispersive systems. The IR of the quasi-sparse system is an acoustic feedback path between the microphone and the loudspeaker of a hearing aid device that was measured from a real-world scenario. It represents a typical IR of many practical system identification problems where certain degree of structural sparsity exists. The sparse and dispersive IRs were artificially generated. Each of these IRs has 256 taps. We conducted experiments to obtain the MSE learning curves (i.e., the ensemble average of $e_n^2$ as a function of iteration n) for performance comparison. The ensemble averaging was performed over 1000 independent Monte Carlo runs for obtaining each curve. In all experiments, the adaptive filter coefficients were initialized with all zeros. For the input signal, we mainly consider for theoretical analysis two types of u_n: i) a zero mean, unit variance white Gaussian process and ii) a first order autoregressive (AR) process according to u_n = ρu_n−1 + η_n, where ρ = 0.8 and η_n is i.i.d. according to $\mathcal{N}(0,1)$. We also include results of speech inputs for demonstrating the algorithm performance with non-stationary signals. The system noise v_n is i.i.d. according to $\mathcal{N}(0,0.01)$. Regarding the algorithms, when using (34) for updating W_n, a small positive constant c = 0.001 was always used.

Fig. 1:

IRs of (a) quasi-sparse, (b) sparse, and (c) dispersive systems. The quasi-sparse IR is an acoustic feedback path of a hearing aid that was measured from a real-world scenario. The sparse and dispersive IRs were artificially generated.

A. Comparison of Algorithms with and without AST

Fig. 2 compares the proposed generalized sparse LMS (29), i.e., the AST-based approach, to some existing regularization-based algorithms of (10), i.e., the regular gradient descent without AST. Specifically, we use the p-norm-like penalty $\Vert h{\Vert }_p^p$ with p = 1 and the log-sum penalty ${\sum }_{i=0}^{M-1}\text{log}\left(|h_i|+ϵ\right)$ with ϵ = 0.1 as examples. These two choices of the sparsity-inducing function $\mathcal{I}(h)$ in (10) result in the ZA-LMS and RZA-LMS [28], respectively. We compare them with the corresponding AST-based algorithms obtained from (29), also adopting the two penalty functions for G(h) that lead to (34) and (37) for computing W_n, respectively. We set μ = 0.0025 and λ = 0.001 in all cases and used the white Gaussian process input. Fig. 2 (a) shows the results of identifying the sparse IR and Fig. 2 (b) is the case of estimating the quasi-sparse IR. From the results we see that the AST strategy leads to algorithms (dotted lines) that demonstrate faster convergence than the existing approaches (solid lines).

Fig. 2:

Comparison of algorithms with and without AST for identifying (a) sparse and (b) quasi-sparse IRs with white Gaussian process input. Solid lines are existing approaches as given by (10). Dotted lines are their corresponding AST-based algorithms given by (29). It can be seen that AST leads to improved performance.

B. Effect of Sparsity Control Parameter on SLMS and SNLMS

In this experiment we investigate the effect of the sparsity control parameter on the convergence of SLMS (30) and SNLMS (32). We use the p-norm-like diversity measure $\Vert h{\Vert }_p^p$ within the reweighted ℓ₂ framework, i.e., using (34) for updating W_n, for demonstration purposes. We study the cases of the sparsity control parameter p = 1, 1.2, 1.5, 1.8, 2. We also include the LMS (3) and NLMS (4) performance curves for reference. For LMS and SLMS we used μ = 0.0025. For NLMS and SNLMS we used $\tilde{\mu }=0.5$ and δ = 0.01.

Fig. 3 and Fig. 4 show the resulting MSE curves for SLMS using the white Gaussian noise input and SNLMS using the AR process input, respectively. Recall that the proportionate factors of SLMS/SNLMS using (34) for W_n approximate that of the PNLMS when p → 1, and regenerate the LMS/NLMS when p = 2, as has been discussed in Section IV-B. Therefore, the parameter p plays the role for fitting different sparsity levels and the selection of p can be crucial for obtaining optimal performance for IRs with different sparsity degrees. The results in both Fig. 3 and Fig. 4 suggest that for the quasi-sparse case, the fastest convergence is given by p ∈ [1.2, 1.5], which seems reasonable in terms of finding a balance between PNLMS (p → 1) and LMS/NLMS (p = 2). On the other hand, for the sparse system, p ∈ [1, 1.2] gives the best results, which is also reasonable since as the sparsity level increases, a more PNLMS-like algorithm can be more favorable. Finally, for the dispersive system we see that p ∈ [1.8, 2] results in the fastest convergence and is comparable to, if not better than, the LMS and NLMS. This indicates that a more LMS/NLMS-like algorithm is preferable when the system IR is far from sparse. To conclude, the results show that the algorithms exploit the underlying system structure in the way we expect.

Fig. 3:

Effect of sparsity control parameter p on convergence of SLMS for (a) quasi-sparse, (b) sparse, and (c) dispersive IRs with white Gaussian process input. It can be seen that the optimal p value varies with the sparsity degree.

Fig. 4:

Effect of sparsity control parameter p on convergence of SNLMS for (a) quasi-sparse, (b) sparse, and (c) dispersive IRs with AR process input. In the colored input case here we have similar observations to the white input case of Fig. 3.

C. Effect of Step Size on SLMS and SNLMS

Fig. 5 studies the effect of the step size on the convergence behavior of the SLMS and SNLMS. We again used (34) for updating W_n. Fig.5 (a) shows the resulting MSE curves obtained by running the SLMS with p = 1.2 on the sparse IR with various μ values, using the white Gaussian noise input. Fig. 5 (b) shows the resulting MSE curves obtained by running the SNLMS with p = 1.5 on the quasi-sparse IR with various $\tilde{\mu }$ values, using the AR process input. The dotted lines indicate the theoretical steady-state MSE levels computed from (40) using (44) and (48) for SLMS and SNLMS, respectively. We can see that similar to the well-known trade-off in LMS and NLMS, a larger step size results in faster convergence while at the expense of steady-state performance. We also see that as the step size increases the theoretical prediction becomes less accurate; this is probably due to the approximation made based on the small step size assumption for arriving at (44) and (48). Nevertheless, the prediction agrees well with the stead-state MSE in most cases for a small step size. In addition, though several assumptions have been made to arrive at (40), (44) and (48), the results show that they predict reasonably well in the case of white input and even for correlated input.

Fig. 5:

Effect of step size μ or $\widehat{\mu }$ on convergence of (a) SLMS for the sparse IR with white Gaussian process input and (b) SNLMS for the quasi-sparse IR with AR process input. Dotted lines indicate the theoretical steady-state MSE levels. It can be seen that the theoretical prediction agrees reasonably well with the experimental results especially for a small step size.

D. Comparison with Existing Algorithms

We compare the proposed SLMS and SNLMS using (34) for W_n with existing LMS-type and NLMS-type algorithms. To see how the algorithms behave in a changing environment, in each of the following experiments, a change in the underlying system was introduced by shifting the IR to the right by 16 samples in the middle of the adaptation process [47].

Fig. 6 compares the LMS-type algorithms using the white Gaussian process input. Fig. 6 (a) and Fig. 6 (b) show the MSE curves obtained with the quasi-sparse and sparse IRs, respectively. For LMS we used μ = 0.0025. For ZA-LMS, RZA-LMS, and ℓ₀-LMS we fixed μ = 0.0025 and then experimentally optimized the remaining parameters to obtain the best performance in each case. For SLMS we used p = 1.5 and μ = 0.002 in the quasi-sparse case and p = 1.2 and μ = 0.0005 in the sparse case. From the results we can see that all the sparsity-aware algorithms outperform the LMS, with SLMS demonstrating the best result. Comparing Fig. 6 (a) and Fig. 6 (b), we also see that the benefit brought by existing sparsity-aware algorithms becomes limited when the system is less sparse, while the SLMS still provides significant improvement.

Fig. 6:

Comparison of LMS-type algorithms with white Gaussian process input on (a) quasi-sparse and (b) sparse IRs. One can see that the proposed SLMS outperforms all the other approaches in both cases.

Fig. 7 compares the NLMS-type algorithms using the AR process input. Fig. 7 (a) and Fig. 7 (b) show the MSE curves obtained with the quasi-sparse and sparse IRs, respectively. For all the algorithms we used $\tilde{\mu }=0.5$. For NLMS we used δ = 0.01. For PNLMS, IPNLMS, and IPNLMS-ℓ₀ we set δ = 0.01/M according to [47], and experimentally optimized the remaining parameters to obtain the best performance in each case. For SNLMS we used p = 1.5 in the quasi-sparse case and p = 1.2 in the sparse case. We used δ = 0.01 for SNLMS, same as NLMS.¹³ From the results we again observe the benefit of using sparsity-aware adaptation. In addition, the SNLMS demonstrates performance as good as, if not better than, the other proportionate algorithms.

Fig. 7:

Comparison of NLMS-type algorithms with AR process input on (a) quasi-sparse and (b) sparse IRs. One can see that the proposed SNLMS performs better than all the other approaches in both cases.

Fig. 8 considers a more practical scenario where we used a speech signal as the input and the quasi-sparse IR, which represents an acoustic channel of practical interest, as the underlying system. The input signal-to-noise ration (SNR) was set to 20 dB using white Gaussian noise. For the SLMS and SNLMS we used p = 1.5 which is a suitable choice for quasi-sparse systems. For evaluation we compare the normalized misalignment ${\Vert h^o-h_n\Vert }_2^2/{\Vert h^o\Vert }_2^2$. In Fig. 8 (a) we see that SLMS performs much better than the LMS, while the ℓ₀-LMS fails to provide any improvement. This may be due to the fact that existing regularization-based algorithms tend to enforce sparsity in a more aggressive manner as they work with λ > 0, and this may not be beneficial, if not harmful, when the underlying system is not truly sparse. In Fig.8 (b) we see that SNLMS demonstrates superior convergence behavior than the NLMS, and is also better than the IPNLMS and IPNLMS-ℓ_0.

Fig. 8:

Comparison of (a) LMS-type and (b) NLMS-type algorithms for identifying the quasi-sparse acoustic channel response with speech input at 20 dB SNR. In can be seen that the SLMS and SNLMS perform the best in both cases.

Fig. 9 shows the results for a noisier environment, i.e., 0 dB input SNR, for the same experimental setting of Fig. 8 (only the step size parameters were further tuned due to the stronger noise). We see that in Fig.9 (a) the SLMS significantly outperforms the LMS, while the ℓ₀-LMS performs worse. The SNLMS in Fig. 9 (b), on the other hands, still performs better than the NLMS, and is comparable to other proportionate algorithms. From the results we see that our observation on the SLMS and SNLMS performance may be robust to the noise condition.

Fig. 9:

Comparison of (a) LMS-type and (b) NLMS-type algorithms for identifying the quasi-sparse acoustic channel response with speech input at 0 dB SNR. In the noisier setting here the SLMS and SNLMS perform comparably, if not better than, other competing algorithms.

VII. Conclusion

In this paper, we developed a mathematical framework for rigorously deriving adaptive filters that exploit the sparse structure of the underlying system response. We started with the regularized objective framework of SSR and developed algorithms that are of the proportionate type. As a result, the adaptive algorithms are quite general and can accommodate a range of regularization functions. The framework utilizes the AST methodology within the iterative reweighted ℓ₂ and ℓ₁ frameworks for deriving algorithms. We showed that the AST is crucial for obtaining improved adaptive filtering performance over existing approaches when gradient descent approaches are used. We further introduced the SLMS and SNLMS by adopting a zero regularization coefficient in our framework, which take advantage of, though do not strictly enforce, the sparsity of the underlying system if it already exists. Note that the proposed framework is not limited to the algorithms that we have presented so far. Any other penalty function that satisfies the conditions imposed on the diversity measure can potentially be a good candidate for obtaining effective adaptive algorithms by utilizing the framework.

Appendix A

Proof of Theorem 3.1

The proof follows the idea in [48]. We wish to show that the regularized objective function J^G(h) in (9) is decreased in each iteration when optimized via (20) and (21). Before proceeding, we need the following lemmas:

Lemma A.1:

For the general diversity measure $G(h)={\sum }_{i=0}^{M-1}g\left(h_i\right)$ that satisfies Properties 1–4 in Section II-B, with g(z) being strictly concave in z² for Property 4, we have:

$$G\left(h_{n+1}\right)-G\left(h_n\right)<{\Vert W_n^{-1}{h}_{n+1}\Vert }_2^2-{\Vert W_n^{-1}{h}_n\Vert }_2^2,$$ (51)

where W_n = diag{w_i,n} with w_i,n given by (13).

Proof:

Since g(z) is strictly concave in z², it satisfies g(z) = f(z²) where f(z) is concave for $z\in {ℝ}_{+}$. Due to the concavity, we have the following inequality:

$$f\left(z_2\right)-f\left(z_1\right)<f^{\prime }\left(z_1\right)\left(z_2-z_1\right)$$ (52)

hold for some z₁, $z_2\in {ℝ}_{+}$. Note that we use f′(z₁) to denote the first order derivative of f(z) w.r.t. z evaluated at z = z₁.

Substituting $z_1=h_{i,n}^2$ and $z_2=h_{i,n+1}^2$ into (52) gives:

$$f\left(h_{i,n+1}^2\right)-f\left(h_{i,n}^2\right)<f^{\prime }\left(h_{i,n}^2\right)\left(h_{i,n+1}^2-h_{i,n}^2\right).$$ (53)

Noting that $f\left(h_{i,n+1}^2\right)=g\left(h_{i,n+1}\right)$ and $f\left(h_{i,n}^2\right)=g\left(h_{i,n}\right)$, we have:

$$g\left(h_{i,n+1}\right)-g\left(h_{i,n}\right)<f^{\prime }\left(h_{i,n}^2\right)\left(h_{i,n+1}^2-h_{i,n}^2\right).$$ (54)

From (13) we have $f^{\prime }\left(h_{i,n}^2\right)=w_{i,n}^{-2}$. Therefore,

$$g\left(h_{i,n+1}\right)-g\left(h_{i,n}\right)<w_{i,n}^{-2}\left(h_{i,n+1}^2-h_{i,n}^2\right).$$ (55)

Summing over i = 0, 1, …, M − 1 on both sides of (55) justifies (51) of Lemma A.1.

Lemma A.2:

For the general diversity measure $G(h)={\sum }_{i=0}^{M-1}g\left(h_i\right)$ that satisfies Properties 1–4 in Section II-B, with g(z) being strictly concave in |z| for Property 4, we have:

$$G\left(h_{n+1}\right)-G\left(h_n\right)<{\Vert W_n^{-1}{h}_{n+1}\Vert }_1-{\Vert W_n^{-1}{h}_n\Vert }_1,$$ (56)

where W_n = diag{w_i,n} with w_i,n given by (14).

Proof:

Since g(z) is strictly concave in |z|, it satisfies g(z) = f(|z|) where f(z) is concave for $z\in {ℝ}_{+}$. Again, the inequality (52) holds due to the concavity of f(z).

Substituting z₁ = |h_i,n| and z₂ = |h_i,n+1| into (52) gives:

$$f\left(|h_{i,n+1}|\right)-f\left(|h_{i,n}|\right)<f^{\prime }\left(|h_{i,n}|\right)\left(|h_{i,n+1}|-|h_{i,n}|\right).$$ (57)

Noting that f(|h_i,n+1|) = g(h_i,n+1) and f(|h_i,n|) = g(h_i,n), we have:

$$g\left(h_{i,n+1}\right)-g\left(h_{i,n}\right)<f^{\prime }\left(|h_{i,n}|\right)\left(|h_{i,n+1}|-|h_{i,n}|\right).$$ (58)

From (14) we have $f^{\prime }\left(|h_{i,n}|\right)=w_{i,n}^{-1}$. Therefore,

$$g\left(h_{i,n+1}\right)-g\left(h_{i,n}\right)<w_{i,n}^{-1}\left(|h_{i,n+1}|-|h_{i,n}|\right).$$ (59)

Summing over i = 0, 1, …, M − 1 on both sides of (59), we have (56) of Lemma A.2 justified.

Now we are ready to show that J^G(h) decreases in each iteration by using the update recursions (20) and (21).

First, for the reweighted ℓ₂ framework with $J_n^{{\mathcal{l}}_2}(q)$ in (16), we have:

$$\begin{gathered} J^G\left(h_{n+1}\right)-J^G\left(h_n\right)=\left[J\left(h_{n+1}\right)+\lambda G\left(h_{n+1}\right)\right]-\left[J\left(h_n\right)+\lambda G\left(h_n\right)\right] \\ <\left[J\left(h_{n+1}\right)+\lambda {\Vert W_n^{-1}{h}_{n+1}\Vert }_2^2\right]-\left[J\left(h_n\right)+\lambda {\Vert W_n^{-1}{h}_n\Vert }_2^2\right] \\ =\left[J\left(W_n{q}_{n+1\mid n}\right)+\lambda {\Vert q_{n+1\mid n}\Vert }_2^2\right]-\left[J\left(W_n{q}_{n\mid n}\right)+\lambda {\Vert q_{n\mid n}\Vert }_2^2\right] \\ =J_n^{{\mathcal{l}}_2}\left(q_{n+1\mid n}\right)-J_n^{{\mathcal{l}}_2}\left(q_{n\mid n}\right). \end{gathered}$$ (60)

The inequality follows from Lemma A.1. The AST relationships (18) and (19) are also utilized. As we perform optimization of (16) gradient descent, we can have $J_n^{{\mathcal{l}}_2}(q)$ decrease in each iteration n, i.e., $J_n^{{\mathcal{l}}_2}\left(q_{n+1\mid n}\right)-J_n^{{\mathcal{l}}_2}\left(q_{n\mid n}\right)<0$ using some μ_n. Therefore, the choice of ${\left\{{\mu }_n\right\}}_{n=0}^{\infty }$ ensures the decrease in J^G(h) according to (60), and the update recursion (20) monotonically converges to a local minimum (or saddle point) of (9) under a WSS environment.

On the other hand, for the reweighted ℓ₁ framework with $J_n^{{\mathcal{l}}_1}(q)$ in (17), we have:

$$\begin{gathered} J^G\left(h_{n+1}\right)-J^G\left(h_n\right)=\left[J\left(h_{n+1}\right)+\lambda G\left(h_{n+1}\right)\right]-\left[J\left(h_n\right)+\lambda G\left(h_n\right)\right] \\ <\left[J\left(h_{n+1}\right)+\lambda {\Vert W_n^{-1}{h}_{n+1}\Vert }_1\right]-\left[J\left(h_n\right)+\lambda {\Vert W_n^{-1}{h}_n\Vert }_1\right] \\ =\left[J\left(W_n{q}_{n+1\mid n}\right)+\lambda {\Vert q_{n+1\mid n}\Vert }_1\right]-\left[J\left(W_n{q}_{n\mid n}\right)+\lambda {\Vert q_{n\mid n}\Vert }_1\right] \\ =J_n^{{\mathcal{l}}_1}\left(q_{n+1\mid n}\right)-J_n^{{\mathcal{l}}_1}\left(q_{n\mid n}\right). \end{gathered}$$ (61)

The inequality follows from Lemma A.2. The AST relationships (18) and (19) are also utilized. Similar to the above argument of the reweighted ℓ₂ case, there exists a choice of ${\left\{{\mu }_n\right\}}_{n=0}^{\infty }$ that ensures the decrease in J^G(h) according to (61), and the update recursion (21) monotonically converges to a local minimum (or saddle point) of (9) under a WSS environment.

Appendix B

Proof of Theorem 5.1

The proof follows the discussion in [4], [5], [25]. Substituting $e_n=d_n-u_n^T{h}_n$ into (38) we have:

$$h_{n+1}=h_n-\mu S{u}_n{u}_n^T{h}_n+\mu S{u}_n{d}_n.$$ (62)

Using the fact that $d_n=u_n^T{h}^o+v_n$, we have:

$$h_{n+1}=h_n+\mu S{u}_n{u}_n^T\left(h^o-h_n\right)+\mu S{u}_n{v}_n.$$ (63)

Define the misalignment vector ε_n as:

$${\epsilon }_n=h^o-h_n.$$ (64)

Then from (63) we have:

$${\epsilon }_{n+1}=\left(I-\mu S{u}_n{u}_n^T\right){\epsilon }_n-\mu S{u}_n{v}_n.$$ (65)

Next, based on (65) we have:

$${\epsilon }_{n+1}{\epsilon }_{n+1}^T=\left(I-\mu S{u}_n{u}_n^T\right){\epsilon }_n{\epsilon }_n^T\left(I-\mu u_n{u}_n^{T}S\right)+{\mu }^2{v}_n^{2}S{u}_n{u}_n^{T}S+\Xi ,$$ (66)

where Ξ represents the remaining cross terms whose expectations are zero.

Let ${\Omega }_n=\text{E}\left[{\epsilon }_n{\epsilon }_n^T\right]$. Taking expectation on both sides of (66) we have:

$${\Omega }_{n+1}=\underset{\triangleq \Theta }{\underbrace{\text{E}\left[\left(I-\mu S{u}_n{u}_n^T\right){\epsilon }_n{\epsilon }_n^T\left(I-\mu u_n{u}_n^{T}S\right)\right]}}+{\mu }^2{\sigma }_v^{2}SRS.$$ (67)

Note that:

$$\Theta ={\Omega }_n-\mu SR{\Omega }_n-\mu {\Omega }_{n}RS+{\mu }^{2}S\text{E}\left[u_n{u}_n^T{\epsilon }_n{\epsilon }_n^T{u}_n{u}_n^T\right]S.$$ (68)

With Assumptions 1 and 2 it can be shown that [4]:

$$\text{E}\left[u_n{u}_n^T{\epsilon }_n{\epsilon }_n^T{u}_n{u}_n^T\right]=2R{\Omega }_{n}R+R\text{tr}\left(R{\Omega }_n\right).$$ (69)

Thus,

$$\Theta ={\Omega }_n-\mu SR{\Omega }_n-\mu {\Omega }_{n}RS+2{\mu }^{2}SR{\Omega }_{n}RS+{\mu }^{2}SR\text{tr}\left(R{\Omega }_n\right)S.$$ (70)

Then, with $R={\sigma }_u^{2}I$, in steady-state, i.e., n → ∞,

$${\Omega }_{\infty }={\Omega }_{\infty }-\mu {\sigma }_u^{2}S{\Omega }_{\infty }-\mu {\sigma }_u^2{\Omega }_{\infty }S+2{\mu }^2{\sigma }_u^{4}S{\Omega }_{\infty }S+{\mu }^2{\sigma }_u^{4}S\text{tr}\left({\Omega }_{\infty }\right)S+{\mu }^2{\sigma }_u^2{\sigma }_v^2{S}^2.$$ (71)

This implies:

$${\omega }_{\infty }={\omega }_{\infty }-2\mu {\sigma }_u^{2}S{\omega }_{\infty }+2{\mu }^2{\sigma }_u^4{S}^2{\omega }_{\infty }+{\mu }^2{\sigma }_u^4{s}^2{1}^T{\omega }_{\infty }+{\mu }^2{\sigma }_u^2{\sigma }_v^2{s}^2,$$ (72)

where ω_∞ and s are the vectors consisting of the diagonal elements of Ω_∞ and S, respectively, and s² denotes the element-wise squares of the vector s.

The steady-state excess MSE is then:

$$\begin{gathered} J_{\text{ex}}\triangleq \underset{n\to \infty }{\text{lim}}\text{E}\left[{\left(u_n^T\left(h^o-h_n\right)\right)}^2\right]=\underset{n\to \infty }{\text{lim}}\text{E}\left[{\left(u_n^T{\epsilon }_n\right)}^2\right] \\ =\underset{n\to \infty }{\text{lim}}\text{tr}\left({\Omega }_n\text{E}\left[u_n{u}_n^T\right]\right)={\sigma }_u^2\text{tr}\left({\Omega }_{\infty }\right)={\sigma }_u^2{1}^T{\mathit{\omega }}_{\infty }. \end{gathered}$$ (73)

Using (73) for (72) and rearrange the equation, we have:

$${\omega }_{i,\infty }=\frac{{\mu }^2{\sigma }_u^2{s}_i^2{J}_{\text{ex}}+{\mu }^2{\sigma }_u^2{\sigma }_v^2{s}_i^2}{2\mu {\sigma }_u^2{s}_i-2{\mu }^2{\sigma }_u^4{s}_i^2}.$$ (74)

Then it leads to:

$$J_{\text{ex}}={\sigma }_u^2\sum _{i=0}^{M-1}{\omega }_{i,\infty }=\sum _{i=0}^{M-1}\frac{{\mu }^2{\sigma }_u^2{s}_i^2{J}_{\text{ex}}+{\mu }^2{\sigma }_u^2{\sigma }_v^2{s}_i^2}{2\mu s_i-2{\mu }^2{\sigma }_u^2{s}_i^2},$$ (75)

which yields:

$$J_{\text{ex}}=\frac{\mu {\sum }_{i=0}^{M-1}\frac{{\sigma }_u^2{s}_i}{2-2\mu {\sigma }_u^2{s}_i}}{1-\mu {\sum }_{i=0}^{M-1}\frac{{\sigma }_u^2{s}_i}{2-2\mu {\sigma }_u^2{s}_i}}{\sigma }_v^2.$$ (76)

This justifies Theorem 5.1.

Appendix C

Proof of Theorem 5.2

Note that Assumption 1 ensures that h_n, u_n, and v_n are mutually independent. Thus, taking expectation of both sides of (65) gives:

$$\text{E}\left[{\epsilon }_{n+1}\right]=(I-\mu SR)\text{E}\left[{\epsilon }_n\right].$$ (77)

Therefore, the following condition is sufficient for convergence in the mean sense [4]:

$$|{\lambda }_{\text{max}}\{I-\mu SR\}|<1.$$ (78)

With $R={\sigma }_u^{2}I$, Theorem 5.2-i) is justified.

From (76) we see, by requiring:

$$1-\mu \sum _{i=0}^{M-1}\frac{{\sigma }_u^2{s}_i}{2-2\mu {\sigma }_u^2{s}_i}>0,$$ (79)

we obtain the stability bound for μ as:

$$0<\mu <{\left(\sum _{i=0}^{M-1}\frac{{\sigma }_u^2{s}_i}{2-2\mu {\sigma }_u^2{s}_i}\right)}^{-1},$$ (80)

which justifies Theorem 5.2-ii).