AMP-B-SBL: An algorithm for clustered sparse signals using approximate message passing

Abstract

Recently, we proposed an algorithm for the single measurement vector problem where the underlying sparse signal has an unknown clustered pattern. The algorithm is essentially a sparse Bayesian learning (SBL) algorithm simplified via the approximate message passing (AMP) framework. Treating the cluster pattern is controlled via a knob that accounts for the amount of clumpiness in the solution. The parameter corresponding to the knob is learned using expectation-maximization algorithm. In this paper, we provide further study by comparing the performance of our algorithm with other algorithms in terms of support recovery, mean-squared error, and an example in image reconstruction in a compressed sensing fashion.

I. Introduction

Compressive sensing (CS) is a promising technique in processing compressible or sparse signals by requiring far few samples than the Nyquist rate [1], [2]. This technique is essentially composed of two main parts. The first part is devoted to the design of the sensing device to be able to measure a compressible signal via very few samples such that all the important information of the underlying signal is captured. The sensing structure is required to satisfy some conditions such as restricted isometry property (RIP) [3]. Under the assumption that all the informational value of the signal has already been compressed into very few samples, the problem then is how to extract all the information and reconstruct the underlying signal again in an efficient manner.

The single measurement vector (SMV) problem is one of the benchmark problems with many applications in CS [4]–[9]. In the SMV problem, a set of linear noisy measurements from a sparse signal x is modeled as y = Ax + e, where x ∈ ℝ^N is the signal that we seek to reconstruct, and e denotes the measurement noise. In this model, A ∈ ℝ^M×N is a known sensing matrix with M ≪ N, which satisfies RIP. There exist many algorithms to estimate the sparse solution for the SMVs categorized mainly into either greedy-based algorithms [4], [10] or sparse Bayesian learning (SBL) algorithms [7], [11]. In addition to sparsity, there are many applications in which the sparse signal x exhibits a clustered pattern. Applications can be found in gene expression analysis [12] and direction of arrival (DOA) [13]. This feature has been referred to as block-sparsity or sparse clustered pattern in the literature [7], [8], [14], [15]. Regarding the SBL algorithms, two main priors have been considered to encourage the sparsity. The first category involves using Gaussian-inverse-Gamma prior on the entries of the solution. In this model, a zero-mean Gaussian distribution is assumed on each element of the solution to encourage the sparsity. The variances of the Gaussian distributions are assumed to be drawn from inverse-Gamma priors [16]. The reason for using either Gamma or inverse-Gamma hyperpriors is due to the requirement for the non-negative outcomes for variances and also to use the conjugacy property of the distributions for the simplification purposes. The second category is referred to as Bernoulli-Gaussian or spike-and-slab priors. In this case, the solution vector is usually separated into two parts; the support leaning vector and the solution value vector [8], [9], [15]. Although SBL algorithms have been shown to be very promising in estimating sparse signals, they are slow in terms of run-time of the algorithm. Recently, it has been shown that using approximate message passing (AMP) techniques can reduce the complexity of SBL algorithms while maintaining almost the same performance as the regular SBLs [16]–[18]. In [19], we provided an SBL algorithm that uses AMP techniques to solve for the clustered pattern sparse signals, referred to as AMP-B-SBL algorithm. This algorithm is based on AMP-SBL algorithm introduced in [16], into which we incorporated another hyperprior to account for the unknown clustered pattern that may exist in the supports of the solution. We imposed the hyperprior to reflect the amount of clumpiness of the overall supports of the solution. The set of hyper-parameters controlling the clustered pattern of the support vector were learned via the expectation-maximization (EM) algorithm.

In this paper, we continue the study of the performance of AMP-B-SBL algorithm. We compare the performance of our algorithm with some of the other existing algorithms. Below, we first provide a short review on the theory behind AMP-B-SBL, and then we demonstrate the simulation results.

II. AMP-B-SBL: An algorithm for solving SMVs with unknown clustered pattern

In the AMP-B-SBL algorithm for the SMV problem, we assume i.i.d. zero-mean Gaussian priors on the components of the solution as follows

$$x_n~\mathcal{N}(0,{\alpha }_n),\forall n=1,\dots ,N.$$ (1)

As was discussed in section I, having priors of the form (1) promotes sparsity in the solution. We model the measurement noise in the SMV model as e ~ 𝒩(0, σ²I_M). To encourage the clustered pattern in the supports of the solution x, in [19] we defined a measure of clumpiness as follows

$${(\mathrm{\sum }\mathrm{\Delta })}_{(\text{support}\text{of}\mathbf{x})}:=\sum _{n=2}^N\mid b(x_n,T)-b(x_{n-1},T)\mid ,$$ (2)

where the function b(·, ·) is defined as

$$b(x_n,T)=\{\begin{array}{ll}1\hfill & \text{if}\mid x_n\mid >T\hfill \\ 0\hfill & \text{otherwise.}\hfill \end{array}$$ (3)

The threshold T is set to a small value, meaning that the estimated components of x which possesses an amplitude less than T are discarded from the support set and are deemed to be zero. Then, we incorporated the measure of clumpiness into our modeling by adding a hyperprior on each variance α_n as follows [19]

$${\alpha }_n~\mathcal{N}(e^{\left\{\frac{{(\mathrm{\sum }\mathrm{\Delta })\mid }_{b(x_n,.)=0}-{(\mathrm{\sum }\mathrm{\Delta })\mid }_{b(x_n,.)=1}-1}{{\theta }_1}\right\}},{\theta }_2),$$ (4)

where (ΣΔ)|_b(xn,·)=k, ∀k = 0, 1, ∀n = 1, …, N is defined as the measure of clumpiness in the support of x in case where the support corresponded to x_n is set to k. For example (ΣΔ)|_b(xn,·=1 is the case of forcing the index of x_n to be active. Here, we abbreviate (ΣΔ)|_b(xn,·=k to (ΣΔ)_n,k. The parameter θ₁ in (4) is the emphasis parameter on our measure of clumpiness, and is left as a free parameter in our model. The parameter θ₂ is another hyper-parameter and is learned via the EM algorithm. For the rationale behind assuming (4) as a prior on α_n, one can refer to [19]. The joint probability distribution of our model using (1) and (4) becomes

$$P(\mathbf{y},\mathbf{x},\mathit{\alpha },{\theta }_1,{\theta }_2,{\sigma }^2)\propto P(\mathbf{y}\mid \mathbf{x},{\sigma }^2{I}_M)\prod _{n=1}^N\left(P(x_n;0,{\alpha }_n)P({\alpha }_n;e^{\left\{\frac{{(\mathrm{\sum }\mathrm{\Delta })}_{n,0}-{(\mathrm{\sum }\mathrm{\Delta })}_{n,1}-1}{{\theta }_1}\right\}},{\theta }_2)\right).$$ (5)

By using the AMP technique defined in [16], [19], [20] and after some algebra, the posterior inference on x_n is estimated as follows

$$\begin{gathered} P(x_n\mid \mathbf{y})\propto P(x_n;{\alpha }_n)\prod _{m=1}^{M}P(y_m\mid x_n) \\ \propto N(x_n;{\mu }_n,{\nu }_n),\text{where} \end{gathered}$$ (6)

$${\mu }_n=\sum _{m=1}^M{a}_{mn}{z}_{mn}(\frac{{\alpha }_n}{c_n+{\alpha }_n}),\text{and}{\nu }_n=\frac{c_n{\alpha }_n}{c_n+{\alpha }_n}.$$ (7)

In (7), the terms z_mn and c_n are defined as

$$z_{mn}:=y_m-\sum _{q\ne m}{a}_{mq}{\mu }_q\text{and}{c}_n=\frac{1}{M}\sum _{m=1}^M{c}_{mn},$$

where c_mn = σ² + Σ_q≠;n |a_mq|²ν_q. (More explanation can be found in [19].) Finally, the update rules for the hyper-parameters α, σ, and θ₂ were obtained using the EM algorithm as the solution to the optimization problems below.

• Update rule for α_n:

  $${{\alpha }_n}^{[k+1]}=arg\underset{{\alpha }_n}{min}ln({\alpha }_n)+\frac{1}{{\alpha }_n}{\mathbb{E}}_{\mathbf{x}\mid {{\alpha }_n}^{[k]},-}[x_n^2]+\frac{1}{{\theta }_2}{({\alpha }_n-e^{\left\{\frac{{(\mathrm{\sum }\mathrm{\Delta })}_{n,0}-{(\mathrm{\sum }\mathrm{\Delta })}_{n,1}-1}{{\theta }_1}\right\}})}^2.$$
• Update rule for σ²:

  $$\begin{gathered} {\sigma }^{2[k+1]}=arg\underset{{\sigma }^2}{max}{\mathbb{E}}_{\mathbf{x}\mid \mathbf{y},{\theta }_1,{\theta }_2,\mathit{\alpha },{\sigma }^{2[k]}}[P(\mathbf{y},\mathbf{x},\mathit{\alpha },{\theta }_1,{\theta }_2,{\sigma }^2)] \\ =arg\underset{{\sigma }^2}{min}2Mln(\sigma )+\frac{1}{{\sigma }^2}{\mathbb{E}}_{\mathbf{x}\mid \mathbf{y},-}[{\Vert \mathbf{y}-A\mathbf{x}\Vert }_2^2] \end{gathered}$$
• Update rule for θ₂:

  $${{\theta }_2}^{[k+1]}=arg\underset{{\theta }_2}{min}{\mathbb{E}}_{\mathbf{x}\mid \mathbf{y},{{\theta }_2}^{[k]},-}[Nln({\theta }_2)+\frac{1}{{\theta }_2}\sum _{n=1}^N{({\alpha }_n-e^{\{{(\mathrm{\sum }\mathrm{\Delta })}_{n,0}-{(\mathrm{\sum }\mathrm{\Delta })}_{n,1}-1\}})}^2].$$

The simplified version of the above update rules can be found in the pseudo code of the AMP-B-SBL algorithm as described below. The stopping condition of our algorithm can be set based on either a predetermined number of iterations or convergence of the solution to a tight bound.

AMP-B-SBL Algorithm [19]:
• Definitions
$F_n(k_n,{\alpha }_n,c)=k_n\frac{{\alpha }_n}{c+{\alpha }_n}$
$G_n({\alpha }_n,c)=\frac{c{\alpha }_n}{c+{\alpha }_n}$
$F_n^{\prime }({\alpha }_n,c)=\frac{{\alpha }_n}{c+{\alpha }_n}$
• Message updates
For n = 1, 2, …, N
$k_n={\sum }_{m=1}^m{a}_{mn}^{★}{z}_m+{\mu }_n$
μn = Fn(kn, αn, c)
νn = Gn(αn, c)
End
$c={\sigma }^2+\frac{1}{M}{\sum }_{n=1}^N{\nu }_n$
$z_m=y_m-{\sum }_{n=1}^N{a}_{mn}{\mu }_n+\frac{z_m}{M}{\sum }_{n=1}^N{F}_n^{\prime }({\alpha }_n,c)$, ∀m = 1, …, M
• Parameter updates
Updating α: ∀n = 1, 2, …, N, solve for αn in ${\alpha }_n^3-e^{\{\frac{{(\mathrm{\sum }\mathrm{\Delta })}_{n,0}-{(\mathrm{\sum }\mathrm{\Delta })}_{n,1}-1}{{\theta }_1}\}}{\alpha }_n^2+\frac{{\theta }_2}{2}{\alpha }_n-\frac{{\theta }_2({\mu }_n^2+{\nu }_n)}{2}=0$
which is the minimizer of $$f({\alpha }_n)=ln({\alpha }_n)+\frac{{\mu }_n^2+{\nu }_n}{{\alpha }_n}+\frac{1}{{\theta }_2}{({\alpha }_n-e^{\{\frac{{(\mathrm{\sum }\mathrm{\Delta })}_{n,0}-{(\mathrm{\sum }\mathrm{\Delta })}_{n,1}-1}{{\theta }_1}\}})}^2$$
Updating the noise variance σ2: ${\sigma }^{2[k+1]}=\frac{{\Vert \mathbf{y}-A\mathit{\mu }\Vert }_2^2+{\sum }_{n=1}^N{\Vert {\mathbf{a}}_n\Vert }_2^2{\nu }_n}{M}$
Updating the variance of α: ${{\theta }_2}^{[k+1]}=\frac{1}{N}{\sum }_{n=1}^N{({\alpha }_n-e^{\{\frac{{(\mathrm{\sum }\mathrm{\Delta })}_{n,0}-{(\mathrm{\sum }\mathrm{\Delta })}_{n,1}-1}{{\theta }_1}\}})}^2$

III. Simulation Results

In this section, we demonstrate the performance of AMP-B-SBL algorithm compared to orthogonal matching pursuit (OMP) [21], basis pursuit denoising (BPDN) [5], MFOCUSS [22], and AMP-SBL [16] algorithms for solving the SMV problem. In all the simulations, we used the default settings for MFOCUSS algorithm. For the OMP, the stopping condition is set to $\sqrt{0.5N{\sigma }^2}$. In AMP-B-SBL, we set c^[10] = 10, α_n = 2 × 10⁻⁴, ∀n = 1, …, N, T = 0.001, θ₁ = 8, ${\theta }_2^{[0]}=0.6$, and the number of iterations is set to 1000. For the AMP-SBL algorithm, we set c^[0] =10, γ_n =2 × 10⁻⁴, ∀n =1, …, N, and the number of iterations is set to 1000. Finally, for BPDN we stopped the algorithm once ||y − Ax̂|| becomes less than 0.75.

In the first set of experiments, we generate 200 independent trials and then average over the obtained results. In this case, we evaluate the performance of the algorithms via detection and false alarm rate in support estimation, and also the normalized mean-squared error (NMSE) between the true and the estimation solutions. The NMSE is defined as $\text{NMSE}(\text{dB}):=20{log}_{10}\frac{{\Vert \mathbf{x}-\widehat{\mathbf{x}}\Vert }_2}{{\Vert \mathbf{x}\Vert }_2}$, where x and x̂ are the true and the estimated solutions, respectively. In each trial, the true solution vector is x ∈ R¹⁰⁰ with 25 non-zero elements (supports). The supports of the true solution in each trial are randomly drawn so that the support set of the solution exhibits a clustered pattern. The non-zero values of the solution are drawn i.i.d. from a zero-mean Gaussian distribution with variance of 1. Each entry of the sensing matrix A is an outcome of a Gaussian distribution 𝒩(0, 1). After constructing the sensing matrix A, we normalize A with respect to its columns. We vary the number of measurements M to show the performance. The elements of measurement noise are drawn i.i.d. from 𝒩(0, σ²) such that SNR=25 dB for all trials. The SNR is defined as SNR= 20 log₁₀(σ_x/σ).

In Fig. 1, we illustrate the empirical results of detection rate vs. the ratio λ = M/N. In the simulations, we discarded those estimated supports which their corresponding amplitudes become less than 0.01.

Fig. 1.

Comparison in terms of detection rate.

In Fig. 1 we see that AMP-B-SBL shows the highest detection rate for λ∈[0.05, 0.2]. For 0.25≤λ≤0.55 AMP-B-SBL, AMP-SBL, BPDN, and MFOCUSS demonstrate almost the same detection rate. Finally, for λ≥0.55 AMP-B-SBL, AMP-SBL, OMP, and MFOCUSS show approximately the same performance.

In Fig. 2, the false alarm rate (the rate of deciding on wrong supports in the solution) vs. λ is illustrated.

Fig. 2.

Comparison of false alarm rate in support recovery.

In Fig. 2 we observe that our algorithm has a higher false alarm rate compared to the other algorithms for λ = [0, 0.2]. Comparing the detection rate with the false alarm rate of AMP-B-SBL for such range of λ shows the trade off that exists between the detection rate and false alarm rate. For 0.2<λ≤0.3 all the algorithms show the same performance. However, our algorithm illustrates the lowest false alarm rate for λ>0.3. Fig. 3 essentially demonstrates the overall performance of the algorithms as a combination of support detection, false alarm rate, and the number of measurements.

Fig. 3.

Comparison based on the difference between the experimental detection and false alarm rate.

From Fig. 3, we observe that the best performance belongs to the AMP-B-SBL algorithm. This shows that incorporating our measure of contiguity in the supports into the SBL algorithm boosts the recovery performance for the clustered pattern sparse signals. Finally, Fig. 4 shows the performance comparison in terms of normalized mean-squared error between the true and estimated solutions.

Fig. 4.

Comparison in terms of normalized mean-squared error between the true and estimated solution.

According to Fig. 4, AMP-B-SBL outperforms the other algorithms in terms of estimating the true solution.

We further evaluate the performance of the algorithms via the following example. In this case we use the image of 112 × 200 pixels, where we consider the black pixels as the “interesting” locations. The image is shown in Fig. 5.

Fig. 5.

True Image.

In our simulations for this image, we assign the value of 1 to the pixels with black color and 0 to the white ones. We then, reshape the matrix corresponding to the image and construct the vector x∈R^11200×1. The number of measurements are set to 5040, i.e., λ = 0.45. We construct the sensing matrix in the same way we described earlier. Then, the measurements are obtained from y = Ax + e with SNR=25 dB. We feed all the algorithms with the measurement vector y and the sensing matrix A. The reconstructed images are illustrated in Fig. 6. In all the algorithms, we applied the threshold of 0.01, meaning that we discarded those estimated supports that corresponded to non-zero elements with the absolute value of less than 0.01. Below, we also include the reconstruction based on CLUSSMCMC algorithm [8]. In order to compare the reconstructed images illustrated in Fig. 6, we compare the obtained results based on the NMSE and peak-SNR (PSNR) as shown in Tab. I. According to Tab. I, we observe that CLUSSMCMC provides the best performance in terms of NMSE and PSNR. However, it is much slower compared to other algorithms. In contrast, AMP-B-SBL algorithm provides high performance for NMSE, PSNR, and the speed of the algorithm. By comparing the results of AMP-SBL with AMP-B-SBL, we also see that using the measure of clumpiness into the SBL algorithm definitely improves the reconstruction performance.

Fig. 6.

Results of reconstructed images for OMP, BPDN, MFOCUSS, CLUSSMCMC, AMP-SBL, and AMP-B-SBL.

TABLE I.

NMSE and PSNR comparison in image reconstruction.

Algorithm	NMSE (dB)	PSNR (dB)	speed
OMP	−1.7124	10.9425	Very Fast
BPDN	−8.7905	17.0205	Very Fast
MFOCUSS	−10.6522	18.8822	Slow
CLUSSMCMC	−27.42	35.6504	Slow
AMP-SBL	−13.4642	22.6943	Fast
AMP-B-SBL	−24.5600	32.7901	Fast

IV. Conclusion

Here, we provided further study on our recently proposed AMP-B-SBL algorithm. AMP-B-SBL is a sparse Bayesian learning approach simplified via approximate message passing techniques that is devised for the sparse signals those that may exhibit some sort of clustering pattern in their supports. Based on the simulation results, we showed that the algorithm provides high performance in terms of both the support recovery and the estimating the underlying true solution for signals having unknown clustered patterns.