SPARSE BAYESIAN LEARNING BOOSTED BY PARTIAL ERRONEOUS SUPPORT KNOWLEDGE

Abstract

Recovery of sparse signals with unknown clustering pattern in the case of having partial erroneous prior knowledge on the supports of the signal is considered. In this case, we provide a modified sparse Bayesian learning model to incorporate prior knowledge and simultaneously learn the unknown clustering pattern. For this purpose, we add one more layer to support-aided sparse Bayesian learning algorithm (SA-SBL). This layer adds a prior on the shape parameters of Gamma distributions, those modeled to account for the precision of the solution elements. We make the shape parameters depend on the total variations on the estimated supports of the solution. Based on the simulation results, we show that the proposed algorithm is able to modify its erroneous prior knowledge on the supports of the solution and learn the clustering pattern of the true signal by filtering out the incorrect supports from the estimated support set.

1. INTRODUCTION

Compressive sensing (CS) provides tools to represent a sparse or compressible signal from a small set of non-adaptive linear measurements [1]. In linear CS, the high dimensional signal x ∈ ℝ^N is modeled by the linear equation y = Ax+e, where A ∈ ℝ^M×N is a wide sensing matrix with M ≪ N. The case where the sensing matrix is known has been referred to as single measurement vector (SMV) problem [2]. In the CS context, it is assumed that x is sparse (has few non-zero elements) under some proper basis. Besides the sparsity, in some practical applications the nonzero entries of the sparse signal x may appear in clusters. This feature has been referred to as clustered-pattern or block-sparsity in the literature [3, 4]. Moreover, there exist cases where a partial erroneous support set of the solution is available as a prior knowledge. This type of information may be obtained from either of the two cases below. It can be the estimate of the supports inferred from the set of measurements taken from a phenomenon of interest at the last time instant. For example, in magnetoencephalography (MEG) for the brain activities and direction of arrival (DOA) estimation problems, it turns out that when taking successive measurements, the supports of the underlying signal remain almost constant or may experience very small variations [5, 6]. This problem has been treated as a multiple measurement vector (MMV) problem with slowly time-varying supports (sources) in the literature [7, 8].

In this paper we investigate the sparse recovery problem of signals with unknown clustering pattern for the case where some prior information on the supports of the solution is available. More specifically, we assume that we are provided with a partial erroneous support set of the solution. One application of our proposed algorithm is that instead of solving an MMV with slowly time-varying supports directly, one can break the problem into a collection of SMVs and solve each SMV one at a time where they are now boosted by the support set estimated from the solution of the SMV of the previous time instant. As the second case, prior knowledge can be the output of some other CS recovery algorithm with the goal of performing some post processing to improve the overall performance in both increasing the success rate in support recovery and removing the number of indexes that have been incorrectly considered as the active supports of the solution. In both of the above cases, the available estimate of the support set usually contains a subset of the true supports accompanied with some other indexes that are incorrectly assumed to be active or they belonged to the estimated support set inferred from the last time instant set of measurements. This set of supports has been referred to as partial erroneous support set in the literature [9].

In case of having prior knowledge on the supports of the solution, some algorithms such as MBPDN and SA-SBL have been recently proposed [9,10]. MBPDN algorithm is a modified version of basis pursuit de-noising algorithm for the case where a subset of true support set is available [10]. Recently, it has been shown [9] that MBPDN is sensitive to the accuracy of the prior knowledge on the available support set. In [9], Fang et al. proposed a modified version of the conventional sparse Bayesian learning model for the purpose of using prior information on the support set in order to obtain better estimate of the true underlying sparse signal. The conventional SBL algorithm considers a Gaussian-inverse-Gamma distribution on the elements of the solution vector x. In [9], one more layer was added to the conventional SBL. This layer incorporates a prior on the rate parameter of the Gamma distribution to take advantage of the available support knowledge of the solution. Following the same notations as was used in [9], suppose that 𝒯 is the set of all the true supports in the solution and 𝒮 ⊂ 𝒯 is a subset of true supports that is available. Furthermore, assume that ℰ ⊂ 𝒯^c contains the error subset that is incorrectly considered as a part of available true supports. Notice that 𝒯 ∪ 𝒯^c = {1, 2,…, N} and ℘ = 𝒮 ∪ ℰ, where ℘ is the erroneous support set that is available to us. As discussed earlier, one can think of ℘ as the support set of the previous column of the solution matrix in the MMV problem.

Below, we briefly describe the priors that were considered in [9]. Each element x_n of the solution was assumed to be drawn i.i.d. from zero mean Gaussian distribution with corresponding precision α_n, i.e.,

$$x_n~\mathcal{N}(x_n;0,{\alpha }_n^{-1}),\forall n=1,\dots ,N,$$ (1)

where the precisions are random variables defined as

$${\alpha }_n~\text{Gamma}({\alpha }_n;a,b_n),\forall n=1,\dots ,N,$$ (2)

where the shape parameter is fixed to a = 10⁻¹⁰. In order to incorporate the available and probably erroneous support knowledge, Fang et al. [9] defined the two following cases on the rate parameters of the Gamma distributions

$$\{\begin{array}{ll}{b}_n~\text{Gamma}(p,q),\hfill & \text{if}n\in \mathcal{P}\hfill \\ b_n={10}^{-15},\hfill & \text{otherwise}\hfill \end{array}$$ (3)

This means that only when the index n belongs to the set ℘ the corresponding precision α_n will be governed by another Gamma distribution with hyper-parameters p and q. In this case, SA-SBL algorithm was proposed in [9].

Our proposed algorithm is essentially a modified version of SA-SBL, in which we also account for the unknown clustering pattern that may exist in the original signal. For this purpose, we incorporate the measure of clumpiness over the supports of the solution (ΣΔ) proposed in [11] into SA-SBL algorithm. We refer to the proposed algorithm as CSA-SBL where the letter “C” stands for the “clustered” pattern signals. The main difference between our proposed algorithm and SA-SBL is that we further put a prior on the shape parameter of the Gamma distribution defined in (2) while it was set to a constant in SA-SBL [9]. Specifically, we impose that the shape parameter is to be controlled by the estimated measure of contiguity in the supports of the solution i.e., total variation on the supports of the solution. Based on the simulations, we show that this modification improves the overall performance in estimating the supports of the solution.

2. CSA-SBL ALGORITHM FOR SOLVING SMVS

In this section we describe our modified version of the conventional SBL algorithm to solve for x in the SMV problem defined by y = Ax + e. It is assumed that an erroneous support set ℘ = 𝒮 ∪ ℰ is available, where 𝒮 is a subset of the true supports of the solution and ℰ is a set of incorrectly considered supports. The partition 𝒮 and ℰ in ℘ is assumed unknown [9]. In order to account for the clustering pattern that may exist in the solution, we borrow the measure of clumpiness from [11], which is defined as

$${(\mathrm{\sum }\mathrm{\Delta })}_{(\text{support}\text{of}\mathbf{x})}=\sum _{n=2}^N\mid b(x_n,T)-b(x_{n-1},T)\mid ,$$ (4)

where T is a predetermined threshold. The function b(·, ·) in (4) returns a binary value and is defined as follows

$$b(x_n,T)=\{\begin{array}{ll}1\hfill & \text{if}\mid x_n\mid >T\hfill \\ 0\hfill & \text{otherwise.}\hfill \end{array}$$ (5)

The more clustered the solution becomes, the lower value (ΣΔ) in (4) will possess. Based on (5), the entries of x with the amplitude less than the threshold T are deemed to zero and their corresponding index will not be considered as supports of the solution. This is due to the fact that the elements do not have significant contribution in our measurements. We experimentally set T = 10⁻⁶.

Below we describe the prior distributions that we consider in our hierarchical Bayesian model. Similar to SBL and for the purpose of promoting sparsity in the solution, we assume that the elements of the solution are drawn i.i.d. from a zero-mean Gaussian distribution as follows

$$x_n~\mathcal{N}(x_n;0,{\alpha }_n^{-1}),\forall n=1,\dots ,N,$$ (Revisiting (1))

where the precision α_n is modeled as

$${\alpha }_n~\text{Gamma}({\alpha }_n;a_n,b_n),\forall n=1,\dots ,N.$$ (6)

Unlike (2), we do not assign the same shape parameter to the precisions α_n, ∀n = 1,…, N in our model.

In order to incorporate the available erroneous support knowledge, we use the same model as defined [9] for the rate parameters defined in (6).

$$\{\begin{array}{ll}{b}_n~\text{Gamma}(b_n;p,q),\hfill & \text{if}n\in \mathcal{P}\hfill \\ b_n={10}^{-15},\hfill & \text{otherwise}\hfill \end{array},$$ (Revisiting (3))

where p = q = 0.1 as suggested in [9].

The reason for having different shape parameters for the precisions α_n in (6) is to promote the clustered pattern in the support set of the solution, where the pattern is learned the via measure of total variation on the supports defined in (4). In other words, we let each shape parameter a_n be controlled via the estimated (ΣΔ). For this purpose, we add another hyper-prior to our model as follows

$$a_n~\text{Gamma}(a_n;g_n,h),\forall n=1,\dots ,N,$$ (7)

where

$$g_n:=\theta exp\{((\mathrm{\sum }\mathrm{\Delta })-{(\mathrm{\sum }\mathrm{\Delta })}_{n,0})\},$$ (8)

where (ΣΔ) is the initial measure of the clumpiness based on the available erroneous support set ℘ and is computed from (4), and (ΣΔ)_n,0 is the measure of clumpiness when forcing x_n = 0. In (8), θ is an emphasizing parameter on the amount of clumpiness over the supports of the solution and one can make it depend on the ratio M/N. This means that when the number of measurements is very low, we may not wish to emphasize on the clustered solutions. Otherwise, the algorithm may also remove some of the true supports in the set ℘ due to the small number of measurements and the lack of information on the full true support set.

Remark

In the simulations, we set h = 1 but in general, “h” is selected based on the belief on the maximum permissible amplitude of the elements of x.

Finally, the prior on the noise is defined as

$${\sigma }^2~\mathcal{N}(0,{\gamma }^{-1}),\gamma ~\text{Gamma}(c,d),$$ (9)

where we set c = d = 10⁻⁴ as suggested in [9].

According to the above prior distributions, the joint probability distribution of our model becomes

$$P(\mathbf{y},\mathbf{x},\mathit{\alpha },\mathbf{a},\mathbf{b},{\sigma }^2,\gamma )\propto P(\mathbf{y}\mid \mathbf{x},{\sigma }^2{I}_M)P(\mathbf{x};0,{\mathit{\alpha }}^{-1})\times P(\mathit{\alpha };\mathbf{a},\mathbf{b})P(\mathbf{a};\mathbf{g},h)p({\sigma }^2;0,{\gamma }^{-1})p(\gamma ;c,d)\prod _{n\in \mathcal{P}}P(b_n;p,q),$$ (10)

where the measurement noise is assumed to be e ~ 𝒩(0, σ²I_M).

According to (10), the marginalized posterior distributions for the variables of interest can be represented as follows. In these descriptions, conditioning on −, as in (x_n|−), denotes the inference on x_n conditioning upon all relevant variables and the observations.

• P(x|−) ∝ P(y|x, σ²I_M)P(x|α⁻¹). Therefore,

  $$(\mathbf{x}\mid -)~\mathcal{N}({\mathit{\mu }}_{x\mid -},{\mathrm{\sum }}_{x\mid -}),$$ (11)
  where

  $${\mathit{\mu }}_{x\mid -}=\gamma {\mathrm{\sum }}_{x\mid -}{A}^T\mathbf{y},{\mathrm{\sum }}_{x\mid -}={(\gamma A^{T}A+D)}^{-1},$$

  where D is a diagonal matrix with α as its main diagonal i.e., [D]_n,n = α_n.

• P(α|−) ∝ P(x|α⁻¹)P(α; a, b). Therefore,

  $$({\alpha }_n\mid -)~\{\begin{array}{l}\text{Gamma}({\overline{a}}_n,b_n+\frac{{\mu }_{x_n\mid -}^2+{\sigma }_{x_n\mid -}^2}{2}),\text{if}n\in \mathcal{P}\hfill \\ \text{Gamma}({\overline{a}}_n,{10}^{-15}+\frac{{\mu }_{x_n\mid -}^2+{\sigma }_{x_n\mid -}^2}{2}),\text{otherwise}\hfill \end{array},$$ (12)

  In the above equation, ${\overline{a}}_n:=a_n+\frac{1}{2}$ and ${\sigma }_{x_n\mid -}^2:={\alpha }_n^{-1}$.

• P(a_n|−) ∝ P(α_n; a_n, b_n)P(a_n|g_n, h), ∀n = 1,…, N. Hence

  $$(a_n\mid -)~\text{Gamma}(g_n,h+log{\alpha }_n),\forall n=1,\dots ,N,$$ (13)

  where $g_n=\theta exp\{{(\widehat{\mathrm{\sum }\mathrm{\Delta }})}^{[k]}-{(\widehat{\mathrm{\sum }\mathrm{\Delta }})}_{n,0}^{[k]}\}$ and k denotes the kth iteration and ( $\widehat{\mathrm{\sum }\mathrm{\Delta }}$) is the estimated measure of clumpiness.

• P(b_n|−) ∝ P(α_n; a_n, b_n)P(b_n; p, q), ∀n ∈ ℘. As a result,

  $$(b_n\mid -)~\text{Gamma}(p,q+{\alpha }_n),\forall n\in \mathcal{P}.$$ (14)
• Finally, P(γ|−) ∝ P(y|x, γ)P(γ; c, d) and therefore,

  $$(\gamma \mid -)~\text{Gamma}\left(c+\frac{M}{2},d+\frac{1}{2}({\Vert \mathbf{y}-A{\mathit{\mu }}_{x\mid -}\Vert }_2^2+\text{tr}(A^{T}A{\mathrm{\sum }}_{x\mid -}))\right).$$ (15)

3. SIMULATION RESULTS

In this section, we demonstrate the performance of our algorithm compared to SA-SBL algorithm proposed in [9]. For simulation purposes, the supports of the solution are randomly drawn from a Bernoulli distribution in such a way to exhibit a random clustered-sparsity pattern. In all the simulations, the number of true supports is set to |𝒯| = 25. The non-zero elements of solution vector x, corresponding to true supports, are drawn i.i.d. from Gaussian distribution with zero-mean and variance ${\sigma }_x^2=1$. The sensing matrix A is M × 100 and is randomly drawn from 𝒩(0, 1) and then it is normalized with respect to its columns. We vary the number of measurements M from 5 up to N to evaluate the performance. The elements of the noise component are drawn i.i.d. from $e_n~\mathcal{N}(0,{\sigma }_n^2)$. In all the simulations, we set SNR= 25 dB. Finally, the measurement vector y is computed from y = Ax+e. For each run we randomly select 80% of the true supports and collect them in the set 𝒮. Then, 10 more indexes that are not in the true supports are randomly chosen and collected into ℰ. We then feed both our algorithm and SA-SBL with an erroneous support set ℘ = {𝒮 ∪ ℰ}. Neither of the algorithms is aware of the number of true supports in set ℘. For each value of M, we run 500 random cases and then average over all of the obtained results both for our algorithm and SA-SBL.

In Fig. 1, we compare the detection rate (P_D) in the support recovery between our algorithm and SA-SBL.

Figure 1.

Detection rate comparison.

It can be seen in Fig. 1 that when the ratio M/N = 0, the detection rate P_D = 0.8 and it is due to the fact that we provided both algorithms with 80% of the true supports. SA-SBL algorithm provides better performance compared to CSA-SBL algorithm and the reason is it does not wish to remove its prior knowledge. It worth noting that statistically speaking, CSA-SBL provides better performance in cases of having less prior knowledge on the supports.

In Fig. 2, we compare the false alarm rate (P_FA) in the support estimation between our algorithm and SA-SBL. According to the results shown in Fig. 2, we see that our algorithm has a very low false alarm rate, much lower than the one obtained from SA-SBL. The performance is essentially an evidence of the role of incorporating the measure of (ΣΔ) to account for the unknown clustering pattern in the solution. In other words, our algorithm is able to filter out those estimated supports, that cause less clustering solutions among other possible solutions.

Figure 2.

False alarm rate comparison.

In Fig. 3 we illustrate another measure of performance which is based on the difference between the detection and false alarm rate in the estimated support set. This figure essentially illustrates the overall performance of the algorithms based on P_D, P_FA, and M/N, all together. Finally, in Fig. 4, we demonstrate the performance of our modified version of SA-SBL compared to SA-SBL in terms of success rate in estimating the true clustering pattern. Our measure of success, P_r, is defined as follows

Figure 3.

An overall measure of performance in terms of both detection rate and false alarm rate.

Figure 4.

Comparison in terms of pattern recovery.

$$P_r=1-\frac{\mid (\widehat{\mathrm{\sum }\mathrm{\Delta }})-(\mathrm{\sum }\mathrm{\Delta })\mid }{(\mathrm{\sum }\mathrm{\Delta })},$$

where (ΣΔ) and ( $\widehat{\mathrm{\sum }\mathrm{\Delta }}$) are the total number of transitions in the true support set and the estimated solution, respectively. It is clear from Fig. 4 that our propose modified version of SA-SBL (CSA-SBL) outperforms SA-SBL and that is because of the role of incorporating the measure of clumpiness in our algorithm.

Finally, we also compare the performance of SA-SBL and CSA-SBL algorithms via the following example. In this case we use the image of 112 × 200 pixels as illustrated in Fig. 5. In this image, we treat the black pixels as the “interesting” locations.

Figure 5.

True Image.

For the simulation purposes, we assign the value of 1 to the pixels with black color and 0 to the white ones. We then, assume that the matrix corresponding to the image is the true solution matrix X ∈ R^112×200 of an MMV problem, where the supports of the solution (sources) slowly change across the columns of X. We construct the sensing matrix in the same way we described earlier in this section. The measurements are obtained from Y = AX+E with SNR=25 dB. We then break the problem into M = 112 SMVs, where the available information to the “m”th SMV is matrix A, the mth column of Y, and the estimated supports obtained from solving the SMV corresponding to the (m − 1)st column of X. As illustrated in Fig. 5, the support set changes slowly across the columns of X. Therefore, even if we were provided with the true supports of the (m − 1)st column of X, the information still would become the erroneous partial support set that was available for solving the “m”th SMV. In order to solve the SMV corresponding to the first column of X, we feed both SA-SBL and CSA-SBL with a union of 80% of the true supports of the column and 10 randomly chosen incorrect supports. In the simulations, we set the threshold to T = 0.4 for both SA-SBL and CSA-SBL, meaning that we discarded the estimated supports with corresponding absolute value of less than 0.4. The reconstructed images for the sampling ratio of M/N = 0.3, 0.35, 0.4, 0.45, and 0.5 are illustrated in Fig. 6 and Fig. 7. In order to show the quality of the reconstructed images we compare the obtained results based on peak-SNR (PSNR) as shown in Tab. 1. According to Tab. 1, we observe that CSA-SBL provides better reconstruction in terms of PSNR compared to SA-SBL algorithm.

Figure 6.

Results of reconstructed images for SA-SBL for different sampling ratios (M/N).

Figure 7.

Results of reconstructed images for CSA-SBL for different sampling ratios (M/N).

Table 1.

NMSE and PSNR comparison in image reconstruction.

	SA-SBL	CSA-SBL
Sampling ratio (M/N)	PSNR (dB)	PSNR (dB)
M/N = 0.30	12.8467	13.5155
M/N = 0.35	15.1456	15.8384
M/N = 0.40	16.4610	18.1877
M/N = 0.45	18.6170	30.9498
M/N = 0.50	Inf	Inf

4. CONCLUSION AND ON-GOING WORK

We proposed a hierarchical Bayesian model to solve the clustered-pattern sparse signals via single measurement vector model in case where some erroneous information about the support set of the solution is available. The proposed algorithm is essentially a modified version of the conventional sparse Bayesian learning model and SA-SBL algorithm. Based on the simulation results, we demonstrated that making the shape parameter of the Gamma distribution in the SA-SBL algorithm for the problem improves the performance in the support recovery.