SPARSE BAYESIAN LEARNING USING VARIATIONAL BAYES INFERENCE BASED ON A GREEDY-BASED CRITERION

Extended Abstract

Compressive sensing (CS) is an evolving area in signal acquisition and reconstruction with many applications [1–3]. In CS the goal is to efficiently measure and then reconstruct the signal under the assumption that such signal is sparse but the number and location of non-zeros are unknown. A linear CS problem is modeled as y=Ax_s+e, where y ∈ ℝ^M contains measurements, x_s ∈ ℝ^N is the sparse solution, and e is the noise with M ≪ N [4–6]. A = ΦΨ, where Φ is the sensing matrix and Ψ is a proper basis in which x_s is sparse. There are three approaches to solve for x_s i.e, greedy-based, convex-based, and sparse Bayesian learning (SBL) algorithms. Here, we consider the SBL approach. Specifically, we consider Gaussian-Bernoulli prior to promote sparsity in the solution and then use variational Bayes (VB) inference to estimate the variables and parameters of the model. In the Gaussian-Bernoulli model, the sparse solution is defined as x_s =(s∘x), where s is a binary support learning vector, x accounts for the values of the solution, and “∘” is the element-wise product [7, 8].

It turns out that using VB inference for CS problem has the over fitting issue mainly when the number of measurements are low. For example, for the CluSS-VB algorithm, Yu et al. [8] pointed out that the solution may tend to become non-sparse. In this work, we propose a VB-based SBL algorithm which uses a simple criterion to remove such effect and forces the solution to become sparse. We also discuss and compare the update rules obtained from the SBL using fully hierarchical Bayesian approach via Markov chain Monte Carlo (MCMC) [7], expectation-maximization (EM) algorithm, and the VB inference. As expected, there exist a very close relationship between all these algorithms and we provide some intuition on how to turn equations of one approach to another approach. Also, we will provide some simulation results to compare the performance of such algorithms for the CS problems.

G-OSBL (VB): An SBL algorithm using VB

Suppose there is a model with parameters Θ, hidden variables collected in x, and a set of observations denoted by y. Then, the approximation to the joint density p(x,Θ|y) can be represented by p(x,Θ|y) ≈ q_x(x)q_θ(Θ). Then, the lower bound on the model log marginal likelihood can be iteratively optimized by the following updates [9]

$$q_x^{[t+1]}(\mathbf{x})\propto exp\{{\mathrm{E}}_{q_{\theta }^{[t]}}[logp(\mathbf{x},\mathbf{y}\mid \mathrm{\Theta })]\}$$ (1)

$$q_{\mathrm{\Theta }}^{[t+1]}(\mathrm{\Theta })\propto p(\mathrm{\Theta })exp\{{\mathrm{E}}_{q_x^{[t]}}[logp(\mathbf{x},\mathbf{y}\mid \mathrm{\Theta })]\}.$$ (2)

For the inverse CS problem, one can define a set of priors as follows [7]. We model the elements of the binary vector “s” as Bernoulli random variables, whose probabilities are governed by the prior γ; that is,

$$s_n~\text{Bernoulli}({\gamma }_n),{\gamma }_n~\text{Beta}({\alpha }_0,{\beta }_0),n=1,\dots ,N.$$ (3)

In order to promote sparsity structure in s, we set α₀≪β₀. The solution-value vector components are assumed to be distributed as i.i.d. normal-gamma distribution

$$\mathbf{x}~\mathcal{N}(0,{\tau }^{-1}{I}_N),\tau ~\text{Gamma}(a_0,b_0),$$ (4)

where a₀ and b₀ denote the shape and rate of the Gamma distribution, respectively. The entries of the noise component e are assumed outcomes of an i.i.d. Gaussian distribution with the precision ε.

$$\mathbf{e}~\mathcal{N}(0,{\epsilon }^{-1}{I}_M),\epsilon ~\text{Gamma}({\theta }_0,{\theta }_1).$$ (5)

Using the VB algorithm defined in (1) and (2), the update rule of the variables and parameters of the model can be simplified as follows.

• Update rule for the support learning vector s

  $$\left(s_n\mid -\right)~\text{Bernoulli}\left(Q_{s_n}\right),Q_{s_n}=\frac{1}{1+c_n{\kappa }_n},$$
  and therefore

  $${\stackrel{\sim }{s}}_n=Q_{s_n},\forall n=1,\dots ,N,$$ (6)
  where

  $$\begin{gathered} c_n:=e^{\psi ({\beta }_{1,n})-\psi ({\alpha }_{1,n})}, \\ {\kappa }_n:=e^{\frac{1}{2}\stackrel{\sim }{\epsilon }\left({\Vert {\mathbf{a}}_n\Vert }_2^2({\stackrel{\sim }{x_n}}^2+{\sigma }_{\stackrel{\sim }{x_n}}^2)-2\stackrel{\sim }{x_n}{\mathbf{a}}_n^T<{\stackrel{\sim }{\mathbf{y}}}^{-n}{>}_x\right)}, \end{gathered}$$

  and $<{\stackrel{\sim }{y}}_m^{-n}{>}_x=y_m-{\sum }_{l\ne n}^N{a}_{ml}{\stackrel{\sim }{s}}_l{\stackrel{\sim }{x}}_l$.

• Update rule for the solution value matrix x

  $$\begin{gathered} (\mathbf{x}\mid -)~\mathcal{N}(\stackrel{\sim }{\mathbf{x}},{\mathrm{\sum }}_{\stackrel{\sim }{x}}),\text{where} \\ {\mathrm{\sum }}_{\stackrel{\sim }{x}}={\left(\stackrel{\sim }{\tau }{I}_N+\stackrel{\sim }{\epsilon }\stackrel{\sim }{\mathrm{\Phi }}\right)}^{-1}\text{and}\stackrel{\sim }{\mathbf{x}}=\stackrel{\sim }{\epsilon }{\mathrm{\sum }}_{\stackrel{\sim }{x}}\text{diag}(\stackrel{\sim }{\mathbf{s}})A^T\mathbf{y}, \end{gathered}$$ (7)

  where Φ̃:=[(A^TA) ∘ (s̃s̃^T +diag(s̃ ∘ (1 − s̃)))].

• Update rule for γ_n

  $$({\gamma }_n\mid -)~\text{Beta}({\alpha }_{1,n},{\beta }_{1,n}).$$
  Therefore,

  $$\stackrel{\sim }{{\gamma }_n}=\frac{{\alpha }_{1,n}}{{\beta }_{1,n}},\forall n=1,\dots ,N,$$ (8)

  where ${\alpha }_1:={\alpha }_0+\stackrel{\sim }{s_n}$ and β₁ := β₀ + 1 − s̃_n.

• Update rule for the noise precision

  $$\stackrel{\sim }{\epsilon }=\frac{{\theta }_0+\frac{M}{2}}{{\theta }_1+\frac{1}{2}\stackrel{\sim }{\mathrm{\Psi }}},$$ (9)
  where

  $$\stackrel{\sim }{\mathrm{\Psi }}:=\left({\mathbf{y}}^T\mathbf{y}-2{(\stackrel{\sim }{\mathbf{x}}\circ \stackrel{\sim }{\mathbf{s}})}^T{A}^T\mathbf{y}+\text{tr}((\stackrel{\sim }{\mathbf{x}}\stackrel{\sim }{\mathbf{x}}+{\mathrm{\sum }}_{\stackrel{\sim }{x}})\stackrel{\sim }{\mathrm{\Phi }})\right).$$
• Update rule for the solution precision

  $$(\tau \mid -)~\text{Gamma}\left(a_0+\frac{N}{2},b_0+\frac{1}{2}[{\Vert \stackrel{\sim }{\mathbf{x}}\Vert }_2^2+\text{tr}({\mathrm{\sum }}_{\stackrel{\sim }{x}})]\right),$$
  where ${\mathrm{\sum }}_{\stackrel{\sim }{x}}=\text{diag}({\sigma }_{\stackrel{\sim }{x_1}}^2,\dots ,{\sigma }_{\stackrel{\sim }{x_N}}^2)$. Therefore,

  $$\stackrel{\sim }{\tau }=\frac{a_0+\frac{N}{2}}{b_0+\frac{1}{2}[{\Vert \stackrel{\sim }{\mathbf{x}}\Vert }_2^2+\text{tr}({\mathrm{\sum }}_{\stackrel{\sim }{x}})]}.$$ (10)

The stopping criterion of the algorithm can be then made based on the log-marginalized likelihood defined as L := log {p(y|x, s, ε, τ)}. The marginalized likelihood is defined as p(y|x, s, ε, τ) = ∫ p(y|x, s, ε)p(x|τ)dx. After some simplification, the negative log-likelihood will be proportional to −L ∝ log |Σ⁻¹| + y^TΣy, where Σ = (ε⁻¹I_M + τ⁻¹ AS̃²A^T)⁻¹, and S̃ := diag(s̃). Therefore, the stopping condition can be defined as [8]

$$\frac{\mid L^{[t+1]}-L^{[t]}\mid }{\mid L^{[t]}\mid }\le \text{Threshold},$$

for some small value of “Threshold”.

Although for a large number of measurements the above algorithm has a good performance, it turns out that when the number of measurements decreases, the solution becomes very non-sparse in terms of support recovery, while the estimated solution vector $\stackrel{\sim }{{\mathbf{x}}_s}$ becomes almost zero and noninformative. We illustrate this over fitting issue via some examples in Fig. 1. In the simulations, we generated a trial, where the true solution vector is x∈R¹⁰⁰ with 25 non-zero elements. The supports of the true solution are randomly drawn so that the support set of the solution exhibits a clustered pattern. The non-zero values of the solution are drawn i.i.d. from a zero-mean Gaussian distribution with variance of 1. Each entry of the sensing matrix A is an outcome of a Gaussian distribution 𝒩(0, 1). After constructing A, we normalize A with respect to its columns. In Fig. 1, we show the performance for the cases where the number of measurements is set to 85, 70, and 65, respectively. The elements of measurement noise are drawn i.i.d. from 𝒩 (0, σ²) with SNR=25 dB, where SNR:=20 log₁₀(σ_x/σ).

Figure 1.

From top row to the bottom row, we illustrate the results when the sampling ratio is 0.85, 0.7, and 0.65, respectively.

Table Tab. 1 shows the obtained detection rate, false alarm rate, and the normalized mean-squared error (NMSE) between the true and the estimated solution.

Table 1.

Investigating the performance of O-SBL(VB).

Sampling ratio	Detection rate	False alarm rate	NMSE (dB)
M/N = 0.85	0.8	0.00	−13.4015
M/N = 0.7	0.68	0.0133	−10.6168
M/N = 0.65	1.00	1.00	−0.1031

As can be seen from Fig. 1 and Tab. 1, the SBL using VB fails to estimate the solution once the sampling ratio decreases. The reason is essentially due to the overfitting problem since there are many parameters to be learned via low number of measurements and the update rule of the support learning vector s is obtained from a soft thresholding rather than the outcome a Bernoulli distribution for the posterior distribution found in the ordinary sparse Bayesian learning (SBL) approach. Due to the overfitting issue, the estimated precision on the components of the solution increased in our simulations, and as a result the energy of the estimated solution became almost negligible. Notice that, there is a chance to remove this issue by smartly selecting the hyper-parameters on the Gamma prior distribution which has a role on governing the distribution on the supports of the solution. But, when having no information on the support set, even the setting of the hyper-parameters becomes challenging. The main issue of the failures can be found in the update rule of the support learning vector s̃ defined in (6). It becomes important to balance between the terms c_n and κ_n, where c_n imposes the effect of hyperprior on s accompanied by the current estimate of s_n, and κ_n imposes the contribution of the current estimates of noise precision, solution, and other supports in fitting the model to the measurements. Therefore, if we impose a strong effect on the sparsity via c_n, then the solution tends to neglect the effect of κ_n and vice versa. To tackle this issue, we combine the intuition behind the SBL approach with the greedy algorithm of orthogonal basis pursuit (OMP). At each iteration, OMP makes decisions on the supports based on the usefulness defined as the normalized version of the inner product between the residual and the columns of the sensing matrix. The problem with the OMP is that it is sensitive to noise. Also, when the number of measurements become close to the dimension of the signal, the OMP solution becomes nonsparse unless some prior knowledge on the sparsity level of the signal in fed to the algorithm.

The main approach for solving the CS problem is as follows. We need to learn where the non-zero locations are and then what value the active locations possess. If we knew the active locations, then it would be easy to project the original problem to a lower dimension and estimate the sparse signal (e.g., with least-squares), and then based on the nonzero locations turn the obtained estimate of the signal to the original dimension. Suppose that there are K non-zero locations and the length of signal is N, where K ≪ N. If the locations were known, then K measurements would suffice to solve the problem. Since we do not know the non-zero locations, we expect to have more than K measurements i.e., with M greater than K. Therefore, it is not possible to do sparse recovery with M measurements less than K i.e., K number of measurements is the lowest possible value that the algorithm needs to be able to perform exact sparse signal recovery in the ideal case. Since the sparsity level is unknown in almost all real compressive sensing problems, we avoid this issue by rejection of having more than M supports in the iterative estimates of s, meaning that no matter how many more than M non-zero locations exist in the true solution, we put effort on finding only M because we do not have more resources to find more. Even for the case where the true number of supports is less than M, we would still let the algorithm decide on the M most probable supports and then let the algorithm figure out the contribution of each remained estimated support. In this case, the algorithm is given more chance to avoid the over fitting problem, since it will work on the lower dimension (M), where fewer parameters are left to be learned.

The proposed approach can be explained as follows. We first estimate the contribution of each location in fitting the model to the measurements using SBL-VB defined in (6). Then, we only keep those locations with the M highest contribution in the support set and discard the other estimated supports. In this way, we can remove the issue of non-sparse s based on our available resources. It is worth mentioning that in our proposed algorithm we first estimate the supports, then we perform the aforementioned post-processing on the supports, and then update the other variables and parameters of the model. In this way, we remove the misleading effect of changing the estimates of the supports after updating the other parameters and variables. The pseudocode of the proposed algorithm is as follows.

G-OSBL(VB) Algorithm:
[x̂, ŝ] = G-OSBL(Y,A, Iter)
iter= 0
While $\frac{\mid L^{[\text{Iter}]}-L^{[\text{Iter}-1]}\mid }{\mid L^{[\text{Iter}-1]}\mid }\ge {10}^{-6}$
% Support-learning vector component
Compute $\stackrel{\sim }{s_n}$ from (6), ∀n = 1,…, N
% Post processing on s̃
% Keep only the M supports with the highest contribution
[−, Indx]=sort (s̃, “Ascend”)
s̃(1 : N − M) = 0
% Solution-value matrix component
Compute Σx̃ and x̃ from (7)
% Parameters of mixing coefficients
Compute α1 and β1 from (8)
% Precision on the solution components
Compute τ from (10)
% Precision on the measurement noise
Compute ε from (9)
End While

In order to show the effect of the post filtering over the estimated s, in Fig. 2 we illustrate the performance of GOSBL algorithm with the same x, s, y, A, and noise that was used in Fig. 1 for the case of M/N = 0.65.

Figure 2.

Results of G-OSBL(VB) algorithm when the sampling ratio is 0.65.

According to Fig. 2, the algorithm was able to avoid the overfitting problem and provided a good performance in sparse signal recovery. In Tab. 2 we show the obtained detection rate and false alarm rate, and the NMSE using G-OSBL(VB). Comparing Tab. 2 with Tab. 1, we see the improvement in the performance when using G-OSBL(VB).

Table 2.

Investigating the performance of G-OSBL(VB).

Sampling ratio	Detection rate	False alarm rate	NMSE (dB)
M/N = 0.85	0.84	0.00	−20.8901
M/N = 0.7	0.76	0.0133	−11.5602
M/N = 0.65	0.76	0.0133	−8.2291

O-SBL, O-SBL(EM), and O-SBL(VB) Duality

In this section we provide the update rules for the parameters of the O-SBL algorithm proposed in [7], the EM-based version of the O-SBL, and the O-SBL based on variational Bayes inference. In the original O-SBL [7], we usedMCMC inference implemented via Gibbs sampler in order to approximate the posterior of the model variables and parameters. Tables Tab. 3–5 describe the obtained rules.

Table 3.

Comparing update rules of the noise precision.

O-SBL	$\epsilon =\text{Gamma}({\theta }_0+\frac{M}{2},{\theta }_1+\frac{1}{2}{\Vert \mathbf{y}-A(\mathbf{s}\circ \mathbf{x})\Vert }_2^2)$
O-SBL(EM)	$\stackrel{\sim }{\epsilon }=\frac{{\theta }_0+\frac{M}{2}-1}{{\theta }_1+\text{trace}\left(\mathbf{y}{\mathbf{y}}^T-A(\stackrel{\sim }{\mathbf{s}}\circ {\mathit{\mu }}_x){\mathbf{y}}^T+\frac{1}{2}A(\stackrel{\sim }{\mathbf{s}}\circ ({\mathrm{\sum }}_x+{\mathit{\mu }}_x{\mathit{\mu }}_x^T))A^T\right)}$
O-SBL(VB)	$$\begin{gathered} \epsilon ~\text{Gamma}({\theta }_0+\frac{M}{2},{\theta }_1+\frac{1}{2}\stackrel{\sim }{\mathrm{\Psi }}),\stackrel{\sim }{\epsilon }=\frac{{\theta }_0+\frac{M}{2}}{{\theta }_1+\frac{1}{2}\stackrel{\sim }{\mathrm{\Psi }}}, \\ \stackrel{\sim }{\mathrm{\Psi }}:=\left({\mathbf{y}}^T\mathbf{y}-2{(\stackrel{\sim }{\mathbf{x}}\circ \stackrel{\sim }{\mathbf{s}})}^T{A}^T\mathbf{y}+\text{tr}((\stackrel{\sim }{\mathbf{x}}\stackrel{\sim }{\mathbf{x}}+{\mathrm{\sum }}_{\stackrel{\sim }{x}})\stackrel{\sim }{\mathrm{\Phi }})\right) \end{gathered}$$

Table 5.

Comparing update rules of the mixing coefficients.

O-SBL	γn ~ Beta(a0 + sn, b0 + 1 − sn)
O-SBL(EM)	${\stackrel{\sim }{\gamma }}_n=\frac{{\alpha }_0+{\stackrel{\sim }{s}}_n-1}{{\alpha }_0+{\beta }_0}$
O-SBL(VB)	${\gamma }_n~\text{Beta}(a_0+\stackrel{\sim }{s_n},b_0+1-{\stackrel{\sim }{s}}_n),{\stackrel{\sim }{\gamma }}_n=\frac{{\alpha }_0+{\stackrel{\sim }{s}}_n}{{\alpha }_0+{\beta }_0+1}$

According to Tab. 3–5, there is a very close relationship between the update rules in all three aforementioned approaches for the posterior estimates. More specifically, the update rules of EM approach are essentially the same as the variational Bayes estimates, on average. In each update rule, both algorithms incorporate the variations of the estimates of the other related parameters and variables, as well. Similarly, variations are considered in the Gibbs method due to the fact that the estimates of the other parameters and variables in each update rule are drawn from their corresponding approximated posteriors. For investigating the performance of all of these three approaches in CS, we will also apply these approaches to a set of simulated data.

Conclusion and Work In Progress

By combining variational Bayes inference and a greedy-based criterion, we proposed G-OSBL algorithm to perform sparse signal recovery in the compressive sensing problem. Initial simulations showed encouraging performance in detecting the supports and diminishing the over-fitting issue of the model parameters when having low number of measurements. In this case, we will provide a comprehensive study in comparing the overall performance of the algorithm compare to the other existing algorithm in the CS area. Furthermore, we showed the duality between a hierarchical SBL, the EM-based version, and variational Bayes inference approach algorithms. We will also include a performance comparison of the approaches on a simulated data.

Table 4.

Comparing update rules of the solution precision.

O-SBL	$\tau =\text{Gamma}(a_0+\frac{N}{2},b_0+\frac{1}{2}{\Vert \mathbf{x}\Vert }_2^2)$
O-SBL(EM)	$\stackrel{\sim }{\tau }=\frac{a_0+\frac{N}{2}-1}{b_0+\frac{1}{2}\text{trace}({\mathrm{\sum }}_x+{\mathit{\mu }}_x{\mathit{\mu }}_x^T)}$
O-SBL(VB)	$$\begin{gathered} \tau ~\text{Gamma}\left(a_0+\frac{N}{2},b_0+\frac{1}{2}({\Vert \stackrel{\sim }{\mathbf{x}}\Vert }_2^2+\text{trace}({\mathrm{\sum }}_{\stackrel{\sim }{x}}))\right) \\ \stackrel{\sim }{\tau }=\frac{a_0+\frac{N}{2}}{b_0+\frac{1}{2}\left({\Vert \stackrel{\sim }{\mathbf{x}}\Vert }_2^2+\text{trace}({\mathrm{\sum }}_{\stackrel{\sim }{x}})\right)} \end{gathered}$$