Markov Chain Monte Carlo Inference of Parametric Dictionaries for Sparse Bayesian Approximations

Abstract

Parametric dictionaries can increase the ability of sparse representations to meaningfully capture and interpret the underlying signal information, such as encountered in biomedical problems. Given a mapping function from the atom parameter space to the actual atoms, we propose a sparse Bayesian framework for learning the atom parameters, because of its ability to provide full posterior estimates, take uncertainty into account and generalize on unseen data. Inference is performed with Markov Chain Monte Carlo, that uses block sampling to generate the variables of the Bayesian problem. Since the parameterization of dictionary atoms results in posteriors that cannot be analytically computed, we use a Metropolis-Hastings-within-Gibbs framework, according to which variables with closed-form posteriors are generated with the Gibbs sampler, while the remaining ones with the Metropolis Hastings from appropriate candidate-generating densities. We further show that the corresponding Markov Chain is uniformly ergodic ensuring its convergence to a stationary distribution independently of the initial state. Results on synthetic data and real biomedical signals indicate that our approach offers advantages in terms of signal reconstruction compared to previously proposed Steepest Descent and Equiangular Tight Frame methods. This paper demonstrates the ability of Bayesian learning to generate parametric dictionaries that can reliably represent the exemplar data and provides the foundation towards inferring the entire variable set of the sparse approximation problem for signal denoising, adaptation and other applications.

I. Introduction

Signal representations are fundamental for denoising, interpolation, estimation, classification and recognition. Recently there has been an increased focus on sparse representations [1] which model a signal with a small number of components from a large overcomplete set of exemplar atoms, called dictionary. A dictionary D ∈ ℜ^P×K contains K atoms d₁, …, d_K ∈ ℜ^P that constitute the building blocks of a P- dimensional signal x ∈ ℜ^P. Specifically a signal can be expressed as an exact or approximate linear combination of a small number of atoms from the dictionary as x ≈ Dc, where c ∈ ℜ^K contains the coefficients of the corresponding atoms. In the typical case where K > P, an infinite set of solutions arise. This can be addressed by imposing a sparsity constraint on x, according to which x should be represented by the smallest number of dictionary atoms and the sparse representation problem can be expressed as an l0-norm minimization.

The problem of minimizing ‖c‖₀ subject to the constraint x = Dc has been proven to be NP-hard and several directions have been proposed to solve it. One approach includes greedy strategies that abandon exhaustive search in favor of locally optimal updates resulting in sub-optimal solutions. Examples include matching pursuit [2] and orthogonal matching pursuit [3], [4] algorithms. An alternative is the relaxation of the discontinuous l0-norm leading to the more computationally expensive basis pursuit [5] and focal underdetermined system solver [6], [7] reaching global solutions. Bayesian methods with appropriate statistical assumptions have been further used to identify the desired sparse solution [8], [9], [10].

An essential step towards compact and reliable representations is the dictionary selection. Traditionally, analytic predesigned dictionaries comprising Gabor [11], wavelet [12], curvelet [13], or other atoms have been used, because of their localization, directionality and multi-resolution properties. Dictionary learning (DL) focuses on learning atoms from the available training data. It includes several well-known algorithms, such as the K-SVD [14] and the MOD [15], as well as probabilistic approaches [16]. Although non-parametric DL is effective for signal reconstruction [14], restoration [17], and classification [18], it depicts a highly non-convex nature, mostly yields non-structured dictionaries [19] and typically requires a large amount of training data [20].

These disadvantages can be mitigated by imposing a pre-determined structure through the use of carefully selected knowledge-driven parametric functions mapping a parameter space to structured dictionary atoms. This results in parametric dictionaries bridging the gap between pre-defined analytic dictionaries and purely numerical DL [21]. Dictionary atoms are expressed through an application-specific function ϕ of a parameter set, say d_k = ϕ(θ_k), where θ_k ∈ ℜ^Q, Q < P, are the atom parameters optimized with respect to desirable properties [22], [23], [24]. Parametric DL is more likely to converge faster and have more efficient implementations compared to the non-parametric problem [19]. It further provides higher signal interpretability yielding important metainformation [25], [26], [27], [28].

This paper proposes a Bayesian framework for learning the parameters of a dictionary given a predetermined parametric function. The formulation of the sparse representation problem from the Bayesian perspective assumes a probabilistic distribution for all variables in an effort to provide a posterior belief for their values [10], [29]. This allows the estimation of full posteriors rather than single estimates which can result in better handling uncertainty and benefits noise estimation.

Related Work

DL methods usually alternate between sparse decoding and dictionary update [19]. In the context of non-parametric DL, Ophir et al. have proposed a sequential learning algorithm by identifying the orthogonal directions to a data subset [30]. Engan et al. used matrix inversion to compute the dictionary matrix [15], while Aharon et al. introduced K-SVD, which is a constrained optimization approach performing atom-by-atom update [14]. Generalized principal component analysis models the data as a union of low-dimensional subspaces with orthogonal bases [31], while structured dictionaries have been proposed in an effort to enforce additional translation invariant, hierarchical and multiscale properties [32], [33], [34].

Parametric dictionaries depict higher interpretability, lower density of local minima and compact representation [19]. Dictionaries can be learnt as a result of translation of elementary signal segments over space and time [35], [36]. They can also contain atoms of predetermined structure, such as wavelets [37] or Gabor functions [22], whose parameters are adapted with steepest-descent (SD) [22] and other least-squares-based methods [25], [37]. Yanghoobi et al. proposed a method to find dictionaries close to the one with minimum coherence, called the equiangular tight frame (ETF) [24]. Finally, Thanou et al. [21] proposed a parametric DL for signals residing on weighted graphs.

Although the aforementioned deterministic DL methods perform well in various signal processing applications [14], [17], [18], they provide single-point estimates and cannot handle noise uncertainty. Bayesian approaches offer a way to address those disadvantages. They have been proposed for compressed sensing [8], [9], [38], [39] as well as for non-parametric DL.

The problem of ensuring sparseness in Bayesian approximations has been addressed in a variety of ways. Early approaches have used continuous sparse-promoting distributions to the atom coefficients c_n, such as the Laplacian [40], Cauchy [16], [41] and Student-t [42], [10]. Another way is the use of indicator variables that permit to independently sample each atom from the Bernoulli distribution [43], [44], [45]. Despite their computational efficiency, these can yield an arbitrarily large amount of non-zero coefficients, which might not be practical for many applications. More inline with our problem, previous studies have explored the use of appropriate probabilistic assumptions for keeping a constant number of dictionary atoms in the representation with proposal distributions iteratively conditioning on the atoms selected at previous steps [39], [46], [38]. Gaussian assumptions are typically imposed on the dictionary atoms [43], [44] and the signal noise [38], [43], [44], [47], the latter being consistent with biomedical applications [48], [49].

Bayesian inference casts DL into an optimization problem for maximizing the posterior distribution. The overcomplete dictionaries containing many atoms in combination with the large amount of training data yield a high-dimensional framework, for which closed form solutions are usually difficult to derive and approximate inference methods are followed. Common approaches include the evidence maximization [8], [9], relevance vector machine [10], Markov Chain Monte Carlo (MCMC) [38], [44], and variational approximation [43].

Contributions

We propose a Bayesian framework for learning the parameters of dictionary atoms, given a parametric function that maps the dictionary parameters to the actual atoms. Our approach imposes probabilistic distributions to the variables of the sparse representation problem that are estimated through MCMC methods because of their simplicity and ability to fully perform Bayesian inference [50]. Compared to previous Bayesian DL [43], [44], our approach introduces parametric dictionaries where non-closed-form solutions are handled with a combination of Gibbs and Metropolis-Hastings (MH) sampling (MH-within-Gibbs). Our approach differs from previous parametric DL [22], [24] because of its stochastic framework that yields estimation of the full problem variables. This results in parametric dictionaries that take into account the structure of the training data and are less prone to overfitting. The parametric nature of our problem further requires the use of appropriate sparse-imposing priors that keep the selected number of dictionary atoms within a pre-determined range. We perform atom sampling with and without replacement formalized through the Multinomial and the Wallenius’ hypergeometric distribution, respectively.

One key challenge with MCMC is to determine its asymptotic behavior, i.e. whether it provides accurate posterior approximations. The goal is to create an aperiodic and irreducible Markov Chain (MC) with stationary distribution same as the posterior distribution of interest [51]. Irreducibility ensures that any state of the space is accessible, while aperiodicity makes sure that the chain does not return to the same state at regular times. Uniformly ergodic MCs are a special case in which the MC converges to the invariant distribution independently of the initial state. They guarantee geometrically fast convergence and are key sufficient conditions in order to establish central limit theorems for empirical averages and provide consistent estimators of MCMC standard errors [51], [52]. Because of these, we discuss the geometric ergodicity of MCMC in our proposed Bayesian inference framework that ensures convergence. We further perform qualitative and quantitative diagnostics to evaluate the reliability of the generated samples and resulting distributions.

We demonstrate the ability of our algorithm for parallel processing with experiments on synthetic data and real biomedical signals. DL is performed for each sample separately and the resulting dictionaries from each exemplar data are further combined into a unified model. Our results indicate that the proposed approach yields benefits in terms of superior signal reconstruction compared to previous SD [22] and ETF [24] methods. When we have precise a priori knowledge of the optimal parametric function ϕ representing the data, our parametric framework also yields better performance than the classical non-parametric approach of K-SVD [14].

In the following, we provide the problem formulation (Section II) and describe the proposed MCMC approach for learning the parameters for the considered problem (Section III). We describe the parallel implementation framework of our algorithm (Section IV) and discuss the use of various parametric functions and their effect on the sample-generating procedure (Section V). We further analyze the geometric ergodicity properties of the proposed MH-within-Gibbs algorithm yielding uniform convergence (Section VI). In Section VII, we provide experimental results and the results of MCMC diagnostics. Finally, we discuss our results and offer conclusions in Sections VIII and IX.

Notation

We denote matrices with bold uppercase letters X and vectors with bold lowercase letters x. Lowercase letters with appropriate numerical indices will either refer to the columns of a matrix, i.e. X = [x₁ … x_N] or the elements of a vector, i.e. x = [x₁ … x_N]^T. We denote (X)_ij the entry of matrix X corresponding to the i^th row and j^th column. The p^th order norm of a vector is symbolized as ‖x‖_p, while the N × 1 identity vector and N × N identity matrix are noted as 1_N and I_N, respectively. The vectorization of matrix X, obtained by stacking its columns on top of one another, is defined as vec(X) = [x₁^T, …, x_N^T]^T. Finally the gradient and Hessian of a scalar valued function f(x), x ∈ ℜ^N, are denoted as ∇f(x) ∈ ℜ^N×1 and H_f = ∇²f(x) ∈ ℜ^N×N, where ${(\nabla f(\mathbf{x}))}_i=\frac{\theta f}{\theta x_i}$ and ${(H_f)}_{ij}=\frac{{\vartheta }^{2}f}{\vartheta x_i\vartheta x_j}$, respectively.

II. Problem Formulation

Let X = [x₁ … x_N] ∈ ℜ^P×N be a data matrix of N examples x_n ∈ ℜ^P. We formulate the parametric DL problem by assuming an overcomplete dictionary D_n ∈ ℜ^P×K containing K prototypical atoms d_nk ∈ ℜ^P for each exemplar data x_n separately. This approach, also found in similar studies [43], enjoys computational benefits compared to batch methods, since it can yield faster reliable estimates of the considered variables (Section III-B) and can be easily parallelized to run on multiple computational threads (Section III-E).

In the case of parametric dictionaries, the atoms can be expressed with an appropriate domain-specific function ϕ : ℜ^Q → ℜ^P in terms of a parameter vector θ_nk ∈ ℜ^Q, Q < P, as d_nk = ϕ(θ_nk), therefore D_n = [ϕ(θ_n1) … ϕ(θ_nK)]. Typically dictionary atoms have a unitary l2-norm, i.e. ‖ϕ(θ_nk)‖₂ = 1. Each signal x_n can be expressed as a linear combination of a small number of atoms, L ≪ K, with additive noise ε_n ∈ ℜ^P

$${\mathbf{x}}_{\mathbf{n}}={\mathbf{D}}_{\mathbf{n}}{\mathbf{c}}_{\mathbf{n}}+{\mathit{\epsilon }}_{\mathit{n}}$$ (1)

where ε_n is the error and c_n ∈ ℜ^K, ‖c_n‖₀ = L, are the coefficients with non-zero values only for the used atoms. According to the Bayesian framework, each variable in (1) is assumed to follow an underlying probabilistic distribution.

Especially in parametric DL, where we jointly sample the parameters of the selected dictionary atoms (Section III-B), a small number of selected atoms can keep the implementation computationally tractable. For this reason, we are interested in imposing explicit sparseness constraints, similarly to previous studies [38], [39], [46] (Section I). We will describe two different ways to approach this using the Multinomial and the Wallenius’ hypergeometric distribution, that allow sampling with and without replacement.

A. Atom Sampling with Replacement

A straightforward method to sample the dictionary atoms is to relax the l0-sparsity norm constraint into ‖c_n‖₀ ≤ L allowing independent sampling of the dictionary atoms L times with replacement through the Multinomial distribution. Since the population size is much larger than the sample size (L≪K), duplicate atoms are rare [53]. If we assume a discrete multinomial distribution for selecting one dictionary atom out of the possible K, (1) can be re-written as

$${\mathbf{x}}_{\mathbf{n}}={\mathbf{D}}_{\mathbf{n}}\sum _{l=1}^L{s}_{nl}{\mathbf{z}}_{\mathbf{n}\mathbf{l}}+{\mathit{\epsilon }}_{\mathit{n}}$$ (2)

where z_nl ∈ ∪ u_i, u_i = [0, 0, …, 1, …, 0]^T with 1 in the i^th entry (i ≤ K) and 0 in the rest K−1 entries. The vector ‖z_nl‖₀ = 1 is binary activating one dictionary atom at a time and s_n = [s_n1 … s_nL]^T ∈ ℜ^L only contains the coefficients of the selected atoms. If atom d_k is the l^th representation term, then z_nlk′ = 1, z_nlk′ = 0, ∀k′ ≠ k, and s_nl consists the k^th entry of vector c_n in (1), i.e. s_nl = c_nk. The probability of selecting atom k for data x_n is π_nk such that z_nl ~ Multinomial (1, π_n) with π_n = [π_n1, …, π_nK]^T ∈ ℜ^K. If the same atom is selected more than once, the corresponding coefficient is only once estimated.

B. Atom Sampling without Replacement

Sampling without replacement avoids duplicate atoms and keeps the l0-sparsity constraint intact. The problem of selecting L atoms out of the possible K can be formalized similarly to the classical experiment of taking colored balls at random from an urn without replacement [54], [55]. If the balls have a different weight, the result follows the Wallenius’ noncentral hypergeometric distribution [56].

In the considered problem, we can assume K dictionary atoms each of a different type (i.e. each ball in the urn has a different color) and selection probability π = [π₁, …, π_K]. If z_n = [z_n1, …, z_nK] ∈ ℜ^K, z_nk ∈ {0, 1} and ‖z_n‖₀ = L, indicates the selected atoms for x_n, then (1) becomes

$${\mathbf{x}}_{\mathbf{n}}={\mathbf{D}}_{\mathbf{n}}({\mathbf{s}}_{\mathbf{n}}◦{\mathbf{z}}_{\mathbf{n}})+{\mathit{\epsilon }}_{\mathit{n}}$$ (3)

where “◦” represents the Hadamard or entrywise product and s_n ∈ ℜ^K and z_n follows the Wallenius distribution.

C. Additional Probabilistic Assumptions

We assume independent atom coefficients s_n1, …, s_nL each following a normal distribution with mean μ_snl and precision γ_s, i.e. ${\mathbf{s}}_{\mathbf{n}}~\text{Normal }({\mathit{\mu }}_{{\mathit{s}}_{\mathit{n}},}{\gamma }_s^{-1}{\mathbf{I}}_{\mathbf{L}})$. We have considered a different mean μ_snl for each exemplar data n and selected dictionary atom l to account for the various signal energy levels and possible atom configurations. We further hypothesize dictionary parameters of normal distribution θ_nk ~ Normal (g_nk, G_n⁻¹) with mean g_k and precision G_n. Finally, we assume zero mean Gaussian noise with variance ${\gamma }_{{\epsilon }_n}^{-1}$, i.e. ${\mathit{\epsilon }}_{\mathit{n}}~\text{Normal }(\mathbf{0},{\gamma }_{{\epsilon }_n}^{-1}{\mathbf{I}}_{\mathbf{Q}})$. Similar distributions have been also hypothesized in prior work [16], [41], [44] (Section I).

The Bayesian framework further treats the parameters of the above variables as random components in order to better capture uncertainty. We introduce conjugate prior distributions that simplify computations. Specifically, we assume that π_n follows a Dirichlet prior, i.e. π_nk ~ Dirichlet (α), where α = [a₁ … a_K]^T, and the precision of the Gaussian noise follows a Gamma prior, i.e. γ_εn ~ Gamma (e, f). The mean vector g_nk and precision matrix G_n of the dictionary atom parameters are modeled with Gaussian and Wishart distributions, respectively, i.e. g_nk ~ Normal (g₀, G₀⁻¹) and G_n ~ Wishart (ν₀, R₀).

D. Objective

The goal is to find ${\mathbf{y}}_{\mathbf{n}}^{*}$ such that

$${\mathbf{y}}_{\mathbf{n}}^{*}=\text{arg }\underset{{\mathbf{y}}_{\mathbf{n}}}{\text{max }}p({\mathbf{y}}_{\mathbf{n}}|{\mathbf{x}}_{\mathbf{n}},{\mathscr{H}}_n)$$ (4)

$${\mathbf{y}}_{\mathbf{n}}={[{{\mathbf{z}}_{\mathbf{n}}}^T,s_{n1},\dots ,s_{nL},{{\mathit{\theta }}_{\mathit{n}\mathbf{1}}}^T,\dots ,{{\mathit{\theta }}_{\mathit{n}\mathit{K}}}^T,{{\mathbf{g}}_{\mathbf{n}\mathbf{1}}}^T,\dots ,{{\mathbf{g}}_{\mathbf{n}\mathbf{K}}}^T,\text{vec}{({\mathbf{G}}_{\mathbf{n}})}^T,{{\mathit{\epsilon }}_{\mathit{n}}}^T,{{\mathit{\pi }}_{\mathit{n}}}^T,{\gamma }_{{\epsilon }_n}]}^T$$ (5)

$${\mathscr{H}}_n=\{\mathit{\alpha },{\mathbf{g}}_{\mathbf{0}},{\mathbf{G}}_{\mathbf{0}},{\nu }_0,{\mathbf{R}}_{\mathbf{0}},e,f,{\mathit{\mu }}_{{\mathit{s}}_{\mathit{n}}},{\gamma }_s\}$$ (6)

The probabilistic assumptions for (4)–(6) are summarized in Table I.

TABLE I.

Prior distributions of Bayesian dictionary learning variables.

Variable	Type	Expression	(Hyper) Parameters
znl† ∈ ∪ ui	Multinomial (1, πn)	${\prod }_{k=1}^K{\pi }_{nk}^{z_{n{l}_k}}$	πn: outcome probability
zn‡ ∈ ℜK	Wallenius (1K, L, πn)	${\int }_0^1{\prod }_{k=1}^K{(1-t^{{\pi }_{nk}/d})}^{z_{nk}}dt,d={\sum }_{k=1}^K{\pi }_{nk}(1-z_{nk})$
sn ∈ ℜL	Normal (μsn, ${\gamma }_s^{-1}{\mathbf{I}}_{\mathbf{L}}$)	${(2\pi )}^{-L/2}{\gamma }_s^{1/2}\text{exp }[-\frac{{\gamma }_s}{2}{({\mathbf{S}}_{\mathbf{n}}-{\mathit{\mu }}_{{\mathit{s}}_{\mathit{n}}})}^T({\mathbf{S}}_{\mathbf{n}}-{\mathit{\mu }}_{{\mathit{s}}_{\mathit{n}}})]$	μsn: mean γs: precision
θnk ∈ ℜQ	Normal (gnk, Gn−1)	${(2\pi )}^{-Q/2}|{\mathbf{G}}_{\mathbf{n}}{|}^{1/2}\text{exp }[-\frac{1}{2}{({\mathit{\theta }}_{\mathit{n}\mathit{k}}-{\mathit{g}}_{\mathit{n}\mathit{k}})}^T{\mathbf{G}}_{\mathbf{n}}({\mathit{\theta }}_{\mathit{n}\mathit{k}}-{\mathit{g}}_{\mathit{n}\mathit{k}})]$	gnk: mean Gn: precision
εn ∈ ℜP	Normal (0, ${\gamma }_{{\epsilon }_n}^{-1}{\mathbf{I}}_{\mathbf{P}}$)	${(2\pi )}^{-P/2}{\gamma }_{{\epsilon }_n}^{1/2}\text{exp }[-\frac{{\gamma }_{{\epsilon }_n}}{2}{{\mathit{\epsilon }}_{\mathit{n}}}^T{\mathit{\epsilon }}_{\mathit{n}}]$	γεn : precision
πn ∈ ℜK	Dirichlet (α)	$\frac{\mathrm{\Gamma }({\sum }_{k=1}^K{\alpha }_k)}{{\sum }_{k=1}^K\mathrm{\Gamma }({\alpha }_k)}{\prod }_{k=1}^K{\pi }_{nk}^{{\alpha }_{k^{-1}}}$	α: concentration parameters
gnk ∈ ℜQ	Normal (g0, G0−1)	${(2\pi )}^{-Q/2}|{\mathbf{G}}_{\mathbf{0}}{|}^{1/2}\text{exp }[-\frac{1}{2}{({\mathit{g}}_{\mathit{n}\mathit{k}}-{\mathit{g}}_{\mathbf{0}})}^T{\mathbf{G}}_{\mathbf{0}}({\mathit{g}}_{\mathit{n}\mathit{k}}-{\mathit{g}}_{\mathbf{0}})]$	g0: mean G0: precision
Gn ∈ ℜQ×Q	Wishart (ν0, R0)	$\left({|{\mathbf{G}}_{\mathbf{n}}|}^{\frac{{\nu }_0-Q-1}{2}}\text{exp }\right[-\frac{\mathit{\text{tr}}({{\mathbf{R}}_{\mathbf{0}}}^{-1}{\mathbf{G}}_{\mathbf{n}})}{2}\left]\right){(2^{\frac{Q_{{\nu }_0}}{2}}{|{\mathbf{R}}_{\mathbf{0}}|}^{\frac{{\nu }_0}{2}}{\mathrm{\Gamma }}_Q(\frac{{\nu }_0}{2}))}^{-1}$	ν0 > Q − 1: dof R0: scale matrix
γεn ∈ ℜ	Gamma (e, f)	$\frac{f^c}{\mathrm{\Gamma }(e)}{\gamma }_{{\epsilon }_n}^{e-1}\text{exp }[-f{\gamma }_{{\epsilon }_n}]$	e: shape f: rate

Γ, Γ_Q: (multivariate) gamma functions, dof: degrees of freedom, 1_K = [1, …, 1]^T ∈ ℜ^K

^†: atom sampling with replacement (‖z_nl‖₀ = 1), u_i = [0, 0, …, 1, …, 0]^T with 1 in the i^th entry, 0 otherwise

^‡: atom sampling without replacement (‖z_n‖₀ = L)

III. Inference with MCMC Sampling

The inference problem aims at finding solutions ${\mathbf{y}}_{\mathbf{n}}^{*}$, n = 1 …, N, that maximize (4). Since (4) is not analytically tractable, we use MCMC for approximate inference because of its simple and fast implementation. We describe the inference procedure in Section III-A and provide the corresponding derivations in Sections III-B, III-C, and III-D.

A. MCMC Sampling

The large number of variables in our problem renders the simultaneous sampling from the full posterior quite prohibitive. For this reason, we divide the variable space y into blocks, a technique which is usually referred as “block-at-a-time” MCMC [57], [58]. Suppose that the variable space is split into B blocks specified according to the problem characteristics, i.e. y = [y₁^T, …, y_B^T]^T. Without loss of generality, we hypothesize these blocks are sampled sequentially from y₁ to y_B. This is referred as “deterministic-scan” MCMC sampling [59], [60], [61] and will be the focus of our paper.

When the posterior probability of a block b yields a known probabilistic distribution, we use the Gibbs sampler, otherwise we sample based on the MH [62], [63]. MH generates a sample with a candidate-generating (or proposal) density q. The MC transitions to the generated sample with a predefined probability of move. A critical component is the selection of proposal density, based on which MH samplers are divided into two categories. The first is the random-walk [62] with samples generated around the current value of the corresponding variables. Despite its simplicity, this method often depicts slow convergence depending on the variance of q [58], [64]. The second type, called independent MH [63], samples independently of the previous state. Its proposal density is close to the target distribution in a certain sense benefiting convergence. Previous work has considered Student-t distributions tailored to the target density [65], [66], [67], whose long tails ensure that no areas of the state space are left unexplored. The “MH-within-Gibbs” sampler uses MH to generate samples for blocks whose posterior does not yield a known distribution, and Gibbs to generate samples for the remaining blocks.

We will further discuss the use of “MH-within-Gibbs sampler” for our problem and the derivation of posteriors for each variable (Table II). We will assume that y_−yb contains all variables except the ones included in the b^th block (i.e. y_b), which are currently being generated.

TABLE II.

Description of Metropolis-Hastings-within-Gibbs sampling distributions.

Variable	Sampling Distribution/Proposal	Sampling Method
znl† ∈ ℜK	Multinomial (1, pnl) $p_{n{l}_k}={\pi }_{nk}\text{ exp}[-\frac{1}{2}{\gamma }_{{\epsilon }_n}{s}_{nl}(s_{nl}-2{\mathit{\epsilon }}_{\mathbf{n}\mathbf{l}}^T\varphi ({\mathit{\theta }}_{\mathit{n}\mathit{k}})\left)\right]$	Gibbs
zn‡ ∈ ℜK	Wallenius (1k, L, πn)	Metropolis-Hastings
snl ∈ ℜ	$\text{Normal }(\frac{{\gamma }_s{\mu }_{s_{nl}}+{\gamma }_{{\epsilon }_n}{({\mathbf{D}}_{\mathbf{n}}{\mathbf{z}}_{\mathbf{n}\mathbf{l}})}^T{\mathit{\epsilon }}_{\mathbf{n}\mathbf{l}}}{{\gamma }_s+{\gamma }_{{\epsilon }_n}},{({\gamma }_s+{\gamma }_{{\epsilon }_n})}^{-1})$	Gibbs
* ${\tilde{\mathit{\theta }}}_{\mathit{n}}=\text{vec }\left(\right[{\mathit{\theta }}_{\mathit{n}{\mathit{k}}_{\mathbf{1}}^{\mathbf{\prime }}}\dots {\mathit{\theta }}_{\mathit{n}{\mathit{k}}_{\mathit{L}}^{\mathbf{\prime }}}\left]\right)\in {\Re }^{QL}$	Student-t (μ̂θ̃n, V̂θ̃n, ν1)	Metropolis-Hastings
† θnk ∈ ℜQ, k ∉ ℐDn	Normal (gnk, Gn)	Gibbs
εn ∈ ℜP	$\text{Normal }(0,{\gamma }_{{\epsilon }_n}^{-1}{\mathbf{I}}_{\mathbf{P}})$	Gibbs
πn ∈ ℜK	$\text{Dirichlet }\left({[{\alpha }_1+{\sum }_{l=1}^L{z}_{n{l}_1},\dots ,{\alpha }_K+{\sum }_{l=1}^L{z}_{n{l}_K}]}^T\right)$	Gibbs
gnk ∈ ℜQ	$\text{Normal }({({\mathbf{G}}_{\mathbf{n}}+{\mathbf{G}}_{\mathbf{0}})}^{-1}({{\mathbf{G}}_{\mathbf{n}}}^T{\mathit{\theta }}_{\mathit{n}\mathit{k}}+{{\mathbf{G}}_{\mathbf{0}}}^T{\mathbf{g}}_{\mathbf{0}}),{({\mathbf{G}}_{\mathbf{n}}+{\mathbf{G}}_{\mathbf{0}})}^{-1})$	Gibbs
Gn ∈ ℜQ×Q	$\text{Wishart }({\nu }_0+K,{[{{\mathbf{R}}_0}^{-1}+{\sum }_{k=1}^K({\mathit{\theta }}_{\mathit{n}\mathit{k}}-{\mathit{g}}_{\mathit{n}\mathit{k}}){({\mathit{\theta }}_{\mathit{n}\mathit{k}}-{\mathit{g}}_{\mathit{n}\mathit{k}})}^T]}^{-1})$	Gibbs
γεn ∈ ℜ	$\text{Gamma }(e+\frac{1}{2},f+\frac{1}{2}{\Vert {\mathbf{x}}_{\mathbf{n}}-{\sum }_{l=1}^L{s}_{nl}\varphi ({\mathit{\theta }}_{\mathit{n}{\mathit{k}}_{\mathit{l}}^{\mathbf{\prime }}})\Vert }^2)$	Gibbs

^*${\mathbf{x}}_{\mathbf{n}}={\sum }_{l=1}^L{s}_{nl}\varphi ({\mathit{\theta }}_{\mathit{n}{\mathit{k}}_{\mathit{l}}^{\mathbf{\prime }}})+{\mathit{\epsilon }}_{\mathit{n}}$,

^†${ℐ}_{D_n}=\{k_1^{\prime },\dots ,k_L^{\prime }\}$, 1_K = [1, …, 1]^T ∈ ℜ^K

^{†, ‡}: atom sampling with/without replacement

B. Sampling Dictionary Parameters

Let ${ℐ}_{D_n}=\{k_1^{\prime },\dots ,k_L^{\prime }\}$ be the indices of dictionary atoms that represent signal x_n and ${\mathit{\theta }}_{\mathit{n}{\mathit{k}}_{\mathbf{1}}^{\mathbf{\prime }}}\dots {\mathit{\theta }}_{\mathit{n}{\mathit{k}}_{\mathit{L}}^{\mathbf{\prime }}}$ the corresponding atom parameters. The joint posterior of ${\tilde{\mathit{\theta }}}_{\mathit{n}}=\text{vec}([{\mathit{\theta }}_{\mathit{n}{\mathit{k}}_{\mathbf{1}}^{\mathbf{\prime }}}\dots {\mathit{\theta }}_{\mathit{n}{\mathit{k}}_{\mathit{L}}^{\mathbf{\prime }}}])\in {\Re }^{QL}$ can be written as

$$p({\tilde{\mathit{\theta }}}_{\mathit{n}}|{\mathbf{y}}_{-{\tilde{\mathit{\theta }}}_{\mathit{n}}},{\mathbf{x}}_{\mathbf{n}},{\mathscr{H}}_n)\propto \prod _{k\in {ℐ}_{D_n}}p({\mathit{\theta }}_{\mathit{n}\mathit{k}}|{\mathbf{g}}_{\mathbf{n}\mathbf{k}},{\mathbf{G}}_{\mathbf{n}})·p({\mathbf{x}}_{\mathbf{n}}-\sum _{l=1}^L{s}_{nl}\varphi ({\mathit{\theta }}_{\mathit{n}{\mathit{k}}_{\mathit{l}}^{\mathbf{\prime }}})|{\gamma }_{{\epsilon }_n})\propto \text{exp}[-\frac{{\gamma }_{{\epsilon }_n}}{2}{\Vert {\mathbf{x}}_{\mathbf{n}}-\sum _{l=1}^L{s}_{nl}\varphi ({\mathit{\theta }}_{\mathit{n}{\mathit{k}}_{\mathit{l}}^{\mathbf{\prime }}})\Vert }_2^2\left]\text{exp}\right[-\frac{1}{2}\sum _{k\in {ℐ}_{D_n}}{({\mathit{\theta }}_{\mathit{n}\mathit{k}}-{\mathbf{g}}_{\mathbf{n}\mathbf{k}})}^T{\mathbf{G}}_{\mathbf{n}}({\mathit{\theta }}_{\mathit{n}\mathit{k}}-{\mathbf{g}}_{\mathbf{n}\mathbf{k}})]\propto \text{exp}[-\frac{{\gamma }_{{\epsilon }_n}}{2}{\Vert {\mathbf{x}}_{\mathbf{n}}-{\mathbf{S}}_{\mathbf{n}}{\tilde{\varphi }}_{\mathit{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}})\Vert }_2^2\left]\text{exp}\right[-\frac{1}{2}{({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\tilde{\mathbf{g}}}_{\mathbf{n}})}^T{\tilde{\mathbf{G}}}_{\mathbf{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\tilde{\mathbf{g}}}_{\mathbf{n}})]$$ (7)

where ${\tilde{\varphi }}_{\mathit{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}})=\text{vec }([\varphi ({\mathit{\theta }}_{\mathit{n}{\mathit{k}}_{\mathbf{1}}^{\mathbf{\prime }}})\dots \varphi ({\mathit{\theta }}_{\mathit{n}{\mathit{k}}_{\mathit{L}}^{\mathbf{\prime }}})])\in {\Re }^{PL},{\tilde{\mathbf{g}}}_{\mathbf{n}}=\text{vec }([{\mathbf{g}}_{\mathbf{n}{\mathbf{k}}_{\mathbf{1}}^{\prime }}\dots {\mathbf{g}}_{\mathbf{n}{\mathbf{k}}_{\mathbf{L}}^{\prime }}])\in {\Re }^{QL}$ and

$${\tilde{\mathbf{G}}}_{\mathbf{n}}=\left(\begin{array}{ccc}{\mathbf{G}}_{\mathbf{n}}\hfill & \hfill & \hfill \\ \hfill & \ddots \hfill & \hfill \\ \hfill & \hfill & {\mathbf{G}}_{\mathbf{n}}\hfill \end{array}\right)\in {\Re }^{QL\times QL}$$

Because of ϕ, we cannot find a known probabilistic distribution for (7), therefore we use MH for sampling θ̃_n. The proposal density is a multivariate Student-t distribution with location μ̂_θ̃n tailored to the target density with ν₁ degrees of freedom and identity scale matrix V̂_θ̃n, defined as

$$q({\tilde{\mathit{\theta }}}_{\mathit{n}}|{\mathbf{x}}_{\mathbf{n}},{\mathscr{H}}_n,{\mathbf{y}}_{-{\tilde{\mathit{\theta }}}_{\mathit{n}}})=\frac{\mathrm{\Gamma }\left(\frac{{\nu }_1+Q}{2}\right)}{\mathrm{\Gamma }\left(\frac{{\nu }_1}{2}\right)}{[\pi ({\nu }_1+Q)]}^{-\frac{Q}{2}}{\left|{\hat{\mathbf{V}}}_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}\right|}^{-\frac{1}{2}}{[1+\frac{1}{{\nu }_1+Q}{({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\hat{\mathit{\mu }}}_{{\tilde{\mathit{\theta }}}_{\mathit{n}}})}^T{\hat{\mathbf{V}}}_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}^{-1}({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\hat{\mathit{\mu }}}_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}\left)\right]}^{-\frac{{\nu }_1+Q}{2}}$$ (8)

$${\hat{\mathit{\mu }}}_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}=\text{arg }\underset{{\tilde{\mathit{\theta }}}_{\mathit{n}}}{\text{max}}\text{ ln }(p({\tilde{\mathit{\theta }}}_{\mathit{n}}|{\mathbf{y}}_{-{\tilde{\mathit{\theta }}}_{\mathit{n}}},{\mathbf{x}}_{\mathbf{n}},{\mathscr{H}}_n))$$ (9)

$${\hat{\mathit{V}}}_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}={\tau }^2{\mathbf{I}}_{\mathbf{Q}\mathbf{L}}$$ (10)

The choice of Student-t is crucial for the uniform ergodicity of MCMC (Section VI). In the experiments (Section VII), we find a numerical solution to (9) using the Nelder Mead’s simplex [68] because of its simplicity compared to gradient-based approaches. From (8–9), we are able to jointly generate the parameters of the selected atoms, rather than generating each one separately. A large number of selected atoms increases the computational cost towards maximizing (9).

The posterior for the remaining atoms θ_nk ∉ ℐ_Dn, generated with Gibbs, is

$$p({\mathit{\theta }}_{\mathit{n}\mathit{k}}|{\mathbf{y}}_{-{\mathit{\theta }}_{\mathit{n}\mathit{k}}},{\mathbf{x}}_{\mathbf{n}},{\mathscr{H}}_n)\propto p({\mathit{\theta }}_{\mathit{n}\mathit{k}}|{\mathbf{g}}_{\mathbf{n}\mathbf{k}},{\mathbf{G}}_{\mathbf{n}})$$ (11)

C. Sampling Atom Indices and Coefficients

The probabilistic selection of dictionary atoms gives the opportunity to test unseen combinations. This can alleviate disadvantages of greedy algorithms, since the randomization might overcome locally optimal solutions, as also observed in [39]. Because the atom indices and coefficients are interdependent, we will describe the sampling procedure for both.

In the case of sampling with replacement, the atom indices and coefficients are generated with the Gibbs sampler, as their posterior can be analytically derived based on the assumptions for their priors (Section II). In the following, we will denote ε_nl = x_n − ∑_l′≠l s_nl′ D_nz_nl′ the error for the exemplar data x_n when we exclude the l^th atom from the representation. Then the total representation error can be expressed as

$${\mathit{\epsilon }}_{\mathit{n}}={\mathit{\epsilon }}_{\mathit{n}\mathit{l}}-s_{nl}{\mathbf{D}}_{\mathbf{n}}{\mathbf{z}}_{\mathbf{n}\mathbf{l}}={\mathit{\epsilon }}_{\mathit{n}\mathit{l}}-s_{nl}\sum _{k=1}^K{z}_{n{l}_k}\varphi ({\mathit{\theta }}_{\mathit{n}\mathit{k}})$$ (12)

The posterior distribution of z_nl can be written as

$$p({\mathbf{z}}_{\mathbf{n}\mathbf{l}}|{\mathbf{y}}_{-\{{\mathbf{z}}_{\mathbf{n}\mathbf{l}}\}},{\mathbf{x}}_{\mathbf{n}},{\mathscr{H}}_n)\propto p({\mathbf{z}}_{\mathbf{n}\mathbf{l}}|{\mathit{\pi }}_{\mathit{n}},L)·p({\mathit{\epsilon }}_{\mathit{n}}|{\gamma }_{{\epsilon }_n})\propto \prod _{k=1}^K{\pi }_{nk}^{z_{n{l}_k}}·\text{exp }(-\frac{{\gamma }_{{\epsilon }_n}}{2}{\Vert {\mathit{\epsilon }}_{\mathit{n}\mathit{l}}-s_{nl}\sum _{k=1}^K{z}_{n{l}_k}\varphi ({\mathit{\theta }}_{\mathit{n}\mathit{k}})\Vert }_2^2)\propto \prod _{k=1}^K{[{\pi }_{nk}\text{ exp }({\gamma }_{{\epsilon }_n}{s}_{nl}{\mathit{\epsilon }}_{\mathbf{n}\mathbf{l}}^T\varphi ({\mathit{\theta }}_{\mathit{n}\mathit{k}}))]}^{z_{n{l}_k}}$$ (13)

For deriving the above expression we took into account that z_nlk ∈ {0, 1} is a binary variable, implying that $z_{n{l}_k}^2=z_{n{l}_k}$ and az_nlk = a^z_nlk, ∀a ∈ ℜ. Also z_nl has unit l0-norm, i.e. ‖z_nl‖₀ = 1, resulting in z_nlkz_nlk′ = 0, ∀k′ ≠ k. As indicated in (13), this update procedure considers the similarity of dictionary atoms to the signal residual and the prior knowledge for selecting atom k. Finally, the posterior of s_nl is

$$p(s_{nl}|{\mathbf{y}}_{-s_{nl}},{\mathbf{x}}_{\mathbf{n}},{\mathscr{H}}_n)\propto p(s_{nl}|{\gamma }_s)·p({\mathit{\epsilon }}_{\mathit{n}}|{\gamma }_{{\epsilon }_n})\propto \text{exp }[-\frac{{\gamma }_s}{2}{(s_{nl}-{\mu }_{s_{nl}})}^2-\frac{{\gamma }_{{\epsilon }_n}}{2}{\Vert {\mathit{\epsilon }}_{\mathit{n}\mathit{l}}-s_{nl}{\mathbf{D}}_{\mathbf{n}}{\mathbf{z}}_{\mathbf{n}\mathbf{l}}\Vert }_2^2]$$ (14)

By completing the square of the above quadratic formula with respect to s_nl and assuming dictionary atoms of unit norm, s_nl can be generated from a normal distribution (Table II).

To the best of our knowledge, we found no conjugate prior for the Wallenius’ hypergeometric distribution [53], therefore we used the independent MH with proposal distribution tailored to the Wallenius prior for sampling without replacement.

D. Sampling the Parameters of the Priors

The conjugate assumptions for the distribution of the parameters of the aforementioned variables allow us to use the Gibbs sampler to generate the corresponding samples. For the sake of completeness, we briefly sketch the derivation of the posteriors with sampling distributions shown in Table II.

$$p({\mathit{\pi }}_{\mathit{n}}|{\mathbf{y}}_{-{\mathit{\pi }}_{\mathit{n}}},-))\propto p({\mathit{\pi }}_{\mathit{n}}|\mathit{\alpha })\prod _{l=1}^{L}p({\mathbf{z}}_{\mathbf{n}\mathbf{l}}|{\mathit{\pi }}_{\mathit{n}},\mathit{\alpha })$$ (15)

$$p({\mathbf{g}}_{\mathbf{n}\mathbf{k}}|{\mathbf{y}}_{-{\mathbf{g}}_{\mathbf{n}\mathbf{k}}},-)\propto p({\mathit{\theta }}_{\mathit{n}\mathit{k}}|{\mathbf{g}}_{\mathbf{n}\mathbf{k}},{\mathbf{G}}_{\mathbf{n}})·p({\mathbf{g}}_{\mathbf{n}\mathbf{k}}|{\mathbf{g}}_{\mathbf{0}},{\mathbf{G}}_{\mathbf{0}})$$ (16)

$$p({\mathbf{G}}_{\mathbf{n}}|{\mathbf{y}}_{-{\mathbf{G}}_{\mathbf{n}}},-)\propto \prod _{k=1}^{K}p({\mathit{\theta }}_{\mathit{n}\mathit{k}}|{\mathbf{g}}_{\mathbf{n}\mathbf{k}},{\mathbf{G}}_{\mathbf{n}})·p({\mathbf{G}}_{\mathbf{n}}|{\nu }_0,{\mathbf{R}}_{\mathbf{0}})$$ (17)

$$p({\gamma }_{{\epsilon }_n}|{\mathbf{y}}_{-{\gamma }_{{\epsilon }_n}},-)\propto p({\gamma }_{{\epsilon }_n}|e,f)·p({\mathbf{x}}_{\mathbf{n}}-{\mathbf{D}}_{\mathbf{n}}\sum _{l=1}^L{s}_{nl}{\mathbf{z}}_{\mathbf{n}\mathbf{l}}|{\gamma }_{{\epsilon }_n})$$ (18)

where the dash “−” denotes {x_n, ℋ_n}

Algorithm 1.

MCMC inference of parametric dictionary learning variables

Require: Data xn, hyperparameters ℋn
1:	for n = 1, …, N do
2:	for m = 1, …, M do
3:	Sample atom indices ${\mathbf{z}}_{\mathbf{n}\mathbf{1}}^{(m)},\dots {\mathbf{z}}_{\mathbf{n}\mathbf{L}}^{(m)}$ or ${\mathbf{z}}_{\mathbf{n}}^{(m)}$
4:	Find ${ℐ}_{D_n}^{(m)}=\{k_1^{\prime },\dots ,k_L^{\prime }\}$ s.t. $z_{n{l}_{k_l^{\prime }}}^{(m)}=1$
5:	Sample atom coefficients $s_{n1}^{(m)},\dots ,s_{nL}^{(m)}$
6:	Sample noise vector ${\mathit{\epsilon }}_{\mathbf{n}}^{(m)}$
7:	Sample dictionary parameters ${\mathit{\theta }}_{\mathbf{n}\mathbf{k}}^{(m)},k\notin {ℐ}_{D_n}^{(m)}$
8:	Sample dictionary priors ${\mathbf{g}}_{\mathbf{n}\mathbf{1}}^{(m)},\dots ,{\mathbf{g}}_{\mathbf{n}\mathbf{K}}^{(m)},{\mathbf{G}}_{\mathbf{n}}^{(m)}$
9:	Sample atom selection probability priors ${\mathit{\pi }}_{\mathbf{n}}^{(m)}$
10:	Sample noise variance ${\gamma }_{{\epsilon }_n}^{(m)}$
11:	Sample ${\tilde{\mathit{\theta }}}_{\mathbf{n}}^{(m)}$ for generating ${\mathit{\theta }}_{\mathbf{n}\mathbf{k}}^{(m)},k\in {ℐ}_{D_n}^{(m)}$
12:	Compute ${\mathbf{D}}_{\mathbf{n}}^{(m)}=\left[\varphi \right({\mathit{\theta }}_{\mathbf{n}\mathbf{1}}^{(m)})\dots \varphi ({\mathit{\theta }}_{\mathbf{n}\mathbf{K}}^{(m)}\left)\right]$
13:	end for
14:	end for

E. Implementation of Bayesian DL

The problem variables (5) are inferred for each exemplar data separately. This method can yield signal-specific estimations of the noise variance, useful for denoising applications. It also allows sharing computational cost in MCMC inference, which typically requires a large amount of iterations to converge. Since more training samples are likely to require more MCMC iterations, if we had trained one dictionary on the entire dataset, the sequential nature of MCMC would have significantly slowed down convergence. It is worth mentioning that similar studies have acknowledged the difficulty of performing Bayesian inference in large datasets [43]. The MCMC inference procedure is outlined in Algorithm 1.

IV. Combination of Generated Dictionaries

The variables of the considered Bayesian problem are inferred for each exemplar data separately yielding sample-specific information useful for denoising and other applications, and allowing parallel implementations (Sections II, III). The latter is beneficial because of the large amount of data in many applications, that render batch DL methods computationally expensive or even prohibitive. However, in order to obtain generalizable dictionaries, that are able to reliably represent unseen data, we need to combine the corresponding results into a unified model (Algorithm 2). While the Bayesian framework aims to maximize the posterior probability of the model parameters, DL is usually evaluated based on the reconstruction error. For this reason, the combination of the dictionaries that result from the Bayesian inference is performed based on a root mean square (RMS) error criterion.

Let ${\mathbf{D}}_{\mathbf{n}}^{(m)}$ be the dictionaries generated with the MH-within-Gibbs sampler (Section III-E). In order to evaluate their performance using signal reconstruction criteria, we need to use a sparse decomposition algorithm, that can represent the original signal based on the inferred dictionaries. We use OMP [3], [4] because of its simplicity and effectiveness. Let ${\mathit{\text{Err}}}_n^{(m)}$ be the relative RMS error that yields from decomposing x_n based on dictionary ${\mathbf{D}}_{\mathbf{n}}^{(m)}$. Also let ${\mathbf{D}}_{\mathbf{n}}^{(m_n^{*})}\in {\Re }^{P\times K}$ be the dictionaries that yield the lowest relative RMS error for each signal n and ${\mathbf{\Theta }}^{(m_n^{*})}=[{\mathit{\theta }}_{\mathbf{n}{\mathbf{k}}_{\mathbf{1}}^{\prime }}^{(m_n^{*})},\dots ,{\mathit{\theta }}_{\mathbf{n}{\mathbf{k}}_{\mathbf{L}}^{\prime }}^{(m_n^{*})}]\in {\Re }^{Q\times L}$ the parameters of the atoms that were selected by OMP based on ${\mathbf{D}}_{\mathbf{n}}^{(m_n^{*})}$. ${\mathbf{\Theta }}^{(m_n^{*})}$ are concatenated into a unified dictionary

$${\mathbf{\Theta }}_{\mathit{U}}=[{\mathbf{\Theta }}^{(m_1^{*})}\dots {\mathbf{\Theta }}^{(m_N^{*})}]\in {\Re }^{Q\times NL}$$

$$m_n^{*}=\text{arg }\underset{m\in [1,M]}{\text{min}}{\mathit{\text{ Err}}}_n^{(m)},n=1,\dots ,N$$

Algorithm 2.

Combination of generated dictionaries

Require: Data xn, generated dictionaries ${\mathbf{D}}_{\mathbf{n}}^{(m)}$ , number of       K-means clusters Nb
1:	for n = 1, …, N do
2:	for m = 1, …, M do
3:	Reconstruct xn based on ${\mathbf{D}}_{\mathbf{n}}^{(m)}$ with OMP
4:	Compute relative RMS error ${\mathit{\text{Err}}}_n^{(m)}$
5:	end for
6:	Find $m_n^{*}$ such that $m_n^{*}=\text{arg}\underset{m=[1,M]}{\text{min}}{\mathit{\text{Err}}}_n^{(m)}$
7:	Retrieve parameters of dictionary atoms selected by   OMP based on ${\mathbf{D}}_{\mathbf{n}}^{(m_n^{*})},{\mathbf{\Theta }}^{(m_n^{*})}=[{\mathit{\theta }}_{\mathbf{n}{\mathbf{k}}_{\mathbf{1}}^{\prime }}^{(m_n^{*})},\dots ,{\mathit{\theta }}_{\mathbf{n}{\mathbf{k}}_{\mathbf{L}}^{\prime }}^{(m_n^{*})}]$
8:	end for
9:	Concatenate ${\mathbf{\Theta }}_{\mathit{U}}=[{\mathbf{\Theta }}^{(m_1^{*})}\dots {\mathbf{\Theta }}^{(m_N^{*})}]$
10:	Quantize dictionary parameters ΘQ=K-means(ΘU)
11:	Compute the final dictionary DQ = ϕ (ΘQ) where ϕ operates on each column of matrix ΘQ

Ideally, we could have used Θ_U as the parameters of our final dictionary. However, in practice the large amount of data renders Θ_U computationally expensive. For this reason, the parameter vectors of Θ_U are further quantized with K-means clustering using N_bin centers. This results in parameter matrix Θ_Q with corresponding final dictionary D_Q. The aforementioned procedure is performed on the training data, while the test data are not seen at all during this step (Section VII-B5).

This approach can be applied to combine dictionaries learnt from any other method, therefore it also provides a unified platform that allows comparison of the considered parametric DL approaches in our paper (Section VII).

V. Choosing the Parametric Dictionary Function

The choice of the parametric dictionary function is not trivial and is usually guided by the application of interest. If designed appropriately, parametric dictionaries can yield interpretable information about meaningful signal characteristics for a variety of applications. Previous studies have used Gabor [69] and Gammatone [24] atoms to represent speech signals because of their good localization properties and similarities to the human auditory system. Other efforts have proposed Gaussian-like functions to efficiently capture spherical stereo images [70], diffusion-based dictionaries to model MRI [25], and other wavelet-like atoms for digitizing fingerprint images [71]. Gabor dictionaries have been used for the electroencephalogram (EEG) [72], spline wavelets for the electrocardiogram (ECG) [73], and sigmoid-exponential functions for the electrodermal activity (EDA) [28].

Our Bayesian parametric DL approach generates the parameters of the dictionary atoms based on the MH sampler (Section III-B). The mean of the corresponding distribution depends on the considered parametric function and is selected so that it maximizes the corresponding posterior distribution (7). We will further show that the concavity of (7) generally depends on function ϕ. It is unusual that a function ϕ is concave with respect to all the parameters of interest, but in practice even the estimation of local optima is enough to generate useful samples (Section VII-C).

The posterior (7) of the dictionary atom parameters is:

$$U({\tilde{\mathit{\theta }}}_{\mathit{n}})=\text{ln}(p({\tilde{\mathit{\theta }}}_{\mathit{n}}|{\mathbf{y}}_{-{\tilde{\mathit{\theta }}}_{\mathit{n}}},\mathbf{X},\mathscr{H}))=\propto -\frac{{\gamma }_{{\epsilon }_n}}{2}{\Vert {\mathbf{x}}_{\mathbf{n}}-{\mathbf{S}}_{\mathbf{n}}{\tilde{\mathit{\varphi }}}_{\mathit{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}})\Vert }_2^2-\frac{1}{2}{({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\tilde{\mathbf{g}}}_{\mathbf{n}})}^T{\tilde{\mathbf{G}}}_{\mathbf{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\tilde{\mathbf{g}}}_{\mathbf{n}})$$ (19)

If we set ψ_n (θ̃_n) = x_n − S_n ϕ̃_n (θ̃_n), then (19) becomes

$$U({\tilde{\mathit{\theta }}}_{\mathit{n}})=-\frac{1}{2}{({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\tilde{\mathbf{g}}}_{\mathbf{n}})}^T{\tilde{\mathbf{G}}}_{\mathbf{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\tilde{\mathbf{g}}}_{\mathbf{n}})-\frac{{\gamma }_{{\epsilon }_n}}{2}{\Vert {\mathit{\psi }}_{\mathit{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}})\Vert }^2$$ (20)

The gradient vector and Hessian matrix of (20) are

$${\nabla }_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}U({\tilde{\mathit{\theta }}}_{\mathit{n}})=-{\tilde{\mathbf{G}}}_{\mathbf{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\tilde{\mathbf{g}}}_{\mathbf{n}})-{\gamma }_{{\epsilon }_n}{({\nabla }_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}{\mathit{\psi }}_{\mathit{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}}))}^T{\mathit{\psi }}_{\mathit{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}})$$ (21)

$${\mathbf{H}}_U={\nabla }_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}^{2}U({\tilde{\theta }}_{\mathbf{n}})=-{\tilde{\mathbf{G}}}_{\mathbf{n}}-\frac{{\gamma }_{{\epsilon }_n}}{2}{\nabla }_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}^2{\Vert {\mathit{\psi }}_{\mathit{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}})\Vert }_2^2$$ (22)

where

$${\left({\nabla }_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}^2{\Vert {\mathit{\psi }}_{\mathit{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}})\Vert }_2^2\right)}_{ij}=\frac{{\vartheta }^2}{\vartheta {\tilde{\theta }}_{n_i}\vartheta {\tilde{\theta }}_{n_j}}{\Vert {\mathit{\psi }}_{\mathit{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}})\Vert }_2^2=\frac{\vartheta }{\vartheta {\tilde{\theta }}_{n_i}}(2\sum _{p=1}^{PL}{\psi }_{n_p}({\tilde{\mathit{\theta }}}_{\mathit{n}})\frac{\vartheta {\psi }_{n_p}({\tilde{\mathit{\theta }}}_{\mathit{n}})}{\vartheta {\tilde{\theta }}_{n_j}})=2\sum _{p=1}^{PL}\frac{\vartheta {\psi }_{n_p}({\tilde{\mathit{\theta }}}_{\mathit{n}})}{\vartheta {\tilde{\theta }}_{n_i}}\frac{\vartheta {\psi }_{n_p}({\tilde{\mathit{\theta }}}_{\mathit{n}})}{\vartheta {\tilde{\theta }}_{n_j}}+2\sum _{p=1}^{PL}{\psi }_{n_p}({\tilde{\mathit{\theta }}}_{\mathit{n}})\frac{{\vartheta }^2{\psi }_{n_p}({\tilde{\mathit{\theta }}}_{\mathit{n}})}{\vartheta {\tilde{\theta }}_{n_i}\vartheta {\tilde{\theta }}_{n_j}}$$ (23)

If H_ψnp is the Hessian of ψ_np, p = 1, …, PL, then from (23)

$${\nabla }_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}^2{\Vert {\mathit{\psi }}_{\mathit{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}})\Vert }_2^2=2{({\nabla }_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}{\mathit{\psi }}_{\mathit{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}}))}^T({\nabla }_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}{\mathit{\psi }}_{\mathit{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}}))+2\sum _{p=1}^P{\psi }_{n_p}({\tilde{\mathit{\theta }}}_{\mathit{n}}){\mathbf{H}}_{{\psi }_{n_p}}$$ (24)

The Hessian of ψ_np can be computed as

$${\left({\mathbf{H}}_{{\psi }_{n_p}}\right)}_{ij}=-\sum _{l=1}^L{s}_{nl}\frac{{\vartheta }^2{\varphi }_p({\mathit{\theta }}_{\mathit{n}{\mathit{k}}_{\mathit{l}}^{\mathbf{\prime }}})}{\vartheta {\theta }_{n{k}_{li}^{\prime }}\vartheta {\theta }_{n{k}_{ll}^{\prime }}}$$ (25)

$${\mathbf{H}}_{{\psi }_{n_p}}=-\sum _{l=1}^L{s}_{nl}{\mathbf{H}}_{{\varphi }_p}{|}_{\mathit{\theta }={\mathit{\theta }}_{\mathit{n}{\mathit{k}}_{\mathit{l}}^{\mathbf{\prime }}}}$$ (26)

From (22), (24) and (26), we get

$${\mathbf{H}}_U=-{\tilde{\mathbf{G}}}_{\mathbf{n}}-{\gamma }_{{\epsilon }_n}({\nabla }_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}{\mathit{\psi }}_{\mathit{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}})){({\nabla }_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}{\mathit{\psi }}_{\mathit{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}}))}^T+{\gamma }_{{\epsilon }_n}\sum _{p=1}^P{\psi }_{n_p}({\tilde{\mathit{\theta }}}_{\mathit{n}})\sum _{l=1}^L{s}_{nl}{\mathbf{H}}_{{\varphi }_p}{|}_{\mathit{\theta }={\mathit{\theta }}_{\mathit{n}{\mathit{k}}_{\mathit{l}}^{\mathbf{\prime }}}}$$ (27)

The first two terms of (27) are postive-definite matrices, whereas the positive-definiteness of the third term depends on ϕ. In our experiments (Section VII) and in the literature mentioned in this section, function ϕ is selected with a knowledge-driven approach based on the application of interest. Although this does not guarantee that the generated atom parameters are the global maxima, in practice the corresponding sampling method yields satisfactory results (Section VII-C).

VI. MCMC Convergence for Bayesian DL

Ergodicity properties of large-dimensional MC have been the subject of several studies [60], [74]. We discuss the uniform ergodicity property of the considered MC which implies convergence independently of the initial state [51], [52]. We focus on the sampling with replacement case, although our discussion can be extended for atom sampling without replacement. We show that the MC used for inferring the variables of the Bayesian DL problem is uniformly ergodic by proving that it meets the conditions of Theorem 6.1. This theorem establishes uniform ergodicity for the MH-within-Gibbs sampler and its proof can be found in [61].

Let P be a Markov transition kernel (Section III-A) and Y^(m), Y^(m+1) be two consecutive MCMC states generating observations y^(m), y^(m+1). We will assume that ${\mathbf{y}}_{\mathbf{b}}^{(m)}$ and ${\mathbf{y}}_{\mathbf{b}}^{(m+1)}$ are the values of block b at the current and previous MCMC state, noted as m+1 and m, respectively. Also, ${\mathbf{y}}_{-{\mathbf{y}}_{\mathbf{b}}}^{(m+1)}$ contains all variables that have already been sampled at state m + 1 and ${\mathbf{y}}_{-{\mathbf{y}}_{\mathbf{b}}}^{(m)}$ the ones from the previous state m.

Definition 6.1 (Conditional Weight Function)

If block b generated with the MH sampler, its conditional weight function is defined as

$$w_{{\mathbf{y}}_{\mathbf{b}}}({{\mathbf{y}}_{\mathbf{b}}}^{(m+1)}|{{\mathbf{y}}_{\mathbf{b}}}^{(m)},{{\mathbf{y}}_{{-\mathbf{y}}_{\mathbf{b}}}}^{(m)},{{\mathbf{y}}_{{-\mathbf{y}}_{\mathbf{b}}}}^{(m+1)})=\frac{p({{\mathbf{y}}_{\mathbf{b}}}^{(m+1)}|{{\mathbf{y}}_{{-\mathbf{y}}_{\mathbf{b}}}}^{(m)},{{\mathbf{y}}_{{-\mathbf{y}}_{\mathbf{b}}}}^{(m+1)})}{q({{\mathbf{y}}_{\mathbf{b}}}^{(m+1)}|{{\mathbf{y}}_{\mathbf{b}}}^{(m)},{{\mathbf{y}}_{{-\mathbf{y}}_{\mathbf{b}}}}^{(m)},{{\mathbf{y}}_{{-\mathbf{y}}_{\mathbf{b}}}}^{(m+1)})}$$ (28)

Theorem 6.1 (MH-within-Gibbs Uniform Ergodicity)

Given p(y|X, ℋ) on state space y = [y₁^T … y_B^T]^T, let P = P₁ … P_B be the Markov kernel of the Gibbs sampler and Q_b, b ∈ ℐ_B′, where ℐ_B′ = {b_i1, …, b_iB′}, B′ < B, be the Markov kernel of the MH sampler with conditional weight w_yb′ as in (28). If each conditional weight w_yb′, b′ ∈ ℐ_B′, of the MH sampler is bounded, sup w_yb′ < ∞, and the Gibbs sampler with Markov kernel P₁ … P_B is uniformly ergodic, then the MH-within-Gibbs sampler, resulting from substituting kernels P_b′ with Q_b′, b′ ∈ ℐ_B′, is also uniformly ergodic.

Based on Theorem 6.1, we show the MCMC uniform ergodicity for our case. Theorem 6.2 describes the minorization condition of the Gibbs sampler. This consists a multivariate extension from Jones et. al [61] (Proposition 2). Lemma 6.1 assists with its proof and ensures the minorization of the first B − 1 blocks. Lemma 6.2 ensures that the conditional weight for θ_k is bounded away from infinity. Finally, Theorem 6.3 combines all the aforementioned to prove the uniform ergodicity of the considered MC. The proofs of the theorems 6.2, 6.3 and lemmas 6.1, 6.2 can be found in Appendix A

Lemma 6.1 (Minorization Condition of B − 1 Blocks)

Let P((Y₁, Y₂, …, Y_B), A₁ × A₂ × … × A_B) be the Markov kernel of a Gibbs sampler, where A₁, …, A_B are elements of the Borel σ-algebra on the variable space Y₁, …, Y_B, respectively. We further assume that all updates of the Gibbs sampler, except the one corresponding to Y_B, are minorisable, in the sense that for b = 1, …, B − 1, there is ε_b > 0 and a probability measure ν_b, such that P_Yb(y_−yb, A_−b) ≥ ε_bν_b(A_−b), where A_−b = A₁ × … × A_b−1 × A_b+1, …, A_B. The Gibbs sampler with Markov kernel P₁ … P_B−1 is minorisable, i.e. there exist ε₀ > 0 and probability measure ν₀ such that

$$P(({\mathbf{Y}}_{\mathbf{1}},{\mathbf{Y}}_{\mathbf{2}},\dots ,{\mathbf{Y}}_{\mathbf{B}-\mathbf{1}}),A_{-B})\ge {\epsilon }_0{\nu }_0(A_{-B})$$

Theorem 6.2 (Partial Minorization Condition for Gibbs)

Let the same assumptions from Lemma 6.1 hold, then the Gibbs sampler with Markov kernel P₁ … P_B is minorisable.

Lemma 6.2 (Bounded Conditional Weight of Dictionary Parameters)

Let the same assumptions from Lemma 6.1 hold. The conditional weight of the B^th block that includes the “super-vector” θ̃_n of dictionary parameters and is generated with MH according to (8)–(10), is bounded, i.e. sup w_θ̃n < ∞.

Theorem 6.3 (MC Uniform Ergodicity for Bayesian DL Inference)

Let y_n be the variables of the parametric DL problem (4) generated with the MH-within-Gibbs sampler (Algorithm 1), according to which all variables except θ̃_n are sampled with Gibbs (first B − 1 blocks), while θ̃_n is sampled with MH (B^th block). Then the corresponding MC {Y⁽¹⁾, Y⁽²⁾, …} is uniformly ergodic.

VII. Experiments

We compare the Bayesian DL model against the previously proposed parametric SD [22] and ETF [24], which are conceptually closer to our approach. We further perform experiments with the non-parametric K-SVD [14] yielding dictionaries of arbitrary structure, therefore not directly comparable to our approach. We use synthetic and real biomedical data from EDA signals. Because of their characteristic structure, these types of signals favor sparse representation approaches with parametric dictionaries providing interpretable information [28].

EDA is decomposed into a slow moving tonic part depicting the general trend and a phasic part which contains fast fluctuations superimposed onto the tonic signal, also called skin conductance responses (SCR). The tonic part is mathematically expressed as a straight line, while SCRs are represented by sigmoid-exponential functions with a steep rise and slow recovery. Taking this into account, dictionaries contain tonic and phasic atoms as shown in Table III. Since SCR shapes typically contain higher variability than the signal level, which remains fairly constant throughout an analysis window, for the sake of simplicity we perform DL on the phasic atoms for learning the parameters θ = [t₀, T_rise, T_decay]^T. The initial dictionaries are created from the combination of all parameters reported in Table III, resulting in 63 tonic and 144 phasic atoms. The analysis window is 5sec, i.e. 160 samples with typical sampling frequency of 32Hz [28].

TABLE III.

Description of EDA-specific dictionary atoms and initial parameters.

Tonic Atoms
ϕ1(t) = Δ0 + Δ · t	Δ0 ∈ {−20, −10, 1} Δ ∈ {−0.01, −0.009, …, 0, 0.01, 0.02, …, 0.1}
Phasic Atoms
${\varphi }_2(t)=\frac{\frac{st-t_0}{e^T\mathit{\text{decay}}}}{{[1+{\left(\frac{st-t_0}{T_{\mathit{\text{rise}}}}\right)}^{-2}]}^2}u(t-t_0)$	Trise ∈ {8, 14, 18} Tdecay ∈ {10, 15, 20} t0 ∈ {0, 10, 20, …, 150}

u(t) = 1, t ≥ 0 and u(t) = 0 otherwise

A critical issue of MCMC is whether a certain number of iterations is enough to stop sampling. In high-dimensional problems, all inferred variables need to converge to the target distribution. Since examining each variable separately is not always feasible, we use a combination of monitoring and diagnostic strategies to quantitatively assess MCMC convergence.

In the following, we describe the data (Section VII-A), the experimental setup (Section VII-B), and the results evaluating the learned dictionaries (Section VII-C) and providing diagnostics for MCMC convergence (Section VII-D).

A. Data Description

1) Synthetic Data

We randomly generate 1000 synthetic signals that simulate the EDA structure. Each signal is expressed as the sum of a constant c and R number of SCRs

$$x(t)=c+\sum _{r=1}^R{\varphi }_2^{(r)}(t)$$ (29)

where ${\varphi }_2^{(r)}$ is given in Table III with parameters $T_{\mathit{\text{rise}}}^{(r)},T_{\mathit{\text{decay}}}^{(r)}$ and $t_0^{(r)}$. In contrast to the rest of the paper, in which superscript “(·)” denotes the MCMC state, here it symbolizes the SCR index. The parameters of the synthetic data are randomly generated within a pre-specified range $t_0^{(r)}\in [1,150]$ samples, T_rise,^(r), $T_{\mathit{\text{decay}}}^{(r)}\in [1,20]$, and R ∈ [1, 5]. Since the number of SCRs for each signal is known a priori, DL was performed with K = R + 1 number of atoms, from which one captures the tonic part and the rest, the phasic.

2) Real Data

We further evaluate the considered DL methods on human EDA data from the database of emotion analysis using physiological signals (DEAP) [75]. DEAP contains 40 one-minute recordings from 32 subjects watching long excerpts of music videos designed to study the relation of multimedia content with mood and temperament [75]. Because of the expected SCR rate in the considered 5sec analysis window [76], training was performed with K = 3 atoms, although similar results yield for different values.

B. Experimental Setup

1) Bayesian DL

The proposed MCMC inference (Algorithm 1) is performed with 1000 and 500 iterations for the synthetic and real data, respectively. The atom indices z_nl, coefficients s_n and selection probabilities π_n are initialized based on the decomposition of each exemplar signal with OMP. The mean μ_sn of the coefficients’ prior was initialized with the average of the coefficients of the selected atoms from OMP. The mean and precision of dictionary atoms’ prior g₀ and G₀ were initialized with the mean and inverse covariance matrix of the initial parameters (Table III). The scale matrix R₀ of the Wishart distribution for sampling G_n was also initialized with the covariance of dictionary parameters. The remaining hyperparameters were empirically set to α_k = 2, e = 1, f = 2, ν₀ = 3, γ_s = 10, and γ_e = 8.

Dictionary combination (Algorithm 2) was performed on the generated vector parameters [T_rise, T_decay]^T with N_b = 100, 225, 400. We did not include the time offset t₀ in this procedure, since it does not contribute to the shape of the dictionary atoms; the final dictionaries should contain the learnt versions of phasic atoms shifted across the entire analysis frame. Therefore, the quantized matrix of dictionary parameters Θ_Q ∈ ℜ^2×N_b is replicated for all values of t₀ (Table III), yielding final dictionaries with 16N_b phasic atoms.

2) Steepest descent DL

Dictionaries are trained in parallel for each exemplar data using the SD [22], in which OMP alternates with a least-squares fit step for estimating the parameters of the selected atoms. SD was run over 250 iterations. We perform the same procedure for combining the learnt parameters, yielding dictionaries of same size with MCMC.

3) Equiangular tight frame DL

This method alternates between finding the dictionary with minimum Frobenius distance from the ETF and updating the corresponding parameters [24]. These steps involve an ETF relaxation constraint value and a gradient descent step, crucial for the overall performance, referred in [24] as α_k and ε. In our experiments, dictionaries were trained with different combinations of α_k ∈ {0.1, 0.5, 0.9} and ε ∈ {0.001, 0.002, 0.003, 0.004}. The dictionary {${\alpha }_k^{*}$, ε*} that showed the lowest relative RMS error on the training data was used to evaluate the corresponding test data. In order to ensure the same dictionary size as the other approaches, parameters are uniformly initialized within the intervals T_rise ∈ [8, 18], T_decay ∈ [10, 20] with $\sqrt{N_b}=10,15,20$ different values.

4) K-SVD

K-SVD is a classical non-parametric DL method generalizing K-means and iteratively estimating each dictionary atom in order to minimize the reconstruction error [14]. The original dictionaries used in this method were the same as the initial ones of the aforementioned approaches.

5) Evaluation

Evaluation of the final dictionaries is performed based on the relative RMS error computed between the original signal and its corresponding approximation through OMP based on the final dictionaries. RMS error is a common metric in related studies [14], [15], [24]. In order to ensure that our results reflect the ability of the algorithms to represent unseen data, all experiments are performed within a 10-fold cross-validation for the synthetic data and a leave-one-subject-out cross-validation setup for the real EDA signals. We note that OMP has two functionalities in our framework. It serves as a decomposition technique, based on which the learned dictionaries are evaluated against the test data, and is also used at the dictionary combination step (Section IV) in order to “prune” the atoms of the generated dictionaries. OMP does not operate on the test data during this dictionary combination framework, but rather at the evaluation step, during which the final dictionary is already created independently of the test set.

Besides signal reconstruction, dictionary coherence is an additional evaluation criterion. It is defined as the absolute value of the largest inner-product between any atoms of the dictionary and is an important property related to signal reconstruction quality [24]. We computed the resulting coherence of the dictionaries after training.

C. Results

Visual inspection of the final dictionaries indicates that SD does not always preserve the initial dictionary structure (Figs. 1a–b), while ETF yields less variable dictionaries (Fig. 1c). MCMC results in a variety of atoms preserving the original shape (Fig. 1e–f). In contrast to parametric DL, non-parametric K-SVD yields very unstructured non-interpretable atoms (Fig. 1d). Further comparison between an exemplar input and the reconstructed signals suggests that our approach depicts superior signal representation (Fig. 2).

Fig. 1.

Example of initial dictionary and dictionaries learnt Steepest Descent (SD), Equiangular Tight Frame (ETF), K-SVD and Markov Chain Monte Carlo Bayesian inference (MCMC) using atom sampling with and without replacement (w-,w-o rplcm). Dictionary combination is performed with N_b = 400. An instance of phasic atoms shifted with t₀ = 30 is shown.

Fig. 2.

Example of input EDA signal (solid cyan line) and reconstructed signals using the original dictionary (blue dashed line) and the dictionaries learnt with Steepest Descent (SD), Equiangular Tight Frame (ETF), and Markov Chain Monte Carlo Bayesian inference (MCMC) (black-triangle, green-asterisk, and red-circled lines, respectively). Reconstruction is performed using orthogonal matching pursuit with 4 iterations.

Dictionaries learnt from our proposed Bayesian DL yield lower reconstruction errors compared to the initial ones and the ones learnt through SD and ETF (Fig. 3). Atom sampling with and without replacement are not significantly different, since the two distributions are very similar for K≫L. Dictionaries trained using SD perform quite poorly on unseen synthetic data (Figs. 3a–c), which might occur because their simple structure causes significant overfitting to least-squares-based methods. Despite the fact that ETF DL is not prone to overfitting, since it does not take into account exemplar data during training, it lacks adaptation to more complex real data (Figs. 3d–f). K-SVD appears more accurate for real signals than the parametric approaches (Figs. 3d–f), indicating its ability to learn more complex patterns, not necessarily of the same structure as the initial dictionaries. Similar results occur for different dictionary sizes, omitted here for the sake of simplicity.

Fig. 3.

Relative root mean square (RMS) error between original and reconstructed signal with respect to the number of (orthogonal) matching pursuit (OMP) iterations. Dictionaries are learnt with Steepest Descent (SD), Equiangular Tight Frame (ETF), K-SVD, and Markov Chain Monte Carlo Bayesian inference (MCMC). During MCMC atom sampling is performed with (w-) and without (w-o) replacement (rplcm).

The large number of atoms generally yields dictionaries of high coherence. All considered methods appear quite equivalent, with ETF achieving the lowest coherence, since this is included as an optimization metric (Table IV).

TABLE IV.

Final dictionary coherence

Method	Synthetic Data	Real Data
Initial	1	1
SD	0.9990	0.9992
ETF	0.9988	0.9990
K-SVD	0.9989	0.9991
MCMC w- rplcm	0.9989	0.9991
MCMC w-o rplcm	0.9989	0.9991

D. MCMC Diagnostics

Besides theoretical analysis (Section VI), another way to examine MCMC convergence is to see how well the MC is mixing, which is usually achieved through visual inspection. Traceplots of several problem variables from an exemplar signal indicate that the considered chains move around the parameter space and are not only limited in certain areas suggesting good mixing (Figs. 4a–d). Density plots of the generated samples further validate that the estimated posteriors are close to the target distributions (Figs. 4e–h).

Fig. 4.

Example MCMC trace plots and generated posteriors for the first element of vectors containing the (a,e) dictionary atoms θ_n, (b,f) atom coefficients s_n, (c,g) atom priors π_n, and (d,h) for the noise precision γ_εn.

Convergence diagnostics can further quantitatively assess if there is a bias from generated samples. We use the Geweke diagnostic [77], because it only requires one running chain and attempts to address issues of both bias and variance [78]. It takes two non-overlapping parts of the MC (usually the first 0.1 and last 0.5) and compares their mean using a difference of means test. If the samples are drawn from the stationary distribution of the chain, the two means are equal. The test statistic is a standard Z-score with values under convergence within two standard deviations from zero, i.e. |z| < 2 for the standard normal distribution. We perform the Geweke test for both datasets and atom sampling methods (Sections II-A,II-B) with the first 100 samples used as a burn-in period.

The high-dimensionality of our problem prohibits us to report diagnostics for each variable separately, therefore we will summarize the results for groups of variables. For each group of variables, we report the proportion of chains for which |z| < 2 (Table V). Most of the variables in our framework succeed on the Geweke diagnostic. The dictionary parameters, generated with MH, usually have a lower success percentages. Although there is a reduced number of cases where the Geweke diagnostic fails, given the large number of variables and the signal reconstruction results (Section VII-C), the performed number of MCMC iterations appears to result in meaningful dictionaries learned through this process.

TABLE V.

Geweke MCMC diagnostic - P(|z| < 2)

		Synthetic Data		Real Data
		w- rplcm	w-o rplcm	w- rplcm	w-o rplcm
cn		0.93	0.98	0.92	0.95
zn		0.93	0.93	0.91	0.94
γεn		0.91	0.98	0.99	0.92
πn		0.93	0.96	0.96	0.92
θn	t0	0.68	0.70	0.72	0.75
	Trise	0.91	0.92	0.87	0.85
	Tdecay	0.92	0.92	0.86	0.85
gn	t0	0.75	0.80	0.78	0.79
	Trise	0.90	0.87	0.89	0.87
	Tdecay	0.90	0.86	0.91	0.85
Gn		0.81	0.75	0.82	0.83

MH acceptance rate refers to the fraction of candidate draws that are accepted and is important for convergence. Very high acceptance rates suggest that the chain is not mixing well, while very low rates might be inefficient. The acceptance rates for the variables of our problem (Table VI) are close to previously proposed optimal values in the literature [79] suggesting a good mixing of the considered MC.

TABLE VI.

Metropolis-Hastings acceptance rates (%)

Synthetic Data		Real Data
w- rplcm	w-o rplcm	w- rplcm	w-o rplcm
23.22	31.52	25.48	38.15

VIII. Discussion

An important benefit of Bayesian methods yields in providing estimates of the entire variable set of a problem. In the considered setup this can help inferring the dictionary size, the optimal number of dictionary atoms for a given signal, and the corresponding noise levels. Estimation of the dictionary size is not as crucial in our greedy sparse representation approach, that is not as prone to overcomplete dictionaries as basis pursuit methods [5]. Inferring the optimal number of dictionary atoms is important, as it can yield high compression rates and help interpret the underlying signal information. An extension of our method could have imposed discrete probabilistic distributions onto the number of atoms appropriately inferred through MH.

Another advantage of Bayesian methods is that they are less prone to overfitting, which usually occurs with deterministic algorithms that might describe the random noise instead of the actual data [81]. This is reflected in our results (Section VII-C), since deterministic SD, that does not include prior knowledge of the signal structure, performs more poorly on unseen data. On the other side, ETF avoids overfitting, but does not learn the morphology of training data. Bayesian inference appears to compromise between the two.

Inherent differences exist between parametric and non-parametric DL methods. The first impose a predetermined structure on the input space (through function ϕ) and learn the parameters that represent this structure from the training data. Their major benefit lies in their interpretability, since the considered dictionary atoms are able to meaningfully capture the characteristics of input signals and can be used for knowledge-driven classification and inference. On the other hand, the exemplar signals learned from non-parametric methods are hardly interpretable and can blindly represent the input space. Since the functionality and scope of these methods is so different, meaningful comparison is challenging. In the case of synthetic data, where function ϕ is a perfect match to the signal (i.e. input signals are built with the same functions as the dictionary atoms), our proposed parametric Bayesian approach is more reliable than K-SVD in learning the hidden atom parameters. This means that given a perfectly constructed dictionary function, Bayesian parametric DL yields better results, even compared to non-parametric approaches. However, in the case of real signals, function ϕ cannot always perfectly selected, therefore parametric approaches seem to perform slightly worse than K-SVD. While more precise function might have yielded lower errors, the problem of finding the optimal mapping function is still active in research [82].

Although our proposed approach is general for learning the parameters of the dictionary atoms, we need to have good knowledge about the appropriate mapping function ϕ between the parameters and the data. As discussed in Section V, the selection of ϕ is usually guided by the application of interest and can vary for different types of signals. In the case of 2D images, for example, the dictionary needs to be constructed using a function ϕ that captures the context of the image, such as a Gabor or wavelet-like function. In order to reduce complexity, we can convert the 2D image into a 1D vector with dimensionality equal to the number of pixels, as typically performed [14], [40], [44]. Given a function ϕ suitable for the image of interest, our Bayesian approach can learn the parameters that efficiently capture the shape of the data.

In this paper, for the sake of simplicity we considered a simple dictionary combination approach with K-means quantization (Section IV). However, there exist more sophisticated approaches of block-coordinate descent with warm restarts [17] and weighted batch averaging [43].

Convergence is a critical component of MCMC in the context of Bayesian inference. In Section VI, we discussed the uniform ergodicity of the corresponding MH-within-Gibbs sampler, ensuring that the MC converges to the invariant distribution. Convergence rate is another important issue, for which previous studies have proposed explicit theoretical bounds in simple scenarios [59], [83]. The computation of these bounds can be quite difficult in such high-dimensional problems [84], therefore it goes beyond the scope of our study.

IX. Conclusions

We propose a sparse Bayesian model for parametric DL, whose problem variables follow appropriately selected probabilistic distributions. We use MH-within-Gibbs to infer the corresponding variables, because of its ability to compensate for posterior distributions that cannot be analytically computed. We further show the uniform ergodicity of the proposed MCMC through the minorization of the corresponding Gibbs sampler and the bounded conditional weights of the MH. Our experimental results performed on synthetic and real biomedical signals indicate that this approach offers benefits in terms of signal reconstruction compared to previously proposed SD and ETF methods and also provides a good tradeoff between learning the signal structure and avoiding overfitting.

Biographies

Theodora Chaspari (S’12) received the diploma in electrical and computer engineering from the National Technical University of Athens, Greece (2010) and the master’s degree from the University of Southern California (2012), where she is currently pursuing a Ph.D. degree. Since 2010 she has been a member of the Signal Analysis and Interpretation Laboratory (SAIL). Her research interests lie in the area of biomedical signal processing, speech analysis and behavioral signal processing.

Ms Chaspari is a recipient of the USC Annenberg Graduate Fellowship, USC WiSE Merit Fellowship, and the IEEE Signal Processing Society Travel Grant.

Andreas Tsiartas (S’10-M’14) received the B.Sc. degree in electronics and computer engineering from the Technical University of Crete, Crete, Greece, in 2006, and the M.Sc. and Ph.D. degrees in the Department of Electrical Engineering, University of Southern California (USC), Los Angeles, in 2014. He is currently a research engineer at SRI International. His main research direction focuses on speech-to-speech translation. Other research interests include acoustic and language modeling for automatic speech recognition (ASR) and voice activity detection.

Dr. Tsiartas received best teaching assistant awards for the years 2009 and 2010 in the Department of Electrical Engineering, USC. In 2006, he was awarded the Viterbi School Deans Doctoral Fellowship from USC.

Panagiotis Tsilifis was born in Aigio, Greece in 1985. He received his Diploma and M.Sc. degrees in Applied Mathematical Sciences from the National Technical University of Athens, Greece, in 2009 and 2011 respectively. In between, he also spent one year as an exchange student at the Royal Institute of Technology (KTH), in Stockholm, Sweden. He also received an M.A. in Applied Mathematics in 2014 from the University of Southern California (USC), Los Angeles, CA, USA. Currently he is pursuing his Ph.D. degree in Applied Mathematics at USC, working under the supervision of Prof. Roger G. Ghanem. His research focuses on Uncertainty Quantification methods including Bayesian approaches to optimal design and inference as well as construction of Polynomial Chaos surrogates with applications in geosciences, reservoir modeling and food microbiology.

Mr. Tsilifis received the Graduate Scholarship by the State Scholarship Foundation (IKY), Greece in 2010 and the Gerondelis Graduate Fellowship in 2013. He is also a SIAM student member and has received twice the SIAM student travel award, in 2015 and 2016.

Shrikanth S. Narayanan (S’88-M’95-SM’02-F’09) is Andrew J. Viterbi Professor of Engineering at the University of Southern California (USC), and holds appointments as Professor of Electrical Engineering, Computer Science, Linguistics, Psychology Neuroscience and Pediatrics and as the founding director of the Ming Hsieh Institute. Prior to USC he was with AT&T Bell Labs and AT&T Research from 1995-2000. At USC he directs the Signal Analysis and Interpretation Laboratory (SAIL). His research focuses on human-centered signal and information processing and systems modeling with an interdisciplinary emphasis on speech, audio, language, multimodal and biomedical problems and applications with direct societal relevance. [http://sail.usc.edu]

Prof. Narayanan is a Fellow of the Acoustical Society of America and the American Association for the Advancement of Science (AAAS) and a member of Tau Beta Pi, Phi Kappa Phi, and Eta Kappa Nu. He is also an Editor in Chief for the IEEE Journal of Selected Topics in Signal Processing and an Editor for the Computer Speech and Language Journal and an Associate Editor for the IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, IEEE TRANSACTIONS ON SIGNAL AND INFORMATION PROCESSING OVER NETWORKS, APSIPA TRANSACTIONS ON SIGNAL AND INFORMATION PROCESSING and the Journal of the Acoustical Society of America. He was also previously an Associate Editor of the IEEE TRANSACTIONS OF SPEECH AND AUDIO PROCESSING (2000–2004), IEEE SIGNAL PROCESSING MAGAZINE (2005–2008) and the IEEE TRANSACTIONS ON MULTIMEDIA (2008–2011). He is a recipient of a number of honors including Best Transactions Paper awards from the IEEE Signal Processing Society in 2005 (with A. Potamianos) and in 2009 (with C. M. Lee) and selection as an IEEE Signal Processing Society Distinguished Lecturer for 2010–2011 and ISCA Distinguished Lecturer for 2015–2016. Papers co-authored with his students have won awards including the 2014 Ten-year Technical Impact Award from ACM ICMI and at Interspeech 2015 Nativeness Detection Challenge, 2014 Cognitive Load Challenge, 2013 Social Signal Challenge, Interspeech 2012 Speaker Trait Challenge, Interspeech 2011 Speaker State Challenge, InterSpeech 2013 and 2010, InterSpeech 2009 Emotion Challenge, IEE DCOSS 2009, IEEE MMSP 2007, IEEE MMSP 2006, ICASSP 2005 and ICSLP 2002. He has published over 650 papers and has been granted seventeen U.S. patents.

Appendix A

MCMC Ergodicity Proofs

We provide the proofs for the theorems and lemmas of Section VI.

Proof of Lemma 6.1

We assume that the sampling order is Y₁, …, Y_B−1. Let $\{{\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}-1}^{\prime }\}$ and {y₁, …, y_B} be the block variables at the current and previous MCMC state, respectively. The conditional probability for sampling the first B − 1 blocks is

$$p({\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}-\mathbf{1}}^{\prime }|{\mathbf{y}}_{\mathbf{1}},\dots ,{\mathbf{y}}_{\mathbf{B}})=p({\mathbf{y}}_{\mathbf{B}-\mathbf{1}}^{\prime }|{\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}-\mathbf{2}}^{\prime },{\mathbf{y}}_{\mathbf{B}-\mathbf{1}},{\mathbf{y}}_{\mathbf{B}})·p({\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}-\mathbf{2}}^{\prime }|{\mathbf{y}}_{\mathbf{1}},\dots ,{\mathbf{y}}_{\mathbf{B}})=p({\mathbf{y}}_{\mathbf{B}-\mathbf{1}}^{\prime }|{\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}-\mathbf{2}}^{\prime },{\mathbf{y}}_{\mathbf{B}-\mathbf{1}},{\mathbf{y}}_{\mathbf{B}})·p({\mathbf{y}}_{\mathbf{B}-\mathbf{2}}^{\prime }|{\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}-\mathbf{3}}^{\prime },{\mathbf{y}}_{\mathbf{B}-\mathbf{2}},{\mathbf{y}}_{\mathbf{B}-\mathbf{1}},{\mathbf{y}}_{\mathbf{B}})··\dots ·p({\mathbf{y}}_{\mathbf{1}}^{\prime }|{\mathbf{y}}_{\mathbf{1}},\dots ,{\mathbf{y}}_{\mathbf{B}})=p({\mathbf{y}}_{\mathbf{B}-\mathbf{1}}^{\prime }|{\mathbf{y}}_{-{\mathbf{y}}_{\mathbf{B}-\mathbf{1}}})·p({\mathbf{y}}_{\mathbf{B}-\mathbf{2}}^{\prime }|{\mathbf{y}}_{-{\mathbf{y}}_{\mathbf{B}-\mathbf{2}}})·p({\mathbf{y}}_{\mathbf{1}}^{\prime }|{\mathbf{y}}_{-{\mathbf{y}}_{\mathbf{1}}})$$

where y_−b contains all variables except y_b. The Markov kernel for the first B − 1 blocks can be written as in (30). The first and second inequalities in (30) occur from the minorization of the (B − 1)^th and (B − 2)^th blocks. Therefore $\exists {\epsilon }_0={\prod }_{b=1}^{B-1}{\epsilon }_b>0$ and ${\nu }_0={\prod }_{b-1}^{B-1}{\nu }_b(A_b)$ (a probability measure) that satisfy the desired inequality.

$$P(({\mathbf{Y}}_{\mathbf{1}},{\mathbf{Y}}_{\mathbf{2}},\dots ,{\mathbf{Y}}_{\mathbf{B}-\mathbf{1}}),A_{-B})=\underset{A_1\dots A_{B-1}}{\int }p({\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}-\mathbf{1}}^{\prime }|{\mathbf{y}}_{\mathbf{1}},\dots ,{\mathbf{y}}_{\mathbf{B}})\mu (d(\mathbf{y}{\prime }_{-\mathbf{B}}))=\underset{A_1\dots A_{B-1}}{\int }p({\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}-\mathbf{2}}^{\prime }|{\mathbf{y}}_{-{\mathbf{y}}_1},\dots ,{\mathbf{y}}_{-{\mathbf{y}}_{\mathbf{B}-\mathbf{2}}})p({\mathbf{y}}_{\mathbf{B}-\mathbf{1}}^{\prime }|{\mathbf{y}}_{-{\mathbf{y}}_{\mathbf{B}-\mathbf{1}}}){\mu }_{B-1}(d{\mathbf{y}}_{\mathbf{B}-\mathbf{1}}^{\prime })\dots {\mu }_1(d{\mathbf{y}}_{\mathbf{1}}^{\prime })=\underset{A_1\dots A_{B-2}}{\int }p({\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}-\mathbf{2}}^{\prime }|{\mathbf{y}}_{-{\mathbf{y}}_{\mathbf{1}}},\dots ,{\mathbf{y}}_{-{\mathbf{y}}_{\mathbf{B}-\mathbf{2}}})(\underset{A_{B-1}}{\int }p({\mathbf{y}}_{\mathbf{B}-\mathbf{1}}^{\prime }|{\mathbf{y}}_{-{\mathbf{y}}_{\mathbf{B}-\mathbf{1}}}){\mu }_{B-1}(d{\mathbf{y}}_{\mathbf{B}-\mathbf{1}}^{\prime })){\mu }_{B-2}(d{\mathbf{y}}_{\mathbf{B}-\mathbf{2}}^{\prime })\dots {\mu }_1(d{\mathbf{y}}_{\mathbf{1}}^{\prime })\ge {\epsilon }_{B-1}{\nu }_{B-1}(A_{B-1})\underset{A_1\dots A_{B-3}}{\int }p({\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}-\mathbf{3}}^{\prime }|\dots )(\underset{A_{B-2}}{\int }p({\mathbf{y}}_{\mathbf{B}-\mathbf{2}}^{\prime }|{\mathbf{y}}_{-{\mathbf{y}}_{\mathbf{B}-\mathbf{2}}}){\mu }_{\mathbf{B}-\mathbf{2}}(d{\mathbf{y}}_{\mathbf{B}-\mathbf{2}}^{\prime })){\mu }_{B-3}(d{\mathbf{y}}_{\mathbf{B}-\mathbf{3}}^{\prime })\dots {\mu }_1(d{\mathbf{y}}_{\mathbf{1}}^{\prime })\ge \dots \ge \prod _{b-1}^{B-1}{\epsilon }_b{\nu }_b(A_b)$$ (30)

Proof of Theorem 6.2

If we assume that the sampling order is Y₁, …, Y_B, the Markov kernel of the Gibbs sampler can be expressed as in (31). The first inequality in (31) results from Lemma 6.1 and can yield to a minorization condition for the entire Gibbs sampler.

$$P(({\mathbf{Y}}_{\mathbf{1}},{\mathbf{Y}}_{\mathbf{2}},\dots ,{\mathbf{Y}}_{\mathbf{B}}),A_1\times A_2\times \dots \times A_B)=\underset{A_1\dots A_B}{\int }p({\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}}^{\prime }|{\mathbf{y}}_{\mathbf{1}},\dots ,{\mathbf{y}}_{\mathbf{B}})\mu (d({\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}}^{\prime }))=\underset{A_B}{\int }p({\mathbf{y}}_{\mathbf{B}}^{\prime }|{\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}-\mathbf{1}}^{\prime },{\mathbf{y}}_{\mathbf{B}})\underset{A_1\dots A_{B-1}}{\int }p({\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}-\mathbf{1}}^{\prime }|{\mathbf{y}}_{\mathbf{1}},\dots ,{\mathbf{y}}_{\mathbf{B}-\mathbf{1}},{\mathbf{y}}_{\mathbf{B}})\mu (d({\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}-\mathbf{1}}^{\prime }))·{\mu }_B(d{\mathbf{y}}_{\mathbf{B}}^{\prime })\ge {\epsilon }_0\underset{A_B}{\int }p({\mathbf{y}}_{\mathbf{B}}^{\prime }|{\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}-\mathbf{1}}^{\prime },{\mathbf{y}}_{\mathbf{B}})\underset{A_1\dots A_{B-1}}{\int }{\nu }_0(d({\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}-\mathbf{1}}^{\prime }))·{\mu }_B(d{\mathbf{y}}_{\mathbf{B}}^{\prime })={\epsilon }_0\underset{A_1\dots A_B-1}{\int }p({\mathbf{y}}_{-{\mathbf{y}}_{\mathbf{B}}},A_b)\nu (d({\mathbf{y}}_{\mathbf{1}}^{\prime },\dots ,{\mathbf{y}}_{\mathbf{B}-\mathbf{1}}^{\prime })$$ (31)

Proof of Lemma 6.2

From Def. 6.1, Table II, and (7–8)

$$w_{{\mathbf{y}}_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}}\propto \text{exp }(-\frac{{\gamma }_{{\epsilon }_n}}{2}{\Vert {\mathit{\epsilon }}_{\mathit{n}}\Vert }^2)\frac{{|{\tilde{\mathbf{G}}}_{\mathbf{n}}|}^{\frac{1}{2}}}{{|{\mathit{V}}_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}|}^{-1/2}}·\frac{\text{exp }[-\frac{1}{2}{({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\tilde{\mathbf{g}}}_{\mathbf{n}})}^T{\tilde{\mathbf{G}}}_{\mathbf{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\tilde{\mathbf{g}}}_{\mathbf{n}})]}{{[1+\frac{1}{{\nu }_1+Q}{({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\hat{\mathit{\mu }}}_{{\tilde{\mathit{\theta }}}_{\mathit{n}}})}^T{\hat{\mathbf{V}}}_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}^{-1}({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\hat{\mathit{\mu }}}_{{\tilde{\mathit{\theta }}}_{\mathit{n}}})]}^{-\frac{{\nu }_1+Q}{2}}}$$ (32)

Using (10), we have

$$\frac{{|{\tilde{\mathbf{G}}}_{\mathbf{n}}|}^{\frac{1}{2}}}{{|{\mathit{V}}_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}|}^{-\frac{1}{2}}}=\tau {|{\tilde{\mathbf{G}}}_{\mathbf{n}}|}^{\frac{1}{2}}<\infty \text{ and }\frac{1}{{\prod }_{k=1}^K{|{\mathit{V}}_{{\mathit{\theta }}_{\mathit{k}}}|}^{-\frac{1}{2}}}={\tau }^K<\infty$$

since G̃_n is the precision matrix of Gaussian distribution, therefore has finite eigenvalues and determinants, and τ < ∞. Moreover $\prod _{n=1}^N\text{exp }(-\frac{{\gamma }_{{\epsilon }_n}}{2}{\Vert {\mathit{\epsilon }}_{\mathit{n}}\Vert }^2)<1$ because $\frac{{\gamma }_{{\epsilon }_n}}{2}{\Vert {\mathit{\epsilon }}_{\mathit{n}}\Vert }^2\ge 0$ for n = 1, …, N and γ_εn > 0. Finally the function

$$f(\widetilde{{\mathit{\theta }}_{\mathit{n}}})=\frac{{[1+\frac{1}{{\nu }_1+Q}{({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\hat{\mathit{\mu }}}_{{\tilde{\mathit{\theta }}}_{\mathit{n}}})}^T{\hat{\mathbf{V}}}_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}^{-1}({\tilde{\mathit{\theta }}}_n-{\hat{\mathit{\mu }}}_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}\left)\right]}^{\frac{{\nu }_1+Q}{2}}}{\text{exp }(\frac{1}{2}{({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\tilde{\mathbf{g}}}_{\mathbf{n}})}^T{\tilde{\mathbf{G}}}_{\mathbf{n}}({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\tilde{\mathbf{g}}}_{\mathbf{n}}))}$$ (33)

is bounded since f(θ̃_n) → 0, ‖θ̃_n‖ → ∞, and f is continuous. Function f remains bounded at ‖θ̃_n‖ → ∞, since the quadratic forms (θ̃_n − g̃_n)^T G̃_n(θ̃_n − g̃_n) and ${({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\tilde{\mathbf{g}}}_{\mathbf{n}})}^T{\hat{\mathbf{V}}}_{{\tilde{\mathit{\theta }}}_{\mathit{n}}}^{-1}({\tilde{\mathit{\theta }}}_{\mathit{n}}-{\tilde{\mathbf{g}}}_{\mathbf{n}})$ increase to infinity at the same rate, and the denominator increases exponentially fast, while the numerator with polynomial rate.

Proof of Theorem 6.3

The first B − 1 blocks of y_n, that are generated with the Gibbs sampler, follow well-known distributions (Table I), therefore it is trivial to show that their updates are minorisable. From Theorem 6.2, since the partial updates for all blocks except the last one are minorisable, the entire Gibbs sampler is minorisable, therefore uniformly ergodic. From Lemma 6.2, the conditional weight of the B^th block y_θ̃n generated with the MH is bounded. Thus the MH-within-Gibbs sampler meets the conditions of Theorem 6.1, therefore it is uniformly ergodic.

Appendix B

Computational Complexity

We analyze the computational complexity of our proposed framework for each MCMC step (Algorithm 1, Table II) using the “𝒪” notation. For an input signal and MCMC iteration, the weight of the Multinomial distribution for each atom is computed with cost 𝒪(P), i.e. 𝒪(PK) for all atoms. Sampling L indices from K dictionary atoms with replacement results in 𝒪(LK), therefore the final cost yields 𝒪(PK + LK). For sampling without replacement, re-adjusting the atom weights requires 𝒪 ((K − 1) + … + (K − L + 1)) ~ 𝒪(LK). Since each of the L iterations takes into account the previously selected atoms [55], the cost of the Wallenius distribution is 𝒪 (K + (K − 1) + … + (K − L + 1)) ~ 𝒪(LK).

Each of the L coefficients is generated from the normal distribution, whose mean requires 𝒪(P), therefore the entire complexity is 𝒪(LP). Regarding the dictionary atom parameters, their posterior (7) requires 𝒪 (L(P + Q²)), while the typical cost of the Nelder Mead’s simplex is 𝒪 (Q²) [80]. Finally, complexity results in 𝒪(1) for the noise ε_n, 𝒪(K + L) for the Dirichlet prior π_n, 𝒪(Q²) for the mean g_nk and precision G_n of the dictionary parameters prior, and 𝒪(P) for the noise precision γ_εn.

Taking these into account, the total complexity when using sampling with and without replacement, respectively, yields:

$$𝒪(PK+LK+2LP+(L+3)Q^2+P+K+L+1)~𝒪(PK+LK+LP+L{Q}^2)$$ (34)

$$𝒪(2LK+2LP+(L+3)Q^2+P+K+L+1)~𝒪(LK+LP+L{Q}^2)$$ (35)

Our approach requires first-order polynomial time with the signal dimensionality P, the number of dictionary atoms K, and the number of selected atoms in the representation L ≪ K, and second-order polynomial cost with respect to the number of dictionary parameters Q. We note that Q ≪ P, L, therefore the latter is not too expensive.

We further compare run-time statistics of all the approaches. We report the average duration of one training iteration and the number of estimated variables for each approach (Table B1). Experiments were performed with an Intel Core i7 Processor with CPU at 2.93Hz and RAM at 7.8GB. SD and K-SVD require a decomposition step, therefore they compute slightly more variables than ETF. Our method estimates the highest number of variables, including the priors of the considered problem. Results indicate that SD and ETF are computationally less expensive than the proposed MCMC. Consistently with previous observations [19], K-SVD also has high computational cost. MCMC sampling with and without replacement yield computation times of the same order.

TABLE B1.

Average computation time of dictionary learning algorithms

Method	# Variables	Computation Time (sec / training iteration)
SD	438	0.1173
ETF	432	0.0616
K-SVD	438	3.9536
MCMC w- rplcm	1024	3.5014
MCMC w-o rplcm	1024	4.9638