Bayesian Inference for Nonnegative Matrix Factorisation Models

Abstract

We describe nonnegative matrix factorisation (NMF) with a Kullback-Leibler (KL) error measure in a statistical framework, with a hierarchical generative model consisting of an observation and a prior component. Omitting the prior leads to the standard KL-NMF algorithms as special cases, where maximum likelihood parameter estimation is carried out via the Expectation-Maximisation (EM) algorithm. Starting from this view, we develop full Bayesian inference via variational Bayes or Monte Carlo. Our construction retains conjugacy and enables us to develop more powerful models while retaining attractive features of standard NMF such as monotonic convergence and easy implementation. We illustrate our approach on model order selection and image reconstruction.

1. Introduction

In machine learning, nonnegative matrix factorisation (NMF) was introduced by Lee and Seung [1] as an alternative to k-means clustering and principal component analysis (PCA) for data analysis and compression (also see [2]). In NMF, given a W × K nonnegative matrix X = {x _ν,τ}, where ν = 1 : W, τ = 1 : K, we seek positive matrices T and V such that

$$x_{\nu \mathrm{,}\tau }\approx {[TV]}_{\nu \mathrm{,}\tau }=\sum _i{t}_{\nu \mathrm{,}i}{v}_{i\mathrm{,}\tau }\mathrm{,}$$ (1)

where i = 1 : I. We will refer to the W × I matrix T as the template matrix, and I × K matrix V the excitation matrix. The key property of NMF is that T and V are constrained to be positive matrices. This is in contrast with PCA, where there are no positivity constraints or k-means clustering where each column of V is constrained to be a unit vector. Subject to the positivity constraints, we seek a solution to the following minimisation problem:

$${(T\mathrm{,}V)}^{*}=\text{arg}\hspace{0.17em}\underset{T\mathrm{,}V>0}{\text{min}\hspace{0.17em}}\hspace{0.17em}D(X\parallel TV).$$ (2)

Here, the function D is a suitably chosen error function. One particular choice for D, on which we will focus here, is the information (Kullback-Leibler) divergence, which we write as

$$D(X\parallel \Lambda )=-{\displaystyle {\displaystyle \sum }_{\nu \mathrm{,}\tau }}\left(x_{\nu \mathrm{,}\tau }\hspace{0.17em}\hspace{0.17em}\text{log}\frac{{\lambda }_{\nu \mathrm{,}\tau }}{x_{\nu \mathrm{,}\tau }}-{\lambda }_{\nu \mathrm{,}\tau }+x_{\nu \mathrm{,}\tau }\right)\mathrm{.}$$ (3)

Using Jensen's inequality [3] and concavity of log x, it can be shown that D(·) is nonnegative and D(X||Λ) = 0 if and only if X = Λ. The objective in (2) could be minimised by any suitable optimisation algorithm. Lee and Seung [1] have proposed a very efficient variational bound minimisation algorithm that has attractive convergence properties and which has been successfully applied in various applications in signal analysis and source separation, for example, [4–6].

The interpretation of NMF, like singular value decomposition (SVD), as a low rank matrix approximation is sufficient for the derivation of a useful inference algorithm; yet this view arguably does not provide the complete picture about assumptions underlying the statistical properties of X. Therefore, we describe NMF from a statistical perspective as a hierarchical model. In our framework, the original nonnegative multiplicative update equations of NMF appear as an expectation-maximisation (EM) algorithm for maximum likelihood estimation of a conditionally Poisson model via data augmentation. Starting from this view, we develop Bayesian extensions that facilitate more powerful modelling and allow more sophisticated inference, such as Bayesian model selection. Inference in the resulting models can be carried out easily using variational (structured mean field) or Markov Chain Monte Carlo (Gibbs sampler). The resulting algorithms outperform existing NMF strategies and open up the way for a full Bayesian treatment for model selection via computation of the marginal likelihoods (the evidence), such as estimating the dimensions of the template matrix or regularising overcomplete representations via automatic relevance determination.

2. The Statistical Perspective

The interpretation of NMF as a low-rank matrix approximation is sufficient for the derivation of an inference algorithm; yet this view arguably does not provide the complete picture. In this section, we describe NMF from a statistical perspective. This view will pave the way for developing extensions that facilitate more realistic and flexible modelling as well as more sophisticated inference, such as Bayesian model selection.

Our first step is the derivation of the information divergence error measure from a maximum likelihood principle. We consider the following hierarchical model:

$$\begin{array}{l}T\sim p(T\hspace{0.17em}\mid \hspace{0.17em}{\Theta }^t),\hspace{1em}\hspace{1em}V\sim p(V\hspace{0.17em}\mid \hspace{0.17em}{\Theta }^v),\end{array}$$ (4)

$$\begin{array}{c}{s}_{\nu \mathrm{,}i\mathrm{,}\tau }\sim 𝒫𝒪(s_{\nu \mathrm{,}i\mathrm{,}\tau };t_{\nu \mathrm{,}i}{v}_{i\mathrm{,}\tau }),\hspace{1em}\hspace{1em}{x}_{\nu \mathrm{,}\tau }={\displaystyle \sum _i}{s}_{\nu \mathrm{,}i\mathrm{,}\tau }\mathrm{.}\end{array}$$ (5)

Here, 𝒫𝒪(s; λ) denotes the Poisson distribution of the random variable s ∈ ℕ ₀ with nonnegative intensity parameter λ, where

$$𝒫𝒪(s;\lambda )=\text{exp}(s\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}\hspace{0.17em}\lambda -\lambda -\text{log}\hspace{0.17em}\Gamma (s+1))$$ (6)

and Γ(s + 1) = s! is the gamma function. The priors p(T ∣ ·) and p(V ∣ ·) will be specified later. We call the variables S _i = {s _ν,i,τ} latent sources. We can analytically marginalise out the latent sources S = {S ₁ ⋯ S _I} to obtain the marginal likelihood

$$\begin{array}{ll}\text{log}\hspace{0.17em}p(X\hspace{0.17em}\mid \hspace{0.17em}T\mathrm{,}V)& =\text{log}\hspace{0.17em}{\displaystyle \sum _S}p(X\hspace{0.17em}\mid \hspace{0.17em}S)p(S\hspace{0.17em}\mid \hspace{0.17em}T\mathrm{,}V)\\ & =\text{log}\hspace{0.17em}\prod _{\nu \mathrm{,}\tau }𝒫𝒪(x_{\nu \mathrm{,}\tau };{\displaystyle \sum _i}{t}_{\nu \mathrm{,}i}\mathrm{,}{v}_{i\mathrm{,}\tau })\\ & ={\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _{\tau }}(x_{\nu \mathrm{,}\tau }\hspace{0.17em}\hspace{0.17em}\text{log}{[TV]}_{\nu \mathrm{,}\tau }-{[TV]}_{\nu \mathrm{,}\tau }-\text{log}\hspace{0.17em}\Gamma (x_{\nu \mathrm{,}\tau }+1\left)\right)\mathrm{.}\end{array}$$ (7)

This result follows from the well-known superposition property of Poisson random variables [7], namely, when s _i ∼ 𝒫𝒪(s _i; λ _i) and x = s ₁ + s ₂ + ⋯ + s _I, then the marginal probability is given by p(x) = 𝒫𝒪(x; ∑_i λ _i). The maximisation of this objective in T and V is equivalent to the minimisation of the information divergence in (3). In the derivation of original NMF in [8], this objective is stated first; the S variables are introduced implicitly later during the optimisation on T and V. In the sequel, we show that this algorithm is actually equivalent to EM, ignoring the priors p(T ∣ ·) and p(V ∣ ·).

2.1. Maximum Likelihood and the EM Algorithm

The log-likelihood of the observed data X can be written as

$$\begin{array}{ll}{ℒ}_X(T\mathrm{,}V)& \equiv \text{log}{\displaystyle \hspace{0.17em}\sum _S\hspace{0.17em}}p(X\hspace{0.17em}\mid \hspace{0.17em}S)p(S\hspace{0.17em}\mid \hspace{0.17em}T\mathrm{,}V)\\ & \ge {\displaystyle \sum _S}q(S)\hspace{0.17em}\text{log}\frac{p(X\mathrm{,}S\hspace{0.17em}\mid \hspace{0.17em}T\mathrm{,}V)}{q(S)}\equiv {ℬ}_{\text{EM}}[q],\end{array}$$ (8)

where q(S) is an instrumental distribution, that is arbitrary provided that the sum on the right exists; q can only vanish at a particular S only when p does so. Note that this defines a lower bound to the log-likelihood. It can be shown via functional derivatives and imposing the normalisation condition ∑_S q(S) = 1 via Lagrange multipliers that the lower bound is tight for the exact posterior of the latent sources, that is,

$$\text{arg}\hspace{0.17em}\underset{q(S)}{\text{max}\hspace{0.17em}}\hspace{0.17em}{ℬ}_{\text{EM}}[q]=p(S\hspace{0.17em}\mid \hspace{0.17em}X\mathrm{,}T\mathrm{,}V)\mathrm{.}$$ (9)

Hence the log-likelihood can be maximised iteratively as follows:

$$\begin{array}{c}\text{E}\hspace{0.17em}\hspace{0.17em}\text{Step}\hspace{1em}q{(S)}^{(n)}=p(S\hspace{0.17em}\mid \hspace{0.17em}X\mathrm{,}{T}^{(n-1)}\mathrm{,}{V}^{(n-1)}),\hfill \\ \text{M}\hspace{0.17em}\hspace{0.17em}\text{Step}\hspace{1em}\left(T^{(n)}\mathrm{,}{V}^{(n)}\right)=\text{arg}\hspace{0.17em}\underset{T\mathrm{,}V}{\text{max}\hspace{0.17em}}\hspace{0.17em}{\langle \hspace{0.17em}\text{log}\hspace{0.17em}p(S\mathrm{,}X\hspace{0.17em}\mid \hspace{0.17em}T\mathrm{,}V)\rangle }_{q{(S)}^{(n)}}\mathrm{.}\hfill \end{array}$$ (10)

Here, 〈f(x)〉_p(x) = ∫ p(x)f(x)dx, the expectation of some function f(x) with respect to p(x). In the E step, we compute the posterior distribution of S. This defines a lower bound on the likelihood

$${ℬ}^{(n)}(T\mathrm{,}V\hspace{0.17em}\mid \hspace{0.17em}{T}^{(n-1)}\mathrm{,}{V}^{(n-1)})={\langle \hspace{0.17em}\text{log}\hspace{0.17em}p(S\mathrm{,}X\hspace{0.17em}\mid \hspace{0.17em}T\mathrm{,}V)\rangle }_{q{(S)}^{(n)}}\mathrm{.}$$ (11)

For many models in the exponential family, which includes (5), the expectation on the right depends on the sufficient statistics of q(S)⁽ⁿ⁾ and is readily available; in fact calculating q(S) should be literally taken as calculating the sufficient statistics of q(S). The lower bound is readily obtained as a function of these sufficient statistics, and maximisation in the M Step yields a fixed point equation.

2.1.1. The E Step

To derive the posterior of the latent sources, we observe that

$$p(S\hspace{0.17em}\mid \hspace{0.17em}X\mathrm{,}T\mathrm{,}V)=\frac{p(S\mathrm{,}X\hspace{0.17em}\mid \hspace{0.17em}T\mathrm{,}V)}{p(X\hspace{0.17em}\mid \hspace{0.17em}T\mathrm{,}V)}.$$ (12)

For the model in (5), we have

$$\begin{array}{ll}& \log p(S\mathrm{,}X\hspace{0.17em}\mid \hspace{0.17em}T\mathrm{,}V)\\ & \hspace{1em}\hspace{1em}={\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _{\tau }}({\displaystyle \sum _i}(-t_{\nu \mathrm{,}i}{v}_{i\mathrm{,}\tau }+s_{\nu \mathrm{,}i\mathrm{,}\tau }\hspace{0.17em}\hspace{0.17em}\text{log}\left(t_{\nu \mathrm{,}i}{v}_{i\mathrm{,}\tau }\right)-\text{log}\hspace{0.17em}\mathrm{\Gamma }(s_{\nu \mathrm{,}i\mathrm{,}\tau }+1))\\ & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\text{log}\hspace{0.17em}\delta (x_{\nu \mathrm{,}\tau }-{\displaystyle \sum _i}{s}_{\nu \mathrm{,}i\mathrm{,}\tau }))\mathrm{.}\end{array}$$ (13)

It follows from (5), (12), (13), and (7)

$$\begin{array}{ll}& \text{log}\hspace{0.17em}p(S\hspace{0.17em}\mid \hspace{0.17em}X\mathrm{,}T\mathrm{,}V)\\ & \hspace{1em}\hspace{1em}={\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _{\tau }}({\displaystyle \sum _i}\left(s_{\nu \mathrm{,}i\mathrm{,}\tau }\hspace{0.17em}\hspace{0.17em}\text{log}\left(\frac{t_{\nu \mathrm{,}i}{v}_{i\mathrm{,}\tau }}{{\sum }_{i\text{'}}{t}_{\nu \mathrm{,}i\text{'}}{v}_{i\text{'},\tau }}\right)-\text{log}\hspace{0.17em}\Gamma (s_{\nu \mathrm{,}i\mathrm{,}\tau }+1)\right)\\ & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\text{log}\hspace{0.17em}\Gamma (x_{\nu \mathrm{,}\tau }+1)+\text{log}\hspace{0.17em}\delta (x_{\nu \mathrm{,}\tau }-{\displaystyle \sum _i}{s}_{\nu \mathrm{,}i\mathrm{,}\tau }))\\ & \hspace{1em}\hspace{1em}={\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _{\tau }}\text{log}\hspace{0.17em}ℳ(s_{\nu \mathrm{,1,}\tau }\mathrm{,}\dots \mathrm{,}{s}_{\nu \mathrm{,}I\mathrm{,}\tau };x_{\nu \mathrm{,}\tau }\mathrm{,}{p}_{\nu \mathrm{,1,}\tau }\mathrm{,}\dots \mathrm{,}{p}_{\nu \mathrm{,}I\mathrm{,}\tau }),\end{array}$$ (14)

where p _ν,i,τ ≡ t _ν,i v _i,τ/∑_i' t _ν,i' v _i',τ are the cell probabilities. Here, ℳ denotes a multinomial distribution defined by

$$\begin{array}{ll}ℳ(\mathbf{s};x\mathrm{,}\mathbf{p})& =\left(\begin{array}{c}x\\ s_1\hspace{0.17em}{s}_2\hspace{0.17em}\cdots \hspace{0.17em}{s}_I\end{array}\right)p_1^{s_1}{p}_2^{s_2}\cdots p_I^{s_I}\delta (x-{\displaystyle \sum _i}{s}_i)\\ & =\delta (x-{\displaystyle \sum _i}{s}_i)x!\prod _{i=1}^I\frac{p_i^{s_i}}{s_i!}\mathrm{,}\end{array}$$ (15)

where s = {s ₁, s ₂,…, s _I}, p = {p ₁, p ₂,…, p _I}, and p ₁ + p ₂ + ⋯ + p _I = 1. Here, p _i, i = 1 ⋯ I are the cell probabilities, and x is the index parameter where s ₁ + s ₂ + ⋯ + s _I = x. The Kronecker delta function is defined by δ(x) = 1 when x = 0, and δ(x) = 0 otherwise. It is a standard result that the marginal mean is

$$\langle s_i\rangle =x{p}_i\mathrm{,}$$ (16)

that is, the expected value of each source s _i is a fraction of the observation, where the fraction is given by the corresponding cell probability.

2.1.2. The M Step

It is indeed a good news that the posterior has an analytic form. Since now the M step can be calculated easily as follows:

$$\begin{array}{ll}& {\langle \text{log}\hspace{0.17em}p(S\mathrm{,}X\hspace{0.17em}\mid \hspace{0.17em}T\mathrm{,}V)\rangle }_{p(S\mid X\mathrm{,}T\mathrm{,}V)}\\ & ={\displaystyle \sum _{\nu }}{\displaystyle \sum _{\tau }}({\displaystyle \sum _i}(-t_{\nu \mathrm{,}i}{v}_{i\mathrm{,}\tau }+\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle \log \left(t_{\nu \mathrm{,}i}{v}_{i\mathrm{,}\tau }\right)-\langle \text{log}\hspace{0.17em}\Gamma (s_{\nu \mathrm{,}i\mathrm{,}\tau }+1)\rangle )\\ & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\langle \text{log}\hspace{0.17em}\delta (x_{\nu \mathrm{,}\tau }-{\displaystyle \sum _i}{s}_{\nu \mathrm{,}i\mathrm{,}\tau })\rangle )\mathrm{.}\end{array}$$ (17)

Fortunately, for maximisation with respect to T and V, the last two difficult terms are merely constant, and we need only to maximise the simpler objective

$$Q(T\mathrm{,}V)={\displaystyle \sum _{\nu }}{\displaystyle \sum _{\tau }}\left({\displaystyle \sum _i}\left(-t_{\nu \mathrm{,}i}{v}_{i\mathrm{,}\tau }+{\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle }^{(n)}\hspace{0.17em}\hspace{0.17em}\text{log}\left(t_{\nu \mathrm{,}i}{v}_{i\mathrm{,}\tau }\right)\right)\right)\mathrm{,}$$ (18)

where we only need the expected value of the sources given by the previous values of the templates and excitations:

$${\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle }^{(n)}=x_{\nu \mathrm{,}\tau }\frac{t_{\nu \mathrm{,}i}^{(n)}{v}_{i\mathrm{,}\tau }^{(n)}}{{\sum }_{i\text{'}}{t}_{\nu \mathrm{,}i\text{'}}^{(n)}{v}_{i\text{'},\tau }^{(n)}}\mathrm{.}$$ (19)

Maximisation of the objective Q and substituting 〈s _ν,i,τ〉⁽ⁿ⁾ give the following fixed point equations:

$$\begin{array}{ll}\frac{\partial Q}{\partial t_{\nu \mathrm{,}i}}& =-{\displaystyle \sum _{\tau }}{v}_{i\mathrm{,}\tau }^{(n)}+\frac{{\sum }_{\tau }{\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle }^{(n)}}{t_{\nu \mathrm{,}i}}\mathrm{,}\\ t_{\nu \mathrm{,}i}^{(n+1)}& =\frac{{\sum }_{\tau }{\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle }^{(n)}}{{\sum }_{\tau }{v}_{i\mathrm{,}\tau }^{(n)}}=t_{\nu \mathrm{,}i}^{(n)}\frac{{\sum }_{\tau }{x}_{\nu \mathrm{,}\tau }{v}_{i\mathrm{,}\tau }^{(n)}/{\sum }_{i\text{'}}{t}_{\nu \mathrm{,}i\text{'}}^{(n)}{v}_{i\text{'},\tau }^{(n)}}{{\sum }_{\tau }{v}_{i\mathrm{,}\tau }^{(n)}}\mathrm{,}\\ \frac{\partial Q}{\partial v_{i\mathrm{,}\tau }}& =-\sum _{\nu }{t}_{\nu \mathrm{,}i}^{(n)}+\frac{{\sum }_{\nu }{\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle }^{(n)}}{v_{i\mathrm{,}\tau }}\mathrm{,}\\ v_{i\mathrm{,}\tau }^{(n+1)}& =\frac{{\sum }_{\nu }{\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle }^{(n)}}{{\sum }_{\nu }{t}_{\nu \mathrm{,}i}^{(n)}}=v_{i\mathrm{,}\tau }^{(n)}\frac{{\sum }_{\nu }{t}_{\nu \mathrm{,}i}^{(n)}{x}_{\nu \mathrm{,}\tau }/{\sum }_{i\text{'}}{t}_{\nu \mathrm{,}i\text{'}}^{(n)}{v}_{i\text{'},\tau }^{(n)}}{{\sum }_{\nu }{t}_{\nu \mathrm{,}i}^{(n)}}\mathrm{.}\end{array}$$ (20)

Equation (20) is identical to the multiplicative update rules of [8]. However, our derivation via data augmentation obtains the same result as an EM algorithm. It is interesting to note that in literature, NMF is often described as EM-like; here, we show that it is actually just an EM algorithm. We see that the efficiency of NMF is due to the fact that the W × I × K object 〈S〉 needs not to be explicitly calculated as we only need its marginal statistics (sums across τ or ν).

We note that our model is valid when X is integer valued. See [9] for a detailed discussion about consequences of this issue. Here, we assume that for nonnegative real valued $\tilde{X}$, we only consider the integer part, that is, we let $\tilde{X}=X+E$, where E is a noise matrix with entries uniformly drawn in [0, 1). In practice, this is not an obstacle when the entries of X are large.

The interpretation of NMF as a maximum likelihood method in a Poisson model is mentioned in the original NMF paper [1] and discussed in more detail by [5, 10]. The equivalence of NMF and probabilistic latent sematic analysis is shown in [11]. Kameoka in [5] focuses on the optimisation and gives an equivalent description using auxiliary function maximisation. In contrast, the auxiliary variables can be viewed as model variables (the sources s) that are analytically integrated out [10]. A general framework is described in [12]. Prior structures are placed on conditionally Gaussian NMF models to enforce sparsity in [13]. However, all of these approaches are based on regularisation, that is, aim at calculating a maximum a posteriori estimate. In contrast, we provided in this article a full Bayesian treatment where the templates and excitations are integrated out.

2.2. Hierarchical Prior Structure

Given the probabilistic interpretation, it is possible to propose various hierarchical prior structures to fit the requirements of an application. Here we will describe a simple choice where we have a conjugate prior as follows:

$$t_{\nu \mathrm{,}i}\sim 𝒢(t_{\nu \mathrm{,}i};a_{\nu \mathrm{,}i}^t\mathrm{,}\frac{b_{\nu \mathrm{,}i}^t}{a_{\nu \mathrm{,}i}^t}),\hspace{1em}\hspace{1em}{v}_{i\mathrm{,}\tau }\sim 𝒢(v_{i\mathrm{,}\tau };a_{i\mathrm{,}\tau }^v\mathrm{,}\frac{b_{i\mathrm{,}\tau }^v}{a_{i\mathrm{,}\tau }^v}).$$ (21)

Here, 𝒢 denotes the density of a gamma random variable x ∈ ℝ₊ with shape a ∈ ℝ₊ and scale b ∈ ℝ₊ defined by

$$𝒢(x;a\mathrm{,}b)=\text{exp}((a-1)\hspace{0.17em}\text{log}\hspace{0.17em}x-\frac{x}{b}-\text{log}\hspace{0.17em}\Gamma (a)-a\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}b).$$ (22)

The primary motivation for choosing a Gamma distribution is computational convenience: Gamma distribution is the conjugate prior to Poisson intensity. The indexing highlights the most general case where there are individual parameters for each element t _ν,i and v _i,τ. Typically, we do not allow many free hyperparameters but tie them depending upon the requirements of an application. See Figure 1 for an example. As an example, consider a model where we tie the hyperparameters such as a _ν,i ^t = a ^t, b _ν,i ^t = b ^t, a _i,τ ^v = a ^v, and b _i,τ ^v = b ^v for i = 1 ⋯ I, ν = 1 ⋯ W, and τ = 1 ⋯ K. This model is simple to interpret, where each component of the templates and the excitations is drawn independently from the Gamma family shown in Figure 2. Qualitatively, the shape parameter a controls the sparsity of the representation. Remember that 𝒢(x; a, b/a) has the mean b and standard deviation $b/\sqrt{a}$. Hence, for large a, all coefficients will have more or less the same magnitude b, and typical representations will be full. In contrast, for small a, most of the coefficients will be very close to zero, and only very few will be dominating, hence favouring a sparse representation. The scale parameter b is adapted to give the expected magnitude of each component.

Figure 1.

(a) A schematic description of the NMF model with data augmentation. (b) Graphical model with hyperparameters. Each source element s _ν,i,τ is Poisson distributed with intensity t _ν,i v _i,τ. The observations are given by x _ν,τ = ∑_i s _ν,i,τ. In matrix notation, we write X = ∑ S _i. We can analytically integrate out over S. Due to superposition property of Poisson distribution, intensities add up, and we obtain 〈X〉 = TV. Given X, the NMF algorithm is shown to seek the maximum likelihood estimates of the templates T and excitations V. In our Bayesian treatment, we further assume that elements of T and V are Gamma distributed with hyperparameters Θ.

Figure 2.

(Left) The family of densities p(v; a, b = 1) = 𝒢(v; a, b/a) with the same mean 〈v〉 = b = 1. Small values of a (for a < 1) enforce sparser representations, and large values of a ≈ 100 tie all values to be close to a nonzero mean (nonsparse representation).

To model missing data, that is, when some of the x _ν,τ are not observed, we define a mask matrix M = {m _ν,τ}, the same size as X where m _ν,τ = 0, if x _ν,τ is missing and 1 otherwise (see Appendix A.4 for details). Using the mask variables, the observation model with missing data can be written as

$$p(X\hspace{0.17em}\mid \hspace{0.17em}S)p(S\hspace{0.17em}\mid \hspace{0.17em}T\mathrm{,}V)={\displaystyle \prod _{\nu \mathrm{,}\tau }}{\left(p(x_{\nu \mathrm{,}\tau }\hspace{0.17em}\mid \hspace{0.17em}{s}_{\nu \mathrm{,1}:I\mathrm{,}\tau })p(s_{\nu \mathrm{,1}:I\mathrm{,}\tau }\hspace{0.17em}\mid \hspace{0.17em}{t}_{\nu \mathrm{,1}:I}\mathrm{,}{v}_{1:I\mathrm{,}\tau })\right)}^{m_{\nu \mathrm{,}\tau }}\mathrm{.}$$ (23)

The hierarchical model in (21) is more powerful than the basic model of (5), in that it allows a lot of freedom for more realistic modelling. First of all, the hyperparameters can be estimated from examples of a certain class of source to capture the invariant features. Another possibility is Bayesian model selection, where we can compare alternative models in terms of their marginal likelihood. This enables one to estimate the model order, for example, the optimum number of templates to represent a source.

3. Full Bayesian Inference

Below, we describe various interesting problems that can be cast to Bayesian inference problems. In signal analysis and feature extraction with NMF, we may wish to calculate the posterior distribution of templates and excitations, given data and hyperparameters Θ ≡ (Θ^t, Θ^v). Another important quantity is the marginal likelihood (also known as the evidence), where

$$p(X\hspace{0.17em}\mid \hspace{0.17em}\Theta )={\displaystyle \int }dT\hspace{0.17em}\hspace{0.17em}dV{\displaystyle \sum _S}p(X\hspace{0.17em}\mid \hspace{0.17em}S)p(S\hspace{0.17em}\mid \hspace{0.17em}T\mathrm{,}V)p(T\mathrm{,}V\hspace{0.17em}\mid \hspace{0.17em}\Theta ).$$ (24)

The marginal likelihood can be used to estimate the hyperparameters, given examples of a source class

$${\Theta }^{*}=\arg \underset{\Theta }{\max }\hspace{0.17em}\hspace{0.17em}p(X\hspace{0.17em}\mid \hspace{0.17em}\Theta )$$ (25)

or to compare two given models via Bayes factors

$$l\left({\Theta }_1\mathrm{,}{\Theta }_2\right)=\frac{p(X\hspace{0.17em}\mid \hspace{0.17em}{\Theta }_1)}{p(X\hspace{0.17em}\mid \hspace{0.17em}{\Theta }_2)}\mathrm{.}$$ (26)

This latter quantity is particularly useful for comparing different classes of models. Unfortunately, the integrations required cannot be computed in closed form. In the sequel, we will describe the Gibbs sampler and variational Bayes as approximate inference strategies.

3.1. Variational Bayes

We sketch here the Variational Bayes (VB) [3, 14] method to bound the marginal log-likelihood as

$$\begin{array}{ll}{ℒ}_X(\mathrm{\Theta })& \equiv \text{log}\hspace{0.17em}p(X\hspace{0.17em}\mid \hspace{0.17em}\mathrm{\Theta })\ge {\displaystyle \sum _S}{\displaystyle \int }d(T\mathrm{,}V)q\hspace{0.17em}\hspace{0.17em}\text{log}\frac{p(X\mathrm{,}S\mathrm{,}T\mathrm{,}V\hspace{0.17em}\mid \hspace{0.17em}\Theta )}{q}\\ & ={\langle \text{log}\hspace{0.17em}p(X\mathrm{,}S\mathrm{,}V\mathrm{,}T\hspace{0.17em}\mid \hspace{0.17em}\Theta )\rangle }_q+H[q]\equiv {ℬ}_{\text{VB}}[q],\end{array}$$ (27)

where q = q(S, T, V) is an instrumental distribution, and H[q] is its entropy. The bound is tight for the exact posterior q(S, T, V) = p(S, T, V ∣ X, Θ) but as this distribution is complex, we assume a factorised form for the instrumental distribution by ignoring some of the couplings present in the exact posterior as follows:

$$\begin{array}{ll}q(S\mathrm{,}T\mathrm{,}V)& =q(S)q(T)q(V)\\ & =\left({\displaystyle \prod }_{\nu \mathrm{,}\tau }q\left(s_{\nu \mathrm{,1}:I\mathrm{,}\tau }\right)\right)\left({\displaystyle \prod }_{\nu \mathrm{,}i}q\left(t_{\nu \mathrm{,}i}\right)\right)\left({\displaystyle \prod }_{i\mathrm{,}\tau }q\left(v_{i\mathrm{,}\tau }\right)\right)\equiv {\displaystyle \prod }_{\mathbf{\alpha }\in 𝒞}{q}_{\mathbf{\alpha }}\mathrm{,}\end{array}$$ (28)

where α ∈ 𝒞 = {{S}, {T}, {V}} denotes set of disjoint clusters. Hence, we are no longer guaranteed to attain the exact marginal likelihood ℒ _X(Θ). Yet, the bound property is preserved, and the strategy of VB is to optimise the bound. Although the best q distribution respecting the factorisation is not available in closed form, it turns out that a local optimum can be attained by the following fixed point iteration:

$$q_{\mathbf{\alpha }}^{(n+1)}\propto \text{exp}\left({\langle \text{log}\hspace{0.17em}p(X\mathrm{,}S\mathrm{,}T\mathrm{,}V\hspace{0.17em}\mid \hspace{0.17em}\Theta )\rangle }_{q_{\neg \mathbf{\alpha }}^{(n)}}\right)\mathrm{,}$$ (29)

where q _¬α = q/q _α. This iteration monotonically improves the individual factors of the q distribution, that is, ℬ[q ⁽ⁿ⁾] ≤ ℬ[q ⁽ⁿ⁺¹⁾] for n = 1, 2,… given an initialisation q ⁽⁰⁾. The order is not important for convergence; one could visit blocks in arbitrary order. However, in general, the attained fixed point depends upon the order of the updates as well as the starting point q ⁽⁰⁾(·). We choose the following update order in our derivations:

$$q{(S)}^{(n+1)}\propto \text{exp}\left({\langle \text{log}\hspace{0.17em}p(X\mathrm{,}S\mathrm{,}T\mathrm{,}V\hspace{0.17em}\mid \hspace{0.17em}\Theta )\rangle }_{q{(T)}^{(n)}q{(V)}^{(n)}}\right)\mathrm{,}$$ (30)

$$q{(T)}^{(n+1)}\propto \text{exp}\left({\langle \text{log}\hspace{0.17em}p(X\mathrm{,}S\mathrm{,}T\mathrm{,}V\hspace{0.17em}\mid \hspace{0.17em}\Theta )\rangle }_{q{(S)}^{(n+1)}q{(V)}^{(n)}}\right)\mathrm{,}$$ (31)

$$q{(V)}^{(n+1)}\propto \text{exp}\left({\langle \text{log}\hspace{0.17em}p(X\mathrm{,}S\mathrm{,}T\mathrm{,}V\hspace{0.17em}\mid \hspace{0.17em}\Theta )\rangle }_{q{(S)}^{(n+1)}q{(T)}^{(n+1)}}\right)\mathrm{.}$$ (32)

3.2. Variational Update Equations and Sufficient Statistics

The expectations of 〈log p(X, S, T, V ∣ Θ)〉 are functions of the sufficient statistics of q (see the expression in the Appendix A.2). The update equation for the latent sources (30) leads to the following:

$$\begin{array}{ll}q\left(s_{\nu \mathrm{,1}:I\mathrm{,}\tau }\right)& \propto \text{exp}({\displaystyle \sum _i}(s_{\nu \mathrm{,}i\mathrm{,}\tau }(\langle \text{log}\hspace{0.17em}{t}_{\nu \mathrm{,}i}\rangle +\langle \text{log}\hspace{0.17em}{v}_{i\mathrm{,}\tau }\rangle )-\text{log}\hspace{0.17em}\mathrm{\Gamma }(s_{\nu \mathrm{,}i\mathrm{,}\tau }+1)))\delta (x_{\nu \mathrm{,}\tau }-{\displaystyle \sum _i}{s}_{\nu \mathrm{,}i\mathrm{,}\tau })\\ & \propto ℳ(s_{\nu \mathrm{,1,}\tau }\mathrm{,}\dots \mathrm{,}{s}_{\nu \mathrm{,}i\mathrm{,}\tau }\mathrm{,}\dots \mathrm{,}{s}_{\nu \mathrm{,}I\mathrm{,}\tau };\\ & \hspace{1em}\hspace{1em}\hspace{1em}{x}_{\nu \mathrm{,}\tau }\mathrm{,}{p}_{\nu \mathrm{,1,}\tau }\mathrm{,}\dots ,p_{\nu \mathrm{,}i\mathrm{,}\tau }\mathrm{,}\dots \mathrm{,}{p}_{\nu \mathrm{,}I\mathrm{,}\tau }),\\ & p_{\nu \mathrm{,}i\mathrm{,}\tau }=\frac{\text{exp}(\langle \text{log\hspace{0.17em}}{t}_{\nu \mathrm{,}i}\rangle +\langle \text{log\hspace{0.17em}}{v}_{i\mathrm{,}\tau }\rangle )}{{\sum }_i\hspace{0.17em}\exp (\langle \text{log\hspace{0.17em}}{t}_{\nu \mathrm{,}i}\rangle +\langle \text{log\hspace{0.17em}}{v}_{i\mathrm{,}\tau }\rangle )},\\ & \hspace{1em}\hspace{1em}\hspace{1em}\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle =x_{\nu \mathrm{,}\tau }{p}_{\nu \mathrm{,}i\mathrm{,}\tau }\mathrm{.}\end{array}$$ (33)

These equations are analogous to the multinomial posterior of EM given in (14); only the computation of cell probabilities is different. The excitation and template distributions and their sufficient statistics follow from the properties of the gamma distribution:

$$\begin{array}{cc}\hfill q\left(t_{\nu \mathrm{,}i}\right)& \propto \text{exp}((a_{\nu \mathrm{,}i}^t+{\displaystyle \sum _{\tau }}\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle -1)\log \left(t_{\nu \mathrm{,}i}\right)\hfill \\ & \hspace{1em}-\left(\frac{a_{\nu \mathrm{,}i}^t}{b_{\nu \mathrm{,}i}}+{\displaystyle \sum _{\tau }}\langle v_{i\mathrm{,}\tau }\rangle \right)t_{\nu \mathrm{,}i})\hfill \\ & \propto 𝒢\left(t_{\nu \mathrm{,}i};{\alpha }_{\nu \mathrm{,}i}^t\mathrm{,}{\beta }_{\nu \mathrm{,}i}^t\right)\mathrm{,}\hfill \\ \hfill {\alpha }_{\nu \mathrm{,}i}^t& \equiv a_{\nu \mathrm{,}i}^t+{\displaystyle \sum _{\tau }}\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle \mathrm{,}\hspace{1em}{\beta }_{\nu \mathrm{,}i}^t\equiv {\left(\frac{a_{\nu \mathrm{,}i}^t}{b_{\nu \mathrm{,}i}^t}+{\displaystyle \sum _{\tau }}\langle v_{i\mathrm{,}\tau }\rangle \right)}^{-1}\mathrm{,}\hfill \\ \hfill \exp \left(\langle \log \hspace{0.17em}{t}_{\nu \mathrm{,}i}\rangle \right)& =\text{exp}\left(\mathrm{\Psi }\right({\alpha }_{\nu \mathrm{,}i}^t\left)\right){\beta }_{\nu \mathrm{,}i}^t\mathrm{,}\hfill \\ \hfill \langle t_{\nu \mathrm{,}i}\rangle & ={\alpha }_{\nu \mathrm{,}i}^t{\beta }_{\nu \mathrm{,}i}^t\mathrm{,}\hfill \\ \hfill q\left(v_{i\mathrm{,}\tau }\right)& \propto \text{exp}((a_{i\mathrm{,}\tau }^v+{\displaystyle \sum _{\nu }}\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle -1)\text{log}\hspace{0.17em}{v}_{i\mathrm{,}\tau }\hfill \\ & \hspace{1em}\hspace{1em}-\left(\frac{a_{i\mathrm{,}\tau }^v}{b_{i\mathrm{,}\tau }^v}+{\displaystyle \sum _{\nu }}\langle t_{\nu \mathrm{,}i}\rangle \right)v_{i\mathrm{,}\tau })\hfill \\ & \propto 𝒢\left(v_{i\mathrm{,}\tau };{\alpha }_{i\mathrm{,}\tau }^v\mathrm{,}{\beta }_{i\mathrm{,}\tau }^v\right)\mathrm{,}\hfill \\ \hfill {\alpha }_{i\mathrm{,}\tau }^v& \equiv a_{i\mathrm{,}\tau }^v+{\displaystyle \sum _{\nu }}\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle \mathrm{,}\hspace{1em}{\beta }_{i\mathrm{,}\tau }^v\equiv {\left(\frac{a_{i\mathrm{,}\tau }^v}{b_{i\mathrm{,}\tau }^v}+{\displaystyle \sum _{\nu }}\langle t_{\nu \mathrm{,}i}\rangle \right)}^{-1}\mathrm{,}\hfill \\ \hfill \text{exp}\left(\langle \text{log\hspace{0.17em}}{v}_{i\mathrm{,}\tau }\rangle \right)& =\text{exp}\left(\mathrm{\Psi }\right({\alpha }_{i\mathrm{,}\tau }^v\left)\right){\beta }_{i\mathrm{,}\tau }^v\mathrm{,}\hfill \\ \hfill \langle v_{i\mathrm{,}\tau }\rangle & ={\alpha }_{i\mathrm{,}\tau }^v{\beta }_{i\mathrm{,}\tau }^v\mathrm{.}\hfill \end{array}$$ (34)

3.3. Efficient Implementation

One of the attractive features of NMF is easy and efficient implementation. In this section, we derive that the update equations of Section 3.2 in compact matrix notation are to illustrate that these attractive properties are retained for the full Bayesian treatment. A subtle but key point in the efficiency of the algorithm is that we can avoid explicitly storing and computing the W × I × K object 〈S〉, as we only need the marginal statistics during optimisation. Consider (33). We can write

$$\begin{array}{ll}\hfill {\displaystyle \sum _{\tau }}\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle & ={\displaystyle \sum _{\tau }}{x}_{\nu \mathrm{,}\tau }{p}_{\nu \mathrm{,}i\mathrm{,}\tau }\hfill \\ & =\text{exp}\left(\langle \text{log\hspace{0.17em}}{t}_{\nu \mathrm{,}i}\rangle \right)\\ & \hspace{1em}\hspace{1em}\times {\displaystyle \sum _{\tau }}\left(\frac{x_{\nu \mathrm{,}\tau }}{\left({\sum }_{i\text{'}}\text{exp}\left(\langle \text{log\hspace{0.17em}}{t}_{\nu \mathrm{,}i\text{'}}\rangle \right)\text{exp}\left(\langle \text{log\hspace{0.17em}}{v}_{i\text{'},\tau }\rangle \right)\right)}\right)\\ & \hspace{1em}\hspace{1em}\times \text{exp}\left(\langle \text{log\hspace{0.17em}}{v}_{i\mathrm{,}\tau }\rangle \right),\hfill \\ \hfill {\Sigma }_t& =L_t\hspace{0.17em}\hspace{0.17em}\mathrm{.}*\left(\right(X\mathrm{.}/\left(L_t{L}_v\right)\left)L_v^{\top }\right).\hfill \end{array}$$ (35)

Here, the denominator has to be nonzero. In the last line, we have represented the expression in compact notation where we define the following matrices:

$$\begin{array}{c}{E}_t=\left\{\langle t_{\nu \mathrm{,}i}\rangle \right\},\hspace{1em}\hspace{1em}{L}_t=\left\{\text{exp}\right(\langle \text{log\hspace{0.17em}}{t}_{\nu \mathrm{,}i}\rangle \left)\right\},\\ {\Sigma }_t=\left\{{\displaystyle \sum _{\tau }}\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle \right\},\hspace{1em}\hspace{1em}{A}_t=\left\{a_{\nu \mathrm{,}i}^t\right\},\\ B_t=\left\{b_{\nu \mathrm{,}i}^t\right\},\hspace{1em}\hspace{1em}{\alpha }_t=\left\{{\alpha }_{\nu \mathrm{,}i}^t\right\},\hspace{1em}\hspace{1em}{\beta }_t=\left\{{\beta }_{\nu \mathrm{,}i}^t\right\},\\ E_v=\left\{\langle v_{i\mathrm{,}\tau }\rangle \right\},\hspace{1em}\hspace{1em}{L}_v=\left\{\text{exp}\right(\langle \text{log}\hspace{0.17em}{v}_{i\mathrm{,}\tau }\rangle \left)\right\},\\ {\Sigma }_v=\left\{{\displaystyle \sum _{\nu }}\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle \right\},\hspace{1em}\hspace{1em}{A}_v=\left\{a_{i\mathrm{,}\tau }^v\right\},\\ B_v=\left\{b_{i\mathrm{,}\tau }^v\right\},\hspace{1em}\hspace{1em}{\alpha }_v=\left\{{\alpha }_{i\mathrm{,}\tau }^v\right\},\hspace{1em}\hspace{1em}{\beta }_v=\left\{{\beta }_{i\mathrm{,}\tau }^v\right\}.\end{array}$$ (36)

The matrices subscripted with t are in ℝ ₊ ^W×I and with v are in ℝ ₊ ^I×K. For notational convenience, we define. ∗ and ./ as elementwise matrix multiplication and division, respectively, and 1 _W as a W × 1 vector of ones. After straightforward substitutions, we obtain the variational nonnegative matrix factorisation algorithm, that can compactly be expressed as in panel Algorithm 1.

Algorithm 1.

Variational nonnegative matrix factorisation.

Similarly, an iterative conditional modes (ICM) algorithm can be derived to compute the maximum a posteriori (MAP) solution (see Appendix A.4):

$$\begin{array}{ll}V& :=(A_v+V\hspace{0.17em}\mathrm{.}*(T^{\top }((M\hspace{0.17em}\mathrm{.}*X)\mathrm{.}/(TV))\left)\right)\\ & \hspace{1em}\mathrm{.}/(A_v\mathrm{.}/B_v+T^{\top }M),\end{array}$$ (37)

$$\begin{array}{ll}T& :=(A_t+T\hspace{0.17em}\mathrm{.}*(((M\hspace{0.17em}\mathrm{.}*X)\mathrm{.}/(TV))V^{\top }\left)\right)\\ & \hspace{1em}\mathrm{.}/(A_t\mathrm{.}/B_t+M{V}^{\top }).\end{array}$$ (38)

Note that when the shape parameters go to zero, that is, A _t, A _v → 0, we obtain the maximum likelihood NMF algorithm.

3.4. Markov Chain Monte Carlo, the Gibbs Sampler

Monte Carlo methods [15, 16] are powerful computational techniques to estimate expectations of form

$$E={\langle f(x)\rangle }_{p(x)}\approx \frac{1}{N}{\displaystyle \sum _{n=1}^N}f\left(x^{(i)}\right)={\tilde{E}}_N\mathrm{,}$$ (39)

where x ⁽ⁱ⁾ are independent samples drawn from p(x). Under mild conditions on f, the estimate ${\tilde{E}}_N$ converges to the true expectation for N → ∞. The difficulty here is obtaining independent samples {x ⁽ⁱ⁾}_{i=1 ⋯ N} from complicated distributions.

The Markov Chain Monte Carlo (MCMC) techniques generate subsequent samples from a Markov chain defined by a transition kernel 𝒯, that is, one generates x ⁽ⁱ⁺¹⁾ conditioned on x ⁽ⁱ⁾ as follows:

$$x^{(i+1)}\sim 𝒯(x\hspace{0.17em}\mid \hspace{0.17em}{x}^{(i)}).$$ (40)

Note that the transition kernel 𝒯 is not needed explicitly in practice; all is needed is a procedure to sample a new configuration, given the previous one. Perhaps surprisingly, even though subsequent samples are correlated, provided that 𝒯 satisfies certain ergodicity conditions, (39) remains still valid, and estimated expectations converge to their true values when number of samples N goes to infinity [15]. To design a transition kernel 𝒯 such that the desired distribution is the stationary distribution, that is, p(x) = ∫ d x′𝒯(x ∣ x′)p(x′), many alternative strategies can be employed [16]. One particularly convenient and simple procedure is the Gibbs sampler where one samples each block of variables from full conditional distributions. For the NMF model, a possible Gibbs sampler is

$$\begin{array}{c}{S}^{(n+1)}\sim p(S\hspace{0.17em}\mid \hspace{0.17em}{T}^{(n)}\mathrm{,}{V}^{(n)}\mathrm{,}X\mathrm{,}\mathrm{\Theta }),\\ T^{(n+1)}\sim p(T\hspace{0.17em}\mid \hspace{0.17em}{V}^{(n)}\mathrm{,}{S}^{(n+1)}\mathrm{,}X\mathrm{,}\mathrm{\Theta }),\\ V^{(n+1)}\sim p(V\hspace{0.17em}\mid \hspace{0.17em}{S}^{(n+1)}\mathrm{,}{T}^{(n+1)}\mathrm{,}X\mathrm{,}\mathrm{\Theta }).\end{array}$$ (41)

Note that this procedure implicitly defines a transition kernel 𝒯(· ∣ ·). It can be shown [15] that the stationary distribution of 𝒯 is the exact posterior p(S, T, V ∣ X, Θ). Eventually, the Gibbs sampler converges regardless of the order that the blocks are visited, provided that each block is visited infinitely often in the limit n → ∞. However, the rate of convergence is very difficult to assess as it depends upon the order of the updates as well as the starting configuration (T ⁽⁰⁾, V ⁽⁰⁾, S ⁽⁰⁾). It is instructive to contrast above (41) with the variational update of (30)–(32): algorithmically the two approaches are quite similar. The pseudo-code is given in Algorithm 2.

Algorithm 2.

Gibbs sampler for nonnegative matrix factorisation.

3.4.1. Marginal Likelihood Estimation with Chib's Method

The marginal likelihood can be estimated from the samples generated by the Gibbs sampler using a method proposed by Chib [17]. Suppose we have run the block Gibbs sampler until convergence and have N samples as follows:

$${\left\{T^{(n)}\right\}}_{n=1\hspace{0.17em}\hspace{0.17em}:\hspace{0.17em}N},\hspace{1em}\hspace{1em}{\left\{V^{(n)}\right\}}_{n=1\hspace{0.17em}\hspace{0.17em}:\hspace{0.17em}N},\hspace{1em}\hspace{1em}{\left\{S^{(n)}\right\}}_{n=1\hspace{0.17em}\hspace{0.17em}:\hspace{0.17em}N}\mathrm{.}$$ (42)

The marginal likelihood is (omitting hyperparameters Θ)

$$p(X)=\frac{p(V\mathrm{,}T\mathrm{,}S\mathrm{,}X)}{p(V\mathrm{,}T\mathrm{,}S\hspace{0.17em}\mid \hspace{0.17em}X)}\mathrm{.}$$ (43)

This equation holds for all points (V, T, S). We choose a point in the configuration space; provided that the distribution is unimodal, a good candidate is a configuration near the mode $(\tilde{T}\mathrm{,}\tilde{V}\mathrm{,}\tilde{S})$. The numerator in (43) is easy to evaluate. The denominator is

$$\begin{array}{ll}p(\tilde{V}\mathrm{,}\tilde{T}\mathrm{,}\tilde{S}\hspace{0.17em}\mid \hspace{0.17em}X)& =p(\tilde{V}\hspace{0.17em}\mid \hspace{0.17em}\tilde{T}\mathrm{,}\tilde{S}\mathrm{,}X)p(\tilde{T}\hspace{0.17em}\mid \hspace{0.17em}\tilde{S}\mathrm{,}X)p(\tilde{S}\hspace{0.17em}\mid \hspace{0.17em}X)\\ & =p(\tilde{V}\hspace{0.17em}\mid \hspace{0.17em}\tilde{T}\mathrm{,}\tilde{S})p(\tilde{T}\hspace{0.17em}\mid \hspace{0.17em}\tilde{S})p(\tilde{S}\hspace{0.17em}\mid \hspace{0.17em}X).\end{array}$$ (44)

The first term is full conditional, so it is available for the Gibbs sampler. The third term is

$$\begin{array}{ll}p(\tilde{S}\hspace{0.17em}\mid \hspace{0.17em}X)& ={\displaystyle \int }dV\hspace{0.17em}\hspace{0.17em}dT\hspace{0.17em}\hspace{0.17em}p(\tilde{S}\hspace{0.17em}\mid \hspace{0.17em}V\mathrm{,}T\mathrm{,}X)p(V\mathrm{,}T\hspace{0.17em}\mid \hspace{0.17em}X)\\ & \approx \frac{1}{N}{\displaystyle \sum _{n=1}^N}p(\tilde{S}\hspace{0.17em}\mid \hspace{0.17em}{V}^{(n)}\mathrm{,}{T}^{(n)}\mathrm{,}X).\end{array}$$ (45)

The second term is trickier

$$p(\tilde{T}\hspace{0.17em}\mid \hspace{0.17em}\tilde{S})={\displaystyle \int }dV\hspace{0.17em}\hspace{0.17em}p(\tilde{T}\hspace{0.17em}\mid \hspace{0.17em}V\mathrm{,}\tilde{S})p(V\hspace{0.17em}\mid \hspace{0.17em}\tilde{S}).$$ (46)

The first term here is full conditional. However, the original Gibbs run gives us only samples from p(V ∣ X), not $p(V\hspace{0.17em}\mid \hspace{0.17em}\tilde{S})$. The idea is to run the Gibbs sampler for M further iterations where we sample from $(V_{\tilde{S}}^{(m)}\mathrm{,}{T}_{\tilde{S}}^{(m)})\sim p(V\mathrm{,}T\hspace{0.17em}\mid \hspace{0.17em}S=\tilde{S})$, that is, with S clamped at $\tilde{S}$. The resulting estimate is

$$p(\tilde{T}\hspace{0.17em}\mid \hspace{0.17em}\tilde{S})\approx \frac{1}{M}{\displaystyle \sum _{m=1}^M}p(\tilde{T}\hspace{0.17em}\mid \hspace{0.17em}{V}_{\tilde{S}}^{(m)}\mathrm{,}\tilde{S}).$$ (47)

Chib's method estimates the marginal likelihood as follows:

$$\begin{array}{ll}\text{log}\hspace{0.17em}p(X\hspace{0.17em}\mid \hspace{0.17em}\mathrm{\Theta })& =\text{log}\hspace{0.17em}p(\tilde{V}\mathrm{,}\tilde{T}\mathrm{,}\tilde{S}\mathrm{,}X\hspace{0.17em}\mid \hspace{0.17em}\mathrm{\Theta })-\text{log}\hspace{0.17em}p(\tilde{V}\mathrm{,}\tilde{T}\mathrm{,}\tilde{S}\hspace{0.17em}\mid \hspace{0.17em}X\mathrm{,}\mathrm{\Theta })\\ & \approx \text{log}\hspace{0.17em}p(\tilde{V}\mathrm{,}\tilde{T}\mathrm{,}\tilde{S}\mathrm{,}X\hspace{0.17em}\mid \hspace{0.17em}\mathrm{\Theta })-\text{log}\hspace{0.17em}p(\tilde{V}\hspace{0.17em}\mid \hspace{0.17em}\tilde{T}\mathrm{,}\tilde{S}\mathrm{,}\mathrm{\Theta })\\ & \hspace{1em}-\text{log}\hspace{0.17em}{\displaystyle \sum _{m=1}^M}p(\tilde{T}\hspace{0.17em}\mid \hspace{0.17em}{V}_{\tilde{S}}^{(m)}\mathrm{,}\tilde{S}\mathrm{,}\mathrm{\Theta })\\ & \hspace{1em}-\log {\displaystyle \sum _{n=1}^N}p(\tilde{S}\hspace{0.17em}\mid \hspace{0.17em}{V}^{(n)}\mathrm{,}{T}^{(n)}\mathrm{,}X\mathrm{,}\mathrm{\Theta })+\text{log}(MN).\end{array}$$ (48)

4. Simulations

Our goal is to illustrate our approach in a model selection context. We first illustrate that the variational approximation to the marginal likelihood is close to the one obtained from the Gibbs sampler via Chib's method. Then, we compare the quality of solutions we obtain via Variational NMF and compare them to the original NMF on a prediction task. Finally, we focus on reconstruction quality in the overcomplete case where the standard NMF is subject to overfitting.

Model Order Determination —

To test our approach, we generate synthetic data from the hierarchical model in (21) with W = 16, K = 10, and the number of sources I _true = 5. The inference task is to find the correct number of sources, given X. The hyperparameters of the true model are set to a _ν,i ^t = a ^t = 10, b _ν,i ^t = b ^t = 1, a _i,τ ^v = a ^v = 1, and b _i,τ ^v = b ^v = 100. In the first experiment, the hyperparameters are assumed to be known and in the second are jointly estimated from data, using hyperparameter adaptation. We evaluate the marginal likelihood for models with the number of templates I = 1 ⋯ 10, with the Gibbs sampler using Chib's method and variational lower bound ℬ via variational Bayes. We run the Gibbs sampler for MAXITER = 10 000 steps following a burn-in period of 5000 steps; then we clamp the sources S and continue the simulation for a further 10 000 steps to estimate quantities required by Chib's method. We run the variational algorithm until convergence of the bound or 10 000 iterations, whichever occurs first. In Figure 3(a), we show a comparison of the variational estimate with the average of 5 independent runs obtained via Chib's method. We observe, that both methods give consistent results. In Figure 4, we show the lower bound as a function of model order I, where for each I, the bound is optimised independently by jointly optimising hyperparameters a _t, b _t, a _v, and b _v using the equations derived in the appendix. We observe, that the correct model order can be inferred even when the hyperparameters are unknown a priori. This is potentially useful for estimation of model order from real data.

Figure 3.

Model selection results. (a) Comparison of model selection by variational bound (squares) and marginal likelihood estimated by Chib's (circles) method. The hyperparameters are assumed to be known. (b) Box-plot of marginal likelihood estimated by Chib's method using 5000, 10 000, and 10 000 iterations for burn-in, free, and clamped sampling. The boxes show the lower quartile, median, and upper quartile values. (c) Model selection by variational bound when hyperparameters are unknown and jointly estimated.

Figure 4.

Model selection using variational bound with adapted hyperparameters on face data 16 × 16 with I* = 27 (a) and 32 × 32 with I* = 42 (b).

As real data, we use a version of the Olivetti face image database (K = 400 images of 64 × 64 pixels available at http://www.cs.toronto.edu/∼roweis/data/olivettifaces.mat). We further downsampled to 16 × 16 or 32 × 32 pixels, hence our data matrix X is 16² × 400 or 32² × 400. We use a model with tied hyperparameters as a _ν,i ^t = a ^t, b _ν,i ^t = b ^t, a _i,τ ^v = a ^v, and b _i,τ ^v = b ^v, where all hyperparameters are jointly estimated. In Figure 4, bottom, we show results of model order determination for this dataset with joint hyperparameter adaptation. Here, we run the variational algorithm for each model order I = 1 ⋯ 100 independently and evaluate the lower bound after optimising the hyperparameters. The Gibbs sampler is not found practical and is omitted here. The lower bound behaves as is expected from marginal likelihood, reflecting the tradeoff between too many and too few templates. Higher resolution implies more templates, consistent with our intuition that detail requires more templates for accurate representation.

We also investigate the nature of the representations (see Figure 5). Here, for each independent run, we fix the values of shape parameters to (a ^t, a ^v) = [(10, 10), (0.1, 0.1), (10, 0.2), (10, 0.5)] and only estimate b ^t and b ^v. This corresponds to enforcing sparse or nonsparse t and v. Each column shows I = 36 templates estimated from the dataset conditioned on hyperparameters. The middle image is the same template image above weighted with the excitations corresponding to the reconstruction (the expected value of the predictive distribution) below. Here, we clearly see the effect of the hyperparameters. In the first condition (a ^t, a ^v) = (10, 10), the prior does not enforces sparsity to the templates and excitations. Hence, for the representation of a given image, there are many active templates. In the second condition, we try to force both matrices to be sparse with (a ^t, a ^v) = (0.1, 0.1). Here, the result is not satisfactory as isolated components of the templates are zeroed, giving a representation that looks like one contaminated by “salt-and-pepper” noise. The third condition ((a ^t, a ^v) = (10, 0.2)) forces only the excitations to be sparse. Here, we observe that the templates correspond to some average face images. Qualitatively, each image is reconstructed using a superposition of a few of these templates. In the final representation, we enforce sparsity in the templates but not in the excitations. Here, our estimate finds templates that correspond to parts of individual face images (eyebrows, lips, etc.). This solution, intuitively corresponding to a parsimonious representation, also is the best in terms of the marginal likelihood. With proper initialisation, our variational procedure is able to find such solutions.

Figure 5.

Templates, excitations for a particular example, and the reconstructions obtained for different hyperparameter settings. B is the lower bound for the whole dataset.

Prediction —

We now compare variational Bayesian NMF with the maximum likelihood NMF on a missing data prediction task.

To illustrate the self regularisation effect, we set up an experiment in which we select a subset of the face data consisting of 50 images. From half of the images, we remove the same patch (Figure 6) and predict the missing pixels. This is a rather small dataset for this task, as we have only 10 images for each of the 5 different persons, and half of these images have missing data at the same spot. We measure the quality of the prediction in terms of signal-to-noise ratio (SNR). The missing values are reconstructed using the mean of the predictive distribution X _pred ≡ 〈X〉_{𝒫𝒪(X;T*V*)} = T*V*, where T* and V* are point estimates of the template and excitation matrix. We compare our variational algorithm with the classical NMF. For each algorithm, we test two different versions. The variational algorithms differ in how we estimate T* and V*. In the first variational algorithm, we use a crude estimate of T* and V* as the mean of the approximating q distribution. In the second condition, after convergence of hyperparameters via VB, we reinitialise T and V randomly and switch to an ICM algorithm (see (38)). This strategy finds a local mode (T*, V*) of the exact posterior distribution. In NMF, we test two initialisation strategies: in the first condition, we initialise the templates randomly; in the second, we set them equal to the images in the dataset with random perturbations.

Figure 6.

Results of a typical run. (a) Example images from the dataset. (b) Comparison of the reconstruction accuracy of different methods in terms of SNR (in dB), organised according to the sparseness of the solution. (c) (from left to right). The ground truth, data with missing pixels. The reconstructions of VB, VB + ICM, and ML-NMF with two initialisation strategies (1 = random, 2 = to image).

In Figure 6, we show the reconstruction results for a typical run, for a model with I = 100 templates. Note that this is an overcomplete model, with twice as many templates as there are images. To characterise the nature of the estimated template and excitation matrices, we use the sparseness criteria [18] of an m × n matrix X, defined as $\text{Sparseness}\hspace{0.17em}(X)=(\sqrt{mn}-({\sum }_{i\mathrm{,}j}|X_{i\mathrm{,}j}|)/({\sum }_{i\mathrm{,}j}{X}_{i\mathrm{,}j}^2{)}^{1/2})/(\sqrt{mn}-1)$. This measure is 1 when the matrix X has only a single nonzero entry and 0 when all entries are equal. We see that the variational algorithms are superior in this case in terms of SNR as well as the visual quality of the reconstruction. This is perhaps not surprising, since with maximum likelihood estimation; if the model order is not carefully chosen, generalisation performance is poor: the “noise” in the observed data is fitted but the prediction quality drops on new data. An interesting observation is that highly sparse solutions (either in templates or excitations) do not give the best result; the solution that balances both seems to be the best in this setting. This example illustrates that sparseness in itself may not be necessarily a good criteria to optimise; for model selection, the marginal likelihood should be used as the natural quantity.

On the same face dataset, we compare the prediction error in terms of the SNR for varying model order I. Our goal is to compare the prediction performance of the full Bayesian approach with the ML-NMF for a range of conditions (under-complete, complete, and overcomplete). The results shown in Figure 7 are averages of several runs with hyperparameter adaptation and different hyperparameter tying. In the simulations, the shape parameters are tied always as a _i,τ ^v = a ^v (and a _ν,i ^t = a ^t). The scale parameters are untied or tied as (b _τ ^v, b _ν ^t) (across sources) or b _i ^v, b _i ^t (different for each source) and jointly optimised. Regardless of the hyperparameter tying structure, the results were quite similar. The best SNR values are attained with untied scale parameters for both excitations and templates.

Figure 7.

Average SNR results for model orders I = 1, 25, 50, 75, 100 covering undercomplete, complete, and overcomplete cases. Comparison of VB, VB + ICM, and ML-NMF with two initialisation strategies (1 = random, 2 = to image).

We observe that, due to the implicit self-regularisation in the Bayesian approach, the prediction performance is not very sensitive to the model order and is immune to overfitting. In contrast, the ML-NMF with random initialisation is prone to overfitting, and prediction performance drops with increasing model order. Interestingly, when we initialise the ML-NMF algorithm to true data points with small perturbations, the prediction performance in terms of SNR improves. Note that this strategy would not be possible for data where the pixels were truly missing. However, visual inspection shows that the interpolation can still be “patchy” (see Figure 6).

We observe that hyperparameter adaptation is crucial for obtaining good prediction performance. In our simulations, results for VB without hyperparameter adaptation were occasionally poorer than the ML estimates. Good initialisation of the shape hyperparameters seems to be also important. We obtain best results when initialising the shape hyperparameters asymmetrically, for example, a ^v < 1 and a ^t > 10 (see 3rd and 4th panels from left in Figure 5). When the shape hyper-parameters are initialised to small a ^v, a ^t ≪ 1, the EM seems to get stuck in a local minima more often. Consequently, the prediction results are poorer. We have also carried out tests with more undercomplete representations when the model order is low I < 10. For these simulations, while the marginal likelihood was in favour of the VB solutions, we have not observed statistically significant differences between VB and ML in terms of SNR. The SNR improvement of VB over ML was on average about 0.1 dB only.

5. Discussion and Conclusions

In this paper, we have investigated KL-NMF from a statistical perspective. We have shown that KL minimisation formulation the original algorithm can be derived from a probabilistic model where the observations are superposition of I independent Poisson-distributed latent sources. Here, the template and excitation matrices turn out to be latent intensity parameters. The interpretation of NMF as a maximum likelihood method in a Poisson model is mentioned in the original NMF paper [1] and discussed in more detail by [5, 10], and [5] focuses on the optimisation and gives an equivalent description using auxiliary function maximisation. In contrast, [10] illustrates that the auxiliary variables can be viewed as model variables (the sources s) that are analytically integrated out. The relationship between KL divergence and the Poisson distribution is not just a lucky coincidence. There exists a duality between divergence functions and exponential family distributions. If a cost function is a Bregman divergence, there exists a regular exponential family where minimising the cost corresponds to maximum likelihood parameter estimation [19]; also see [12] it in the context of matrix factorisation models.

The novel observation in the current article is the exact characterisation of the approximating distribution q(S) or full conditionals p(S ∣ T, V, X) as a product of multinomial distributions, leading to a richer approximation distribution than a naive mean field or single site Gibbs (which would freeze due to deterministic p(X ∣ S)). This conjugate form leads to significant simplifications in full Bayesian integration. Apart from the conditionally Gaussian case, NMF with KL objective seems to be unique in this respect. For several other distance metrics D(·||·), we find that full Bayesian inference not as practical as p(S ∣ T, V, X) is not standard.

We have also shown that the standard KL-NMF algorithm with multiplicative update rules is in fact an EM algorithm with data augmentation. Extending upon this observation, we have developed an hierarchical model with conjugate Gamma priors. We have developed a variational Bayes algorithm and a Gibbs sampler for inference in this hierarchical model. We have also developed methods for estimating the marginal likelihood for model selection. This is an additional feature that is lacking in existing NMF approaches with regularisation, where only MAP estimates are obtained, such as [13, 18, 20].

Our simulations suggest that the variational bound seems to be a reasonable approximation to the marginal likelihood and can guide model selection for NMF. The computational requirements are comparable to the ML-NMF. A potentially time-consuming step in the implementation of the variational algorithm is the evaluation of the Ψ function but this step can also be replaced by a simple piecewise polynomial approximation since exp(Ψ(x)) ≈ x − 0.5 for x > 5.

We first compare the variational inference with a Gibbs sampler. In our simulations, we observe that both algorithms give qualitatively very similar results, both for inference of templates and excitations as well as model order selection. We find the variational approach somewhat more practical as it can be expressed as simple matrix operations, where both the fixed point equations as well as the bound can be compactly and efficiently implemented using matrix computation software. In contrast, our Gibbs sampler is computationally more demanding, and the calculation of marginal likelihood is somewhat more tricky. With our implementation of both algorithms, the variational method is faster by a factor of around 13. Reference implementations of both algorithms in Matlab are available from the following url: http://www.cmpe.boun.edu.tr/∼cemgil/bnmf/.

In terms of computational requirements, the variational procedure has several advantages. First, we circumvent sampling from multinomial variables, which is the main computational bottleneck with the Gibbs sampler. Whilst efficient algorithms are developed for multinomial sampling [21], the procedure is time consuming when the number of latent sources I is large. In contrast, the variational method estimates the expected sufficient statistics directly by elementary matrix operations. Another advantage is hyperparameter estimation. In principle, it is possible to maximise the marginal likelihood via a Monte Carlo EM procedure [22, 23]; yet this potentially requires many more iterations. In contrast, the evaluation of the derivatives of the lower bound is straightforward and can be implemented without much additional computational cost.

The efficiency of the Gibbs sampler could be improved by working out the distribution of the sufficient statistics of sources directly (namely, quantities ∑_τ s _ν,i,τ or ∑_ν s _ν,i,τ) to circumvent multinomial sampling. Unfortunately, for the sum of binomial random variables with different cell probability parameters, the sum does not have a simple form but various approximations are possible [24].

Inference based on VB is easy to implement but at the end of the day, the fixed point iteration is just a gradient-based lower bound optimisation procedure, and second order Newton methods can provide more efficient alternatives. For NMF models, there exist many conditional independence relations, hence the Hessian matrix has a special block structure [12]. It is certainly interesting to develop efficient inference methods that make use of the special block structure of the Hessian matrix. However, as our primary goal was a practical full Bayesian treatment, we have not investigated this path yet. Another approach in this direction is using alternative deterministic integration techniques such as expectation propagation (EP) [25]. Those techniques work directly on an approximation of the true marginal likelihood rather than a bound. A related approach known as expectation consistent (EC) inference is used with success in related source separation problems [26].

From a modelling perspective, our hierarchical model provides some attractive properties. It is easy to incorporate prior knowledge about individual latent sources via hyperparameters, and one can easily capture variability in the templates and excitations that is potentially useful for developing robust techniques. The prior structure here is qualitatively similar to an entropic prior [20, 27], and we find qualitatively similar representations to the ones found by NMF reported earlier by [1, 18]. However, none of the above mentioned methods provide an estimate of the marginal likelihood, which is useful for model selection. Our generative model formulation can be extended in various ways to suit the specific needs of particular applications. For example, one can enforce more structured prior models such as chains or fields [10]. As a second possibility, the Poisson observation model can be replaced with other models such as clipped Gaussian, Gamma, or Gaussians which lead to alternative source separation algorithms. For example, the case of Gaussian sources where the excitations and templates correspond to the variances is discussed in [28].

Our main contribution here is the development of a principled and practical way to estimate both the optimal sparsity criteria and model order, in terms of marginal likelihood. By maximising the bound on marginal likelihood, we have a method where all the hyperparameters can be estimated from data, and the appropriate sparseness criteria is found automatically. We believe that our approach provides a practical improvement to the highly popular KL-NMF algorithm without incurring much additional computational cost.

Appendix

A. Standard Distributions in Exponential Form, Their Sufficient Statistics and Entropies

1. Gamma

   $$\begin{array}{c}𝒢(\lambda ;a\mathrm{,}b)\equiv \text{exp}\left(+(a-1)\log \lambda -\frac{1}{b}\lambda -\log \Gamma (a)-a\log b\right)\mathrm{,}\hfill \\ \hfill {\langle \lambda \rangle }_{𝒢}=ab\hspace{0.17em}\hspace{0.17em}{\langle \text{log}\hspace{0.17em}\lambda \rangle }_{𝒢}=\mathrm{\Psi }(a)+\text{log}(b),\hfill \\ H_{𝒢}[\lambda ]\equiv -{\langle \text{log}\hspace{0.17em}𝒢\rangle }_{𝒢}=-(a-1)\mathrm{\Psi }(a)+\text{log}\hspace{0.17em}b+a+\text{log}\hspace{0.17em}\mathrm{\Gamma }(a).\hfill \end{array}$$ (A.1)

   Here, Ψ denotes the digamma function defined as Ψ(a) ≡ dlogΓ(a)/da.
2. Poisson

   $$\begin{array}{ll}\hfill 𝒫𝒪(s;\lambda )& =\text{exp}(s\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}\hspace{0.17em}\lambda -\lambda -\text{log}\hspace{0.17em}\mathrm{\Gamma }(s+1)),\\ \hfill {\langle s\rangle }_{𝒫𝒪}& =\lambda \mathrm{.}\end{array}$$ (A.2)
3. Multinomial

   $$ℳ(\mathbf{s};x\mathrm{,}\mathbf{p})=\delta (x-{\displaystyle \sum _i}{s}_i)\text{exp}\left(\text{log}\hspace{0.17em}\Gamma (x+1)+\sum _{i=1}^I\left(s_i\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}{p}_i-\text{log}\hspace{0.17em}\Gamma (s_i+1)\right)\right)\mathrm{,}{\langle s_i\rangle }_{ℳ}=x{p}_i\mathrm{.}$$ (A.3)

   Here, s = {s ₁, s ₂,…, s _I}, p = {p ₁, p ₂,…, p _I}, and p ₁ + p ₂ + ⋯ + p _I = 1. Here, p _i, i = 1 ⋯ I are the cell probabilities, and x is the index parameter where s ₁ + s ₂ + ⋯ + s _I = x. The entropy is given as follows:

   $$\begin{array}{ll}{H}_{ℳ}\left[s_{\nu \mathrm{,1}:I\mathrm{,}\tau }\right]& =-\text{log}\hspace{0.17em}\mathrm{\Gamma }(x_{\nu \mathrm{,}\tau }+1)-{\displaystyle \sum _{i=1}^I}\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle \text{log}\hspace{0.17em}{p}_{\nu \mathrm{,}i\mathrm{,}\tau }\\ & \hspace{1em}+{\displaystyle \sum _{i=1}^I}\langle \text{log\hspace{0.17em}}\mathrm{\Gamma }(s_{\nu \mathrm{,}i\mathrm{,}\tau }+1)\rangle -\langle \text{log}\hspace{0.17em}\delta (x_{\nu \mathrm{,}\tau }-{\displaystyle \sum _i}{s}_{\nu \mathrm{,}i\mathrm{,}\tau })\rangle .\end{array}$$ (A.4)

A closed form expression for the entropy is not known due to 〈logΓ(s + 1)〉 terms but asymptotic expansions exist [29, 30]. Computationally efficient sampling from a multinomial distribution is not trivial; see [21] for a comparison of various methods and detailed discussion of tradeoffs.

A.1. Summary of the Generative Model

We have the following. Indices:

• i = 1 ⋯ I, source index;

• ν = 1 ⋯ W, Row (frequency bin) index;

• τ = 1 ⋯ K, Column (time frame) index;

t _ν,i: template variable at νth row of the ith source

$$t_{\nu \mathrm{,}i}\sim 𝒢(t_{\nu \mathrm{,}i};a_{\nu \mathrm{,}i}^t\mathrm{,}\frac{b_{\nu \mathrm{,}i}^t}{a_{\nu \mathrm{,}i}^t});$$ (A.5)

v _i,τ: excitation variable of the ith source at τth column

$$v_{i\mathrm{,}\tau }\sim 𝒢(v_{i\mathrm{,}\tau };a_{i\mathrm{,}\tau }^v\mathrm{,}\frac{b_{i\mathrm{,}\tau }^v}{a_{i\mathrm{,}\tau }^v});$$ (A.6)

s _ν,i,τ: source variable of ith source at νth row (frequency bin) and τth column (time frame)

$$s_{\nu \mathrm{,}i\mathrm{,}\tau }\sim 𝒫𝒪(s_{\nu \mathrm{,}i\mathrm{,}\tau };t_{\nu \mathrm{,}i}{v}_{i\mathrm{,}\tau });$$ (A.7)

x _ν,τ: observation at νth row (frequency bin) and τth column (time frame)

$$x_{\nu \mathrm{,}\tau }\sim {\displaystyle \sum _i}{s}_{\nu \mathrm{,}i\mathrm{,}\tau }\mathrm{.}$$ (A.8)

A.2. Expression of the Full Joint Distribution

Here, ϕ ≡ p(X, S, T, V ∣ Θ) = p(X ∣ S)p(S ∣ T, V)p(V ∣ Θ^v)p(V ∣ Θ^t),

$$\begin{array}{ll}\text{log}\hspace{0.17em}\hspace{0.17em}\varphi & ={\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _i\hspace{0.17em}}{\displaystyle \sum _{\tau }}\left(-t_{\nu \mathrm{,}i}{v}_{i\mathrm{,}\tau }+s_{\nu \mathrm{,}i\mathrm{,}\tau }\hspace{0.17em}\hspace{0.17em}\text{log}\left(t_{\nu \mathrm{,}i}{v}_{i\mathrm{,}\tau }\right)-\text{log}\hspace{0.17em}\Gamma (s_{\nu \mathrm{,}i\mathrm{,}\tau }+1)\right)\\ & \hspace{1em}+{\displaystyle \sum _{\nu }}{\displaystyle \sum _{\tau }}\text{log}\hspace{0.17em}\delta (x_{\nu \mathrm{,}\tau }-{\displaystyle \sum _i}{s}_{\nu \mathrm{,}i\mathrm{,}\tau })\\ & \hspace{1em}+{\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _i}+(a_{\nu \mathrm{,}i}^t-1)\text{log}\hspace{0.17em}{t}_{\nu \mathrm{,}i}-\frac{a_{\nu \mathrm{,}i}^t}{b_{\nu \mathrm{,}i}^t}{t}_{\nu \mathrm{,}i}-\text{log}\hspace{0.17em}\mathrm{\Gamma }\left(a_{\nu \mathrm{,}i}^t\right)\\ & \hspace{1em}-a_{\nu \mathrm{,}i}^t\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}\left(\frac{b_{\nu \mathrm{,}i}^t}{a_{\nu \mathrm{,}i}^t}\right)\\ & \hspace{1em}+{\displaystyle \sum _{\tau }}\hspace{0.17em}{\displaystyle \sum _i}+(a_{i\mathrm{,}\tau }^v-1)\text{log}\hspace{0.17em}{v}_{i\mathrm{,}\tau }-\frac{a_{i\mathrm{,}\tau }^v}{b_{i\mathrm{,}\tau }^v}{v}_{i\mathrm{,}\tau }-\text{log}\hspace{0.17em}\mathrm{\Gamma }\left(a_{i\mathrm{,}\tau }^v\right)\\ & \hspace{1em}-a_{i\mathrm{,}\tau }^v\hspace{0.17em}\hspace{0.17em}\text{log}\left(\frac{b_{i\mathrm{,}\tau }^v}{a_{i\mathrm{,}\tau }^v}\right).\end{array}$$ (A.9)

A.3. The Variational Bound

The variational bound in (27) can be written as

$${ℒ}_X(\Theta )\equiv \text{log}\hspace{0.17em}p(X\hspace{0.17em}\mid \hspace{0.17em}\Theta )\ge {\langle \text{log}\hspace{0.17em}\varphi \rangle }_q+H[q]={ℬ}_{\text{VB}}\mathrm{,}$$ (A.10)

where the energy term is given by the expectation of the expression in Appendix A.2, and H[q] denotes the entropy of the variational approximation distribution q where the individual entropies are defined in Appendix A:

$$\begin{array}{ll}H[q]& =-\langle \text{log}\hspace{0.17em}q\rangle \\ & ={\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _{\tau }}{H}_{ℳ}\left[s_{\nu \mathrm{,1}:I\mathrm{,}\tau }\right]+{\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _i}{H}_{𝒢}\left[t_{\nu \mathrm{,}i}\right]+{\displaystyle \sum _i}\hspace{0.17em}{\displaystyle \sum _{\tau }}{H}_{𝒢}\left[v_{i\mathrm{,}\tau }\right].\end{array}$$ (A.11)

One potential problem is that this expression requires the entropy of a multinomial distribution for which there is no known simple expression. This is due to terms of form 〈logΓ(s + 1)〉 where only asymptotic expansions are known. Fortunately, the difficult terms in the energy term can be canceled by the corresponding terms in the entropy term, and one obtains the following expression that only depends on known sufficient statistics:

$$\begin{array}{ll}ℬ& =-{\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _{\tau }}\hspace{0.17em}{\displaystyle \sum _i}\langle t_{\nu \mathrm{,}i}\rangle \langle v_{i\mathrm{,}\tau }\rangle \\ & \hspace{1em}+{\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _i}\langle \text{log}\hspace{0.17em}{t}_{\nu \mathrm{,}i}\rangle \left(a_{\nu \mathrm{,}i}^t-1+{\displaystyle \sum _{\tau }}\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle \right)\\ & \hspace{1em}+{\displaystyle \sum _{\tau }}\hspace{0.17em}{\displaystyle \sum _i}\langle \text{log}\hspace{0.17em}{v}_{i\mathrm{,}\tau }\rangle \left(a_{i\mathrm{,}\tau }^v-1+{\displaystyle \sum _{\nu }}\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle \right)\\ & \hspace{1em}+{\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _i}-\frac{a_{\nu \mathrm{,}i}^t}{b_{\nu \mathrm{,}i}}\langle t_{\nu \mathrm{,}i}\rangle -\text{log}\hspace{0.17em}\mathrm{\Gamma }\left(a_{\nu \mathrm{,}i}^t\right)-a_{\nu \mathrm{,}i}^t\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}\left(\frac{b_{\nu \mathrm{,}i}}{a_{\nu \mathrm{,}i}^t}\right)\\ & \hspace{1em}+{\displaystyle \sum _{\tau }}\hspace{0.17em}{\displaystyle \sum _i}-\frac{a_{i\mathrm{,}\tau }^v}{b_{i\mathrm{,}\tau }^v}\langle v_{i\mathrm{,}\tau }\rangle -\text{log}\hspace{0.17em}\mathrm{\Gamma }\left(a_{i\mathrm{,}\tau }^v\right)-a_{i\mathrm{,}\tau }^v\hspace{0.17em}\hspace{0.17em}\text{log}\left(\frac{b_{i\mathrm{,}\tau }^v}{a_{i\mathrm{,}\tau }^v}\right)\\ & \hspace{1em}+{\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _{\tau }}\left(-\text{log}\hspace{0.17em}\Gamma (x_{\nu \mathrm{,}\tau }+1)-{\displaystyle \sum _{i=1}^I}\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle \text{log}\hspace{0.17em}{p}_{\nu \mathrm{,}i\mathrm{,}\tau }\right)\\ & \hspace{1em}+{\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _i}\left(-({\alpha }_{\nu \mathrm{,}i}^t-1)\Psi \left({\alpha }_{\nu \mathrm{,}i}^t\right)+\text{log}\hspace{0.17em}{\beta }_{\nu \mathrm{,}i}^t+{\alpha }_{\nu \mathrm{,}i}^t+\text{log}\hspace{0.17em}\Gamma \left({\alpha }_{\nu \mathrm{,}i}^t\right)\right)\\ & \hspace{1em}+{\displaystyle \sum _i}\hspace{0.17em}{\displaystyle \sum _{\tau }}\left(-({\alpha }_{i\mathrm{,}\tau }^v-1)\Psi \left({\alpha }_{i\mathrm{,}\tau }^v\right)+\text{log}\hspace{0.17em}{\beta }_{i\mathrm{,}\tau }^v+{\alpha }_{i\mathrm{,}\tau }^v+\text{log}\hspace{0.17em}\Gamma \left({\alpha }_{i\mathrm{,}\tau }^v\right)\right)\mathrm{.}\end{array}$$ (A.12)

After some careful manipulations, the following expression is obtained where log L denotes here elementwise logarithm of matrix L:

$$\begin{array}{ll}ℬ& ={\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _{\tau }}\left(-E_t{E}_v-\text{log}\hspace{0.17em}\Gamma (X+1)\right)\\ & \hspace{1em}+{\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _{\tau }}-X\hspace{0.17em}\mathrm{.}*(\left(\right(L_t\hspace{0.17em}\hspace{0.17em}\mathrm{.}*\hspace{0.17em}\text{log}\left(L_t\right))L_v+L_t(L_v\hspace{0.17em}\mathrm{.}*\hspace{0.17em}\hspace{0.17em}\text{log}\left(L_v\right)\left)\right)\\ & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\mathrm{.}/\left(L_t{L}_v\right)-\text{log}\left(L_t{L}_v\right))\\ & \hspace{1em}+{\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _i}-(A_t\mathrm{.}/B_t)\hspace{0.17em}\mathrm{.}*E_t-\text{log}\hspace{0.17em}\mathrm{\Gamma }\left(A_t\right)+A_t\hspace{0.17em}\mathrm{.}*\text{log}(A_t\mathrm{.}/B_t)\\ & \hspace{1em}+{\displaystyle \sum _{\nu }}\hspace{0.17em}{\displaystyle \sum _i}{\mathbf{\alpha }}_t\hspace{0.17em}\mathrm{.}*(\text{log}\hspace{0.17em}{\mathbf{\beta }}_t+1)+\text{log}\hspace{0.17em}\mathrm{\Gamma }\left({\mathbf{\alpha }}_t\right)\\ & \hspace{1em}+{\displaystyle \sum _i}\hspace{0.17em}{\displaystyle \sum _{\tau }}-(A_v\mathrm{.}/B_v)\hspace{0.17em}\mathrm{.}*E_v-\text{log}\hspace{0.17em}\mathrm{\Gamma }\left(A_v\right)+A_v\hspace{0.17em}\hspace{0.17em}\mathrm{.}*\text{log}(A_v\mathrm{.}/B_v)\\ & \hspace{1em}+{\displaystyle \sum _i}\hspace{0.17em}{\displaystyle \sum _{\tau }}{\mathbf{\alpha }}_v\hspace{0.17em}\mathrm{.}*(\log \hspace{0.17em}\hspace{0.17em}{\mathbf{\beta }}_v+1)+\log \hspace{0.17em}\hspace{0.17em}\mathrm{\Gamma }\left({\mathbf{\alpha }}_v\right).\end{array}$$ (A.13)

A.4. Handling Missing Data and Map Estimation

When there is missing data, that is, when some of the x _ν,τ are not observed, computation is still straightforward in our framework and can be accomplished by a simple modification to the original algorithm. We first define a mask matrix M = {m _ν,τ}, same size as X, where

$$m_{\nu \mathrm{,}\tau }=\{\begin{array}{cc}\mathrm{0,}& x_{\nu \mathrm{,}\tau }\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{is}\hspace{0.17em}\hspace{0.17em}\text{missing},\hfill \\ \mathrm{1,}& \text{otherwise}.\hfill \end{array}$$ (A.14)

Using the mask variables, the observation model with missing data can be written as follows:

$$p(X\hspace{0.17em}\mid \hspace{0.17em}S)p(S\hspace{0.17em}\mid \hspace{0.17em}T\mathrm{,}V)={\displaystyle \prod _{\nu \mathrm{,}\tau }}{\left(p(x_{\nu \mathrm{,}\tau }\hspace{0.17em}\mid \hspace{0.17em}{s}_{\nu \mathrm{,1}:I\mathrm{,}\tau })p(s_{\nu \mathrm{,1}:I\mathrm{,}\tau }\hspace{0.17em}\mid \hspace{0.17em}{t}_{\nu \mathrm{,1}:I}\mathrm{,}{v}_{1:I\mathrm{,}\tau })\right)}^{m_{\nu \mathrm{,}\tau }}\mathrm{.}$$ (A.15)

The prior is not affected. Hence, we merely replace the first two lines of the expression for the full joint distribution (given in the Appendix A.2) as follows:

$$\begin{array}{ll}\text{log}\hspace{0.17em}\varphi & ={\displaystyle \sum _{\nu }}{\displaystyle \sum _{\tau }}{m}_{\nu \mathrm{,}\tau }{\displaystyle \sum _i}\left(-t_{\nu \mathrm{,}i}{v}_{i\mathrm{,}\tau }+s_{\nu \mathrm{,}i\mathrm{,}\tau }\hspace{0.17em}\hspace{0.17em}\text{log}(t_{\nu \mathrm{,}i}{v}_{i\mathrm{,}\tau }\right)\\ & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-\text{log}\hspace{0.17em}\mathrm{\Gamma }(s_{\nu \mathrm{,}i\mathrm{,}\tau }+1))\\ & \hspace{1em}+{\displaystyle \sum _{\nu }}{\displaystyle \sum _{\tau }}{m}_{\nu \mathrm{,}\tau }\hspace{0.17em}\hspace{0.17em}\text{log}\hspace{0.17em}\delta (x_{\nu \mathrm{,}\tau }-{\displaystyle \sum _i}{s}_{\nu \mathrm{,}i\mathrm{,}\tau })+\cdots \mathrm{.}\end{array}$$ (A.16)

Consequently, it is easy to see that

$$\begin{array}{cc}\hfill q(v_{i\mathrm{,}\tau })& \propto \text{exp}((a_{i\mathrm{,}\tau }^v+{\displaystyle \sum _{\nu }}{m}_{\nu \mathrm{,}\tau }\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle -1)\log v_{i\mathrm{,}\tau }\hfill \\ & \hspace{1em}\hspace{1em}-\left(\frac{a_{i\mathrm{,}\tau }^v}{b_{i\mathrm{,}\tau }^v}+{\displaystyle \sum _{\nu }}{m}_{\nu \mathrm{,}\tau }\langle t_{\nu \mathrm{,}i}\rangle \right)v_{i\mathrm{,}\tau })\hfill \\ & \propto 𝒢\left(v_{i\mathrm{,}\tau };{\alpha }_{i\mathrm{,}\tau }^v\mathrm{,}{\beta }_{i\mathrm{,}\tau }^v\right)\mathrm{,}\hfill \\ \hfill {\alpha }_{i\mathrm{,}\tau }^v& \equiv a_{i\mathrm{,}\tau }^v+{\displaystyle \sum _{\nu }}{m}_{\nu \mathrm{,}\tau }\langle s_{\nu \mathrm{,}i\mathrm{,}\tau }\rangle \mathrm{,}\hfill \\ \hfill {\beta }_{i\mathrm{,}\tau }^v& \equiv {\left(\frac{a_{i\mathrm{,}\tau }^v}{b_{i\mathrm{,}\tau }^v}+{\displaystyle \sum _{\nu }}{m}_{\nu \mathrm{,}\tau }\langle t_{\nu \mathrm{,}i}\rangle \right)}^{-1}\mathrm{.}\hfill \end{array}$$ (A.17)

By a derivation analogous to one detailed in Section 3.3, we see that the excitation update equations in Algorithm 1, line 4, can be written using matrix notation as follows:

$$\begin{array}{c}{\Sigma }_v=L_v\hspace{0.17em}\mathrm{.}*\left(L_t^{\top }\right((\mathbf{M}\hspace{0.17em}\mathrm{.}*X)\mathrm{.}/\left(L_t{L}_v\right)\left)\right)\\ {\mathbf{\alpha }}_v=A_v+{\Sigma }_v,\hspace{1em}\hspace{1em}{\mathbf{\beta }}_v=1./\left(A_v\mathrm{.}/B_v+E_t^{\top }\mathbf{M}\right)\mathrm{.}\end{array}$$ (A.18)

The update rules for the templates are similar. Note that when there is no missing data, we have M = 1 _W 1 _K ^⊤ which gives the original algorithm. The bound in (A.13) can also be easily modified for handling missing data. We merely replace X ← M. ∗X and the first term E _t E _v ← M. ∗E _t E _v.

We conclude this subsection by noting that the standard NMF update equations, given in (20), can be also rewritten to handle missing data as follows:

$$\begin{array}{l}{V}^{(n+1)}=V^{(n)}\hspace{0.17em}\mathrm{.}*\left(T^{\top }\right((\mathbf{M}\hspace{0.17em}\mathrm{.}*X)\mathrm{.}/(TV)\left)\right)\mathrm{.}/\left(T^{\top }\mathbf{M}\right),\\ T^{(n+1)}=T^{(n)}\hspace{0.17em}\mathrm{.}*(((\mathbf{M}\hspace{0.17em}\mathrm{.}*X)\mathrm{.}/(TV))V^{\top })\mathrm{.}/\left(\mathbf{M}{V}^{\top }\right).\end{array}$$ (A.19)

Here, the denominator has to be nonzero. Similarly, an iterative conditional modes (ICM) algorithm can be derived to compute the maximum a posteriori (MAP) solution as follows:

$$\begin{array}{ll}{V}^{(n+1)}& =(A_v+V^{(n)}\hspace{0.17em}\mathrm{.}*(T^{\top }((\mathbf{M}\hspace{0.17em}\mathrm{.}*X)\mathrm{.}/(TV))\left)\right)\\ & \hspace{1em}\mathrm{.}/(A_v\mathrm{.}/B_v+T^{\top }\mathbf{M}),\\ T^{(n+1)}& =(A_t+T^{(n)}\hspace{0.17em}\mathrm{.}*(((\mathbf{M}\hspace{0.17em}\mathrm{.}*X)\mathrm{.}/(TV))V^{\top }\left)\right)\\ & \hspace{1em}\mathrm{.}/(A_t\mathrm{.}/B_t+\mathbf{M}{V}^{\top }).\end{array}$$ (A.20)

Note that when the shape parameters go to zero, that is, A _t, A _v → 0, we obtain the maximum likelihood NMF algorithm.

A.5. Hyperparameter Optimisation

The hyperparameters Θ = (Θ^t, Θ^v) can be estimated by maximising the bound in 13. Below, we will derive the results for the excitations; the results for templates are similar. The solution for shape parameters involves finding the zero of a function f(a) − c, where

$$\begin{array}{c}f(a)=\text{log}\hspace{0.17em}a-\mathrm{\Psi }(a)+\mathrm{1,}\\ a^{*}=f^{-1}(c).\end{array}$$ (A.21)

The solution can be found by Newton's method by iteration of the following fixed point equation:

$$\begin{array}{ll}{a}^{(n+1)}& =a^{(n)}-\frac{f\left(a^{(n)}\right)-c}{f^{\prime }\left(a^{(n)}\right)}\\ & =a^{(n)}-\frac{\text{log}\left(a^{(n)}\right)-\Psi \left(a^{(n)}\right)+1-c}{1/a^{(n)}-{\Psi }^{\prime }\left(a^{(n)}\right)}=a^{(n)}-{\Delta }^{(n)}\mathrm{.}\end{array}$$ (A.22)

It is well known that Newton iterations can diverge if started away from the root. Occasionally, we observe that a can become negative. If this is the case, we set Δ⁽ⁿ⁾ ← Δ⁽ⁿ⁾/2, and try again. The digamma Ψ function and its derivative Ψ′ are available in numeric computation libraries (e.g., in Matlab as $\mathrm{psi(0,a)}$ and $\mathrm{psi(1,a)}$, resp.).

The derivation of the hyperparameter update equations is straightforward:

$$\begin{array}{cc}\hfill \frac{\partial ℬ}{\partial a_{i\mathrm{,}\tau }^v}& =\langle \text{log}\hspace{0.17em}{v}_{i\mathrm{,}\tau }\rangle -\frac{1}{b_{i\mathrm{,}\tau }^v}\langle v_{i\mathrm{,}\tau }\rangle -\mathrm{\Psi }\left(a_{i\mathrm{,}\tau }^v\right)\hfill \\ & \hspace{1em}-\text{log}\hspace{0.17em}{b}_{i\mathrm{,}\tau }^v+\text{log}\hspace{0.17em}{a}_{i\mathrm{,}\tau }^v+1=0,\hfill \\ \hfill c_{i\mathrm{,}\tau }& =\text{log}\hspace{0.17em}{a}_{i\mathrm{,}\tau }^v-\mathrm{\Psi }\left(a_{i\mathrm{,}\tau }^v\right)+\mathrm{1,}\hfill \\ \hfill c_{i\mathrm{,}\tau }& \equiv \frac{\langle v_{i\mathrm{,}\tau }\rangle }{b_{i\mathrm{,}\tau }^v}-(\langle \text{log}\hspace{0.17em}{v}_{i\mathrm{,}\tau }\rangle -\text{log}\hspace{0.17em}{b}_{i\mathrm{,}\tau }^v),\hfill \\ \hfill \frac{\partial ℬ}{\partial b_{i\mathrm{,}\tau }^v}& =\frac{a_{i\mathrm{,}\tau }^v}{{\left(b_{i\mathrm{,}\tau }^v\right)}^2}\langle v_{i\mathrm{,}\tau }\rangle -a_{i\mathrm{,}\tau }^v\frac{1}{b_{i\mathrm{,}\tau }^v}=0,\hfill \\ \hfill b_{i\mathrm{,}\tau }^v& =\langle v_{i\mathrm{,}\tau }\rangle \mathrm{.}\hfill \end{array}$$ (A.23)

Tying parameters across τ as a _i ^v = a _i,τ ^v and b _i ^v = b _i,τ ^v yields

$$\begin{array}{cc}\hfill \frac{\partial ℬ}{\partial a_i^v}& ={\displaystyle \sum _{\tau }}\langle \log v_{i\mathrm{,}\tau }\rangle -{\displaystyle \sum _{\tau }}\frac{1}{b_{i\mathrm{,}\tau }^v}\langle v_{i\mathrm{,}\tau }\rangle -K\mathrm{\Psi }\left(a_i^v\right)\hfill \\ & \hspace{1em}-{\displaystyle \sum _{\tau }}\text{log}\hspace{0.17em}{b}_{i\mathrm{,}\tau }^v+K=0,\hfill \\ c_i& =\text{log}\hspace{0.17em}{a}_i^v-\mathrm{\Psi }\left(a_i^v\right)+\mathrm{1,}\hfill \\ \hfill c_i& =\frac{1}{K}{\displaystyle \sum _{\tau }}\left(\frac{\langle v_{i\mathrm{,}\tau }\rangle }{b_{i\mathrm{,}\tau }^v}-\left(\langle \text{log\hspace{0.17em}}{v}_{i\mathrm{,}\tau }\rangle -\text{log}\hspace{0.17em}{b}_{i\mathrm{,}\tau }^v\right)\right)\mathrm{,}\hfill \\ \hfill \frac{\partial ℬ}{\partial b_i^v}& ={\displaystyle \sum _{\tau }}\frac{a_{i\mathrm{,}\tau }^v}{{\left(b_i^v\right)}^2}\langle v_{i\mathrm{,}\tau }\rangle -\frac{1}{b_i^v}{\displaystyle \sum _{\tau }}{a}_{i\mathrm{,}\tau }^v\mathrm{,}\hfill \\ \hfill b_i^v& =\frac{{\sum }_{\tau }\hspace{0.17em}{a}_{i\mathrm{,}\tau }^v\langle v_{i\mathrm{,}\tau }\rangle }{{\sum }_{\tau }\hspace{0.17em}{a}_{i\mathrm{,}\tau }^v}\mathrm{.}\hfill \end{array}$$ (A.24)

Tying parameters across τ and i, a ^v = a _i,τ ^v, and b ^v = b _i,τ ^v yields

$$\begin{array}{c}c=\text{log}\hspace{0.17em}{a}^v-\mathrm{\Psi }\left(a^v\right)+\mathrm{1,}\\ c=\frac{1}{KI}{\displaystyle \sum _{\tau }}\hspace{0.17em}{\displaystyle \sum _i}\left(\frac{\langle v_{i\mathrm{,}\tau }\rangle }{b_{i\mathrm{,}\tau }^v}-(\langle \text{log\hspace{0.17em}}{v}_{i\mathrm{,}\tau }\rangle -\text{log}\hspace{0.17em}{b}_{i\mathrm{,}\tau }^v)\right)\mathrm{,}\\ \frac{\partial ℬ}{\partial b^v}={\displaystyle \sum _i}\hspace{0.17em}{\displaystyle \sum _{\tau }}\frac{a_{i\mathrm{,}\tau }^v}{{\left(b^v\right)}^2}\langle v_{i\mathrm{,}\tau }\rangle -\frac{1}{b_i^v}{\displaystyle \sum _i}\hspace{0.17em}{\displaystyle \sum _{\tau }}{a}_{i\mathrm{,}\tau }^v\mathrm{,}\\ b^v=\frac{{\sum }_i\hspace{0.17em}{\sum }_{\tau }\hspace{0.17em}{a}_{i\mathrm{,}\tau }^v\langle v_{i\mathrm{,}\tau }\rangle }{{\sum }_i\hspace{0.17em}{\sum }_{\tau }\hspace{0.17em}{a}_{i\mathrm{,}\tau }^v}\mathrm{.}\end{array}$$ (A.25)

The derivation of the template parameters is exactly analogous. We can express the update equations once again in compact matrix notation as follows:

$$\begin{array}{ll}\hfill Z& ⟵E_v^{(n)}\mathrm{.}/B_v^{(n)}-\text{log}(L_v^{(n)}\mathrm{.}/B_v^{(n)}),\hfill \\ \hfill C& ⟵\{\begin{array}{cc}Z\mathrm{,}\hfill & \text{Not}\hspace{0.17em}\hspace{0.17em}\text{tied}\mathrm{,}\hfill \\ \frac{\left(Z{\mathbf{1}}_K\right)}{K}\mathrm{,}\hfill & \text{Tie}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{columns}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}(\text{over}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\tau ),\hfill \\ \frac{\left({\mathbf{1}}_I^{\top }Z\right)}{I}\mathrm{,}\hfill & \text{Tie}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{rows}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}(\text{over}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i),\hfill \\ \frac{\left({\mathbf{1}}_I^{\top }Z{\mathbf{1}}_K\right)}{(KI)},\hfill & \text{Tie}\hspace{0.17em}\hspace{0.17em}\text{all}\hspace{0.17em}\hspace{0.17em}(\text{over}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\tau \hspace{0.17em}\hspace{0.17em}\text{and}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i),\hfill \end{array}\hfill \\ \hfill A_v^{(n+1)}& ⟵\mathrm{SolveByNewton}\hspace{0.17em}\hspace{0.17em}\left(A_v^{(n)}\mathrm{,}C\right),\hfill \\ \hfill B_v^{(n+1)}& ⟵\{\begin{array}{cc}{E}_v^{(n)}\mathrm{,}\hfill & \text{Not}\hspace{0.17em}\hspace{0.17em}\text{tied}\mathrm{,}\hfill \\ \left(\right((A_v^{(n)}\hspace{0.17em}\hspace{0.17em}\mathrm{.}*E_v^{(n)}){\mathbf{1}}_K)\hfill & \\ \hspace{0.17em}\hspace{0.17em}\mathrm{.}/\left(A_v^{(n)}{\mathbf{1}}_K\right)){\mathbf{1}}_K^{\top }\mathrm{,}\hfill & \text{Tie}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{columns}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}(\text{over}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\tau ),\hfill \\ {\mathbf{1}}_I\left(\right({\mathbf{1}}_I^{\top }(A_v^{(n)}\hspace{0.17em}\mathrm{.}*E_v^{(n)}))\hfill & \\ \mathrm{.}/\left({\mathbf{1}}_I^{\top }{A}_v^{(n)}\right)),\hfill & \text{Tie}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{rows}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}(\text{over}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i),\hfill \\ {\mathbf{1}}_I\left(\right({\mathbf{1}}_I^{\top }(A_v^{(n)}\mathrm{.}*E_v^{(n)}){\mathbf{1}}_K)\hfill & \\ \hspace{0.17em}\hspace{0.17em}\mathrm{.}/\left({\mathbf{1}}_I^{\top }{A}_v^{(n)}{\mathbf{1}}_K\right)){\mathbf{1}}_K^{\top },\hfill & \text{Tie}\hspace{0.17em}\hspace{0.17em}\text{all}\hspace{0.17em}\hspace{0.17em}(\text{over}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\tau \hspace{0.17em}\hspace{0.17em}\text{and}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}i)\mathrm{.}\hfill \end{array}\hfill \end{array}$$ (A.26)

Here, we assume $\mathrm{SolveByNewton}(A_0\mathrm{,}C)$ is a matrix-valued function that finds root C _i,j = f(A _i,j) for each element of A, starting from the initial matrix A ₀. If C is a scalar or vector, it is repeated over the missing index to implement parameter tying. For example, if C is a I × 1 vector and A ₀ is I × K, we assume C _i = c _i,τ for all τ = 1 ⋯ K, and the output is the same size as A ₀. This is only a notational convenience; an actual implementation can be achieved more efficiently. Again, the implementation of the template parameters is exactly analogous; merely replace above the subscripts as v ← t, (i, τ) ← (ν, i) and (I, K) ← (W, I).