Mixture models with a prior on the number of components

Abstract

A natural Bayesian approach for mixture models with an unknown number of components is to take the usual finite mixture model with symmetric Dirichlet weights, and put a prior on the number of components—that is, to use a mixture of finite mixtures (MFM). The most commonly-used method of inference for MFMs is reversible jump Markov chain Monte Carlo, but it can be nontrivial to design good reversible jump moves, especially in high-dimensional spaces. Meanwhile, there are samplers for Dirichlet process mixture (DPM) models that are relatively simple and are easily adapted to new applications. It turns out that, in fact, many of the essential properties of DPMs are also exhibited by MFMs—an exchangeable partition distribution, restaurant process, random measure representation, and stick-breaking representation—and crucially, the MFM analogues are simple enough that they can be used much like the corresponding DPM properties. Consequently, many of the powerful methods developed for inference in DPMs can be directly applied to MFMs as well; this simplifies the implementation of MFMs and can substantially improve mixing. We illustrate with real and simulated data, including high-dimensional gene expression data used to discriminate cancer subtypes.

1 Introduction

Mixture models are used in a wide range of applications, including population structure (Pritchard et al., 2000), document modeling (Blei et al., 2003), speaker recognition (Reynolds et al., 2000), computer vision (Stauffer and Grimson, 1999), phylogenetics (Pagel and Meade, 2004), and gene expression profiling (Yeung et al., 2001), to name a few prominent examples. A common issue with finite mixtures is that it can be difficult to choose an appropriate number of mixture components, and many methods have been proposed for making this choice (e.g., Henna, 1985; Keribin, 2000; Leroux, 1992; Ishwaran et al., 2001; James et al., 2001).

From a Bayesian perspective, perhaps the most natural approach is to treat the number of components k like any other unknown parameter and put a prior on it. When the prior on the mixture weights is Dirichlet_k(γ, …, γ) given k, we refer to such a model as a mixture of finite mixtures (MFM), for short. Several inference methods have been proposed for this type of model (Nobile, 1994; Phillips and Smith, 1996; Richardson and Green, 1997; Stephens, 2000; Nobile and Fearnside, 2007; McCullagh and Yang, 2008), the most commonly-used method being reversible jump Markov chain Monte Carlo (Green, 1995; Richardson and Green, 1997). Reversible jump is a very general technique, and has been successfully applied in many contexts, but it can be difficult to use since applying it to new situations requires one to design good reversible jump moves, which is often nontrivial, particularly in high-dimensional parameter spaces. Advances have been made toward general methods of constructing good reversible jump moves (Brooks et al., 2003; Hastie and Green, 2012), but these approaches add another level of complexity to the algorithm design process and/or the software implementation.

Meanwhile, infinite mixture models such as Dirichlet process mixtures (DPMs) have become popular, partly due to the existence of relatively simple and generic Markov chain Monte Carlo (MCMC) algorithms that can easily be adapted to new applications (Neal, 1992, 2000; MacEachern, 1994, 1998; MacEachern and Müller, 1998; Bush and MacEachern, 1996; West, 1992; West et al., 1994; Escobar and West, 1995; Liu, 1994; Dahl, 2003, 2005; Jain and Neal, 2004, 2007). These algorithms are made possible by the variety of computationally-tractable representations of the Dirichlet process, including its exchangeable partition distribution, the Blackwell–MacQueen urn process (a.k.a. the Chinese restaurant process), the random discrete measure formulation, and the Sethuraman–Tiwari stick-breaking representation (Ferguson, 1973; Antoniak, 1974; Blackwell and MacQueen, 1973; Aldous, 1985; Pitman, 1995, 1996; Sethuraman, 1994; Sethuraman and Tiwari, 1981).

It turns out that there are MFM counterparts for each of these properties: an exchangeable partition distribution, an urn/restaurant process, a random discrete measure formulation, and in certain cases, an elegant stick-breaking representation. The main point of this paper is that the MFM versions of these representations are simple enough, and parallel the DPM versions closely enough, that many of the inference algorithms developed for DPMs can be directly applied to MFMs. Interestingly, the key properties of MFMs hold for any choice of prior distribution on the number of components.

There has been a large amount of research on efficient inference methods for DPMs, and an immediate consequence of the present work is that most of these methods can also be used for MFMs. Since several DPM sampling algorithms are designed to have good mixing performance across a wide range of applications—for instance, the Jain–Neal split-merge samplers (Jain and Neal, 2004, 2007), coupled with incremental Gibbs moves (MacEachern, 1994; Neal, 1992, 2000)—this greatly simplifies the implementation of MFMs for new applications. Further, these algorithms can also provide a major improvement in mixing performance compared to the usual reversible jump approaches, particularly in high dimensions.

This work resolves an open problem discussed by Green and Richardson (2001), who noted that it would be interesting to be able to apply DPM samplers to MFMs:

“In view of the intimate correspondence between DP and [MFM] models discussed above, it is interesting to examine the possibilities of using either class of MCMC methods for the other model class. We have been unsuccessful in our search for incremental Gibbs samplers for the [MFM] models, but it turns out to be reasonably straightforward to implement reversible jump split/merge methods for DP models.”

The paper is organized as follows. In Sections 2 and 3, we formally define the MFM and show that it gives rise to a simple exchangeable partition distribution closely paralleling that of the Dirichlet process. In Section 4, we describe the Pólya urn scheme (restaurant process), random discrete measure formulation, and stick-breaking representation for the MFM. In Section 5, we establish some asymptotic results for MFMs. In Section 6, we show how the properties in Sections 3 and 4 lead to efficient inference algorithms for the MFM. In Section 7, we compare the mixing performance of reversible jump versus our proposed algorithms, then apply the model to high-dimensional gene expression data, and finally, provide an empirical comparison between MFMs and DPMs.

2 Model

We consider the following well-known model:

$$\begin{gathered} K~p_K,\text{where}{p}_K\text{is}\mathrm{a}\text{p.m.f.}\text{on}\{1,2,\dots \} \\ ({\pi }_1,\dots ,{\pi }_k)~{\text{Dirichlet}}_k(\gamma ,\dots ,\gamma ),\text{given}K=k \\ {Z}_1,\dots ,Z_n\stackrel{\text{iid}}{\sim }\pi ,\text{given}\pi \\ {\theta }_1,\dots ,{\theta }_k\stackrel{\text{iid}}{\sim }H,\text{given}K=k \\ {X}_j~f_{{\theta }_{Z_j}}\text{independently}\text{for}j=1,\dots ,n,\text{given}{\theta }_{1:K},Z_{1:n}. \end{gathered}$$ (2.1)

Here, H is a prior or “base measure” on Θ ⊂ ℝ^ℓ, and {f_θ : θ ∈ Θ} is a family of probability densities with respect to a sigma-finite measure ζ on 𝒳 ⊂ ℝ^d. (As usual, we give Θ and 𝒳 the Borel sigma-algebra, and assume (x, θ) ↦ f_θ(x) is measurable.) We denote x_1:n = (x₁, …, x_n). Typically, X₁, …, X_n would be observed, and all other variables would be hidden/latent. We refer to this as a mixture of finite mixtures (MFM) model.

It is important to note that we assume a symmetric Dirichlet with a single parameter γ not depending on k. This assumption is key to deriving a simple form for the partition distribution and the other resulting properties. Assuming symmetry in the distribution of π is quite natural, since the distribution of X₁, …, X_n under any asymmetric distribution on π would be the same as if this asymmetric distribution was replaced by its symmetrization, i.e., if the entries of π were uniformly permuted (although this would no longer necessarily be a Dirichlet distribution). Assuming the same γ for all k is a genuine restriction, albeit a fairly natural one, often made in such models even when not strictly necessary (Nobile, 1994; Phillips and Smith, 1996; Richardson and Green, 1997; Green and Richardson, 2001; Stephens, 2000; Nobile and Fearnside, 2007). Note that prior information about the relative sizes of the weights π₁, …, π_k can be introduced through γ—roughly speaking, small γ favors lower entropy π’s, while large γ favors higher entropy π’s. Also note that sometimes it is useful to put a hyperprior on the parameters of H (Richardson and Green, 1997).

Meanwhile, we put very few restrictions on p_K, the distribution of the number of components. For practical purposes, we need the infinite series ${\sum }_{k=1}^{\infty }{p}_K(k)$ to converge to 1 reasonably quickly, but any choice of p_K arising in practice should not be a problem. For certain theoretical purposes, it is desirable to have p_K(k) > 0 for all k ∈ {1, 2, …}, e.g., this is necessary to have consistency for the number of components (Nobile, 1994).

For comparison, the Dirichlet process mixture (DPM) model with concentration parameter α > 0 and base measure H is defined as follows, using the stick-breaking representation (Sethuraman, 1994):

$$\begin{gathered} B_1,B_2,\dots \stackrel{\text{iid}}{\sim }\text{Beta}(1,\alpha ) \\ {Z}_1,\dots ,Z_n\stackrel{\text{iid}}{\sim }\pi ,\text{given}\pi =({\pi }_1,{\pi }_2,\dots )\text{where}{\pi }_i=B_i{\prod }_{j=1}^{i-1}(1-B_j) \\ {\theta }_1,{\theta }_2,\dots \stackrel{\text{iid}}{\sim }H \\ {X}_j~f_{{\theta }_{Z_j}}\text{independently}\text{for}j=1,\dots ,n,\text{given}{\theta }_{1:\infty },Z_{1:n}. \end{gathered}$$

3 Exchangeable partition distribution

The primary observation on which our development relies is that the distribution on partitions induced by an MFM takes a form which is simple enough that it can be easily computed. Let 𝒞 denote the partition of [n] := {1, …, n} induced by Z₁, …, Z_n; in other words, 𝒞 = {E_i : |E_i| > 0} where E_i = {j : Z_j = i} for i ∈ {1, 2, …}.

Theorem 3.1

Under the MFM (Equation 2.1), the probability mass function of 𝒞 is

$$p(\mathcal{C})=V_n(t)\prod _{c\in \mathcal{C}}{\gamma }^{(\mid c\mid )}$$ (3.1)

where t = |𝒞| is the number of parts/blocks in the partition, and

$$V_n(t)=\sum _{k=1}^{\infty }\frac{k_{(t)}}{{(\gamma k)}^{(n)}}{p}_K(k).$$ (3.2)

Here, x^(m) = x(x+1) ··· (x+m−1) and x_(m) = x(x−1) ··· (x−m+1), with x⁽⁰⁾ = 1 and x₍₀₎ = 1 by convention; note that x_(m) = 0 when x is a positive integer less than m. Theorem 3.1 can be derived from the well-known formula for p(𝒞|k), which can found in, e.g., Green and Richardson (2001) or McCullagh and Yang (2008); we provide a proof for completeness. All proofs have been collected in the supplementary material. We discuss computation of V_n(t) in Section 3.2. For comparison, under the DPM, the partition distribution induced by Z₁, …, Z_n is $p_{\text{DP}}(\mathcal{C}\mid \alpha )=\frac{{\alpha }^t}{{\alpha }^{(n)}}{\prod }_{c\in \mathcal{C}}(\mid c\mid -1)!$ where t = |𝒞| (Antoniak, 1974). When a prior is placed on α, as in the experiments in Section 7, the DPM partition distribution becomes

$$p_{\text{DP}}(\mathcal{C})=V_n^{\text{DP}}(t)\prod _{c\in \mathcal{C}}(\mid c\mid -1)!$$ (3.3)

where $V_n^{\text{DP}}(t)=\int \frac{{\alpha }^t}{{\alpha }^{(n)}}p(\alpha )d\alpha$.

Viewed as a function of the block sizes (|c| : c ∈ 𝒞), Equation 3.1 is an exchangeable partition probability function (EPPF) in the terminology of Pitman (1995, 2006), since it is a symmetric function of the block sizes. (Some authors also require EPPFs to have self-consistent marginals—see Section 3.3—but we follow the usage of Pitman (2006), who does not require this.) Consequently, 𝒞 is an exchangeable random partition of [n]; that is, its distribution is invariant under permutations of [n] (alternatively, this can be seen directly from the definition of the model, since Z₁, …, Z_n are exchangeable). In fact, Equation 3.1 induces an exchangeable random partition of the positive integers; see Section 3.3.

Further, Equation 3.1 is a member of the family of Gibbs partition distributions (Pitman, 2006); this is also implied by the results of Gnedin and Pitman (2006) characterizing the extreme points of the space of Gibbs partition distributions. Results on Gibbs partitions are provided by Ho et al. (2007), Lijoi et al. (2008), Cerquetti (2011), Cerquetti (2013), Gnedin (2010), and Lijoi and Prünster (2010). However, the utility of this representation for inference in mixture models with a prior on the number of components does not seem to have been previously explored in the literature.

Due to Theorem 3.1, we have the following equivalent representation of the model:

$$\begin{gathered} \mathcal{C}~p(\mathcal{C}),\text{with}p(\mathcal{C})\text{as}\text{in}\text{Equation}3.1 \\ {\varphi }_c\stackrel{\text{iid}}{\sim }H\text{for}c\in \mathcal{C},\text{given}\mathcal{C} \\ {X}_j~f_{{\varphi }_c}\text{independently}\text{for}j\in c,c\in \mathcal{C},\text{given}\varphi ,\mathcal{C}, \end{gathered}$$ (3.4)

where ϕ = (ϕ_c : c ∈ 𝒞) is a tuple of t = |𝒞| parameters ϕ_c ∈ Θ, one for each block c ∈ 𝒞.

This representation is particularly useful for doing inference, since one does not have to deal with component labels or empty components. (If component-specific inferences are desired, a method such as that provided by Papastamoulis and Iliopoulos (2010) may be useful.) The formulation of models starting from a partition distribution has been a fruitful approach, exemplified by the development of product partition models (Hartigan, 1990; Barry and Hartigan, 1992; Quintana and Iglesias, 2003; Dahl, 2009; Park and Dunson, 2010; Müller and Quintana, 2010; Müller et al., 2011) and the loss-based approach of Lau and Green (2007).

3.1 Basic properties

We list here some basic properties of the MFM model. See the supplementary material for proofs. Denoting x_c = (x_j : j ∈ c) and m(x_c) = ∫_Θ [Π_j∈c f_θ(x_j)] H(dθ) (with the convention that m(x_∅) = 1), we have

$$p(x_{1:n}\mid \mathcal{C})=\prod _{c\in \mathcal{C}}m(x_c).$$ (3.5)

While some authors use the terms “cluster” and “component” interchangeably, we use cluster to refer to a block c in a partition 𝒞, and component to refer to a probability distribution f_θi in a mixture ${\sum }_{i=1}^k{\pi }_i{f}_{{\theta }_i}(x)$. The number of components K and the number of clusters T = |𝒞| are related by

$$p(t\mid k)=\frac{k_{(t)}}{{(\gamma k)}^{(n)}}\sum _{\mathcal{C}:\mid \mathcal{C}\mid =t}\prod _{c\in \mathcal{C}}{\gamma }^{(\mid c\mid )},$$ (3.6)

$$p(k\mid t)=\frac{1}{V_n(t)}\frac{k_{(t)}}{{(\gamma k)}^{(n)}}{p}_K(k),$$ (3.7)

where in Equation 3.6, the sum is over partitions 𝒞 of [n] such that |𝒞| = t. Note that p(t|k) and p(k|t) depend on n. Wherever possible, we use capital letters (such as K and T) to denote random variables as opposed to particular values (k and t). The formula for p(k|t) is required for doing inference about the number of components K based on posterior samples of 𝒞; fortunately, it is easy to compute. We have the conditional independence relations

$$\mathcal{C}\perp K\mid T,$$ (3.8)

$$X_{1:n}\perp K\mid T.$$ (3.9)

3.2 The coefficients V_n(t)

The following recursion is a special case of a more general result for Gibbs partitions (Gnedin and Pitman, 2006).

Proposition 3.2

The numbers V_n(t) (Equation 3.2) satisfy the recursion

$$V_{n+1}(t+1)=V_n(t)/\gamma -(n/\gamma +t)V_{n+1}(t)$$ (3.10)

for any 0 ≤ t ≤ n and γ > 0.

This is easily seen by plugging the identity

$$k_{(t+1)}=(\gamma k+n)k_{(t)}/\gamma -(n/\gamma +t)k_{(t)}$$

into the expression for V_n+1(t + 1). In the case of γ = 1, Gnedin (2010) has discovered a beautiful example of a distribution on K for which both p_K(k) and V_n(t) have closed-form expressions.

In previous work on the MFM model, p_K has often been chosen to be proportional to a Poisson distribution restricted to the positive integers or a subset thereof (Phillips and Smith, 1996; Stephens, 2000; Nobile and Fearnside, 2007), and Nobile (2005) has proposed a theoretical justification for this choice. Interestingly, the model has some nice mathematical properties if one instead chooses K − 1 to be given a Poisson distribution, that is, p_K(k) = Poisson(k − 1|λ) for some λ > 0. For instance, it turns out that if p_K(k) = Poisson(k − 1|λ) and γ = 1 then $V_n(0)=(1-{\sum }_{k=1}^n{p}_K(k))/{\lambda }^n$.

However, to do inference, it is not necessary to choose p_K to have any particular form—we just need to be able to compute p(𝒞), and in turn, we need to be able to compute V_n(t). To this end, note that k_(t)/(γk)⁽ⁿ⁾ ≤ k^t/(γk)ⁿ, and thus the infinite series for V_n(t) (Equation 3.2) converges rapidly when t ≪ n. It always converges to a finite value when 1 ≤ t ≤ n; this is clear from the fact that p(𝒞) ∈ [0, 1]. This finiteness can also be seen directly from the series since k^t/(γk)ⁿ ≤ 1/γⁿ, and in fact, this shows that the series for V_n(t) converges at least as rapidly (up to a constant) as the series ${\sum }_{k=1}^{\infty }{p}_K(k)$ converges to 1. Hence, for any reasonable choice of p_K (i.e., not having an extraordinarily heavy tail), V_n(t) can easily be numerically approximated to a high level of precision. In practice, all of the required values of V_n(t) can be precomputed before MCMC sampling, and this takes a negligible amount of time relative to MCMC (see the supplementary material).

3.3 Self-consistent marginals

For each n = 1, 2, …, let q_n(𝒞) denote the MFM distribution on partitions of [n] (Equation 3.1). This family of partition distributions is preserved under marginalization, in the following sense.

Proposition 3.3

If m < n then q_m coincides with the marginal distribution on partitions of [m] induced by q_n.

In other words, drawing a sample from q_n and removing elements m + 1, …, n from it yields a sample from q_m. This can be seen directly from the model definition (Equation 2.1), since 𝒞 is the partition induced by the Z’s, and the distribution of Z_1:m is the same when the model is defined with any n ≥ m. This property is sometimes referred to as consistency in distribution (Pitman, 2006).

By Kolmogorov’s extension theorem (e.g., Durrett, 1996), it is well-known that this implies the existence of a unique probability distribution on partitions of the positive integers ℤ_>0 = {1, 2, …} such that the marginal distribution on partitions of [n] is q_n for all n ∈ {1, 2, …} (Pitman, 2006).

4 Equivalent representations

4.1 Pólya urn scheme / Restaurant process

Pitman (1996) considered a general class of urn schemes, or restaurant processes, corresponding to exchangeable partition probability functions (EPPFs). The following scheme for the MFM falls into this general class.

Theorem 4.1

The following process generates partitions 𝒞₁, 𝒞₂, … such that for any n ∈ {1, 2, …}, the probability mass function of 𝒞_n is given by Equation 3.1.

• Initialize with a single cluster consisting of element 1 alone: 𝒞₁ = {{1}}.

• For n = 2, 3, …, element n is placed in …

  • an existing cluster c ∈ 𝒞_n−1 with probability ∝ |c| + γ

  • a new cluster with probability $\propto \frac{V_n(t+1)}{V_n(t)}\gamma$

  where t = |𝒞_n−1|.

Clearly, this bears a close resemblance to the Chinese restaurant process (i.e., the Blackwell–MacQueen urn process), in which the nth element is placed in an existing cluster c with probability ∝ |c| or a new cluster with probability ∝ α (the concentration parameter) (Blackwell and MacQueen, 1973; Aldous, 1985).

4.2 Random discrete measures

The MFM can also be formulated starting from a distribution on discrete measures that is analogous to the Dirichlet process. With K, π, and θ_1:K as in Equation 2.1, let

$$G=\sum _{i=1}^K{\pi }_i{\delta }_{{\theta }_i}$$

where δ_θ is the unit point mass at θ. Let us denote the distribution of G by ℳ(p_K, γ,H). Note that G is a random discrete measure over Θ. If H is continuous (i.e., H({θ}) = 0 for all θ ∈ Θ), then with probability 1, the number of atoms is K; otherwise, there may be fewer than K atoms. If we let X₁, …, X_n|G be i.i.d. from the resulting mixture, i.e., from

$$f_G(x):=\int f_{\theta }(x)G(d\theta )=\sum _{i=1}^K{\pi }_i{f}_{{\theta }_i}(x),$$

then the distribution of X_1:n is the same as in Equation 2.1. So, in this notation, the MFM model is:

$$\begin{gathered} G~\mathcal{M}(p_K,\gamma ,H) \\ {X}_1,\dots ,X_n\stackrel{\text{iid}}{\sim }{f}_G,\text{given}G. \end{gathered}$$

This random discrete measure perspective is connected to work on species sampling models (Pitman, 1996) in the following way. When H is continuous, we can construct a species sampling model by letting G ~ ℳ(p_K, γ,H) and modeling the observed data as β₁, …, β_n ~ G given G. We refer to Pitman (1996), Hansen and Pitman (2000), Ishwaran and James (2003), Lijoi et al. (2005, 2007), and Lijoi et al. (2008) for more background on species sampling models and further examples. Pitman derived a general formula for the posterior predictive distribution of a species sampling model; the following result is a special case. Note the close relationship to the restaurant process (Theorem 4.1).

Theorem 4.2

If H is continuous, then β₁ ~ H and the distribution of β_n given β₁, …, β_n−1 is proportional to

$$\frac{V_n(t+1)}{V_n(t)}\gamma H+\sum _{i=1}^t(n_i+\gamma ){\delta }_{{\beta }_i^{\ast }},$$ (4.1)

where ${\beta }_1^{\ast },\dots ,{\beta }_t^{\ast }$ are the distinct values taken by β₁, …, β_n−1, and $n_i=\#\{j\in [n-1]:{\beta }_j={\beta }_i^{\ast }\}$.

For comparison, when G ~ DP(αH) instead, the distribution of β_n given β₁, …, β_n−1 is proportional to $\alpha H+{\sum }_{j=1}^{n-1}{\delta }_{{\beta }_j}=\alpha H+{\sum }_{i=1}^t{n}_i{\delta }_{{\beta }_i^{\ast }}$, since $G\mid {\beta }_1,\dots ,{\beta }_{n-1}~\text{DP}(\alpha H+{\sum }_{j=1}^{n-1}{\delta }_{{\beta }_j})$ (Ferguson, 1973; Blackwell and MacQueen, 1973).

4.3 Stick-breaking representation

The Dirichlet process has an elegant stick-breaking representation for the mixture weights π₁, π₂, … (Sethuraman, 1994; Sethuraman and Tiwari, 1981). This extraordinarily clarifying perspective has inspired a number of nonparametric models (MacEachern, 1999, 2000; Hjort, 2000; Ishwaran and Zarepour, 2000; Ishwaran and James, 2001; Griffin and Steel, 2006; Dunson and Park, 2008; Chung and Dunson, 2009; Rodriguez and Dunson, 2011; Broderick et al., 2012), has provided insight into the properties of related models (Favaro et al., 2012; Teh et al., 2007; Thibaux and Jordan, 2007; Paisley et al., 2010), and has been used to develop efficient inference algorithms (Ishwaran and James, 2001; Blei and Jordan, 2006; Papaspiliopoulos and Roberts, 2008; Walker, 2007; Kalli et al., 2011).

In a certain special case—namely, when p_K(k) = Poisson(k − 1|λ) and γ = 1—we have noticed that the MFM also has an interesting representation that we describe using the stick-breaking analogy, although it is somewhat different in nature. Consider the following procedure:

Take a unit-length stick, and break off i.i.d. Exponential(λ) pieces until you run out of stick.

In other words, let ${\epsilon }_1,{\epsilon }_2,\dots \stackrel{\text{iid}}{\sim }\text{Exponential}(\lambda )$, define $\stackrel{\sim }{K}=min\{j:{\sum }_{i=1}^j{\epsilon }_i\ge 1\}$, let π̃_i = ε_i for i = 1, …, K̃ − 1, and let ${\stackrel{\sim }{\pi }}_{\stackrel{\sim }{K}}=1-{\sum }_{i=1}^{\stackrel{\sim }{K}-1}{\stackrel{\sim }{\pi }}_i$.

Proposition 4.3

The stick lengths π̃ have the same distribution as the mixture weights π in the MFM model when p_K(k) = Poisson(k − 1|λ) and γ = 1.

This is a consequence of a standard construction for Poisson processes. This suggests a way of generalizing the MFM model: take any sequence of positive random variables (ε₁, ε₂, …) (not necessarily independent or identically distributed) such that ${\sum }_{i=1}^{\infty }{\epsilon }_i>1$ with probability 1, and define K̃ and π̃ as above. Although the distribution of K̃ and π̃ may be complicated, in some cases it might still be possible to do inference based on the stick-breaking representation. This might be an interesting way to introduce different kinds of prior information on the mixture weights, however, we have not explored this possibility.

5 Asymptotics

In this section, we consider the asymptotics of V_n(t), the asymptotic relationship between the number of components and the number of clusters, and the approximate form of the conditional distribution on cluster sizes given the number of clusters.

5.1 Asymptotics of V_n(t)

Recall that $V_n(t)={\sum }_{k=1}^{\infty }\frac{k_{(t)}}{{(\gamma k)}^{(n)}}{p}_K(k)$ (Equation 3.2) for 1 ≤ t ≤ n, where γ > 0 and p_K is a p.m.f. on {1, 2, …}. When a_n, b_n > 0, we write a_n ~ b_n to mean a_n/b_n → 1 as n→∞.

Theorem 5.1

For any t ∈ {1, 2, …}, if p_K(t) > 0 then

$$V_n(t)~\frac{t_{(t)}}{{(\gamma t)}^{(n)}}{p}_K(t)~\frac{t!}{n!}\frac{\mathrm{\Gamma }(\gamma t)}{n^{\gamma t-1}}{p}_K(t)$$ (5.1)

as n→∞.

In particular, V_n(t) has a simple interpretation, asymptotically—it behaves like the k = t term in the series. All proofs have been collected in the supplementary material.

5.2 Relationship between the number of clusters and number of components

In the MFM, it is perhaps intuitively clear that, under the prior at least, the number of clusters T = |𝒞| behaves very similarly to the number of components K when n is large. It turns out that under the posterior they also behave very similarly for large n.

Theorem 5.2

Let x₁, x₂, … ∈ 𝒳 and k ∈ {1, 2, …}. If p_K(1), …, p_K(k) > 0 then

$$\mid p(T=k\mid x_{1:n})-p(K=k\mid x_{1:n})\mid \to 0$$

as n→∞.

5.3 Distribution of the cluster sizes under the prior

Here, we examine one of the major differences between the MFM and DPM priors. Roughly speaking, under the prior, the MFM prefers all clusters to be the same order of magnitude, while the DPM prefers having a few large clusters and many very small clusters. In the following calculations, we quantify the preceding statement more precisely. (See Green and Richardson (2001) for informal observations along these lines.) Interestingly, these prior influences remain visible in certain aspects of the posterior, even in the limit as n goes to infinity.

Let 𝒞 be the random partition of [n] in the MFM model (Equation 3.1), and let A = (A₁, …, A_T) be the ordered partition of [n] obtained by randomly ordering the blocks of 𝒞, uniformly among the T! possible choices, where T = |𝒞|. Then

$$p(A)=\frac{p(\mathcal{C})}{\mid \mathcal{C}\mid !}=\frac{1}{t!}{V}_n(t)\prod _{i=1}^t{\gamma }^{(\mid A_i\mid )},$$

where t = |𝒞|. Now, let S = (S₁, …, S_T) be the vector of block sizes of A, that is, S_i = |A_i|. Then

$$p(S=s)=\sum _{A:S(A)=s}p(A)=V_n(t)\frac{n!}{t!}\prod _{i=1}^t\frac{{\gamma }^{(s_i)}}{s_i!}$$

for s ∈ Δ_t, t ∈ {1, …, n}, where Δ_t = {s ∈ ℤ_t : Σ_i s_i = n, s_i ≥ 1 ∀i} (i.e., Δ_t is the set of t-part compositions of n). For any x > 0, by writing x^(m)/m! = Γ(x + m)/(m! Γ(x)) and using Stirling’s approximation, we have x^(m)/m! ~ m^x−1/Γ(x) as m→∞. This yields the approximations

$$p(S=s)\approx \frac{V_n(t)}{\mathrm{\Gamma }{(\gamma )}^t}\frac{n!}{t!}\prod _{i=1}^t{s}_i^{\gamma -1}\approx \frac{p_K(t)}{n^{\gamma t-1}}\frac{\mathrm{\Gamma }(\gamma t)}{\mathrm{\Gamma }{(\gamma )}^t}\prod _{i=1}^t{s}_i^{\gamma -1}$$

(using Theorem 5.1 in the second step), and

$$p(S=s\mid T=t)\approx \kappa \prod _{i=1}^t{s}_i^{\gamma -1}$$

for s ∈ Δ_t, where κ is a normalization constant. Thus, p(s|t) has approximately the same shape as a symmetric t-dimensional Dirichlet distribution, except it is discrete. This would be obvious if we were conditioning on the number of components K, and it makes intuitive sense when conditioning on T, since K and T are essentially the same for large n.

It is interesting to compare p(s) and p(s|t) to the corresponding distributions for the Dirichlet process. For the DP with a prior on α, we have $p_{\text{DP}}(\mathcal{C})=V_n^{\text{DP}}(t){\prod }_{c\in \mathcal{C}}(\mid c\mid -1)!$ by Equation 3.3, and p_DP(A) = p_DP(𝒞)/|𝒞|! as before, so for s ∈ Δ_t, t ∈ {1, …, n},

$$p_{\text{DP}}(S=s)=V_n^{\text{DP}}(t)\frac{n!}{t!}{s}_1^{-1}\cdots s_t^{-1}$$

and

$$p_{\text{DP}}(S=s\mid T=t)\propto s_1^{-1}\cdots s_t^{-1},$$

which has the same shape as a t-dimensional Dirichlet distribution with all the parameters taken to 0 (noting that this is normalizable since Δ_t is finite). Asymptotically in n, p_DP(s|t) puts all of its mass in the “corners” of the discrete simplex Δ_t, while under the MFM, p(s|t) remains more evenly dispersed.

6 Inference algorithms

Many approaches have been used for posterior inference in the MFM model (Equation 2.1). Nobile (1994) approximates the marginal likelihood p(x_1:n|k) of each k in order to compute the posterior on k, and uses standard methods given k. Phillips and Smith (1996) and Stephens (2000) use jump diffusion and point process approaches, respectively, to sample from p(k, π, θ|x_1:n). Richardson and Green (1997) employ reversible jump moves to sample from p(k, π, θ, z|x_1:n). In cases with conjugate priors, Nobile and Fearnside (2007) use various Metropolis–Hastings moves on p(k, z|x_1:n), and McCullagh and Yang (2008) develop an approach that could in principle be used to sample from p(k, 𝒞|x_1:n).

Due to the results of Sections 3 and 4, much of the extensive body of work on MCMC samplers for DPMs can be directly applied to MFMs. This leads to a variety of MFM samplers, of which we consider two main types: (a) sampling from p(𝒞|x_1:n) in cases where the marginal likelihood m(x_c) = ∫_Θ [Π_j∈c f_θ(x_j)] H(dθ) can easily be computed (typically, this means H is a conjugate prior), and (b) sampling from p(𝒞, ϕ|x_1:n) (in the notation of Equation 3.4) in cases where m(x_c) is intractable or when samples of the component parameters are desired.

When m(x_c) can easily be computed, the following Gibbs sampling algorithm can be used to sample from the posterior on partitions, p(𝒞|x_1:n). Given a partition 𝒞, let 𝒞 \ j denote the partition obtained by removing element j from 𝒞.

1. Initialize 𝒞 = {[n]} (i.e., one cluster).

2. Repeat the following steps N times, to obtain N samples.

   • For j = 1, …, n: Remove element j from 𝒞, and place it …

     • in c ∈ 𝒞 \ j with probability $\propto (\mid c\mid +\gamma )\frac{m(x_{c\cup j})}{m(x_c)}$

     • in a new cluster with probability $\propto \gamma \frac{V_n(t+1)}{V_n(t)}m(x_j)$

     where t = |𝒞 \ j|.

This is a direct adaptation of “Algorithm 3” for DPMs (MacEachern, 1994; Neal, 1992, 2000). The only differences are that in Algorithm 3, |c| + γ is replaced by |c|, and γV_n(t + 1)/V_n(t) is replaced by α (the concentration parameter). Thus, the differences between the MFM and DPM versions of the algorithm are precisely the same as the differences between their respective urn/restaurant processes. In order for the algorithm to be valid, the Markov chain needs to be irreducible, and to achieve this it is necessary that {t ∈ {1, 2, …} : V_n(t) > 0} be a block of consecutive integers containing 1. It turns out that this is always the case, since for any k such that p_K(k) > 0, we have V_n(t) > 0 for all t = 1, …, k.

Note that one only needs to compute V_n(1), …, V_n(t^*) where t^* is the largest value of t visited by the sampler. In practice, it is convenient to precompute these values using a guess at an upper bound t_pre on t^*, and this takes a negligible amount of time compared to running the sampler (see the supplementary material). Just to be clear, t_pre is only introduced for computational expedience—the model allows unbounded t, and in the event that the sampler visits t > t_pre, it is trivial to increase t_pre and compute the required values of V_n(t) on demand.

When m(x_c) cannot easily be computed, a clever auxiliary variable technique referred to as “Algorithm 8” can be used for inference in the DPM (Neal, 2000; MacEachern and Müller, 1998). Making the same substitutions as above, we can apply Algorithm 8 for inference in the MFM as well, to sample from p(𝒞, ϕ|x_1:n).

A well-known issue with incremental Gibbs samplers such as Algorithms 3 and 8, when applied to DPMs, is that mixing can be somewhat slow, since it may take a long time to create or destroy substantial clusters by moving one element at a time. With MFMs, this issue seems to be exacerbated, since compared with DPMs, MFMs tend to put small probability on partitions with tiny clusters (see Section 5.3), making it difficult for the sampler to move through these regions of the space.

To deal with this issue, split-merge samplers for DPMs have been developed, in which a large number of elements can be reassigned in a single move (Dahl, 2003, 2005; Jain and Neal, 2004, 2007). In the same way as the incremental samplers, one can directly apply these split-merge samplers to MFMs as well, by simply plugging-in the MFM partition distribution in place of the DPM partition distribution. More generally, it seems likely that any partition-based MCMC sampler for DPMs could be applied to MFMs as well. In Section 7, we apply the Jain–Neal split-merge samplers, coupled with incremental Gibbs samplers, to do inference for MFMs in both conjugate and non-conjugate settings.

As usual in mixture models, the MFM is invariant under permutations of the component labels, and this can lead to so-called “label-switching” issues if one naively attempts to estimate quantities that are not invariant to labeling—such as the mean of a given component—by averaging MCMC samples. In the experiments below, we only estimate quantities that are invariant to labeling, such as the density, the number of components, and the probability that two given points belong to the same cluster. Consequently, the label-switching issue does not arise here. If inference for non-invariant quantities is desired, methods for this purpose have been proposed (Papastamoulis and Iliopoulos, 2010).

In addition to the partition-based algorithms above, several DPM inference algorithms are based on the stick-breaking representation, such as the slice samplers for mixtures (Walker, 2007; Kalli et al., 2011). Although we have not explored this approach, it should be possible to adapt these to MFMs using the stick-breaking representation described in Section 4.3.

7 Empirical demonstrations

In Section 7.1, we compare the mixing performance of reversible jump MCMC versus the Jain–Neal split-merge algorithms proposed in Section 6 for inference in the MFM. Section 7.2 applies an MFM model to discriminate cancer subtypes based on gene expression data. In Section 7.3, we illustrate some of the similarities and differences between MFMs and DPMs. All of the examples below happen to involve Gaussian component densities, but of course our approach is not limited to mixtures of Gaussians.

7.1 Comparison with reversible jump MCMC

We compare two methods of inference for the MFM: the usual reversible jump MCMC (RJMCMC) approach, and our proposed approach based on the Jain–Neal split-merge algorithms. This purpose of this is (1) to compare the mixing performance of our method versus RJMCMC, and (2) to demonstrate the validity of our method by showing agreement with results obtained using RJMCMC. In addition to having better mixing performance, our method has the significant advantage of providing generic algorithms that apply to a wide variety of component distributions and priors, whereas RJMCMC requires one to design specialized reversible jump moves.

RJMCMC is a very general framework and an enormous array of MCMC algorithms could be considered to be a special case of it. The most commonly-used instantiation of RJMCMC for mixtures seems to be the original algorithm of Richardson and Green (1997) for univariate Gaussian mixtures, so it makes sense to compare with this. For multivariate Gaussian mixtures, several RJMCMC algorithms have been proposed (Marrs, 1998; Zhang et al., 2004; Dellaportas and Papageorgiou, 2006), however, the model we consider below uses diagonal covariance matrices, and in this case, the Richardson and Green (1997) moves can simply be extended to each coordinate. More efficient algorithms almost certainly exist within the vast domain of RJMCMC, but the goal here is to compare with “plain-vanilla RJMCMC” as it is commonly-used.

7.1.1 Comparison with RJMCMC on the galaxy dataset

The galaxy dataset (Roeder, 1990) is a standard benchmark for mixture models, consisting of measurements of the velocities of 82 galaxies in the Corona Borealis region. To facilitate the comparison with RJMCMC, we use exactly the same model as Richardson and Green (1997). The component densities are univariate normal, f_θ(x) = f_μ,λ(x) = 𝒩(x|μ, λ⁻¹), and the base measure (prior) H on θ = (μ, λ) is $\mu ~\mathcal{N}({\mu }_0,{\sigma }_0^2)$, λ ~ Gamma(a, b) independently (where Gamma(λ|a, b) ∝ λ^a−1e^−bλ). Further, a hyperprior is placed on b, by taking b ~ Gamma(a₀, b₀). The remaining parameters are set to μ₀ = (max{x_i} + min{x_i})/2, σ₀ = max{x_i} − min{x_i}, a = 2, a₀ = 0.2, and $b_0=10/{\sigma }_0^2$. Note that the parameters μ₀, σ₀, and b₀ are functions of the observed data x₁, …, x_n, making this a data-dependent prior. Note further that taking μ and λ to be independent results in a non-conjugate prior; this is appropriate when the location of the data is not informative about the scale (and vice versa). See Richardson and Green (1997) for the rationale behind these parameter choices. (Note: This choice of σ₀ may be a bit too large, affecting the posteriors on the number of clusters and components, however, we stick with it to facilitate the comparison.) Following Richardson and Green (1997), we take K ~ Uniform{1, …, 30} and set γ = 1 for the finite-dimensional Dirichlet parameters.

To implement our proposed inference approach, we use the split-merge sampler of Jain and Neal (2007) for non-conjugate priors, coupled with Algorithm 8 of Neal (2000) (using a single auxiliary variable) for incremental Gibbs updates to the partition. Specifically, following Jain and Neal (2007), we use the (5,1,1,5) scheme: 5 intermediate scans to reach the split launch state, 1 split-merge move per iteration, 1 incremental Gibbs scan per iteration, and 5 intermediate moves to reach the merge launch state. We use Gibbs sampling to handle the hyperprior on b (i.e., append b to the state of the Markov chain, run the sampler given b as usual, and periodically sample b given everything else). More general hyperprior structures can be handled similarly. We do not restrict the parameter space in any way (e.g., forcing the component means to be ordered to obtain identifiability, as was done by Richardson and Green, 1997). All of the quantities we consider are invariant to the labeling of the clusters; see Jasra et al. (2005) for discussion on this point.

Results for RJMCMC were obtained using the Nmix software provided by Peter Green,¹ which implements the algorithm of Richardson and Green (1997).

Results

First, we demonstrate agreement between inference results obtained using RJMCMC and using our method based on the Jain–Neal algorithm. Table 1 compares estimates of the MFM posterior on the number of components k using both Jain–Neal and RJMCMC. Each algorithm was run for 10⁶ iterations, the first 10⁵ of which were discarded as burn-in. The two methods are in very close agreement, empirically verifying that they are both correctly sampling from the MFM posterior.

Table 1.

Estimates of the MFM posterior on k for the galaxy dataset.

k	1	2	3	4	5	6	7
Jain–Neal	0.000	0.000	0.060	0.134	0.187	0.194	0.158
RJMCMC	0.000	0.000	0.059	0.131	0.187	0.197	0.160

8	9	10	11	12	13	14	15
0.110	0.069	0.040	0.023	0.012	0.007	0.004	0.002
0.110	0.068	0.039	0.022	0.012	0.006	0.003	0.002

To compare the mixing performance of the two methods, Figure 1 (top) shows traceplots of the number of clusters over the first 5000 iterations. The number of clusters is often used to assess mixing, since it is usually one of the quantities for which mixing is slowest. One can see that the Jain–Neal split-merge algorithm appears to explore the space more quickly than RJMCMC. Figure 1 (bottom left) shows estimates of the autocorrelation functions for the number of clusters, scaled by iteration. The autocorrelation of Jain–Neal decays significantly more rapidly than that of RJMCMC, confirming the visual intuition from the traceplot. This makes sense since the Jain–Neal algorithm makes splitting proposals in an adaptive, data-dependent way. The effective sample size (based on the number of clusters) is estimated to be 1.6% for Jain–Neal versus 0.6% for RJMCMC; in other words, 100 Jain–Neal samples are equivalent to 1.6 independent samples (Kass et al., 1998).

Figure 1.

Results on the galaxy dataset. Top: Traceplots of the number of clusters at each iteration. Bottom: Estimated autocorrelation function of the number of clusters, scaled by iteration (left) and time in milliseconds (right).

However, each iteration of the Jain–Neal algorithm takes approximately 2.9 × 10⁻⁶ n seconds, where n = 82 is the sample size, compared with 1.3 × 10⁻⁶ n seconds for RJM-CMC, which is a little over twice as fast. (All experiments in this paper were performed using a 2.80 GHz processor with 6 GB of RAM.) In both algorithms, one iteration consists of a reassignment move for each datapoint, updates to component parameters and hyperparameters, and a split or merge move, but it makes sense that Jain–Neal would be a bit slower per iteration since its reassignment moves and split-merge moves require a few more steps. Nonetheless, it seems that this is compensated for by the improvement in mixing per iteration: Figure 1 (bottom right) shows that when rescaled to account for computation time, the estimated autocorrelation functions are nearly the same. Thus, in this experiment, the two methods perform roughly equally well.

7.1.2 Comparison with RJMCMC as dimensionality increases

In this subsection, we show that our approach can provide a massive improvement over RJMCMC when the parameter space is high-dimensional. We illustrate this with a model that will be applied to gene expression data in Section 7.2.

We simulate data of increasing dimensionality, and assess the effect on mixing performance. Specifically, for a given dimensionality d, we draw $X_1,\dots ,X_n~\frac{1}{3}\mathcal{N}(m,I)+\frac{1}{3}\mathcal{N}(0,I)+\frac{1}{3}\mathcal{N}(-m,I)$, where $m=(3/\sqrt{d},\dots ,3/\sqrt{d})\in {ℝ}^d$, and then normalize each dimension to have zero mean and unit variance. The purpose of the $1/\sqrt{d}$ factor is to prevent the posterior from concentrating too quickly as d increases, so that one can still see the sampler exploring the space.

To enable scaling to the high-dimensional gene expression data in Section 7.2, we use multivariate Gaussian component densities with diagonal covariance matrices (i.e., for each component, the dimensions are independent univariate Gaussians), and we place independent conjugate priors on each dimension. Specifically, for each component, for i = 1, …, d, dimension i is $\mathcal{N}({\mu }_i,{\lambda }_i^{-1})$ given (μ_i, λ_i), and λ_i ~ Gamma(a, b), μ_i|λ_i ~ 𝒩(0, (cλ_i)⁻¹). We set a = 1, b = 1, c = 1 (recall that the data are zero mean, unit variance in each dimension), K ~ Geometric(r) (p_K(k) = (1 − r)^k−1r for k = 1, 2, …) with r = 0.1, and γ = 1. We compare the mixing performance of three samplers:

1. Collapsed Jain–Neal. Given the partition C of the data into clusters, the parameters can be integrated out analytically since the prior is conjugate. Thus, we can use the split-merge sampler of Jain and Neal (2004) for conjugate priors, coupled with Algorithm 3 of Neal (2000). Following Jain and Neal (2004), we use the (5,1,1) scheme: 5 intermediate scans to reach the split launch state, 1 split-merge move per iteration, and 1 incremental Gibbs scan per iteration.

2. Jain–Neal. Even though the prior is conjugate, we can still use the Jain and Neal (2007) split-merge algorithm for non-conjugate priors, coupled with Algorithm 8 of Neal (2000). As in Section 7.1.1, we use the (5,1,1,5) scheme.

3. RJMCMC. For reversible jump MCMC, we use a natural multivariate extension of the moves from Richardson and Green (1997). Specifically, for the split move, we propose splitting the mean and variance for each dimension i according to the formulas in Richardson and Green (1997), using independent variables u_2i and u_3i. For the birth move, we make a proposal from the prior. The remaining moves are essentially the same as in Richardson and Green (1997).

Results

For each d ∈ {4, 8, 12}, a dataset of size n = 100 was generated as described above, and each of the three algorithms was run for 10⁶ iterations. For each algorithm and each d, Figure 2 shows the estimated autocorrelation function for the number of clusters (scaled to milliseconds), as well as a traceplot of the number of clusters over the first 20 seconds of sampling. We see that as the dimensionality increases, mixing performance degrades for all three algorithms, which is to be expected. However, it degrades significantly more rapidly for RJMCMC than for the Jain–Neal algorithms, especially the collapsed version. This difference can be attributed to the clever data-dependent split-merge proposals in the Jain–Neal algorithms, and the fact that the collapsed version benefits further from the effective reduction in dimensionality resulting from integrating out the parameters.

Figure 2.

Top: Estimated autocorrelation functions for the number of clusters, for each algorithm, for each d ∈ {4, 8, 12}. Bottom: Traceplots of the number of clusters over the first 20 seconds of sampling.

As the dimensionality increases, the advantage over RJMCMC becomes ever greater. This demonstrates the major improvement in performance that can be obtained using the approach we propose. In fact, in the next section, we apply our method to very high-dimensionality gene expression data on which the time required for RJMCMC to mix becomes unreasonably long, making it impractical. Meanwhile, our approach using the collapsed Jain–Neal algorithm still mixes well.

7.2 Discriminating cancer subtypes using gene expression data

In cancer research, gene expression profiling—that is, measuring the amount that each gene is expressed in a given tissue sample—enables the identification of distinct subtypes of cancer, leading to greater understanding of the mechanisms underlying cancers as well as potentially providing patient-specific diagnostic tools. In gene expression datasets, there are typically a small number of very high-dimensional data points, each consisting of the gene expression levels in a given tissue sample under given conditions.

One approach to analyzing gene expression data is to use Gaussian mixture models to identify clusters which may represent distinct cancer subtypes (Yeung et al., 2001; McLachlan et al., 2002; Medvedovic and Sivaganesan, 2002; Medvedovic et al., 2004; de Souto et al., 2008; Rasmussen et al., 2009; McNicholas and Murphy, 2010). In fact, in a comparative study of seven clustering methods on 35 cancer gene expression datasets with known ground truth, de Souto et al. (2008) found that finite mixtures of Gaussians provided the best results, as long as the number of components k was set to the true value. However, in practice, choosing an appropriate value of k can be difficult. Using the methods developed in this paper, the MFM provides a principled approach to inferring the clusters even when k is unknown, as well as doing inference for k, provided that the components are well-modeled by Gaussians.

The purpose of this example is to demonstrate that our approach can work well even in very high-dimensional settings, and may provide a useful tool for this application. It should be emphasized that we are not cancer scientists, so the results reported here should not be interpreted as scientifically relevant, but simply as a proof-of-concept.

7.2.1 Setup: data, model, and inference

We apply the MFM to gene expression data collected by Armstrong et al. (2001) in a study of leukemia subtypes. Armstrong et al. (2001) measured gene expression levels in samples from 72 patients who were known to have one of two leukemia types, acute lymphoblastic leukemia (ALL) or acute myelogenous leukemia (AML), and they found that a previously undistinguished subtype of ALL, which they termed mixed-lineage leukemia (MLL), could be distinguished from conventional ALL and AML based on the gene expression profiles.

We use the preprocessed data provided by de Souto et al. (2008), which they filtered to include only genes with expression levels differing by at least 3-fold in at least 30 samples, relative to their mean expression level across all samples. The resulting dataset consists of 72 samples and 1081 genes per sample, i.e., n = 72 and d = 1081. Following standard practice, we take the base-2 logarithm of the data before analysis, and normalize each dimension to have zero mean and unit variance.

We use the same model as in Section 7.1.2. Note that these parameter settings are simply defaults and have not been tailored to the problem; a careful scientific investigation would involve thorough prior elicitation, sensitivity analysis, and model checking.

The collapsed Jain–Neal algorithm described in Section 7.1.2 was used for inference. The sampler was run for 1,000 burn-in iterations, and 19,000 sample iterations. This appears to be many more iterations than required for burn-in and mixing in this particular example—in fact, only 5 to 10 iterations are required to separate the clusters, and the results are indistinguishable from using only 10 burn-in and 190 sample iterations. The full state of the chain was recorded every 20 iterations. Each iteration took approximately 1.3 × 10⁻³ n seconds, where n = 72 is the number of observations in the dataset.

7.2.2 Results

The posterior on the number of clusters t is concentrated at 3 (see Figure 3), in agreement with the division into ALL, MLL, and AML determined by Armstrong et al. (2001). The posterior on k is shifted slightly to the right because there are a small number of observations; this accounts for uncertainty regarding the possibility of additional components that were not observed in the given data.

Figure 3.

Results on the leukemia gene expression dataset. Left: Posteriors on the number of clusters and components. Right: Posterior similarity matrix. See the text for discussion.

Figure 3 also shows the posterior similarity matrix, that is, the matrix in which entry (i, j) is the posterior probability that data points i and j belong to the same cluster; in the figure, white is probability 0 and black is probability 1. The rows and columns of the matrix are ordered according to ground truth, such that 1–24 are ALL, 25–44 are MLL, and 45–72 are AML. The model has clearly separated the subjects into these three groups, with a small number of exceptions: subject 41 is clustered with the AML subjects instead of MLL, subject 52 with the MLL subjects instead of AML, and subject 17 is about 50% ALL and 50% MLL. Thus, the MFM has successfully clustered the subjects according to cancer subtype. This demonstrates the viability of our approach in high-dimensional settings.

7.3 Similarities and differences between MFMs and DPMs

In the preceding sections, we established that MFMs share many of the mathematical properties of DPMs. Here, we empirically compare MFMs and DPMs, using simulated data from a three-component bivariate Gaussian mixture. Our purpose is not to argue that either model should be preferred over the other, but simply to illustrate that in certain respects they are very similar, while in other respects they differ. In general, we would not say that either model is uniformly better than the other; rather, one should choose the model that is best suited to the application at hand—specifically, if one believes there to be infinitely many components, then an infinite mixture such as a DPM is more appropriate, while if one expects finitely many components, an MFM is likely to be a better choice. See Section A.1 for further discussion. For additional empirical analysis of MFMs versus DPMs, we refer to Green and Richardson (2001), who present different types of comparisons than we consider here.

7.3.1 Setup: data, model, and inference

Consider the data distribution ${\sum }_{i=1}^3{w}_i\mathcal{N}({\mu }_i,C_i)$ where w = (0.45, 0.3, 0.25), ${\mu }_1=\left(\begin{array}{c}4\\ 4\end{array}\right),{\mu }_2=\left(\begin{array}{c}7\\ 4\end{array}\right),{\mu }_3=\left(\begin{array}{c}6\\ 2\end{array}\right),C_1=\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right),C_2=R\left(\begin{array}{cc}2.5& 0\\ 0& 0.2\end{array}\right)R^{\mathrm{T}}$ where $R=\left(\begin{array}{cc}cos\rho & -sin\rho \\ sin\rho & cos\rho \end{array}\right)$ with ρ = π/4, and $C_3=\left(\begin{array}{cc}3& 0\\ 0& 0.1\end{array}\right)$. See Figure 4 (bottom).

Figure 4.

Density estimates for MFM (left) and DPM (right) on increasing amounts of data from the bivariate example (bottom). As n increases, the estimates appear to be converging to the true density, as expected.

For the model, we use multivariate normal component densities f_θ(x) = f_μ,Λ(x) = 𝒩(x|μ,Λ⁻¹), and for the base measure H on θ = (μ, Λ), we take μ ~ 𝒩 (μ̂, Ĉ), Λ ~ Wishart_d(V, ν) independently, where μ̂ is the sample mean, Ĉ is the sample covariance, ν = d = 2, and V = Ĉ ⁻¹/ν. Here, ${\text{Wishart}}_d(\mathrm{\Lambda }\mid V,\nu )\propto {\mid det\mathrm{\Lambda }\mid }^{(\nu -d-1)/2}exp\left(-\frac{1}{2}\text{tr}(V^{-1}\mathrm{\Lambda })\right)$.

For the MFM, we take K ~ Geometric(0.1) and γ = 1. For the DPM, we put an Exponential(1) prior on the concentration parameter, α. This makes the priors on the number of clusters t roughly comparable for the range of n values considered below; see Figure 7.

Figure 7.

Prior and posterior on the number of clusters t for the MFM (left) and DPM (right), and the posterior on the number of components k for the MFM (bottom), on increasing amounts of data from the bivariate example. Each prior on t is approximated from 10⁶ samples from the prior.

For both the MFM and DPM, we use the split-merge sampler of Jain and Neal (2007) for non-conjugate priors, coupled with Algorithm 8 of Neal (2000), as in Section 7.1.1. The same inference algorithm as in Section 7.1.1 is used, with the sole exception that in this case there is no hyperprior on the parameters of H. Gibbs updates to the DPM concentration parameter α are made using Metropolis–Hastings moves.

Five independent datasets were used for each n ∈ {50, 100, 250, 1000, 2500}, and for each model (MFM and DPM), the sampler was run for 5,000 burn-in iterations and 95,000 sample iterations (for a total of 100,000). Judging by traceplots and running averages of various statistics, this appeared to be sufficient for mixing. The cluster sizes were recorded after each iteration, and to reduce memory storage requirements, the full state of the chain was recorded only once every 100 iterations. For each run, the seed of the random number generator was initialized to the same value for both the MFM and DPM. The sampler used for these experiments took approximately 8 × 10⁻⁶ n seconds per iteration.

7.3.2 Density estimation

For certain nonparametric density estimation problems, both the DPM and MFM have been shown to exhibit posterior consistency at the minimax optimal rate, up to logarithmic factors (Kruijer et al., 2010; Ghosal and Van der Vaart, 2007). In fact, even for small sample sizes, we observe empirically that density estimates under the two models are remarkably similar; for example, see Figure 4. (See the supplementary material for details on computing the density estimates.) As the amount of data increases, the density estimates for both models appear to be converging to the true density, as expected. Indeed, Figure 5 indicates that the Hellinger distance between the estimated density and the true density is going to zero, and further, that the rate of convergence appears to be nearly identical for the DPM and the MFM. This suggests that for density estimation, the behavior of the two models seems to be essentially the same.

Figure 5.

Left: Hellinger distance between the estimated and true densities, for MFM (red, left) and DPM (blue, right) density estimates, on the bivariate example. Right: Bayes factors in favor of the MFM over the DPM. For each n ∈ {50, 100, 250, 1000, 2500}, five independent datasets of size n were used, and the lines connect the averages for each n.

On the other hand, Figure 5 shows that the Bayes factors p(x_1:n|MFM)/p(x_1:n|DPM) are increasing as n grows, indicating that the MFM is a better fit to this data than the DPM, in the sense that it has a higher marginal likelihood. This makes sense, since the MFM is correctly specified for data from a finite mixture, while the DPM is not. See Section A.2 for details on computing these Bayes factors.

7.3.3 Clustering

Frequently, mixture models are used for clustering and latent class discovery, rather than for density estimation. MFMs have a partition distribution that takes a very similar form to that of the Dirichlet process, as discussed in Section 3. Despite this similarity, the MFM partition distribution differs in two fundamental respects.

1. The prior on the number of clusters t is very different. In an MFM, one has complete control over the prior on the number of components k, which in turn provides control over the prior on t. Further, as the sample size n grows, the MFM prior on t converges to the prior on k. In contrast, in a Dirichlet process, the prior on t takes a particular parametric form and diverges at a log n rate.

2. Given t, the prior on the cluster sizes is very different. In an MFM, most of the prior mass is on partitions in which the sizes of the clusters are all the same order of magnitude, while in a Dirichlet process, most of the prior mass is on partitions in which the sizes vary widely, with a few large clusters and many very small clusters (see Section 5.3).

These prior differences carry over to noticeably different posterior clustering behavior. Figure 6 displays a few representative clusterings sampled from the posterior for the three-component bivariate Gaussian example. When n = 250, around 92% of the MFM samples look like (a) and (b), while around 2/3 of the DPM samples look like (d). Since the MFM is consistent for k, the proportion of MFM samples with three clusters tends to 100% as n increases, however, we do not expect this to occur for the DPM (see Section 7.3.4); indeed, when n = 2500, it is 98% for the MFM and still around 2/3 for the DPM. Many of the DPM samples have tiny “extra” clusters consisting of only a few points, as seen in Figure 6(e) and (f). Meanwhile, when the MFM samples have extra clusters, they tend to be large, as in (c). All of this is to be expected, due to items (1) and (2) above, along with the fact that these data are drawn from a finite mixture over the assumed family, and thus the MFM is correctly specified, while the DPM is not.

Figure 6.

Representative sample clusterings from the posterior for the MFM and DPM, on n = 250 data points from the bivariate example. The bottom plot shows the true component assignments. See discussion in the text. (Best viewed in color.)

7.3.4 Number of components and the mixing distribution

Assuming that the data are from a finite mixture, it is also sometimes of interest to infer the number of components or the mixing distribution, subject to the caveat that these inferences are meaningful only to the extent that the component distributions are correctly specified and the model is finite-mixture identifiable. (In the notation of Section 4.2, the mixing distribution is G, and we say the model is finite-mixture identifiable if for any discrete measures G, G′ supported on finitely-many points, f_G = f_G′ a.e. implies G = G′.) While Nguyen (2013) has shown that under certain conditions, DPMs are consistent for the mixing distribution (in the Wasserstein metric), Miller and Harrison (2013, 2014) have shown that the DPM posterior on the number of clusters is typically not consistent for the number of components, at least when the concentration parameter is fixed; the question remains open when using a prior on the concentration parameter, but we conjecture that it is still not consistent. On the other hand, MFMs are consistent for the mixing distribution and the number of components (for Lebesgue almost-all values of the true parameters) under very general conditions (Nobile, 1994); this is a straightforward consequence of Doob’s theorem. The relative ease with which this consistency can be established for MFMs is due to the fact that in an MFM, the parameter space is a countable union of finite-dimensional spaces, rather than an infinite-dimensional space.

These consistency/inconsistency properties are readily observed empirically—they are not simply large-sample phenomena. As seen in Figure 7, the tendency of DPM samples to have tiny extra clusters causes the number of clusters t to be somewhat inflated, apparently making the DPM posterior on t fail to concentrate, while the MFM posterior on t concentrates at the true value (by Section 5.2 and Nobile, 1994). In addition to the number of clusters t, the MFM also permits inference for the number of components k in a natural way (Figure 7), while in the DPM the number of components is always infinite. One might wonder whether the DPM would perform better if the data were drawn from a mixture with an additional one or two components with much smaller weight, however, this doesn’t seem to make a difference; see the supplementary material. On the other hand, on data from an infinite mixture, one would expect the DPM to perform better than the MFM. See the supplementary material for the formulas used to compute the posterior on k.

Although the MFM enables consistent inference for k in principle (Nobile, 1994), there are several issues that need to be carefully considered in practice; see Section A.3.

A Appendix

A.1 Examples where MFMs may be more suitable than DPMs

In many real-world situations, it is clear a priori that there is a finite (but unknown) number of components, due to the physics of the data generating process. In neural spike sorting, for instance, the observed spikes come from a mixture in which each spike is generated by one of many neurons/units, and “regardless of the number of putative spikes detected, the number of different single units one could conceivably discriminate from a single electrode is upper bounded due to the conductive properties of the tissue” (Carlson et al., 2014). In genetics, “a straightforward statistical genetics argument shows that the problem of haplotype inference can be formulated as a mixture model, where the set of mixture components corresponds to the pool of ancestor haplotypes, or founders, of the population… however, the size of this pool is unknown; indeed, knowing the size of the pool would correspond to knowing something significant about the genome and its history” (Xing et al., 2006). In cancer tumor phylogenetics, each datapoint corresponds to a locus in the genome, and the data come from a mixture of tumor cell populations within a single tissue sample. The number of cell populations (k) should not increase indefinitely as the number of loci assayed (n) grows (Landau et al., 2013).

Furthermore, the DPM is often mistakenly described in the literature as a mixture with a prior on the number of components, i.e., it is mistakenly described as an MFM. For instance, da Silva (2007) writes, “the DP mixture model is a Bayesian nonparametric methodology that relies on Markov chain Monte Carlo simulations for exploring mixture models with an unknown number of components.” Medvedovic and Sivaganesan (2002) write, “this [DPM] model does not require specifying the number of mixture components, and the obtained clustering is a result of averaging over all possible numbers of mixture components.” Xing et al. (2007) write, “methods for fitting the genotype mixture must crucially address the problem of estimating a mixture with an unknown number of mixture components… we present a Bayesian approach to this problem based on a nonparametric prior known as the Dirichlet process.” We suspect that in many such cases an MFM is a more natural model.

There are a variety of examples where researchers initially used an MFM, and later began using a DPM only because it was more computationally convenient. For instance, Tadesse et al. (2005) used MFMs with RJMCMC for a novel Bayesian variable selection method, but in follow-on work (Kim et al., 2006) they used DPMs because “with the Dirichlet process mixture models, the creation and deletion of clusters is naturally taken care of in the process of updating the sample allocations.” Meanwhile, in some cases, researchers have attempted to force DPMs to behave more like MFMs by choosing the DPM concentration parameter as a function of n to prevent the number of clusters from growing with n (Landau et al., 2013). With our new MFM algorithms, researchers need not default to DPMs for computational considerations; they can use the model that is best suited to the application at hand.

A.2 Computing Bayes factors for MFM versus DPM

In principle, the Bayes factor for the MFM versus the DPM could be used as an empirical criterion for choosing between the two models, and in fact, it is quite easy to compute an approximation to the Bayes factor using importance sampling. However, at this time we are not comfortable with using this as a method for choosing between MFM and DPM because when n is small, this Bayes factor can be sensitive to the choice of hyperparameters, and when n is large, the importance sampling approximation might have high variance since the two partition distributions could be concentrating in different regions of the space. Nonetheless, we include the details of the calculation here, since it may be of interest.

Given a pair of partition distributions f(C) = f̃(C)/Z_f and g(C) = g̃(C)/Z_g with common support, importance sampling can be used to approximate the ratio of normalization constants: $Z_f/Z_g\approx \frac{1}{N}{\sum }_{i=1}^N\stackrel{\sim }{f}({\mathcal{C}}_i)/\stackrel{\sim }{g}({\mathcal{C}}_i)$ where C₁,…, C_N ~ g. In particular, if f̃(C) = p(x_1:n|C)p_MFM(C) and g̃(C) = p(x_1:n|C)p_DP(C), then f is the MFM posterior and g is the DPM posterior, and we can use samples C₁, …, C_N from g to approximate the Bayes factor in favor of the MFM over the DPM:

$$\text{BF}=\frac{p(x_{1:n}\mid \text{MFM})}{p(x_{1:n}\mid \text{DPM})}=\frac{Z_f}{Z_g}\approx \frac{1}{N}\sum _{i=1}^N\frac{\stackrel{\sim }{f}({\mathcal{C}}_i)}{\stackrel{\sim }{g}({\mathcal{C}}_i)}=\frac{1}{N}\sum _{i=1}^N\frac{p_{\text{MFM}}({\mathcal{C}}_i)}{p_{\text{DP}}({\mathcal{C}}_i)},$$

where we have used the fact that f̃(C)/g̃(C) = p_MFM(C)/p_DP(C) since the likelihood p(x_1:n|C) is the same in the MFM and the DPM. This approximation to the Bayes factor is easy to compute using the output of an MCMC sampler for the DPM, since all we need to do is compute the ratio of the prior probabilities p_MFM(C_i)/p_DP(C_i) for each partition C_i sampled from the DPM posterior. Alternatively, one could use the output of an MCMC sampler for the MFM, by swapping the roles of the DPM and the MFM throughout.

A.3 Issues with estimating the number of components

Due to the fact that inference for the number of components is a topic of high interest in many communities, it seems prudent to make some cautionary remarks in this regard. Many approaches have been proposed for estimating the number of components (Henna, 1985; Keribin, 2000; Leroux, 1992; Ishwaran et al., 2001; James et al., 2001; Henna, 2005; Woo and Sriram, 2006, 2007). In theory, the MFM model provides a Bayesian approach to consistently estimating the number of components, making it a potentially attractive method of assessing the heterogeneity of the data. However, there are several possible pitfalls to consider, some of which are more obvious than others.

An obvious potential issue is that in many applications, the clusters which one wishes to distinguish are purely notional (for example, perhaps, clusters of images or documents), and a mixture model is used for practical purposes, rather than because the data are actually thought to arise from a mixture. Clearly, in such cases, inference for the “true” number of components is meaningless. On the other hand, in some applications, the data definitely come from a mixture (for example, extracellular recordings of multiple neurons), so there is in reality a true number of components, although usually the form of the mixture components is far from clear.

More subtle issues are that the posteriors on k and t can be (1) strongly affected by the base measure H, and (2) sensitive to misspecification of the family of component distributions {f_θ}. Issue (1) can be seen, for instance, in the case of normal mixtures: it might seem desirable to choose the prior on the component means to have large variance in order to be less informative, however, this causes the posteriors on k and t to favor smaller values (Richardson and Green, 1997; Stephens, 2000; Jasra et al., 2005). The basic mechanism at play here is the same as in the Bartlett–Lindley paradox, and shows up in many Bayesian model selection problems. With some care, this issue can be dealt with by varying the base measure H and observing the effect on the posterior—that is, by performing a sensitivity analysis—for instance, see Richardson and Green (1997).

Issue (2) is more serious. In practice, we typically cannot expect our choice of {f_θ: θ ∈ Θ} to contain the true component densities (assuming the data are even from a mixture). When the model is misspecified in this way, the posteriors of k and t can be severely affected and depend strongly on n. For instance, if the model uses Gaussian components, and the true data distribution is not a finite mixture of Gaussians, then the posteriors of k and t can be expected to diverge to infinity as n increases. Consequently, the effects of misspecification need to be carefully considered if these posteriors are to be used as measures of heterogeneity. Steps toward addressing the issue of robustness have been taken by Woo and Sriram (2006, 2007) and Rodríguez and Walker (2014), however, this is an important problem demanding further study.

Despite these issues, sample clusterings and estimates of the number of components or clusters can provide a useful tool for exploring complex datasets, particularly in the case of high-dimensional data that cannot easily be visualized. It should always be borne in mind, though, that the results can only be interpreted as being correct to the extent that the model assumptions are correct.