Defining Predictive Probability Functions for Species Sampling Models

Abstract

We review the class of species sampling models (SSM). In particular, we investigate the relation between the exchangeable partition probability function (EPPF) and the predictive probability function (PPF). It is straightforward to define a PPF from an EPPF, but the converse is not necessarily true. In this paper we introduce the notion of putative PPFs and show novel conditions for a putative PPF to define an EPPF. We show that all possible PPFs in a certain class have to define (unnormalized) probabilities for cluster membership that are linear in cluster size. We give a new necessary and sufficient condition for arbitrary putative PPFs to define an EPPF. Finally, we show posterior inference for a large class of SSMs with a PPF that is not linear in cluster size and discuss a numerical method to derive its PPF.

1. INTRODUCTION

The status of the Dirichlet process (Ferguson, 1973) (DP) among nonparametric priors is comparable to that of the normal distribution among finite-dimensional distributions. This is in part due to the marginalization property: a random sequence sampled from a random probability measure with a Dirichlet process prior forms marginally a Polya urn sequence (Blackwell and MacQueen, 1973). Markov chain Monte Carlo simulation based on the marginalization property has been the central computational tool for the DP and facilitated a wide variety of applications. See MacEachern (1994), Escobar and West (1995) and MacEachern and Müller (1998), to name just a few. In Pitman (1995, 1996), the species sampling model (SSM) is proposed as a generalization of the DP. SSMs can be used as flexible alternatives to the popular DP model in nonparametric Bayesian inference. The SSM is defined as the directing random probability measure of an exchangeable species sampling sequence which is defined as a generalization of the Polya urn sequence. The SSM has a marginalization property similar to the DP. It therefore enjoys the same computational advantage as the DP while it defines a much wider class of random probability measures. For its theoretical properties and applications, we refer to Ishwaran and James (2003), Lijoi, Mena and Prünster (2005), Lijoi, Prünster and Walker (2005), James (2008), Navarrete, Quintana and Müller (2008), James, Lijoi and Prünster (2009) and Jang, Lee and Lee (2010).

Suppose (X₁, X₂, …) is a sequence of random variables. In a traditional application the sequence arises as a random sample from a large population of units, and X_i records the species of the ith individual in the sample. This explains the name SSM. Let X̃_j be the j th distinct species to appear. Let n_jn be the number of times the j th species X̃_j appears in (X₁, …, X_n), j = 1, 2, …, and

$${\mathbf{n}}_n=(n_{jn},j=1,\dots ,k_n),$$

where k_n = k_n (n_n) = max{j: n_jn > 0} is the number of different species to appear in (X₁, …, X_n). The sets {i ≤ n: X_i = X̃_j} define clusters that partition the index set {1, …, n}. When n is understood from the context we just write n_j, n and k or k (n).

We now give three alternative characterizations of species sampling sequences: (i) by the predictive probability function, (ii) by the driving measure of the exchangeable sequence, and (iii) by the underlying exchangeable partition probability function.

PPF

Let ν be a diffuse (or nonatomic) probability measure on a complete separable metric space 𝒳 equipped with Borel σ -field. An exchangeable sequence (X₁, X₂, …) is called a species sampling sequence (SSS) if X₁ ~ ν and

$$X_{n+1}|X_1,\dots ,X_n~\sum _{j=1}^{k_n}{p}_j({\mathbf{n}}_n){\delta }_{{\mathrm{X̃}}_j}+p_{k_n+1}({\mathbf{n}}_n)\nu ,$$ (1)

where δ_x is the degenerate probability measure at x. Examples of SSS include the Pólya urn sequence (X₁, X₂, …) whose distribution is the same as the marginal distribution of independent observations from a Dirichlet random distribution F, that is, $X_1,X_2,\dots |F\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{~}F$ with F ~ DP(αν), where α > 0. The conditional distribution of the Pólya urn sequence is

$$X_{n+1}|X_1,\dots ,X_n~\sum _{j=1}^{k_n}\frac{n_j}{n+\alpha }{\delta }_{{\mathrm{X̃}}_j}+\frac{\alpha }{n+\alpha }\nu .$$

This marginalization property has been a central tool for posterior simulation in DP mixture models, which benefit from the fact that one can integrate out F using the marginalization property. The posterior distribution becomes then free of the infinite-dimensional object F. Thus, Markov chain Monte Carlo algorithms for DP mixtures do not pose bigger difficulties than the usual parametric Bayesian models (MacEachern, 1994; MacEachern and Müller, 1998). Similarly, alternative discrete random distributions have been considered in the literature and proved computationally attractive due to analogous marginalization properties; see, for example, Lijoi, Mena and Prünster (2005, 2007).

The sequence of functions (p₁, p₂, …) in (1) is called a sequence of predictive probability functions (PPF). These are defined on ${\mathbb{N}}^{*}={\cup }_{k=1}^{\infty }{\mathbb{N}}^k$, where ℕ is the set of natural numbers, and satisfy the conditions

$$p_j(\mathbf{n})\ge 0\text{and}\sum _{j=1}^{k_n+1}{p}_j(\mathbf{n})=1\text{for all}\mathbf{n}\in {\mathbb{N}}^{*}.$$ (2)

Motivated by these properties of PPFs, we define a sequence of putative PPFs as a sequence of functions (p_j, j = 1, 2, …) defined on ℕ* which satisfies (2). Note that not all putative PPFs are PPFs, because (2) does not guarantee exchangeability of (X₁, X₂, …) in (1). Note that the weights p_j (·) depend on the data only indirectly through the cluster sizes n_n. The widely used DP is a special case of a species sampling model, with p_j (n_n) ∝ n_j and p_k+1 (n_n) ∝ α for a DP with total mass parameter α. The use of p_j in (1) implies

$$p_j(\mathbf{n})=\mathbb{P}(X_{n+1}={\mathrm{X̃}}_j|X_1,\dots ,X_n),j=1,\dots ,k_n,p_{k_n+1}(\mathbf{n})=\mathbb{P}(X_{n+1}\notin \{{\mathrm{X̃}}_1,\dots ,{\mathrm{X̃}}_{k_n}\}|X_1,\dots ,X_n).$$

In words, p_j is the probability of the next observation being the j th species (falling into the j th cluster) and p_{kn + 1} is the probability of a new species (starting a new cluster).

An important point in the above definition is that a sequence X_i can be a SSS only if it is exchangeable.

SSM

Alternatively, a SSS can be characterized by the following defining property. An exchangeable sequence of random variables (X₁, X₂, …) is a species sampling sequence if and only if X₁, X₂, … | G is a random sample from G where

$$G=\sum _{h=1}^{\infty }{P}_h{\delta }_{m_h}+R\nu$$ (3)

for some sequence of positive random variables (P_h) and R such that $1-R={\sum }_{h=1}^{\infty }{P}_h\le 1$ with probability 1, (m_h) is a sequence of independent variables with distribution ν, and (P_i) and (m_h) are independent. See Pitman (1996). The result is an extension of de Finetti’s theorem and characterizes the directing random probability measure of the species sample sequence. We call the directing random probability measure G in equation (3) the SSM of the SSS (X_i).

EPPF

A third alternative definition of a SSS and corresponding SSM is in terms of the implied probability model on a sequence of random partitions.

Suppose a SSS (X₁, X₂, …) is given. Since the de Finetti measure (3) is partly discrete, there are ties among X_i ’s. The ties among (X₁, X₂, …, X_n) for a given n induce an equivalence relation in the set [n] = {1, 2, …, n}, that is, i ~ j if and only if X_i = X_j. This equivalence relation on [n], in turn, induces the partition ∏_n of [n]. Due to the exchangeability of (X₁, X₂, …), it can be easily seen that the random partition ∏_n is an exchangeable random partition on [n], that is, for any partition {A₁, A₂, …, A_k} of [n], the probability P (∏_n = {A₁, A₂, …, A_k}) is invariant under any permutation on [n] and can be expressed as a function of n = (n₁, n₂, …, n_k), where n_i is the cardinality of A_i for i = 1, 2, …, k. Extending the above argument to the entire SSS, we can get an exchangeable random partition on the natural numbers ℕ from the SSS. Kingman (1978, 1982) showed a remarkable result, called Kingman’s representation theorem, that in fact every exchangeable random partition can be obtained by a SSS.

For any partition {A₁, A₂, …, A_k} of [n], we can represent P (∏_n = {A₁, A₂, …, A_k}) = p (n) for a symmetric function p: ℕ* → [0, 1] satisfying

$$p(1)=1,p(\mathbf{n})=\sum _{j=1}^{k(\mathbf{n})+1}p({\mathbf{n}}^{j+})\text{for all}\mathbf{n}\in {\mathbb{N}}^{*},$$ (4)

where n^j+ is the same as n except that the j th element is increased by 1. This function is called an exchangeable partition probability function (EPPF) and characterizes the distribution of an exchangeable random partition on ℕ.

We are now ready to pose the problem for the present paper. It is straightforward to verify that any EPPF defines a PPF by

$$p_j(\mathbf{n})=\frac{p({\mathbf{n}}^{j+})}{p(\mathbf{n})},j=1,2,\dots ,k+1.$$ (5)

The converse is not true. Not every putative p_j (n) defines an EPPF and thus a SSM and a SSS. For example, it is easy to show that $p_j(\mathbf{n})\propto n_j^2+1$, j = 1, …, k (n), does not. In Bayesian data analysis it is often convenient, or at least instructive, to elicit features of the PPF rather than the joint EPPF. Since the PPF is crucial for posterior computation, applied Bayesians tend to focus on it to specify the species sampling prior for a specific problem. For example, the PPF defined by a DP prior implies that the probability of joining an existing cluster is proportional to the cluster size. This is not always desirable. Can the user define an alternative PPF that allocates new observations to clusters with probabilities proportional to alternative functions f (n_j) and still define a SSS? In general, the simple answer is no. We already mentioned that a PPF implies a SSS if and only if it arises as in (5) from an EPPF. But this result is only a characterization. It is of little use for data analysis and modeling since it is difficult to verify whether or not a given PPF arises from an EPPF. In this paper we develop some conditions to address this gap. We consider methods to define PPFs in two different directions. First we give an easily verifiable necessary condition for a putative PPF to arise from an EPPF (Lemma 1) and a necessary and sufficient condition for a putative PPF to arise from an EPPF. A consequence of this result is an elementary proof of the characterization of all possible PPFs with form p_j (n) ∝ f (n_j). This result has been proved earlier by Gnedin and Pitman (2006). Although the result in Section 2 gives necessary and sufficient conditions for a putative PPF to be a PPF, the characterization is not constructive. It does not give any guidance in how to create a new PPF for a specific application. In Section 3 we propose an alternative approach to define a SSM based on directly defining a joint probability model for the P_h in (3). We develop a numerical algorithm to derive the corresponding PPF. This facilitates the use of such models for nonparametric Bayesian data analysis. This approach can naturally create PPFs with very different features than the well-known PPF under the DP.

The literature reports some PPFs with closed-form analytic expressions other than the PPF under the DP prior. There are a few directions which have been explored for constructing extensions of the DP prior and deriving PPFs. The normalization of complete random measures (CRM) has been proposed in Kingman (1975). A CRM such as the generalized gamma process (Brix, 1999), after normalization, defines a discrete random distribution and, under mild assumptions, a SSM. Developments and theoretical results on this approach have been discussed in a series of papers; see, for example, Perman, Pitman and Yor (1992), Pitman (2003) and Regazzini, Lijoi and Prünster (2003). Normalized CRM models have also been studied and applied in Lijoi, Mena and Prünster (2005), Nieto-Barajas, Prünster and Walker (2004) and more recently in James, Lijoi and Prünster (2009). A second related line of research considered the so-called Gibbs models. In these models the analytic expressions of the PPFs share similarities with the DP model. An important example is the Pitman–Yor process. Contributions include Gnedin and Pitman (2005), Lijoi, Mena and Prünster (2007), Lijoi, Prünster and Walker (2008a, 2008b) and Gnedin, Haulk and Pitman (2010). Lijoi and Prünster (2010) provide a recent overview on major results from the literature on normalized CRM and Gibbs-type partitions.

2. WHEN DOES A PPF IMPLY AN EPPF?

Suppose we are given a putative PPF (p_j). Using equation (5), one can attempt to define a function p: ℕ* → [0, 1] inductively by the following mapping:

$$p(1)=1,p({\mathbf{n}}^{j+})=p_j(\mathbf{n})p(\mathbf{n})\text{for all}\mathbf{n}\in \mathbb{N}\text{and}j=1,2,\dots ,k(\mathbf{n})+1.$$ (6)

In general, equation (6) does not lead to a unique definition of p (n) for each n ∈ ℕ*. For example, let n = (2, 1). Then, p (2, 1) could be computed in two different ways as p₂ (1) p₁ (1, 1) and p₁ (1) p₂ (2) which correspond to partitions {{1, 3}, {2}} and {{1, 2}, {3}}, respectively. If p₂ (1) p₁ (1, 1) ≠ p₁ (1) p₂ (2), equation (6) does not define a function p: ℕ* → [0, 1]. The following lemma shows a condition for a PPF for which equation (6) leads to a valid unique definition of p: ℕ* → [0, 1].

Suppose ∏ = {A₁, A₂, …, A_k} is a partition of [n] with clusters indexed in the order of appearance. For 1 ≤ m ≤ n, let ∏_m be the restriction of ∏ on [m]. Let n (∏) = (n₁, …, n_k), where n_i is the cardinality of A_i, and let ∏ (i) be the class index of element i in partition ∏ and ∏ ([n]) = (∏ (1), …, ∏ (n)).

Lemma 1. If and only if a putative PPF (p_j) satisfies

$$p_i(\mathbf{n})p_j({\mathbf{n}}^{i+})=p_j(\mathbf{n})p_i({\mathbf{n}}^{j+})\mathit{\text{for all}}\mathbf{n}\in {\mathbb{N}}^{*},i,j=1,2,\dots ,k(\mathbf{n})+1,$$ (7)

then p defined by (6) is a function from ℕ* to [0, 1], that is, p in (6) is uniquely defined.

Proof. Let n = (n₁, …, n_k) ${\sum }_{i=1}^k{n}_i=n$ and ∏ and Ω be two partitions of [n] with n (∏) = n (Ω) = n. Let $p^{\mathrm{\Pi }}(\mathbf{n})={\mathrm{\Pi }}_{i=1}^{n-1}{p}_{\mathrm{\Pi }(i+1)}(\mathbf{n}({\mathrm{\Pi }}_i))$ and $p^{\mathrm{\Omega }}(\mathbf{n})={\mathrm{\Pi }}_{i=1}^{n-1}{p}_{\mathrm{\Omega }(i+1)}(\mathbf{n}({\mathrm{\Omega }}_i))$. We need to show that p^∏ (n) = p^Ω (n).Without loss of generality, we can assume ∏ ([n]) = (1, …, 1, 2, …, 2, …, k, …, k), where i is repeated n_i times for i = 1, …, k. Note that Ω ([n]) is just a certain permutation of ∏ ([n]) and by a finite times of swapping two consecutive elements in Ω ([n]), one can change Ω ([n]) to ∏ ([n]). Thus, it suffices to show when Ω ([n]) is different from ∏ ([n]) in only two consecutive positions. But, this is guaranteed by condition (7).

The opposite is easy to show. Assume p_j defines a unique p (n). Consider (7) and multiply on both sides with p(n). By assumption, we get on either side p (n^{i + j +}). This completes the proof.

Note that the conclusion of Lemma 1 is not (yet) that p is an EPPF. The missing property is exchangeability, that is, invariance of p with respect to permutations of the group indices j = 1, …, k (n). When the function p, recursively defined by expression (6), satisfies the balance imposed by equation (7) it is called the partially exchangeable probability function (Pitman, 1995, 2006) and the resulting random partition of ℕ is termed partially exchangeable. In Pitman (1995), it is proved that a p: ℕ* → [0, 1] is a partially exchangeable probability function if and only if it exists as a sequence of nonnegative random variables P_i, i = 1,…, with ∑_i P_i ≤ 1 such that

$$p(n_1,\dots ,n_k)=E\left[\prod _{i=1}^k{P}_i^{n_i-1}\prod _{i=1}^{k-1}\right(1-\sum _{j=1}^i{P}_i\left)\right],$$ (8)

where the expectation is with respect to the distribution of the sequence (P_i). We refer to Pitman (1995) for an extensive study of partially exchangeable random partitions.

It is easily checked whether or not a given PPF satisfies the condition of Lemma 1. Corollary 1 describes all possible PPFs that have the probability of cluster memberships depend on a function of the cluster size only. This result is part of a theorem in Gnedin and Pitman (2006), but we give here a more straightforward proof.

Corollary 1. Suppose a putative PPF (p_j) satisfies (7) and

$$p_j(n_1,\dots ,n_k)\propto \{\begin{array}{cc}f(n_j),\hfill & j=1,\dots ,k,\hfill \\ \theta ,\hfill & j=k+1,\hfill \end{array}$$ (9)

where f (k) is a function from ℕ to (0, ∞) and θ > 0. Then, f (k) = ak for all k ∈ ℕ for some a > 0.

Proof. Note that for any n = (n₁, …, n_k) and i = 1, …, k + 1,

$$p_i(n_1,\dots ,n_k)=\{\begin{array}{cc}\hfill \frac{f(n_i)}{{\sum }_{u=1}^{k}f(n_u)+\theta },\hfill & \hfill i=1,\dots ,k,\hfill \\ \hfill \frac{\theta }{{\sum }_{u=1}^{k}f(n_u)+\theta },\hfill & \hfill i=k+1.\hfill \end{array}$$

Equation (7) with 1 ≤ i ≠ j ≤ k implies

$$\frac{f(n_i)}{{\sum }_{u=1}^{k}f(n_u)+\theta }\frac{f(n_j)}{{\sum }_{u\ne i}^{k}f(n_u)+f(n_i+1)+\theta }=\frac{f(n_j)}{{\sum }_{u=1}^{k}f(n_u)+\theta }\frac{f(n_i)}{{\sum }_{u\ne j}^{k}f(n_u)+f(n_j+1)+\theta },$$

which in turn implies

$$f(n_i)+f(n_j+1)=f(n_j)+f(n_i+1)$$

or

$$f(n_j+1)-f(n_j)=f(n_i+1)-f(n_i).$$

Since this holds for all n_i and n_j, we have for all k ∈ ℕ

$$f(m)=am+b$$ (10)

for some a, b ∈ ℝ.

Now consider i = k + 1 and 1 ≤ j ≤ k. Then,

$$\frac{\theta }{{\sum }_{u=1}^{k}f(n_u)+\theta }\frac{f(n_j)}{{\sum }_{u=1}^{k}f(n_u)+f(1)+\theta }=\frac{f(n_j)}{{\sum }_{u=1}^{k}f(n_u)+\theta }\frac{\theta }{{\sum }_{u\ne j}^{k}f(n_u)+f(n_j+1)+\theta },$$

which implies f (n_j) + f (1) = f (n_j + 1) for all n_j. This together with (10) implies b = 0. Thus, we have f (k) = ak for some a > 0.

For any a > 0, the putative PPF

$$p_i(n_1,\dots ,n_k)\propto \{\begin{array}{cc}{an}_i,\hfill & i=1,\dots ,k,\hfill \\ \theta ,\hfill & i=k+1,\hfill \end{array}$$

defines a function p: ℕ → [0, 1],

$$p(n_1,\dots ,n_k)=\frac{{\theta }^{k-1}{a}^{n-k}}{{[\theta +1]}_{n-1;a}}\prod _{i=1}^k(n_i-1)!,$$

where [θ]_k;a = θ (θ + a) … (θ + (k − 1)a). Since this function is symmetric in its arguments, it is an EPPF. This is the EPPF for a DP with total mass θ/a. Thus, Corollary 1 implies that the EPPF under the DP is the only EPPF that satisfies (9). The corollary shows that it is not an entirely trivial matter to come up with a putative PPF that leads to a valid EPPF. A version of Corollary 1 is also well known as Johnson’s Sufficientness postulate (Good, 1965). See also the discussion in Zabell (1982).

We now give a necessary and sufficient condition for the function p defined by (5) to be an EPPF, without any constraint on the form of p_j (as were present in the earlier results). Suppose σ is a permutation of [k] and n = (n₁, …, n_k) ∈ ℕ*. Define σ (n) = σ (n₁, …, n_k) = (n_{σ (1)}, n_{σ (2)}, …, n_{σ (k)}). In words, σ is a permutation of group labels and σ (n) is the corresponding permutation of the group sizes n.

Theorem 1. Suppose a putative PPF (p_j) satisfies (7) as well as the following condition: for all n = (n₁, …, n_k) ∈ ℕ*, and permutations σ on [k] and i = 1, …, k,

$$p_i(n_1,\dots ,n_k)=p_{{\sigma }^{-1}(i)}(n_{\sigma (1)},n_{\sigma (2)},\dots ,n_{\sigma (k)}).$$ (11)

Then, p defined by (6) is an EPPF. The condition is also necessary; if p is an EPPF, then (11) holds.

Proof. Fix n = (n₁, …, n_k) ∈ ℕ* and a permutation on [k], σ. We wish to show that for the function p defined by (6)

$$p(n_1,\dots ,n_k)=p(n_{\sigma (1)},n_{\sigma (2)},\dots ,n_{\sigma (k)}).$$ (12)

Let ∏ be the partition of [n] with n (∏) = (n₁, …, n_k) such that

$$\prod ([n])=(1,2,\dots ,k,1,\dots ,1,2,\dots ,2,\dots ,k,\dots ,k),$$

where after the first k elements 1, 2, …, k, i is repeated n_i − 1 times for all i = 1, …, k. Then,

$$p(\mathbf{n})=\prod _{i=2}^k{p}_i({\mathbf{1}}_{(i-1)})\times \prod _{i=k}^{n-1}{p}_{\Pi (i+1)}(\mathbf{n}({\Pi }_i)),$$

where 1_(j) is the vector of length j whose elements are all 1’s.

Now consider a partition Ω of [n] with n(Ω) = (n_{σ (1)}, n_{σ (2)}, …, n_{σ (k)}) such that

$$\mathrm{\Omega }([n])=(1,2,\dots ,k,{\sigma }^{-1}(1),\dots ,{\sigma }^{-1}(1),{\sigma }^{-1}(2),\dots ,{\sigma }^{-1}(2),\dots ,{\sigma }^{-1}(k),\dots ,{\sigma }^{-1}(k)),$$

where after the first k elements 1, 2, …, k, σ⁻¹ (i) is repeated n_i − 1 times for all i = 1, …, k. Then,

$$p(n_{\sigma (1)},n_{\sigma (2)},\dots ,n_{\sigma (k)})=\prod _{i=2}^k{p}_i({\mathbf{1}}_{(i-1)})\times \prod _{i=k}^{n-1}{p}_{\mathrm{\Omega }(i+1)}(\mathbf{n}({\mathrm{\Omega }}_i))=\prod _{i=2}^k{p}_i({\mathbf{1}}_{(i-1)})\times \prod _{i=k}^{n-1}{p}_{{\sigma }^{-1}(\mathrm{\Omega }(i+1))}(\sigma (\mathbf{n}({\mathrm{\Omega }}_i)))=\prod _{i=2}^k{p}_i({\mathbf{1}}_{(i-1)})\times \prod _{i=k}^{n-1}{p}_{\Pi (i+1)}(\mathbf{n}({\Pi }_i))=p(n_1,\dots ,n_k),$$

where the second equality follows from (11). This completes the proof of the sufficient direction.

Finally, we show that every EPPF p satisfies (6) and (11). By Lemma 1, every EPPF satisfies (6). Condition (12) is true by the definition of an EPPF, which includes the condition of symmetry in its arguments. And (12) implies (11).

Fortini, Ladelli and Regazzini (2000) prove results related to Theorem 1. They provide sufficient conditions for a system of predictive distributions p (X_n | X₁, …, X_n−1), n = 1,…, of a sequence of random variables (X_i) that imply exchangeability. The relation between these conditions and Theorem 1 becomes apparent by constructing a sequence (X_i) that induces a p-distributed random partition of ℕ. Here, it is implicitly assumed the mapping of (X_i) to the only partition such that i, j ∈ ℕ belongs to the same subset if and only if X_i = X_j.

A second more general example, which extends the predictive structure considered in Corollary 1, includes the so-called Gibbs random partitions. Within this class of models

$$p(n_1,n_2,\dots ,n_k)=V_{n,k}\prod _{i=1}^k{W}_{n_i},$$ (13)

where (V_n,k) and (W_ni) are sequences of positive real numbers. In this case the predictive probability of a novel species is a function of the sample size n and of the number of observed species k. See Lijoi, Mena and Prünster (2007) for related distributional results on Gibbs type models. Gnedin and Pitman (2006) obtained sufficient conditions for the sequences (V_n,k) and (W_ni), which imply that p is an EPPF.

3. SSMS BEYOND THE DP

3.1 The SSM(p, ν)

We know that an SSM with a nonlinear PPF, that is, p_j different from the PPF of a DP, cannot be described as a function p_j ∝ f (n_j) of n_j only. It must be a more complicated function f (n). Alternatively, one could try to define an EPPF and deduce the implied PPF. But directly specifying a symmetric function p (n) such that it complies with (4) is difficult. As a third alternative we propose to consider the weights P = {P_h, h = 1, 2, …} in (3). Figure 1(a) illustrates p (P) for a DP model. The sharp decline is typical. A few large weights account for most of the probability mass. The stick breaking construction for a DP prior with total mass θ implies E (P_h) = θ^h−1 (1 + θ)^−h. Such geometrically decreasing mean weights are inappropriate to describe prior information in many applications. The weights can be interpreted as asymptotic relative cluster sizes. A typical application of the DP prior is, for example, a partition of patients in a clinical study into clusters. However, if clusters correspond to disease subtypes defined by variations of some biological process, then one would rather expect a number of clusters with a priori comparable size. Many small clusters with very few patients are implausible and would also be of little clinical use. This leads us to propose the use of alternative SSMs.

Fig. 1.

The lines in each panel show 10 draws P ~ p(P) for the DP (left) and for the SSM defined in (16) below (right). The P_h are defined for integers h only. We connect them to a line for presentation only. Also, for better presentation we plot the sorted weights. The thick line shows the prior mean. For comparison, a dashed thick line plots the prior mean of the unsorted weights. Under the DP the sorted and unsorted prior means are almost indistinguishable.

Figure 1(b) shows an alternative probability model p(P). There are many ways to define p(P); we consider, for h = 1, 2, …,

$$P_h\propto u_h\text{or}{P}_h=\frac{u_h}{{\sum }_{i=1}^{\infty }{u}_i},$$

where u_h are independent and nonnegative random variables with

$$\sum _{i=1}^{\infty }{u}_i<\infty \mathrm{a}.\mathrm{s}.$$ (14)

A sufficient condition for (14) is

$$\sum _{i=1}^{\infty }E(u_i)<\infty$$ (15)

by the monotone convergence theorem. Note that when the unnormalized random variables u_h are defined as the sorted atoms of a nonhomogeneous Poisson process on the positive real line, under mild assumptions, the above (P_h) construction coincides with the Poisson–Kingman models. Ferguson and Klass (1972) provide a detailed discussion on the outlined mapping of a Poisson process into a sequence of unnormalized positive weights. In this particular case the mean of the Poisson process has to satisfy minimal requirements (see, e.g., Pitman, 2003) to ensure that the sequence (P_i) is well defined.

As an illustrative example in the following discussion, we define, for h = 1, 2, …,

$$P_h\propto e^{X_h}\text{with}{X}_h~N(\text{log}(1-{\{1+e^{b-ah}\}}^{-1}),{\sigma }^2),$$ (16)

where a, b, σ² are positive constants. The existence of such random probabilities is guaranteed by (15), which is easy to check.

The S-shaped nature of the random distribution (16), when plotted against h, distinguishes it from the DP model. The first few weights are a priori of equal size (before sorting). This is in contrast to the stochastic ordering of the DP and the Pitman–Yor process in general. In panel (a) of Figure 1 the prior mean of the sorted and unsorted weights is almost indistinguishable, because the prior already implies strong stochastic ordering of the weights.

The prior in Figure 1(b) reflects prior information of an investigator who believes that there should be around 5 to 10 clusters of comparable size in the population. This is in sharp contrast to the (often implausible) assumption of one large dominant cluster and geometrically smaller clusters that is reflected in panel (a). Prior elicitation can exploit such readily interpretable implications of the prior choice to propose models like (16).

We use SSM(p, ν) to denote a SSM defined by p (P) for the weights P_h and $m_h\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{~}\nu$. The attraction of defining the SSM through P is that by (3) any joint probability model p (P) such that P (∑_h P_h = 1) defines a proper SSM. There are no additional constraints as for the PPF p_j (n) or the EPPF p (n). However, we still need the implied PPF to implement posterior inference and also to understand the implications of the defined process. Thus, a practical use of this second approach requires an algorithm to derive the PPF starting from an arbitrarily defined p (P).

3.2 An Algorithm to Determine the PPF

Recall definition (3) for an SSM random probability measure. Assuming a proper SSM, we have

$$G=\sum _{h=1}^{\infty }{P}_h{\delta }_{m_h}.$$ (17)

Let P = (P_h, h ∈ ℕ) denote the sequence of weights. Recall the notation X̃_j for the j th unique value in the SSS {X_i, i = 1, …, n}. The algorithm requires indicators that match the X̃_j with the m_h, that is, that match the clusters in the partition with the point masses of the SSM. Let π_j = h if X̃_j = m_h, j = 1, …, k_n. In the following discussion it is important that the latent indicators π_j are only introduced up to j = k. Conditional on m_h, h ∈ ℕ and X̃_j, j ∈ ℕ, the indicators π_j are deterministic. After marginalizing with respect to the m_h or with respect to the X̃_j, the indicators become latent variables. Also, we use cluster membership indicators s_i = j for X_i = X̃_j to simplify notation. We use the convention of labeling clusters in the order of appearance, that is, s₁ = 1 and s_i+1 ∈ {1, …, k_i, k_i + 1}.

In words, the algorithm proceeds as follows. We write the desired PPF p_j (n) as an expectation of the conditional probabilities p (X_n+1 = X̃_j | n, π, P). The expectation is with respect to p (P, π | n). Next we approximate the integral with respect to p (P,π | n) by a weighted Monte Carlo average over samples (P^(ℓ), π^(ℓ)) ~ p (P^(ℓ)) p (π^(ℓ) | P^(ℓ)) from the prior. Note π and P together define the size-biased permutation of (P_j),

$${\mathrm{P̃}}_j=P_{{\pi }_j},j=1,2,\dots .$$

The size-biased permutation (P̃_j) of (P_j) is a resampled version of (P_j) where sampling is done with probability proportional to P_j and without replacement. Once the sequence (P_j) is simulated, it is computationally straightforward to get (P̃_j). Note also that the properties of the random partition can be characterized by the distribution on P only. The point masses m_h are not required.

Using the cluster membership indicators s_i and the size-biased probabilities P̃_j, we write the desired PPF as

$$p_j(\mathbf{n})=p(s_{n+1}=j|\mathbf{n})=\int p(s_{n+1}=j|\mathbf{n},\mathbf{P̃})p(\mathbf{P̃}|\mathbf{n})\mathrm{d}\mathbf{P̃}\propto \int p(s_{n+1}=j|\mathbf{n},\mathbf{P̃})p(\mathbf{n}|\mathbf{P̃})p(\mathbf{P̃})\mathrm{d}\mathbf{P̃}\approx \frac{1}{L}\sum p(s_{n+1}=j|\mathbf{n},{\mathbf{P̃}}^{(\ell )})p(\mathbf{n}|{\mathbf{P̃}}^{(\ell )}).$$ (18)

The Monte Carlo sample P̃^(ℓ) or, equivalently, (P^(ℓ), π^(ℓ)), is obtained by first generating P^(ℓ) ~ p(P) and then $p({\pi }_j^{(\ell )}=h|{\mathbf{P}}^{(\ell )},{\pi }_1^{(\ell )},\dots ,{\pi }_{j-1}^{(\ell )})\propto p_h^{(\ell )},h\notin \{{\pi }_1^{(\ell )},\dots ,{\pi }_{j-1}^{(\ell )}\}$. In actual implementation the elements of P^(ℓ) and π^(ℓ) are only generated as and when needed.

The terms in the last line of (18) are easily evaluated. The first factor is given as predictive cluster membership probabilities

$$p(s_{n+1}=j|\mathbf{n},\mathbf{P̃})=\{\begin{array}{cc}{\mathrm{P̃}}_j,\hfill & j=1,\dots ,k_n,\hfill \\ (1-\sum _{j=1}^{k_n}{\mathrm{P̃}}_j),\hfill & j=k_n+1.\hfill \end{array}$$ (19)

The second factor is evaluated as

$$p(\mathbf{n}|\mathbf{P̃})=\prod _{j=1}^k{\mathrm{P̃}}_j^{n_j-1}\prod _{j=1}^{k-1}(1-\sum _{i=1}^j{\mathrm{P̃}}_i).$$

Note that the second factor coincides with the previously mentioned [cf. expression (8)] Pitman’s representation result for partially exchangeable partitions.

Figure 2 shows an example. The figure plots p (s_i+1 = j | s) against cluster size n_j. In contrast, the DP Polya urn would imply a straight line. The plotted probabilities are averaged with respect to all other features of s, in particular, the multiplicity of cluster sizes, etc. The figure also shows probabilities (19) for specific simulations.

Fig. 2.

Panel (a) shows the PPF (19) for a random probability measure G ~ SSM(p, ν), with P_h as in (16). The thick line plots p (s_n+1 = j | s) against n_j, averaging over multiple simulations. In each simulation we used the same simulation truth to generate s and stop simulation at n = 100. The 10 thin lines show p_j (n) for 10 simulations with different n. In contrast, under the DP Polya urn the curve is a straight line and there is no variation across simulations [panel (b)].

3.3 A Simulation Example

Many data analysis applications of the DP prior are based on DP mixtures of normals as models for a random probability measure F. Applications include density estimation, random effects distributions, generalizations of a probit link, etc. We consider a stylized example that is chosen to mimic typical features of such models.

In this section we show posterior inference conditional on the data set (y₁, y₂, …, y₉) = (−4, −3, −2, …, 4). The use of these data highlights the differences in posterior inference between the SSM and DP priors. Assume $y_i\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{~}F$, with a semi-parametric mixture of normal prior on F,

$$y_i\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{~}F\text{with}F(y_i)=\int N(y_i;\mu ,{\sigma }^2)\mathrm{d}G(\mu ,{\sigma }^2).$$

Here N (x; m, s²) denotes a normal distribution with moments (m, s²) for the random variable x. We estimate F under two alternative priors,

$$G~\mathrm{S}\mathrm{S}\mathrm{M}(p,\nu )\text{or}G~\mathrm{D}\mathrm{P}(M,\nu ).$$

The distribution p of the weights for the SSM(p, ·) prior is defined as in (16). The total mass parameter M in the DP prior is fixed to match the prior mean number of clusters, E (k_n), implied by (16). We find M = 2.83. Let Ga (x; a, b) indicate that the random variable x has a Gamma distribution with shape parameter a and inverse scale parameter b. For both prior models we use

$$\nu (\mu ,1/{\sigma }^2)=N(x;{\mu }_0,c{\sigma }^2)\mathrm{G}\mathrm{a}(1/{\sigma }^2;a/2,b/2).$$

We fix µ₀ = 0, c = 10 and a = b = 4. The model can alternatively be written as $y_i~N({\mu }_i,{\sigma }_i^2)$ and $X_i=({\mu }_i,1/{\sigma }_i^2)~G$.

Figures 3 and 4 show some inference summaries. Inference is based on Markov chain Monte Carlo (MCMC) posterior simulation with 1000 iterations. Posterior simulation is for (s₁, …, s_n) only. The cluster-specific parameters $({\mathrm{\mu ̃}}_j,{\mathrm{\sigma ̃}}_j^2)$, j = 1, …, k_n, are analytically marginalized. One of the transition probabilities (Gibbs sampler) in the MCMC requires the PPF under SSM(p, ν). It is evaluated using (18).

Fig. 3.

Posterior estimated sampling model F̅ = E (F | data) = p (y_n+1 | data) under the SSM(p, ν) prior and a comparable DP prior. The triangles along the x-axis show the data.

Fig. 4.

Co-clustering probabilities p (s_i = s_j | data) under the two prior models.

Figure 3 shows the posterior estimated sampling distributions F. The figure highlights a limitation of the DP prior. The single total mass parameter M controls both, the number of clusters and the prior precision. A small value for M favors a small number of clusters and implies low prior uncertainty. LargeM implies the opposite. Also, we already illustrated in Figure 1 that the DP prior implies stochastically ordered cluster sizes, whereas the chosen SSM prior allows for many approximately equal size clusters. The equally spaced grid data (y₁, …, y_n) implies a likelihood that favors a moderate number of approximately equal size clusters. The posterior distribution on the random partition is shown in Figure 4. Under the SSM prior the posterior supports a moderate number of similar size clusters. In contrast, the DP prior shrinks the posterior toward a few dominant clusters. Let n₍₁₎ ≡ max_{j = 1, …, kn} n_j denote the leading cluster size. Related evidence can be seen in the marginal posterior distribution (not shown) of k_n and n₍₁₎. We find E (k_n | data) = 6.4 under the SSM model versus E (k_n | data) = 5.1 under the DP prior. The marginal posterior modes are k_n = 6 under the SSM prior and k_n = 5 under the DP prior. The marginal posterior modes for n₍₁₎ is n₍₁₎ = 2 under the SSM prior and n₍₁₎ = 3 under the DP prior.

3.4 Analysis of Sarcoma Data

We analyze data from of a small phase II clinical trial for sarcoma patients that was carried out in the M. D. Anderson Cancer Center. The study was designed to assess efficacy of a treatment for sarcoma patients across different subtypes. We consider the data accrued for 8 disease subtypes that were classified as having overall intermediate prognosis, as presented in Table 1. Each table entry indicates the total number of patients for each sarcoma subtype and the number of patients who reported a treatment success. See further discussion in León-Novelo et al. (2012).

Table 1.

Sarcoma data. For each disease subtype (top row) we report the total number of patients and the number of treatment successes. See León-Novelo et al. (2012) for a discussion of disease subtypes

Sarcoma	LEI	LIP	MFH	OST	Syn	Ang	MPNST	Fib
	6/28	7/29	3/29	5/26	3/20	2/15	1/5	1/12

One limitation of these data is the small sample size, which prevents separate analysis for each disease subtype. On the other hand, it is not clear that we should simply treat the subtypes as exchangeable. We deal with these issues by modeling each table entry as a binomial response and adopt a hierarchical framework for the success probabilities. The hierarchical model includes a random partition of the subtypes. Conditional on a given partition, data across all subtypes in the same cluster are pooled, thus allowing more precise inference on the common success probabilities for all subtypes in this cluster. We consider two alternative models for the random partition, based on a DP (M, ν) prior versus a SSM(p, ν) prior. Specifically, we consider the following models:

$$y_i|{\pi }_i~\text{Bin}(n_i,{\pi }_i),{\pi }_i|G~G,G~\mathrm{D}\mathrm{P}(M,\nu )\text{or}\mathrm{S}\mathrm{S}\mathrm{M}(p,\nu ),$$

where ν is a diffuse probability measure on [0, 1] and p is again defined as in (16).

The hierarchical structure of the data and the aim of clustering subpopulations in order to achieve borrowing of strength is in continuity with a number of applied contributions. Several of these, for instance, are meta analyses of medical studies (Berry and Christensen, 1979), with subpopulations defined by medical institutions or by clinical trials. In most cases the application of the DP is chosen for computational advantages and (in some cases) due to the easy implementation of strategies for prior specification (Liu, 1996). With a small number of studies, as in our example, ad hoc construction of alternative SSM combines hierarchical modeling with advantageous posterior clustering. The main advantage is the possibility of avoiding the exponential decrease typical of the ordered DP atoms.

In this particular analysis, we used M = 2.83 and chose ν to be the Beta (0.15, 0.85) distribution, which was designed to match the prior mean of the observed data and has prior equivalent sample size of 1. The total mass M = 2.83 for the DP prior was selected to achieve matching prior expected number of clusters under the two models. The DP prior on G favors the formation of large clusters (with matched prior mean number of clusters) which leads to less posterior shrinkage of cluster-specific means. In contrast, under the SSM prior the posterior puts more weight on several smaller clusters.

Figure 5 shows the estimated posterior probabilities of pairwise co-clustering for model (16) in the left panel and for the DP case (right panel). Clearly, compared to the DP model, the chosen SSM induces a posterior distribution with more clusters, as reflected in the lower posterior probabilities p (s_i = s_j | y) for all i, j.

Fig. 5.

Posterior probabilities of pairwise co-clustering, p_ij = p(s_i = s_j | y). The grey scales in the two panels are scaled as black for p_ij = 0 to white for p_ij = max_r,s p_rs. The maxima are indicated in the right top of the plots.

Figure 6 shows the posterior distribution of the number of clusters under the SSM and DP mixture models. Under the DP (right panel) includes high probability for a single cluster, k = 1, with n₁ = 8. The high posterior probability for few large clusters also implies high posterior probabilities p̂_ij of co-clustering. Under the SSM (left panel) the posterior distribution on ρ retains substantial uncertainty. Finally, the same pattern is confirmed in the posterior distribution of sizes of the largest cluster, p (n₁ | y), shown in Figure 7. The high posterior probability for a single large cluster of all n = 8 sarcoma subtypes seems unreasonable for the given data.

Fig. 6.

Posterior distribution on the number of clusters.

Fig. 7.

Posterior distribution on the size of the largest cluster.

4. DISCUSSION

We have reviewed alternative definitions of SSMs. We also reviewed the fact that all SSMs with a PPF of the form p_j (n) = f (n_j) must necessarily be a linear function of n_j and provided a new elementary proof. In other words, the PPF p_j (n) depends on the current data only through the cluster sizes. The number of clusters and any other aspect of the partition ∏_n do not change the prediction. This is an excessively simplifying assumption for most data analysis problems.

We provide an alternative class of models that allows formore general PPFs. These models are obtained by directly specifying the distribution of unnormalized weights u_h. The proposed approach for defining SSMs allows the incorporation of the desired qualitative properties concerning the decrease of the ordered clusters cardinalities. This flexibility comes at the cost of additional computation required to implement the algorithm described in Section 3.2, compared to the standard approaches under DP-based models. Nevertheless, the benefits obtained in the case of data sets that require more flexible models compensate the increase in computational effort. A different strategy for constructing discrete random distributions has been discussed in Trippa and Favaro (2012). In several applications, the scope for which SSMs are to be used suggests these desired qualitative properties. Nonetheless, we see the definition of a theoretical framework supporting the selection of a SSM as an open problem.

R code for an implementation of posterior inference under the proposed new model is available at http://math.utexas.edu/users/pmueller/.