Nonparametric Bayes Classification and Hypothesis Testing on Manifolds

Abstract

Our first focus is prediction of a categorical response variable using features that lie on a general manifold. For example, the manifold may correspond to the surface of a hypersphere. We propose a general kernel mixture model for the joint distribution of the response and predictors, with the kernel expressed in product form and dependence induced through the unknown mixing measure. We provide simple sufficient conditions for large support and weak and strong posterior consistency in estimating both the joint distribution of the response and predictors and the conditional distribution of the response. Focusing on a Dirichlet process prior for the mixing measure, these conditions hold using von Mises-Fisher kernels when the manifold is the unit hypersphere. In this case, Bayesian methods are developed for efficient posterior computation using slice sampling. Next we develop Bayesian nonparametric methods for testing whether there is a difference in distributions between groups of observations on the manifold having unknown densities. We prove consistency of the Bayes factor and develop efficient computational methods for its calculation. The proposed classification and testing methods are evaluated using simulation examples and applied to spherical data applications.

1. Introduction

Classification is one of the fundamental problems in statistics and machine learning. Let (X, Y) denote a pair of random variables with X ∈ the predictors and Y ∈ = {1, …, L} the response. The focus in classification is on predicting Y given features X. From a model-based perspective, one can address this problem by first estimating p(y, x) = Pr(Y = y |X = x), for y = 1, …, L and all x ∈ , based on a training sample of n subjects. Then, under a 0–1 loss function, the optimal predictive value for y_n+1, the unknown response for an unlabeled (n+1)st subject, is simply the value of y that maximizes the estimate p̂(y, x_n+1) for y ∈ {1, …, L}. The model-based perspective has the advantage of providing measures of uncertainty in classification. However, performance will be critically dependent on obtaining an accurate estimate of p(y, x).

A common strategy for addressing this problem is to use a discriminant analysis approach, which lets

$$p(y,x)=Pr(Y=y\mid X=x)=\frac{Pr(Y=y)f(x\mid Y=y)}{{\sum }_{j=1}^{L}Pr(Y=j)f(x\mid Y=j)},y=1,\dots ,L,$$

with Pr(Y = y) the marginal probability of having label y, and f(x |Y = y) the conditional density of the features (predictors) for subjects in class y. Then, one can simply use the proportion of subjects in class y as an estimate of Pr(Y = y), while applying a multivariate density estimator to learn f(x |Y = y) separately within each class. For example, [25] proposed a popular approach which estimates f(x |Y = y) using a mixture of multivariate Gaussians (refer also to [19]). [1] instead use mixtures of von Mises-Fisher (vMF) distributions, but for unsupervised clustering on the unit hypersphere instead of classification. [5] extends the idea to features lying on a general manifold and uses mixtures of complex Watson distributions on the shape space.

Even when features can be assumed to have support on ℜ^p, there are two primary issues that arise. Firstly, the number of mixture components is typically unknown and can be difficult to estimate reliably, and secondly it may be difficult to accurately estimate a multivariate density specific to each class unless p is small and there are abundant training data in each class. To address these issues, a nonparametric Bayes discriminant analysis approach can be used in which the prior incorporates dependence in the unknown class-specific densities ([13]; [14]). An important challenge in implementing nonparametric Bayes methods is showing that the prior is sufficiently flexible to accurately approximate any classification function p(y, x). A primary goal of this article is to provide methods for classification that allow the predictors to have support on a manifold, utilizing priors with full support that lead to posterior consistency for the classification function.

As noted in [23] it is routine in many applications areas, ranging from genomics to computer vision, to normalize the data prior to analysis to remove artifacts. This leads to feature vectors that lie on the surface of the unit hypersphere, though due to the lack of straightforward methods for the analysis of spherical data, Gaussian approximations in Euclidean space are typically used. [23] show that treating spherical features as Euclidean can lead to poor performance in classification if the feature vectors are not approximately spherical-homoscedastic. We will propose a class of product kernel mixture models that can be designed to have full support on the set of densities on a manifold, and that lead to strong posterior consistency. In important special cases, such as spherical data, these models also facilitate computationally convenient Gibbs sampling algorithms for posterior computation.

A closely related problem to the classification problem is testing for differences in the distribution of features across groups. In the testing setting, the nonparametric Bayes literature is surprisingly limited perhaps due to the computational challenges that arise in calculating Bayes factors. For recent articles on nonparametric testing of differences between groups, refer to [15], [32] and [26]. The former two articles considered interval null hypotheses, while the later article considered a point null for testing differences in two groups using Polya tree priors. Here, we modify the methodology developed for the classification problem to obtain an easy to implement approach for nonparametric Bayes testing of differences between groups, with the data within each group constrained to lie on a compact metric space or Riemannian manifold, and prove consistency of this testing procedure.

Here is a very brief overview of the sections to follow. §2 describes the general modeling framework for classification on manifolds. §3 adapts the methodology to the testing problem. §4 focuses on the special case in which the features lie on the surface of a unit hypersphere and adapts the theory and methodology of the earlier sections to this manifold. §5 contains results from simulation studies where the developed methods of classification and testing are compared with existing ones. §6 applies the methods to spherical data applications, and §7 discusses the results. Proofs are included in an Appendix (§8).

2. Nonparametric Bayes Classification

2.1. Kernel mixture model

Let ( , ρ) be a compact metric space, ρ being the distance metric and = {1, …, L} a finite set. Consider a pair of random variables (X, Y) taking values in × . To induce a flexible model on the classification function p(y, x), we propose to model the joint distribution of the (X, Y) pair. The approach of inducing a flexible model on the conditional distribution through a nonparametric model for the joint distribution was proposed by [31]. In particular, they used a Dirichlet process mixture (DPM) of multivariate Gaussians for (X, Y) to induce a flexible prior on E(Y |X = x). [35, 24] recently generalized this approach beyond Gaussian mixtures. [9] used a joint DPM for random effects underlying a functional predictor and a response to induce a flexible model for prediction.

As an alternative to the joint modeling approach, one can directly place a non-parametric Bayesian model on the conditional p(y, x) = Pr(Y = y|X = x). There is a rich applied literature on this topic, with most approaches focusing on categorical predictors and/or predictors in a Euclidean space without consideration of theoretical properties. [22] showed posterior consistency in the = {1, 2} binary case using a Gaussian process prior mapped to the unit interval. [12] showed minimax optimal rates of posterior contraction up to a log factor for a binary response kernel regression model, with Gaussian priors on the weights leading to a conditionally Gaussian process. These and other theoretical articles on nonparametric Bayes classification focus on being a compact d-dimensional subset of ℜ^d, with the optimal rate for α-smooth functions being n^{−α/(d+2α)}.

Our focus is on the case in which Y is an unordered categorical variable and X is constrained to have support on a compact metric space, with non-Euclidean Riemannian manifolds of particular interest. Potentially, one could induce a conditional model for p(y, x) without modeling the marginal distribution of the predictors p(x) through appropriate mappings of Gaussian processes defined over . However, Gaussian processes lead to well-known computational challenges even in Euclidean spaces, and there are advantages of the joint modeling approach in terms of simplifying theory, computation and modifications (e.g., to the testing problem considered in §3) while also allowing missing predictors. Most of the literature on nonparametric Bayes asymptotic theory follows a similar path, with some early papers on the type of strategy we will follow including [2, 20]. However, it is far from straightforward to establish the necessary conditions in new settings, and essentially all of the nonparametric Bayes asymptotic literature has focused on Euclidean spaces and relatively simple problems.

Assume that the joint distribution of (X, Y) has a joint density with respect to some fixed base measure λ on × . Let $\lambda ={\lambda }_1\otimes {\sum }_{j=1}^L{\delta }_j$ where δ denotes the Dirac delta measure. If is a Riemannian manifold, the natural choice for λ₁ will be the Riemannian volume form V. The distance ρ will be chosen to maintain the topology of the manifold. Letting ( × ) denote the space of all densities with respect to λ, we propose the following joint density model

$$f(x,y;P,\kappa )={\int }_{\mathbb{X}\times S_{L-1}}{\nu }_{y}K(x;\mu ,\kappa )P(d\mu d\nu ),(x,y)\in \mathbb{X}\times \mathbb{Y},$$ (2.1)

where ν = (ν₁, …, ν_L)′ ∈ S_L−1 is a probability vector on the simplex S_L−1 = {ν ∈ [0, 1]^L: Σν_j = 1}, K(·; μ, κ) is a kernel located at μ ∈ with precision or inverse-scale κ ∈ ℜ⁺, and P ∈ ( × S_L−1) is a mixing measure, with ( × S_L−1) denoting the space of all probability measures on × S_L−1.

One can interpret this model in the following hierarchical way. Draw (μ, ν) from P. Given (μ, ν, κ), X and Y are conditionally independent with X having the conditional density K(.; μ, κ) with respect to λ₁ and

$$Pr(Y=l\mid \mu ,\nu ,\kappa )={\nu }_l,1\le l\le L.$$

If K(.; μ, κ) is a valid probability kernel, i.e.

$${\int }_{\mathbb{X}}K(x;\mu ,\kappa ){\lambda }_1(dx)=1,\text{for}\text{all}(\mu ,\kappa )\in \mathbb{X}\times {\mathfrak{R}}^{+},$$

one can show that f(x, y; P, κ) is a valid probability density with respect to λ.

To justify model (2.1), it is necessary to show flexibility in approximating any joint density in ( × ), and hence in approximating p(y, x). Our focus is on nonparametric Bayes methods that place a prior on the joint distribution of the measure P and the precision κ to induce a prior over ( × ). Flexibility of the model is quantified in terms of the size of the support of this prior. In particular, our goal is to choose a specification that leads to full L^∞ and Kullback-Leibler (KL) support, meaning that the prior assigns positive probability in arbitrarily small neighborhoods of any density in ( × ). This property will not necessarily hold for arbitrarily chosen kernels, and one of our primary theoretical contributions is to provide sufficient conditions under which KL support and posterior consistency hold. This is not just of theoretical interest, as it is important to verify that the model is sufficiently flexible to approximate any classification function, with the accuracy of the estimate improving as the amount of training data grows. This is not automatic for nonparametric models in which there is often concern about over-fitting.

Remark 2.1

In the joint model (2.1), one may also mix across the precision parameter and make the model more flexible. Posterior computation with such a model is a straight-forward extension of this one which is illustrated in §2.3.

Remark 2.2

One can allow = ⊗ · · · ⊗ , with each corresponding to a different type of space (e.g., a hypersphere, a subset of ℜ^d, a discrete space), by replacing K(x; μ, κ) with a product of kernels appropriate to each space.

2.2. Support of the prior and consistency

Assume that the joint distribution of (X, Y) has a density f_t(x, y) = g_t(x)p_t(y, x) with respect to λ, where g_t is the true marginal density of X and p_t(y, x) is the true Pr(Y = y|X = x). For Bayesian inference, we choose a prior Π₁ on (P, κ) in (2.1), with one possible choice corresponding to DP(w₀P₀) ⊗ Gam(a, b), with DP (w₀P₀) denoting a Dirichlet process prior with precision w₀ and base probability measure P₀ ∈ ( × S_L−1) and Gam denoting the gamma distribution. The prior Π₁ induces a corresponding prior Π on the space ( × ) through (2.1). Under minor assumptions on Π₁ and hence Π, the theorem below shows that the prior probability of any uniform neighborhood of a continuous true density is positive.

Theorem 2.1

Under the assumptions

• A1:K is continuous in its arguments,
• A2:
  For any continuous function φ from to ℜ,

  $$\underset{\kappa \to \infty }{lim}\underset{x\in \mathbb{X}}{sup}\left|\phi (x)-{\int }_{\mathbb{X}}K(x;\mu ,\kappa )\phi (\mu ){\lambda }_1(d\mu )\right|=0,$$
• A3:
  For any κ > 0, there exists κ̃ ≥ κ such that (P_t, κ̃) ∈ supp(Π₁) where P_t ∈ ( × S_L−1) is defined as

  $$P_t(d\mu d\nu )=\sum _{j\in \mathbb{Y}}{f}_t(\mu ,j){\lambda }_1(d\mu ){\delta }_{e_j}(d\nu ),$$

  with e_j ∈ ℜ^L denoting a zero vector with a single one in position j,

• A4:
  f_t(., j) is continuous for all j ∈ , given any ε > 0,

  $$\mathrm{\Pi }\left(\{f\in \mathcal{D}(\mathbb{X}\times \mathbb{Y}):\underset{x\in \mathbb{X},y\in \mathbb{Y}}{sup}\mid f(x,y)-f_t(x,y)\mid <\epsilon \}\right)>0.$$

Assumption A4 restricts the true conditional density of X given Y = j to be continuous for all j. Assumptions A1 and A2 place minor regularity conditions on the kernel K. If K(x; μ, κ) is symmetric in x and μ, as will be the case in most examples, A2 implies that K(.; μ, κ) converges to δ_μ in the weak sense uniformly in μ as κ → ∞. This justifies the names ‘location’ and ‘precision’ for the parameters. Assumption A3 provides a minimal condition on the support of the prior for (P, κ). These assumptions provide general sufficient conditions for the induced prior Π on the joint density of (X, Y) to have full L^∞ support.

Although full uniform support is an appealing property, much of the theoretical work on asymptotic properties of nonparametric Bayes estimators relies on KL support. The following corollary shows that KL support follows from A1–A4 and the additional assumption that the true density is everywhere positive. The proof is very much on the same lines of Corollary 1, [5]. The KL divergence of a density f from f_t is defined as $KL(f_t;f)={\int }_{\mathbb{X}\times \mathbb{Y}}{f}_{t}log\frac{f_t}{f}\lambda (\mathit{dxdy})$. Given ε > 0, K_ε(f_t) = {f: KL(f_t; f) < ε} will denote an ε-sized KL neighborhood of f_t. The prior Π is said to satisfy the KL condition at f_t, or f_t is said to be in its KL support, if Π{K_ε(f_t)} > 0 for any ε > 0.

Corollary 2.2

Under assumptions A1–A4 and

• A5:f_t(x, y) > 0 for all x, y,

  f_t is in the KL support of Π.

Suppose we have an independent and identically distributed (iid) sample $({\mathbf{x}}_n,{\mathbf{y}}_n)\equiv {(x_i,y_i)}_{i=1}^n$ from f_t. Since f_t is unobserved, we take the likelihood function to be ${\mathrm{\Pi }}_{i=1}^{n}f(x_i,y_i;P,\kappa )$. Using the prior Π on f and the observed sample, we find the posterior distribution of f, denoted by Π(.|x_n, y_n). Using the Schwartz theorem ([33]), Corollary 2.2 implies weak posterior consistency. This in turn implies that for any measurable subset A of , with λ₁(A) > 0, λ₁(∂A) = 0, and y ∈ , the posterior conditional probability of Y being y given X in A converges to the true conditional probability almost surely. Here ∂A denotes the boundary of A.

To give more flexibility to the classification function, we may replace the location-scale kernel by some broader family of parametric distributions on such as K(.; μ, κ, Θ) with Θ denoting additional kernel parameters. When performing posterior computations, we may set hyperpriors on the parameters of the prior. Then the conclusions of Results 2.1 and 2.2 hold and hence weak consistency follows as long as the assumptions are verified given the hyperparameters over a set of positive prior probability. This is immediate and is verified in Lemma 1, [41].

Under stronger assumptions on the kernel and the prior, we prove strong posterior consistency for the joint model. We will illustrate how these conditions are met for a von Mises-Fisher (vMF) mixture model for hyperspherical data through Proposition 4.1.

Theorem 2.3

Under assumptions A1–A5 and

• A6:
  There exist positive constants , a₁, A₁ such that for all ≥ , μ, ν ∈ ,

  $$\underset{m\in M,\kappa \in [0,\mathcal{K}]}{sup}|K(m;\mu ,\kappa )-K(m;\nu ,\kappa )|\le A_1{\mathcal{K}}^{a1}\rho (\mu ,\nu ).$$
• A7:
  There exists positive constants a₂, A₂ such that for all κ₁, κ₂ ∈ [0, ], ≥ ,

  $$\underset{m,\mu \in M}{sup}|K(m;\mu ,{\kappa }_1)-K(m;\mu ,{\kappa }_2)|\le A_2{\mathcal{K}}^{a_2}\mid {\kappa }_1-{\kappa }_2\mid .$$
• A8:There exist positive constants a₃, A₃, A₄ such that given any ε > 0, M can be covered by at-most A₃ε^−a₃ + A₄ many subsets of diameter at-most ε.
• A9:Π₁( (M) × (n^a, ∞)) is exponentially small for some a < (a₁a₃)⁻¹, the posterior probability of any total variation neighborhood of f_t converges to 1 almost surely.

Given the training data, we can classify a new subject based on their features using the posterior mean classification function p̂. As a corollary to Theorem 2.3, we show that p̂ converges to p_t in an L¹ sense.

Corollary 2.4

(a) Strong consistency for the posterior of f implies that

$$\mathrm{\Pi }\left\{f:\underset{y\in \mathbb{Y}}{max}{\int }_{\mathbb{X}}\mid p(y,x)-p_t(y,x)\mid g_t(x){\lambda }_1(dx)<\epsilon |{\mathbf{x}}_{\mathbf{n}},{\mathbf{y}}_{\mathbf{n}}\right\}$$ (2.2)

converges to 1 as n → ∞ a.s. (b) Under assumptions A4–A5 on f_t, this implies

$$\mathrm{\Pi }\left\{f:\underset{y\in \mathbb{Y}}{max}{\int }_{\mathbb{X}}\mid p(y,x)-p_t(y,x)\mid w(x){\lambda }_1(dx)<\epsilon |{\mathbf{x}}_{\mathbf{n}},{\mathbf{y}}_{\mathbf{n}}\right\}$$

converges to 1 a.s. for any non-negative function w with sup_x w(x) < ∞.

Remark 2.3

Part (a) of Corollary 2.4 holds even when is non-compact. It just needs strong posterior consistency for the joint model.

From part (b) of Corollary 2.4, it would seem intuitive that point-wise posterior consistency can be obtained for the predictive probability function. However, this is not immediate because the convergence rate may depend on the choice of w.

Assumption A9 is hard to satisfy, especially when the feature space is high dimensional. This type of problem was mentioned by [42] in a different setting. Then a₁ and a₃ turn out to be very big, so that the prior is required to have very light tails and place small mass at high precisions. This is undesirable in applications and instead we can let Π₁ depend on the sample size n and obtain weak and strong consistency under weaker assumptions.

Theorem 2.5

Let Π₁ = Π₁₁ ⊗ π_n where π_n is a sequence of densities on ℜ⁺. Assume the following.

• A10:The prior Π₁₁ has full support.
• A11:
  For any β > 0, there exists a κ₀ ≥ 0, such that for all κ ≥ κ₀,

  $$\underset{n\to \infty }{liminf}exp(n\beta ){\pi }_n(\kappa )=\infty .$$
• A12:
  For some β₀ > 0 and a < (a₁a₃)⁻¹,

  $$\underset{n\to \infty }{lim}exp(n{\beta }_0){\pi }_n\{(n^a,\infty )\}=0.$$

  1. Under assumptions A1–A2 on the kernel, A10–A11 on the prior and A4–A5 on f_t, the posterior probability of any weak neighborhood of f_t converges to one a.s.

  2. Under assumptions A1–A2, A4–A8 and A10–A12, the posterior probability of any total variation neighborhood of f_t converges to 1 a.s.

The proof is similar to that of Theorems 2.6 and 2.9 in [6] and hence is omitted. With Π₁₁ = DP (ω₀P₀) and π_n = Gam(a, b_n), the conditions in Theorem 2.5 are satisfied (for example) when P₀ has full support and b_n = b₁n/{log(n)}^b₂ for any b₁, b₂ > 0. Then from Corollary 2.4, we have L¹ consistency of the estimated classification function.

2.3. Computation

Given the training sample (x_n, y_n), we classify a new subject based on the predictive probability of allocating it to category j, which is expressed as

$$Pr(y_{n+1}=j\mid x_{n+1},{\mathbf{x}}_n,{\mathbf{y}}_n),j\in \mathbb{Y},$$ (2.3)

where x_n+1 denotes the feature for the new subject and y_n+1 its unknown class label. It follows from Theorem 2.1 and Corollary 2.4 that the classification rule is consistent if the kernel and prior are chosen correctly. Following the recommendation in §2.2, for the prior, we let P ~ DP(w₀P₀) independently of κ ~ π, with P₀ = P₀₁ ⊗ P₀₂, P₀₁ a distribution on , P₀₂ a Dirichlet distribution Diri(a) (a = (a₁, …, a_L)) on S_L−1, and π a base distribution on ℜ⁺.

Since it is not possible to get a closed form expression for the predictive probability, we use a Markov chain Monte Carlo (MCMC) algorithm. Specifically, we rely on a simple-to-implement Gibbs sampling algorithm that utilizes slice sampling. Slice sampling was proposed for posterior computation in DPMs by [38], with efficiency later improved by Papaspiliopoulos (2008, unpublished technical report). Our approach is related to that used by [43], with [28] providing a general class of algorithms for slice sampling in mixture models.

Using the stick-breaking representation of [34] and introducing cluster allocation indices S = (S₁, …, S_n) (S_i ∈ {1, …, ∞}), the generative model (2.1) can be expressed in hierarchical form as

$$\begin{array}{l}{x}_i~K({\mu }_{S_i},\kappa ),y_i~\text{Multi}(1,\dots ,L;{\nu }_{S_i}),\\ S_i~\sum _{j=1}^{\infty }{w}_j{\delta }_{{\theta }_j},{\theta }_j=({\mu }_j,{\nu }_j),\end{array}$$ (2.4)

where w_j = V_jΠ_h<j(1 − V_h) is the probability that subject i is allocated to cluster S_i = j, θ_j is the vector of parameters specific to cluster j, and V_j ~ Beta(1, w₀), μ_j ~ P₀₁ and ν_j ~ Diri(a) are mutually independent for j = 1, …, ∞.

The joint posterior density of $V={\{V_j\}}_{j=1}^{\infty }$, $\theta ={\{{\theta }_j\}}_{j=1}^{\infty }=(\mu ,\nu )$, S and κ given the training data is proportional to

$$\{{\mathrm{\Pi }}_{i=1}^{n}K(x_i;{\mu }_{S_i},\kappa ){\nu }_{S_i{y}_i}{w}_{S_i}\}\{{\mathrm{\Pi }}_{j=1}^{\infty }\text{Beta}(V_j;1,w_0)P_{01}(d{\mu }_j)\text{Diri}({\nu }_j;\mathbf{a})\}\pi (\kappa ).$$

To avoid the need for posterior computation for infinitely-many unknowns, we introduce slice sampling latent variables $u={\{u_i\}}_{i=1}^n$ drawn iid from Unif(0,1) such that the augmented posterior density is proportional to

$$\begin{array}{r}\pi (u,V,\theta ,S,\kappa \mid {\mathbf{x}}_n,{\mathbf{y}}_n)\propto \left\{\prod _{i=1}^{n}K(x_i;{\mu }_{S_i},\kappa ){\nu }_{S_i{y}_i}I(u_i<w_{S_i})\right\}\times \\ \left\{\prod _{j=1}^{\infty }\text{Beta}(V_j;1,w_0)P_{01}(d{\mu }_j)\text{Diri}({\nu }_j;\mathbf{a})\right\}\pi (\kappa ).\end{array}$$ (2.5)

Letting S_max = max{S_i}, the conditional posterior distribution of {V_j, θ_j, j > S_max} is the same as the prior, and we can use this to bypass the need for updating infinitely-many unknowns in the Gibbs sampler. After choosing initial values, the sampler iterates through the following steps.

1. Update S_i, for i = 1, …, n, given (u, V, θ, κ, x_n, y_n) by sampling from the multinomial distribution with

   $$Pr(S_i=j)\propto K(x_i;{\mu }_j,\kappa ){\nu }_{{jy}_i}\text{for}j\in A_i=\{j:1\le j\le J,w_j>u_i\},$$

   with J being the smallest index satisfying $1-min(u)<{\sum }_{j=1}^J{w}_j$. In implementing this step, draw V_j ~ Beta(1, w₀) and θ_j ~ P₀ for j > S_max as needed.

2. Update the scale parameter κ by sampling from the full conditional posterior which is proportional to

   $$\pi (\kappa )\prod _{i=1}^{n}K(x_i;{\mu }_{S_i},\kappa ).$$

   If direct sampling is not possible, rejection sampling or Metropolis-Hastings (MH) sampling can be used.

3. Update the atoms θ_j, j = 1, …, S_max from the full conditional posterior distribution, which is equivalent to independently sampling from

   $$\begin{array}{l}\pi ({\mu }_j\mid -)\propto P_{01}(d{\mu }_j)\prod _{i:S_i=j}K(x_i;{\mu }_j,\kappa )\\ ({\nu }_j\mid -)~\text{Diri}\left(a_1+\sum _{i:S_i=j}I(y_i=1),\dots ,a_L+\sum _{i:S_i=j}I(y_i=L)\right).\end{array}$$

   If P₀₁ is not conjugate, then rejection or MH sampling can be used to update μ_j.

4. Update the stick-breaking random variables V_j, for j = 1, …, S_max, from their conditional posterior distributions given the cluster allocation S but marginalizing out the slice sampling latent variables u. In particular,

   $$V_j~\text{Beta}\left(1+\sum _{i}I(S_i=j),w_0+\sum _{i}I(S_i>j)\right).$$
5. Update the slice sampling latent variables from their conditional posterior by letting

   $$u_i~\text{Unif}(0,w_{S_i}),i=1,\dots ,n.$$

These steps are repeated a large number of iterations, with a burn-in discarded to allow convergence. Given a draw from the posterior, the predictive probability of allocating a new observation to category l, l ≤ L, as defined through (2.3) is proportional to

$$\sum _{j=1}^{S_{\mathit{max}}}{w}_j{\nu }_{jl}K(x_{n+1};{\mu }_j,\kappa )+w_{S_{max}+1}\underset{\mathbb{X}\times S_{L-1}}{\int }{\nu }_{l}K(x_{n+1};{\mu }_j,\kappa )P_0(d\mu d\nu )$$ (2.6)

where $w_{S_{\mathit{max}}+1}=1-{\sum }_{j=1}^{S_{\mathit{max}}}{w}_j$. We can average these conditional predictive probabilities across the MCMC iterations after burn-in to estimate predictive probabilities. For moderate to large numbers of training samples n, ${\sum }_{j=1}^{S_{\mathit{max}}}{w}_j\approx 1$ with high probability, so that an accurate approximation can be obtained by setting the final term equal to zero and hence bypassing need to calculate the integral.

3. Nonparametric Bayes Testing

3.1. Hypotheses and Bayes factor

The previous section focused on the classification problem in which there is interest in predicting a class label y_n+1 for a new subject based on training data (x_n, y_n). In many applications, the primary focus is instead on conducting inferences on differences in the distribution of features X across groups Y. For example, Y may correspond to a patient’s ethnicity and X to health outcomes in an epidemiology study, with interest being in assessing evidence in the data of differences in the distribution of health outcomes across ethnic groups. This testing problem is somewhat orthogonal to the classification problem considered in §2.3, with classification being of primary interest in certain applications and testing of primary interest in others. However, as we demonstrate in this section, the models, theory and computational algorithms we developed for the classification problem can be adapted to accommodate testing. Although testing of group differences is one of the canonical problems in statistics, the literature addressing this problem from a nonparametric Bayes perspective while providing theoretical guarantees is essentially non-existent. Hence, the material in this section is a major contribution of the paper, which should hopefully stimulate additional research.

Although our methods can allow testing of pairwise differences between groups, we focus for simplicity in exposition on the case in which the null hypothesis corresponds to homogeneity across the groups. Formally, the alternative hypothesis H₁ corresponds to any joint density in ( × ) excluding densities of the form

$$H_0:f(x,y)=g(x)p(y)$$ (3.1)

for all (x, y) outside of a λ-null set. Note that model (2.1) will in general assign zero probability to H₀, and hence is an appropriate model for the joint density under H₁.

As a model for the joint density under the null hypothesis H₀ in (3.1), we replace P(dμdν) in (2.1) with P₁(dμ)P₂(dν) so that the joint density becomes

$$f(x,y;P_1,P_2,\kappa )=g(x;P_1,\kappa )p(y;P_2)\text{where}$$ (3.2)

$$g(x;P_1,\kappa )={\int }_{\mathbb{X}}K(x;\mu ,\kappa )P_1(d\mu ),p(y;P_2)={\int }_{S_{L-1}}{\nu }_y{P}_2(d\nu ).$$ (3.3)

We set priors Π₁ and Π₀ for the parameters in the models under H₁ and H₀, respectively. The Bayes factor in favor of H₁ over H₀ is then the ratio of the marginal likelihoods under H₁ and H₀,

$$BF(H_1:H_0)=\frac{{\int }_{\mathcal{M}(\mathbb{X}\times S_{L-1})\times {\mathfrak{R}}^{+}}{\prod }_{i=1}^{n}f(x_i,y_i;P,\kappa ){\mathrm{\Pi }}_1(\mathit{dPd}\kappa )}{{\int }_{\mathcal{M}(\mathbb{X})\times \mathcal{M}(S_{L-1})\times {\mathfrak{R}}^{+}}{\prod }_{i=1}^{n}g(x_i;P_1,\kappa )p(y_i;P_2){\mathrm{\Pi }}_0({dP}_1{dP}_{2}d\kappa )}$$

The priors should be suitably constructed so that we get consistency of the Bayes factor and computation is straightforward and efficient. [3] propose an approach for calculating Bayes factors for comparing Dirichlet process mixture (DPM) models, but their algorithm is quite involved to implement and is limited to DPM models with a single DP prior on an unknown mixture distribution. Simple conditions for consistency of Bayes factors for testing a point null versus a nonparametric alternative have been provided by [11], but there has been limited work on consistency of Bayes tests in more complex cases, such as we are faced with here. [21] develop general theory on nonparametric Bayesian model selection and averaging, with their Corollary 3.1 providing conditions for Bayes factor consistency. [29] shows Bayes factor and model selection consistency in the setting of semiparametric linear models with nonparametric residual distributions. Our theory takes a different direction appropriate to the group testing problem.

The prior Π₁ on (P, κ) under H₁ can be constructed as in §2. To choose a prior Π₀ for (P₁, P₂, κ) under H₀, we take (P₁, κ) to be independent of P₂ so that the marginal likelihood becomes a product of the X and Y marginals if H₀ is true. Dependence in the priors for the mixing measures would induce dependence between the X and Y densities, and it is important to maintain independence under H₀. In addition, as in parametric Bayes model selection [10], it is important to maintain compatibility of the prior specifications under H₀ and H₁. In the parametric literature, compatibility is often obtained through first specifying a prior for the unknowns in the larger model and then marginalizing to induce priors for the unknowns in smaller nested models. Following this idea, we let the prior for (P₁, κ) under H₀ correspond to the marginal on (P₁, κ) from prior Π₁ on (P, κ) assuming P = P₁ ⊗ P₂. It remains to specify a prior for P₂ under H₀. Expression (3.3) suggests that under H₀ the density of Y depends on P₂ only through

$$p={(p(1;P_2),p(2;P_2),\dots ,p(L;P_2))}^{\prime }\in S_{L-1}.$$

Hence, it is sufficient to choose a prior for p, such as Diri(b) with b = (b₁, …, b_L)′, instead of specifying a full prior for P₂. For compatibility, the Dirichlet hyperparameters can potentially be chosen to approximately match the first two moments of the marginal prior on p induced from Π₁.

Under prior Π₀ noting that P = P₁ ⊗ P₂, the marginal likelihood under H₀ is

$$\begin{array}{r}\underset{\mathcal{M}(\mathbb{X}\times S_{L-1})\times {\mathfrak{R}}^{+}}{\int }\prod _{i=1}^{n}g(x_i;P_1,\kappa ){\mathrm{\Pi }}_1(\mathit{dPd}\kappa )\underset{S_{L-1}}{\int }\prod _{j=1}^L{p}_j^{{\sum }_{i=1}^{n}I(y_i=j)}\text{Diri}(dp;\mathbf{b})\\ =\frac{D({\mathbf{b}}_n)}{D(\mathbf{b})}\underset{\mathcal{M}(\mathbb{X}\times S_{L-1})\times {\mathfrak{R}}^{+}}{\int }\prod _{i=1}^{n}g(x_i;P_1,\kappa ){\mathrm{\Pi }}_1(\mathit{dPd}\kappa ),\end{array}$$ (3.4)

with b_n being the L-dimensional vector with j^th coordinate $b_j+{\sum }_{i=1}^{n}I(y_i=j)$, 1 ≤ j ≤ L, D being the normalizing constant for the Dirichlet distribution given by $D(\mathbf{a})=\frac{{\prod }_{j=1}^L\mathrm{\Gamma }(a_j)}{\mathrm{\Gamma }({\sum }_{j=1}^L{a}_j)}$ and Γ denoting the gamma function. The marginal likelihood under H₁ is

$$\int \prod _{i=1}^{n}f(x_i,y_i;P,\kappa ){\mathrm{\Pi }}_1(\mathit{dPd}\kappa ).$$ (3.5)

The Bayes factor in favor of H₁ against H₀ is the ratio of the marginal (3.5) over (3.4).

3.2. Consistency of the Bayes factor

Let Π be the prior induced on the space of all densities ( × ) through Π₁. For any density f (x, y), let g(x) = Σ_j f (x, j) denote the marginal density of X while p(y) = ∫f (x, y) λ₁(dx) denotes the marginal probability vector of Y. Let f_t, g_t and p_t be the corresponding values for the true distribution of (X, Y). The Bayes factor in favor of the alternative, as obtained in the last section, can be expressed as

$$BF=\frac{D(\mathbf{b})}{D({\mathbf{b}}_n)}\frac{\int {\prod }_{i}f(x_i,y_i)\mathrm{\Pi }(df)}{\int {\prod }_{i}g(x_i)\mathrm{\Pi }(df)}.$$ (3.6)

Theorem 3.1 proves consistency of the Bayes factor at an exponential rate if the alternative hypothesis of dependence holds.

Theorem 3.1

If X and Y are not independent under the true density f_t and if the prior Π satisfies the KL condition at f_t, then there exists a β₀ > 0 for which lim inf_n→∞ exp(−nβ₀)BF = ∞ a.s. $f_t^{\infty }$.

3.3. Computation

Seemingly, one of the major reasons for the lack of methodology literature on Bayesian nonparametric testing is the substantial computational hurdles involved in accurately calculating Bayes factors for comparing hypotheses. Except in very simple models, marginal likelihoods cannot be calculated in closed form, accurate analytic approximations are not available and Monte Carlo methods are prohibitively expensive computationally when run sufficiently long to produce an accurate estimate. Outside of conjugate models, one of the most successful strategies for calculating Bayes factors based on MCMC algorithms relies on an encompassing approach in which a single MCMC algorithm is designed, which moves freely between the models. There has been limited success in designing such algorithms for nonparametric Bayes testing, but we devise an efficient data augmentation algorithm that appears to accurately produce Bayes factors and posterior hypothesis probabilities based on a single MCMC chain.

We introduce a latent variable z = I(H₁ is true) which takes value 1 if H₁ is true and 0 if H₀ is true. Assuming equal prior probabilities for H₀ and H₁, the conditional likelihood of (x_n, y_n) given z is

$$\begin{array}{l}\mathrm{\Pi }({\mathbf{x}}_n,{\mathbf{y}}_n\mid z=0)=\frac{D({\mathbf{b}}_n)}{D(\mathbf{b})}\int \prod _{i=1}^{n}g(x_i;P_1,\kappa ){\mathrm{\Pi }}_1(\mathit{dPd}\kappa )\text{and}\\ \mathrm{\Pi }({\mathbf{x}}_n,{\mathbf{y}}_n\mid z=1)=\int \prod _{i=1}^{n}f(x_i,y_i;P,\kappa ){\mathrm{\Pi }}_1(\mathit{dPd}\kappa ).\end{array}$$

In addition, the Bayes factor can be expressed as

$$BF=\frac{Pr(z=1\mid {\mathbf{x}}_n,{\mathbf{y}}_n)}{Pr(z=0\mid {\mathbf{x}}_n,{\mathbf{y}}_n)}.$$ (3.7)

Next introduce latent parameters μ, ν, V, S, κ as in §2.3 such that

$$\begin{array}{r}\mathrm{\Pi }({\mathbf{x}}_n,{\mathbf{y}}_n,\mu ,V,S,\kappa ,z=0)=\frac{D({\mathbf{b}}_n)}{D(\mathbf{b})}\pi (k)\prod _{i=1}^n\{w_{S_i}K(x_i;{\mu }_{S_i},\kappa )\}\times \\ \prod _{j=1}^{\infty }\{\text{Be}(V_j;1,w_0)P_{01}(d{\mu }_j)\},\end{array}$$ (3.8)

$$\begin{array}{r}\mathrm{\Pi }({\mathbf{x}}_n,{\mathbf{y}}_n,\mu ,\nu ,V,S,\kappa ,z=1)=\pi (k)\prod _{i=1}^n\{w_{S_i}{\nu }_{S_i{y}_i}K(x_i;{\mu }_{S_i},\kappa )\}\times \\ \prod _{j=1}^{\infty }\{\text{Be}(V_j;1,w_0)P_0(d{\mu }_{j}d{\nu }_j)\}.\end{array}$$ (3.9)

Marginalize out ν from equation (3.9) to get

$$\begin{array}{r}\mathrm{\Pi }({\mathbf{x}}_n,{\mathbf{y}}_n,\mu ,V,S,\kappa ,z=1)=\pi (k)\prod _{j=1}^{\infty }\frac{D(\mathbf{a}+{\stackrel{\sim }{a}}_j(S))}{D(\mathbf{a})}\times \\ \prod _{i=1}^n\{w_{S_i}K(x_i;{\mu }_{S_i},\kappa )\}\prod _{j=1}^{\infty }\text{Be}(V_j;1,w_0)P_{01}(d{\mu }_j)\},\end{array}$$ (3.10)

with ã_j(S), 1 ≤ j < ∞ being L-dimensional vectors with l^th coordinate Σ_i:Si=j I(y_i = l), l ∈ . Integrate out z by adding equations (3.8) and (3.10) and the joint distribution of (μ, V, S, κ) given the data becomes

$$\begin{array}{r}\mathrm{\Pi }(\mu ,V,S,\kappa \mid {\mathbf{x}}_n,{\mathbf{y}}_n)\propto \{C_0+C_1(S)\}\pi (\kappa )\prod _{i=1}^n\{w_{S_i}K(x_i;{\mu }_{S_i},\kappa )\}\times \\ \prod _{j=1}^{\infty }\{\text{Be}(V_j;1,w_0)P_{01}(d{\mu }_j)\}\\ \text{with}{C}_0=\frac{D({\mathbf{b}}_n)}{D(\mathbf{b})}\text{and}{C}_1(S)=\prod _{j=1}^{\infty }\frac{D(\mathbf{a}+{\stackrel{\sim }{a}}_j(S))}{D(\mathbf{a})}.\end{array}$$ (3.11)

To estimate the Bayes factor, first make repeated draws from the posterior in (3.11). For each draw, compute the posterior probability distribution of z from equations (3.8) and (3.10) and take their average after discarding a suitable burn-in. The averages estimate the posterior distribution of z given the data, from which we can get an estimate for BF from (3.7). The sampling steps are accomplished as follows

1. Update the cluster labels S given (μ, V, κ) and the data from their joint posterior which is proportional to

   $$\{C_0+C_1(S)\}\pi (k)\prod _{i=1}^n\{w_{S_i}K(x_i;{\mu }_{S_i},\kappa )\}.$$ (3.12)

   Introduce slice sampling latent variables u as in §2.3 and replace w_Si by I(u_i < w_Si) to make the total number of possible states finite. However unlike in §2.3, the S_is are no more conditionally independent. We propose to use a Metropolis-Hastings block update step in which a candidate for (S₁, …, S_n), or some subset of this vector if n is large, is sampled independently from multinomials with Pr(S_i = j) ∝ K(x_i; μ_j, κ), for j ∈ A_i where A_i = {j: 1 ≤ j ≤ J, w_j > u_i} and J is the smallest index satisfying $1-min(u)<{\sum }_{j=1}^J{w}_j$. In implementing this step, draw V_j ~ Be(1, w₀) and μ_j ~ P₀₁ for j > S_max as needed. The acceptance probability is simply the ratio of C₀ + C₁(S) calculated for the candidate value and the current value of S.

2. Update $\kappa ,{\{{\mu }_j\}}_{j=1}^{S_{\mathit{max}}},{\{V_j\}}_{j=1}^{S_{\mathit{max}}},{\{u_i\}}_{i=1}^n$ as in Steps (2) – (5) of the algorithm in §2.3.

3. Compute the full conditional posterior distribution of z which is given by

   $$Pr(z\mid \mu ,S,{\mathbf{x}}_n,{\mathbf{y}}_n)\propto \{\begin{array}{l}\frac{D({\mathbf{b}}_n)}{D(\mathbf{b})}\text{if}z=0,\hfill \\ {\prod }_{j=1}^{S_{\mathit{max}}}\frac{D(\mathbf{a}+{\stackrel{\sim }{a}}_j(S))}{D(\mathbf{a})}\text{if}z=1.\hfill \end{array}$$

4. Application to the unit sphere S^d

4.1. vMF Kernel Mixture Models

For classification using predictors X lying on the hypersphere

$$\mathbb{X}=S^d=\left\{x\in {\mathfrak{R}}^{d+1}:{\left|\right|x\left|\right|}^2\equiv \sum _{j=1}^{d+1}{x}_j^2=1\right\},$$

we recommend using a von Mises-Fisher (vMF) kernel in the mixture model (2.1) to induce a prior over (S^d × ). Although the other distributions on S^d described in [30] could be used, the vMF kernel provides a relatively simple and computationally tractable choice. As shown in Proposition 4.1, this kernel also satisfies the assumptions in §2.2 for building a flexible joint density model and for posterior consistency. For a proof, see [6].

The vMF distribution has the density ([37], [18], [39])

$$\text{vMF}(x;\mu ,\kappa )=c^{-1}(\kappa )exp(\kappa x^{\prime }\mu )(x,\mu \in S^d,\kappa \in [0,\infty ))$$ (4.1)

with respect to the invariant volume form on S^d, where

$$c(\kappa )=\frac{2{\pi }^{d/2}}{\mathrm{\Gamma }(\frac{d}{2})}{\int }_{-1}^{1}exp(\kappa t){(1-t^2)}^{d/2-1}dt$$

is its normalizing constant. This distribution has a unique extrinsic mean (as defined in [8]) of μ, thereby μ can be interpreted as the kernel location. The parameter κ is a measure of concentration with κ = 0 corresponding to the uniform distribution while as κ diverges to ∞, it converges weakly to δ_μ uniformly in μ. Sampling from vMF is straightforward using results in [36] and [40].

Proposition 4.1

(a) The vMF kernel K as defined in (4.1) satisfies assumptions A1 and A2. It satisfies A6 with a₁ = d/2+1 and A7 with a₂ = d/2. The compact metric-space S^d endowed with chord distance satisfies A8 with a₃ = d.

In the sequel, we will apply the general methods for classification and testing developed earlier in the paper to hyperspherical features.

4.2. MCMC Details

When features lie on S^d and we choose vMF kernels and priors as in §2.3, simplifications result in the MCMC steps for updating κ and μ. Letting P₀₁ = vMF(μ₀, κ₀), we obtain conjugacy for the full conditional of the kernel locations,

$$({\mu }_j\mid -)\stackrel{\mathcal{D}}{=}\text{vMF}({\overline{\mu }}_j,{\kappa }_j)$$

with μ̄_j = v_j/||v_j||, κ_j = ||v_j||, v_j = κ₀μ₀ + κX_j and X_j = Σ_i x_iI(S_i = j). As a default, we can set μ₀ equal to the x_n sample extrinsic mean or we can choose κ₀ = 0 to induce a uniform P₀₁. The full posterior of κ is

$$\pi (\kappa )c^{-n}(\kappa ){\kappa }^{-nd/2}{e}^{n\kappa }{\kappa }^{nd/2}exp\{-\kappa (n-\sum _i{x}_{S_i}^{\prime }{\mu }_{S_i})\}.$$ (4.2)

If we set

$$\pi (\kappa )\propto c^n(\kappa ){\kappa }^{a+\frac{nd}{2}-1}{e}^{-\kappa (n+b)},a,b>0,$$ (4.3)

the posterior simplifies to

$$\text{Gam}\left(a+\frac{nd}{2},b+n-\sum _i{x}_i^{\prime }{\mu }_{S_i}\right).$$ (4.4)

We can make the MCMC more efficient by marginalizing out μ while updating κ. In particular

$$\pi (\kappa \mid S)\propto c^{-n}(\kappa )\pi (\kappa )\prod _{j:ms(j)>0}c(||\kappa X_j+{\kappa }_0{\mu }_0||)$$

with ms(j) = Σ_i I(S_i = j). This is easy to compute in the case d = 2, κ₀ = 0 and π ~ Gam(a, b). Then it simplifies to

$$\begin{array}{r}\pi (\kappa \mid S)\propto \text{Gam}(\kappa ;n-\sum _{j}I(ms(j)>0)+a,n-\sum _j\left|\right|X_j\left|\right|+b)\\ \times {\{1-exp(-2\kappa )\}}^{-n}\prod _{j:ms(j)>0}\{1-exp(-2\kappa \left|\right|X_j\left|\right|)\}.\end{array}$$

For this choice, we suggest a Metropolis-Hastings proposal that corresponds to the gamma component. This leads to a high acceptance probability when κ is high, and has good performance in general cases we have considered.

In the predictive probability expression in (2.6), the integral simplifies to

$$\frac{a_j}{\sum a_i}{c}^{-1}(\kappa )c^{-1}({\kappa }_0)c(||\kappa X_{n+1}+{\kappa }_0{\mu }_0||).$$

5. Simulation Examples

5.1. Classification

We draw iid samples on S⁹ × , = {1, 2, 3} from

$$f_t(x,y)=(1/3)\sum _{l=1}^{3}I(y=l)\text{vMF}(x;{\mu }_l,200)$$

where μ₁ = (1, 0, …)^T, μ_j = cos(0.2)μ₁ + sin(0.2)v_j, j = 2, 3, v₂ = (0, 1, …)^T and $v_3={(0,0.5,\sqrt{0.75},0,\dots )}^T$. Hence, the three response classes y ∈ {1, 2, 3} are equally likely and the distribution of the features within each class is a vMF on S⁹ with distinct location parameters. We purposely chose the separation between the kernel locations to be small, so that the classification task is challenging.

We implemented the approach described in §2.3 to perform nonparametric Bayes classification. The hyperparameters were chosen to be w₀ = 1, P₀ = vMF(μ_n, 10) ⊗ Diri(1, 1, 1), μ_n being the feature sample extrinsic mean, and π as in (4.3) with a = 1, b = 0.1. Cross-validation is used to assess classification performance, with posterior computation applied to data from a training sample of size 200, and the results used to predict y given the x values for subjects in a test sample of size 100. The MCMC algorithm was run for 5 × 10⁴ iterations after a 10⁴ iteration burn-in. Based on examination of trace plots for the predictive probabilities of y for representative test subjects, the proposed algorithm exhibits good rates of convergence and mixing. Note that we purposely avoid examining trace plots for component-specific parameters due to label switching. The out-of-sample misclassification rates for categories y = 1, 2 and 3 were 18.9%, 9.7% and 12.5%, respectively, with the overall rate being 14%.

As an alternative method for flexible model-based classification, we considered a discriminant analysis approach, which models the conditional density of x given y as a finite mixture of 10-dimensional Gaussians. In the literature it is very common to treat data lying on a hypersphere as if the data had support in a Euclidean space to simplify the analysis. Using the EM algorithm to fit the finite mixture model, we encountered singularity problems when allowing more than two Gaussian components per response class. Hence, we present the results only for mixtures of one or two multivariate Gaussian components. In the one component case, we obtained class-specific misclassification rates of 27%, 12.9% and 18.8%, with the overall rate being 20%. The corresponding results for the two component mixture were 21.6%, 16.1% and 28.1% with an overall misclassification rate of 22%.

Hence, the results from a parametric Gaussian discriminant analysis and a mixture of Gaussians classifier were much worse than those for our proposed Bayesian nonparametric approach. There are several possible factors contributing to the improvement in performance. Firstly, the discriminant analysis approach requires separate fitting of different mixture models to each of the response categories. When the amount of data in each category is small, it is difficult to reliably estimate all these parameters, leading to high variance and unstable estimates. In contrast our approach of joint modeling of f_t using a DPM favors a more parsimonious representation. Secondly, inappropriately modeling the data as having support on a Euclidean space has some clear drawbacks. The size of the space over which the densities are estimated is increased from a compact subset S⁹ to an unbounded space ℜ¹⁰. This can lead to an inflated variance and difficulties with convergence of EM and MCMC algorithms. In addition, the properties of the approach are expected to be poor even in larger samples. As Gaussian mixtures give zero probability to the embedded hypersphere, one cannot expect strong posterior consistency.

5.2. Hypothesis Testing

We draw an iid sample of size 100 on S⁹ × , = {1, 2, 3}, from the distribution

$$f_t(x,y)=(1/3)\sum _{l=1}^{3}I(y=l)\sum _{j=1}^3{w}_{lj}\text{vMF}(x;{\mu }_j,200),$$

where μ_j, j = 1, 2, 3 are as in the earlier example and the weights {w_lj} are chosen so that w₁₁ = 1 and w_lj = 0.5 for l = 2, 3 and j = 2, 3. Hence, in group y = 1, the features are drawn from a single vMF density, while in groups y = 2 and 3, the feature distributions are equally weighted mixtures of the same two vMFs.

Letting f_j denote the conditional density of X given Y = j for j = 1, 2, 3, respectively, the global null hypothesis of no difference in the three groups is H₀: f₁ = f₂ = f₃, while the alternative H₁ is that they are not all the same. We set the hyperparameters as w₀ = 1, P₀ = vMF(μ_n, 10) ⊗ Diri(a), μ_n being the X-sample extrinsic mean, b = a = p̂ = (0.28, 0.36, 0.36) - the sample proportion of observations from each group, and a prior π on κ as in 4.3 with a = 1 and b = 0.1. We run the proposed MCMC algorithm for calculating the Bayes factor (BF) in favor of H₁ over H₀ for 6 ×10⁴ iterations updating cluster labels S in 4 blocks of 25 each every iteration. The starting value of S is obtained by the k-means algorithm (k = 10) applied to the X component of the sample using geodesic distance on S⁹ and we started with κ = 200. The trace plots exhibit good rate of convergence of the algorithm. After discarding a burn-in of 4 × 10⁴ iterations, the estimated BF was 2.23 × 10¹⁵, suggesting strong evidence in the data in favor of H₁. We tried multiple starting points and different hyperparameter choices and found the conclusions to be robust, with the estimated BFs not exactly the same but within an order of magnitude. We also obtained similar estimates using substantially shorter and longer chains.

We can also use the proposed methodology for pairwise hypothesis testing of H_0,ll′: f_l = f_l′ against the alternative H_1,ll′: f_l ≠ f_l′ for any two pairs, l, l′, with l ≠ l^′. The analysis is otherwise implemented exactly as in the global hypothesis testing case. The resulting BF in favor of H_1,ll′ over H_0,ll′ for the different possible choices of (l, l′) are shown in Table 1. We obtain very large BFs in testing differences between groups 1 and 2 and 1 and 3, but a moderately small BF for testing a difference between groups 2 and 3, suggesting mild evidence that these two groups are equal. These conclusions are all consistent with the truth. We have noted a general tendency for the BF in favor of the alternative to be large when the alternative is true even in modest sample sizes, suggesting a rapid rate of convergence under the alternative in agreement with our theoretical results. When the null is true, the BF appears to converge to zero based on empirical results in our simulations, but at a slow rate.

Table 1.

Nonparametric Bayes and frequentist test results for data simulated for three groups with the second and third groups identical.

groups	BF	p-value
(1,2,3)	2.3 × 1015	2 × 10−6
(1,2)	2.4 × 104	1.8 × 10−4
(1,3)	1.7 × 106	1.5 × 10−5
(2,3)	0.235	0.43

For comparison, we also considered a frequentist nonparametric test for detecting differences in the groups based on comparing the sample extrinsic means of the f_ls. The test statistic used has an asymptotic ${\mathcal{X}}_{d(L-1)}^2$ distribution where d = 9 is the feature space dimension and L is the number of groups that we are comparing. It takes the expression $n{\sum }_{j=1}^L{\widehat{p}}_j{\overline{X}}_j^{\prime }B{(B^{\prime }\widehat{\sum }B)}^{-1}{B}^{\prime }{\overline{X}}_j$ where n = 100 is the sample size, X̄_j is the X sample mean for group j, X̄ denotes the overall sample mean, Σ̂ is the sample covariance, and B is an orthonormal basis for the tangent space of S⁹ at μ_n = X̄/||X̄||. It is obtained via a Taylor expansion of the map x ↦ x/||x|| from ℜ^d+1 to S^d. For more details, see §4.8 of [4]. The corresponding p-values are shown in Table 1. The conclusions are all consistent with those from the nonparametric Bayes approach.

5.3. Testing with No Differences in Mean

In this example, we draw iid samples on S² × , = {1, 2} from the distribution

$$f_t(x,y)=(1/2)\sum _{l=1}^{2}I(y=l)\sum _{j=1}^3{w}_{lj}\text{vMF}(x;{\mu }_j,200),$$

where $w=\left[\begin{array}{ccc}1& 0& 0\\ 0& 0.5& 0.5\end{array}\right]$, μ₁ = (1, 0, 0)^T, μ_j = cos(0.2)μ₁ + sin(0.2)v_j (j = 2, 3) and v₂ = − v₃ = (0, 1, 0)^T. In this case the features are drawn from two groups equally likely, one of them is a vMF, while the other is a equally weighted mixture of two different vMFs. The locations μ_j are chosen such that both the groups have the same extrinsic mean μ₁.

We draw 10 samples of 50 observations each from the model f_t and carry out hypothesis testing to test for association between X and Y via our method and the frequentist one. The prior, hyperparameters and the algorithm for Bayes Factor (BF) computation are as in the earlier example. In each case we get insignificant p-values, often over 0.5, but very high BFs, often exceeding 10⁶. The values are listed in Table 2.

Table 2.

Nonparametric Bayes and frequentist test results for 10 simulations of 50 observations each for two groups with same population means.

BF	6.1e9	6.4e8	1.3e9	4.3e8	703.1	4.4e7	42.6	4.7e6	1.9e6	379.1
p-value	1.00	0.48	0.31	0.89	0.89	0.49	0.71	0.53	0.56	0.60

The reason for the failure of the frequentist test is because it relies on comparing the group specific sample extrinsic means and in this example the difference between them is little. Our method on the other hand compares the full conditionals and hence can detect differences that are not in the means.

6. Applications

6.1. Magnetization direction data

In this example from [16], measurements of remanent magnetization in red silts and claystones were made at 4 locations. This results in samples from four group of directions on the sphere S², the sample sizes are 36, 39, 16 and 16. The goal is to compare the magnetization direction distributions across the groups and test for any significant difference. Figure 1 shows the 3D plot of the sample clouds. The plot suggests no major differences. To test that statistically, we calculate the Bayes factor (BF) in favor of the alternative, as in §5.2. As mixing was not quite as good as in the simulated examples, we implemented label switching moves. We updated the cluster configurations in two blocks of size 54 and 53. The estimated BF was ≈1, suggesting no evidence in favor of the alternative hypothesis that the distribution of magnetization directions vary across locations.

Figure 1.

3D coordinates of 4 groups in §6.1: 1(r), 2(b), 3(g), 4(c).

To assess sensitivity to the prior specification, we repeated the analysis with different hyperparameter values of a, b equal to the proportions of samples within each group and P₀₁ corresponding to an uniform on the sphere. In addition, we tried different starting clusterings in the data, with a default choice obtained by implementing k-means with 10 clusters assumed. In each case, we obtain BF ≈1, so the results were robust.

In Example 7.7 of [17], a coordinate-based parametric test was conducted to compare mean direction in these data, producing a p-value of 1 − 1.4205 × 10⁻⁵ based on a ${\mathcal{X}}_6^2$ statistic. They also compared the mean directions for the first two groups and obtained a non-significant p-value. Repeating this two sample test using our Bayesian nonparametric method, we obtained a Bayes factor of 1.00. The nonparametric frequentist test from §5.2 yield p-values of 0.06 and 0.38 for the two tests.

6.2. Volcano location data

The NOAA National Geophysical Data Center Volcano Location Database contains information on locations and characteristics of volcanoes across the globe. The locations using latitude-longitude coordinates are plotted in Figure 2. We are interested in testing if there is any association between the location and type of the volcano. We consider the most common three types which are Strato, Shield and Submarine volcanoes, with data available for 999 volcanoes of these types worldwide. Their location coordinates are shown in Figure 3. Denoting by X the volcano location which lies on S² and by Y its type which takes values from = {1, 2, 3}, we compute the Bayes factor (BF) for testing if X and Y are independent.

Figure 2.

Longitude-Latitude coordinates of volcano locations in §6.2.

Figure 3.

Coordinates of 3 major type volcano locations: Strato(r), Shield(b), Submarine(g). Their sample extrinsic mean locations:1, 2, 3. Full sample extrinsic mean

As should be apparent from the Figures, the volcano data are particularly challenging in terms of density estimation because the locations tend to be concentrated along fault lines. Potentially, data on distance to the closest fault, volcano elevation and other information could be utilized to improve performance, but we do not have access to such data. It would be straight forward to include such additional predictors as explained in Remark 2.2. Without such information, the data present a challenging test case for the methodology in that it is clear that one may need to utilize very many vMF kernels to accurately characterize the density of volcano locations across the globe, with the use of moderate to large numbers of kernels leading to challenging mixing issues. Indeed, we did encounter a sensitivity to the starting cluster configuration in our initial analyses.

We found that one of issues that exacerbated the problem with mixing of the cluster allocation was the ordering in the weights on the stick-breaking representation utilized by the exact block Gibbs sampler. Although label switching moves can lead to some improvements, they proved to be insufficient in this case. Hence, we modified the computational algorithm slightly to instead use the finite Dirichlet approximation to the Dirichlet process proposed in [27]. The finite Dirichlet treats the components as exchangeable so eliminates sensitivity to the indices on the starting clusters, which we obtained using k-means for 50 clusters. We used K = 50 as the dimension of the finite Dirichlet and hence the upper bound on the number of occupied clusters. Another issue that lead to mixing problems was the use of a hyperprior on κ. In particular, when the initial clusters were not well chosen, the kernel precision would tend to drift towards smaller than optimal values and as a result too few clusters would be occupied to adequately fit the data. We did not observe such issues at all in a variety of other simulated and real data applications, but the volcano data are particularly difficult as we note above.

To address this second issue, we chose the kernel precision parameter κ by cross-validation. In particular, we split the sample into training and test sets, and then ran our Bayesian nonparametric analysis on the training data separately for a wide variety of κ values between 0 and 1,000. We chose the value that produced the highest expected posterior log likelihood in the test data, leading to κ̂ = 80. In this analysis and the subsequent analyses for estimating the BF, we chose the prior on the mixture weights to be Diri(w₀/K1_K) (K = 50). The other hyper-parameters were chosen to be w₀ = 1, a = b = (0.71, 0.17, 0.11) = the sample proportion of different volcano types, κ₀ = 10, and μ₀ = the X-sample extrinsic mean. We collected 5 × 10⁴ MCMC iterations after discarding a burn-in of 10⁴. Using a fixed band-width considerably improved the algorithm convergence rate.

Based on the complete data set of 999 volcanoes, the resulting BF in favor of the alternative was estimated to be over 10¹⁰⁰, providing conclusive evidence that the different types of volcanos have a different spatial distribution across the globe. For the same fixed κ̂ value, we reran the analysis for a variety of alternative hyperparameter values and different starting points, obtaining similar BF estimates and the same conclusion. We also repeated the analysis for a randomly selected subsample of 300 observations, obtaining BF = 5.4 × 10¹¹. The testing is repeated for other sub-samples, each resulting in a very high BF. We also obtained a high BF in repeating the analysis with a hyperprior on κ.

For comparison, we perform the asymptotic test as described in §5.2, obtaining a p-value of 3.6 × 10⁻⁷ which again favors H₁. The large sample sizes for the three types (713,172,114) justifies the use of asymptotic theory. However given that the volcanoes are spread all over the globe, the validity of the assumption that the three conditionals have unique extrinsic means may be questioned.

We also perform a coordinate based test by comparing the means of the latitude longitude coordinates of the three sub-samples using a statistic. The three coordinate means are (12.6, 27.9), (21.5, 9.2), and (9.97, 21.5) (latitude, longitude). The value of the statistic is 17.07 and the asymptotic p-value equals 1.9 × 10⁻³ which is larger by orders of magnitude than its coordinate-free counterpart, but still significant. Coordinate based methods, however, can be very misleading because of the discontinuity at the boundaries. They heavily distort the geometry of the sphere which is evident from the figures.

7. Discussion

We have proposed a novel Bayesian approach for classification and testing relying on modeling the joint distribution of the categorical response and continuous predictors as a Dirichlet process product mixture. The product mixture likelihood includes a multinomial for the categorical response and an arbitrary kernel for the predictors, with dependence induced through the DP prior on the unknown joint mixing measure. By modifying the kernel for the predictors, one can modify the support, with multivariate Gaussian kernels for predictors in ℜ^p and von Mises Fisher kernels for predictors on the hypersphere. For other predictor spaces, one can appropriately modify the kernel.

Although our focus has been on hyperspherical predictors for concreteness, the proposed product mixture formulation is broadly applicable to classification problems for predictors in general spaces and we can easily consider predictors having a variety of supports. For example, some predictors can be in a Euclidean space and some on a hypersphere. The framework has some clear practical advantages over frequentist and nonparametric Bayes discriminant analysis approaches, which rely on separately modeling the conditional distributions of the feature (predictor) distributions specific to each response category.

One of our primary contributions was showing theoretical properties, including large support and posterior consistency, in modeling of the classification function. In addition, we have added to the under-developed literature on nonparametric Bayes testing of differences in distributions, not only on ℜ^p but on more general manifolds. We provide a novel computational approach for estimating Bayes factors as well as prove theoretical results on Bayes factor consistency. The proposed method can be implemented in broad applications.

An area of substantial interest in ongoing research is the development of models that reduce dimensionality through projecting predictors X to a lower-dimensional subspace. Such dimensionality reduction is critical in addressing the curse of dimensionality that arises inevitably in estimating classification or regression functions with high-dimensional predictors. [7] develop a promising approach for nonparametric Bayes learning of affine subspaces for density estimation and classification in Euclidean spaces, but it remains to develop related methods in non-Euclidean spaces while also obtaining theory on rates of convergence.

8. Appendix

8.1. Proof of Theorem 2.1

Before proving the Theorem, we prove the following Lemma.

Lemma 8.1

Under assumptions A2 and A4,

$$\underset{\kappa \to \infty }{lim}sup\{\mid f(x,y;P_t,\kappa )-f_t(x,y)\mid :(x,y)\in \mathbb{X}\times \mathbb{Y}\}=0.$$

Proof

From the definition of P_t, we can write

$$f(x,y;P_t,\kappa )=\underset{\mathbb{X}}{\int }K(x;\mu ,\kappa ){\phi }_y)(\mu ){\lambda }_1(d\mu )$$

for φ_y(μ) = f_t(μ, y). Then from A4, it follows that φ_y is continuous for all y ∈ . Hence from A2, it follows that

$$\underset{\kappa \to \infty }{lim}\underset{x\in \mathbb{X}}{sup}\left|f_t(x,y)-\underset{\mathbb{X}}{\int }K(x;\mu ,\kappa )f_t(\mu ,y){\lambda }_1(d\mu )\right|=0$$

for any y ∈ . Since is finite, the proof is complete.

Proof of Theorem 2.1

Throughout this proof we will view as a compact metric space and ( × S_L−1) as a topological space under the weak topology. From Lemma 8.1, it follows that there exists a κ_t ≡ κ_t(ε) > 0 such that

$$\underset{x,y}{sup}\mid f(x,y;P_t,\kappa )-f_t(x,y)\mid <\frac{\epsilon }{3}$$ (8.1)

for all κ ≥ κ_t. From assumption A3, it follows that by choosing κ_t sufficiently large, we can ensure that (P_t, κ_t) ∈ supp(Π₁). From assumption A1, it follows that K is uniformly continuous at κ_t, i.e. there exists an open set W (ε) ⊆ ℜ⁺ containing κ_t s.t.

$$\underset{x,\mu \in \mathbb{X}}{sup}\mid K(x;\mu ,\kappa )-K(x;\mu ,{\kappa }_t)\mid <\frac{\epsilon }{3}\forall \kappa \in W(\epsilon ).$$

This in turn implies that, for all κ ∈ W(ε), P ∈ ( × S_L−1),

$$\underset{x,y}{sup}\mid f(x,y;P,\kappa )-f(x,y;P,{\kappa }_t)\mid <\frac{\epsilon }{3}$$ (8.2)

because the left expression in (8.2) is

$$\underset{x,y}{sup}\left|\int {\nu }_y\{K(x;\mu ,\kappa )-K(x;\mu ,{\kappa }_t)\}P(d\mu d\nu )\right|\le \underset{x,\mu \in \mathbb{X}}{sup}\mid K(x;\mu ,\kappa )-K(x;\mu ,{\kappa }_t)\mid .$$

Since is compact and K(.; ., κ_t) is uniformly continuous on × , we can cover by finitely many open sets: U₁, … U_K s.t.

$$\underset{\mu \in \mathbb{X},x,\stackrel{\sim }{x}\in U_i}{sup}\mid K(x;\mu ,{\kappa }_t)-K(\stackrel{\sim }{x};\mu ,{\kappa }_t)\mid <\frac{\epsilon }{12}$$ (8.3)

for each i ≤ K. For fixed x, y, κ, f(x, y; P, κ) is a continuous function of P. Hence for x_i ∈ U_i, y = j ∈ ,

$${\mathcal{W}}_{ij}(\epsilon )=\{P\in \mathcal{M}(\mathbb{X}\times S_{L-1}):\mid f(x_{i,}j;P,{\kappa }_t)-f(x_i,j;P_t,{\kappa }_t)\mid <\frac{\epsilon }{6}\},$$

1 ≤ i ≤ K, 1 ≤ j ≤ L, define open neighborhoods of P_t. Let (ε) = ∩_i,j (ε) which is also an open neighborhood of P_t. For a general x ∈ , y ∈ , find U_i containing x. Then for any P ∈ (ε),

$$\begin{array}{r}\mid f(x,y;P,{\kappa }_t)-f(x,y;P_t,{\kappa }_t)\mid \le \\ \mid f(x,y;P,{\kappa }_t)-f(x_i,j;P,{\kappa }_t)\mid +\mid f(x_i,j;P,{\kappa }_t)-f(x_i,j;P_t,{\kappa }_t)\mid \\ +\mid f(x_i,j;P_t,{\kappa }_t)-f(x,y;P_t,{\kappa }_t)\mid .\end{array}$$ (8.4)

Denote the three terms to the right in (8.4) as T₁, T₂ and T₃. Since x ∈ U_i, it follows from (8.3) that $T_1,T_3<\frac{\epsilon }{12}$. Since P ∈ (ε), $T_2<\frac{\epsilon }{6}$ by definition of (ε). Hence ${sup}_{x,y}\mid f(x,y;P,{\kappa }_t)-f(x,y;P_t,{\kappa }_t)\mid <\frac{\epsilon }{3}$. Therefore

$${\mathcal{W}}_2(\epsilon )\equiv \{P:\underset{x,y}{sup}\mid f(x,y;P,{\kappa }_t)-f(x,y;P_t,{\kappa }_t)\mid <\frac{\epsilon }{3}\}$$

contains (ε). Since (P_t, κ_t) ∈ supp(Π₁) and (ε) × W (ε) contains an open neighborhood of (P_t, κ_t), therefore

$${\mathrm{\Pi }}_1({\mathcal{W}}_2(\epsilon )\times W(\epsilon ))>0.$$

Let (P, κ) ∈ (ε) × W (ε). Then for (x, y) ∈ × ,

$$\begin{array}{r}\mid f(x,y;P,\kappa )-f_t(x,y)\mid \le \\ \mid f(x,y;P,\kappa )-f(x,y;P,{\kappa }_t)\mid +\mid f(x,y;P,{\kappa }_t)-f(x,y;P_t,{\kappa }_t)\mid \\ +\mid f(x,y;P_t,{\kappa }_t)-f_t(x,y)\mid .\end{array}$$ (8.5)

The first term to the right in (8.5) is $<\frac{\epsilon }{3}$ since κ ∈ W (ε). The second one is $<\frac{\epsilon }{3}$ because P ∈ (ε). The third one is also $<\frac{\epsilon }{3}$ which follows from equation (8.1). Therefore

$${\mathrm{\Pi }}_1\left(\{(P,\kappa ):\underset{x,y}{sup}\mid f(x,y;P,\kappa )-f_t(x,y)\mid <\epsilon \}\right)>0.$$

This completes the proof.

8.2. Proof of Theorem 2.3

The proof uses Proposition 8.2 proved in [6]. Let M be a compact metric-space. Denote by (M) the space of all probability densities on M with respect to some fixed finite base measure τ. Endow it with the total variation distance ||.||. Let ${\mathbf{z}}_n={\{z_i\}}_1^n$ be a iid sample from some density f_t on M. Consider a collection of mixture densities on M given by

$$f(m;P,\kappa )={\int }_{M}K(m;\mu ,\kappa )P(d\mu ),m\in M,\kappa \in {\mathfrak{R}}^{+},P\in \mathcal{M}(M)$$ (8.6)

with ∫_M K(m; μ, κ)τ(dm) = 1. Set a prior Π₁ on (M) × ℜ⁺ which induces a prior Π on (M) through (8.6). For ⊆ (M) and ε > 0, the L₁-metric entropy N(ε, ) is defined as the logarithm of the minimum number of ε-sized (or smaller) L₁ subsets needed to cover .

Proposition 8.2

For a positive sequence {κ_n} diverging to ∞, define

$${\mathcal{D}}_n=\{f(.;P,\kappa ):P\in \mathcal{M}(M),\kappa \in [0,{\kappa }_n]\}.$$

(a) Under assumptions A6–A8, given any ε > 0, for n sufficiently large, $N(\epsilon ,{\mathcal{D}}_n)\le C(\epsilon ){\kappa }_n^{a_1{a}_3}$ for some C(ε) > 0. (b) Under further assumption A9, the posterior probability of any total variation neighborhood of f_t converges to 1 a.s. f_t if f_t is in the KL support of Π.

Proof of Theorem 2.3

For a density f ∈ ( × ), let p(y) be the marginal probability of Y = y and g(x, y) be the conditional density of X = x given Y = y. Then f(x, y) = p(y)g(x, y). For f₁, f₂ ∈ ( × ),

$$\begin{array}{r}\left|\right|f_1-f_2\left|\right|=\int \mid f_1(x,y)-f_2(x,y)\mid \lambda (\mathit{dxdy})=\sum _{j=1}^L\mid p_1(j)g_1(x,j)-p_2(j)g_2(x,j)\mid {\lambda }_1(dx)\\ \le \underset{j}{max}\left|\right|g_1(.,j)-g_2(.,j)\left|\right|+\sum _j\mid p_1(j)-p_2(j)\mid .\end{array}$$ (8.7)

Hence an ε diameter ball in ( × ) contains the intersection of L many ε/2 diameter balls from ( ) with a ε/2 diameter subset of S_L−1. For any f (; P, κ) as in (2.1), its X-conditional g(.; j) for j ∈ can be expressed as

$$\begin{array}{r}g(x,j)=\frac{{\int }_{\mathbb{X}\times S_{L-1}}{\nu }_{j}K(x;\mu ,\kappa )P(d\mu d\nu )}{{\int }_{\mathbb{X}\times S_{L-1}}{\nu }_{j}P(d\mu d\nu )}={\int }_{\mathbb{X}}K(x;\mu ,\kappa )P_j(d\mu )\\ \text{with}{P}_j(d\mu )=\frac{{\int }_{S_{L-1}}{\nu }_{j}P(d\mu d\nu )}{{\int }_{\mathbb{X}\times S_{L-1}}{\nu }_{j}P(d\mu d\nu )}.\end{array}$$

Hence g(., j) is as in (8.6) with M = . Define

$$\begin{array}{r}{\mathcal{D}}_n=\{f(;P,\kappa ):P\in \mathcal{M}(\mathbb{X}\times \mathbb{Y}),\kappa \in [0,{\kappa }_n]\}.\text{Then}\\ {\mathcal{D}}_n=\{f\in \mathcal{D}(\mathbb{X}\times \mathbb{Y}),:g(.;j)\in {\stackrel{\sim }{\mathcal{D}}}_n\forall j\in \mathbb{Y}\}\\ \text{where}{\stackrel{\sim }{\mathcal{D}}}_n=\{g(;P,\kappa ):P\in \mathcal{M}(\mathbb{X}),\kappa \in [0,{\kappa }_n]\}.\end{array}$$ (8.8)

From Proposition 8.2(a), N(ε, ) is of order at-most ${\kappa }_n^{a_1{a}_3}$ and hence from (8.7) and (8.8), $N(\epsilon ,{\mathcal{D}}_n)\le C{\kappa }_n^{a_1{a}_3}$, C depending on ε. Therefore from part (b) of the Proposition, under assumptions A1–A9, strong posterior consistency follows.

8.3. Proof of Corollary 2.4

Proof

1. For any y ∈ ,

   $$\begin{array}{r}{\int }_{\mathbb{X}}\mid p(y,x)-p_t(y,x)\mid g_t(x){\lambda }_1(dx)=\\ {\int }_{\mathbb{X}}\mid f_t(x,y)-f(x,y)+p(y,x)g(x)-p(y,x)g_t(x)\mid {\lambda }_1(dx)\\ \le {\int }_{\mathbb{X}}\mid f_t(x,y)-f(x,y)\mid {\lambda }_1(dx)+{\int }_{\mathbb{X}}\mid g_t(x)-g(x)\mid {\lambda }_1(dx)\\ \le 2\sum _{j=1}^L{\int }_{\mathbb{X}}\mid f(x,j)-f_t(x,j)\mid {\lambda }_1(dx)={\left|\right|f-f_t\left|\right|}_1\end{array}$$

   and hence any neighborhood of p_t of the form {p: max_y∈ ∫|p(y, x)− p_t(y, x)|g_t(x)λ₁(dx) < ε} contains an L¹ neighborhood of f_t. Now part(a) follows from strong consistency of the posterior distribution of f.

2. Since is compact, f_t being continuous and positive implies that c = inf_x∈ g_t(x) > 0. Hence

   $${\int }_{\mathbb{X}}\mid p(y,x)-p_t(y,x)\mid w(x){\lambda }_1(dx)\le c^{-1}sup(w(x)){\int }_{\mathbb{X}}{g}_t(x)\mid p(y,x)-p_t(y,x)\mid {\lambda }_1(dx)$$

   Now the result follows from part (a).

8.4. Proof of Theorem 3.1

The proof uses Lemma 8.3. This lemma is fundamental to proving weak posterior consistency using the Schwartz theorem and its proof can be found in any standard text which contains the theorem’s proof.

Lemma 8.3

(a) If Π includes f_t in its KL support, then

$$lim{inf}_{n\to \infty }exp(n\beta )\int \prod _i\frac{f(x_i,y_i)}{f_t(x_i,y_i)}\mathrm{\Pi }(df)=\infty$$

a.s. $f_t^{\infty }$ for any β > 0. (b) If U is a weak open neighborhood of f_t and Π₀ is a prior on ( × ) with support in U^c, then there exists a β₀ > 0 for which

$${lim}_{n\to \infty }exp(n{\beta }_0)\int \prod _i\frac{f(x_i,y_i)}{f_t(x_i,y_i)}{\mathrm{\Pi }}_0(df)=0$$

a.s. $f_t^{\infty }$.

Proof of Theorem 3.1

Express BF as

$$BF=\{\prod _i{p}_t(y_i)\}\frac{D(\mathbf{b})}{D({\mathbf{b}}_n)}\frac{\int {\prod }_i\frac{f(x_i,y_i)}{f_t(x_i,y_i)}\mathrm{\Pi }(df)}{\int {\prod }_i\frac{g(x_i)p_t(y_i)}{f_t(x_i,y_i)}\mathrm{\Pi }(df)}=T_1{T}_2/T_3$$

with $T_1=\{{\prod }_i{p}_t(y_i)\}\frac{D(\mathbf{b})}{D({\mathbf{b}}_n)},T_2=\int {\prod }_i\frac{f(x_i,y_i)}{f_t(x_i,y_i)}\mathrm{\Pi }(df)$, and $T_3=\int {\prod }_i\frac{g(x_i)p_t(y_i)}{f_t(x_i,y_i)}\mathrm{\Pi }(df)$.

Since Π satisfies the KL condition, Lemma 8.3(a) implies that lim inf_n→∞ exp(nβ)T₂ = ∞ a.s. for any β > 0.

Let U be the space of all dependent densities, that is

$$U^c=\{f\in \mathcal{D}(\mathbb{X}\times \mathbb{Y}):f(x,y)=g(x)p(y)\mathbf{a}.\text{s}.\lambda (\mathit{dxdy})\}.$$

The prior Π induces a prior Π₀ on U^c via f ↦ {Σ_j f (., j)}p_t and T₃ can be expressed as $\int {\prod }_i\frac{f(x_i,y_i)}{f_t(x_i,y_i)}{\mathrm{\Pi }}_0(df)$. It is easy to show that U is open under the weak topology and hence under H₁ is a weak open neighborhood of f_t. Then using Lemma 8.3(b), it follows that lim_n→∞ exp(nβ₀)T₃ = 0 a.s. for some β₀ > 0.

The proof is complete if we can show that lim inf_n→∞ exp(nβ)T₁ = ∞ a.s. for any β > 0 or log(T₁) = o(n) a.s. For a positive sequence a_n diverging to ∞, the Stirling’s formula implies that log Γ(a_n) = a_n log(a_n) − a_n + o(a_n). Express log(T₁) as

$$\sum _{i}log(p_t(y_i))-log(D({\mathbf{b}}_n))+o(n).$$ (8.9)

Since p_t(j) > 0 ∀j ≤ L, by the SLLN,

$$\sum _{i}log(p_t(y_i))=n\sum _j{p}_t(j)log(p_t(j))+o(n)\mathbf{a}.\text{s}.$$ (8.10)

Let b_nj = b_j +Σ_i I(y_i = j) be the jth component of b_n. Then lim_n→∞ b_nj/n = p_t(j), that is b_nj = np_t(j) + o(n) a.s. and hence the Stirling’s formula implies that

$$\begin{array}{r}log(\mathrm{\Gamma }(b_{nj}))=b_{nj}log(b_{nj})-b_{nj}+o(n)\\ ={np}_t(j)log(p_t(j))-{np}_t(j)+log(n)b_{nj}+o(n)\mathbf{a}.\text{s}.\end{array}$$

which implies

$$\begin{array}{r}log(D({\mathbf{b}}_n))=\sum _{j=1}^{L}log(\mathrm{\Gamma }(b_{nj}))-log\mathrm{\Gamma }(\sum _j{b}_j+n)\\ =n\sum _j{p}_t(j)log(p_t(j))+o(n)\mathbf{a}.\text{s}.\end{array}$$ (8.11)

From (8.9), (8.10) and (8.11), log(T₁) = o(n) a.s. and this completes the proof.