Bayesian isotonic density regression

Abstract

Density regression models allow the conditional distribution of the response given predictors to change flexibly over the predictor space. Such models are much more flexible than nonparametric mean regression models with nonparametric residual distributions, and are well supported in many applications. A rich variety of Bayesian methods have been proposed for density regression, but it is not clear whether such priors have full support so that any true data-generating model can be accurately approximated. This article develops a new class of density regression models that incorporate stochastic-ordering constraints which are natural when a response tends to increase or decrease monotonely with a predictor. Theory is developed showing large support. Methods are developed for hypothesis testing, with posterior computation relying on a simple Gibbs sampler. Frequentist properties are illustrated in a simulation study, and an epidemiology application is considered.

1. Introduction

In studying the relationship between a continuous predictor x and a response variable y, it is common to have prior knowledge of stochastic ordering. For example, in environmental epidemiology studies, one can assume that the severity of an adverse health outcome is stochastically nondecreasing with dose of a potentially adverse exposure. Incorporation of stochastic-ordering constraints can improve the efficiency in estimating conditional distributions and in testing for local or global changes as x increases. The focus of this article is on addressing these problems using nonparametric Bayesian methods.

Letting F_x (y) = pr(Y ⩽ y | X = x) denote the conditional distribution function of the response given the predictor, we assume prior knowledge is available that F_x is stochastically smaller than F_x′ for any two points x ⩽ x′ in 𝒳, with 𝒳 the real line or an interval on the real line. Such a restriction is natural when the response is known to increase monotonically with the predictor. We otherwise have substantial uncertainty about {F_x, x ∈ 𝒳}, with the true conditional distribution potentially having any form subject to the stochastic-ordering restriction. It is appealing to choose a prior with full support on the space of stochastically ordered conditional distributions. Full support means that the prior can generate conditional distributions within arbitrarily small neighbourhoods of any true conditional distribution. If the true conditional distribution is not in the support of the prior, the posterior will not concentrate on small neighbourhoods of the truth, and hence an unrealistic and inconsistent estimate will be obtained.

Although using a prior with large support is critical in obtaining robust inferences and appropriately accounting for uncertainty in the true data-generating model, few results are available characterizing the support of the prior in Bayesian density regression models even in the unconstrained case. An important exception is the recent article of Tokdar et al. (2010), who showed large support and posterior consistency in unconstrained density estimation based on a class of logistic Gaussian process priors. Their approach was specific to the logistic Gaussian process, relying heavily on theoretical properties of Gaussian processes. Norets (2010) showed large Kullback–Leibler support for finite mixtures of normal regressions, with the mixing weights potentially dependent on predictors, and unpublished work by D. Pati and D. B. Dunson from Duke University and A. Norets and J. Pelenis from Princeton University independently obtained large support and posterior consistency results for various density regression priors. All of the previously proposed Bayesian models for incorporating monotonicity constraints in nonparametric regression with a continuous predictor violate the full support condition, and make restrictive assumptions about the nature of dependence on the predictor.

For example, the typical focus in the literature has been on nonparametric regression models of the form y_i = f (x_i) + ∊_i, where f is an unknown nondecreasing function and ∊_i ∼ F is a residual. One can then induce a prior on the conditional distribution through independent priors for the regression function f and residual distribution F. Using this general idea, Lavine & Mockus (1995) allowed both f and F to be unknown using Dirichlet process priors (Ferguson, 1973, 1974); Neelon & Dunson (2004) instead used adaptive linear splines for f ; Holmes & Held (2006) used a piecewise constant model; Wang (2008) used approximated monotone functions with cubic splines allowing an unknown number and location of knots; Bornkamp & Ickstadt (2009) modelled the monotone function with a mixture of parametric probability distributions with the mixing measure assigned a stick-breaking prior (Ongaro & Cattaneo, 2004); Shively et al. (2009) proposed Bayesian nonparametric approaches relying on the characterization of monotone functions proposed by Ramsay (1998) and quadratic regression splines. Even choosing flexible priors for f and F leads to a prior on {F_x, x ∈ 𝒳} with small support, as conditional distributions having residual variance or shape varying with x are not supported.

There has been some work on more flexible nonparametric Bayes methods for categorical x. Arjas & Gasbarra (1996) characterized stochastically ordered survival functions in two groups using a piecewise constant hazard model on a fixed grid, with stochastic ordering implying a restriction on the interval-specific hazards. Gelfand & Kottas (2001) instead expressed the distribution functions in two groups as F₁ = G₁ and F₂ = G₁G₂, with independent Dirichlet process (Ferguson, 1973, 1974) mixture priors used for G₁ and G₂. Although this approach enforces stochastic ordering, the implied constraint is more restrictive and many stochastically ordered distributions are not supported. Karabatsos & Walker (2007) proposed alternative models for stochastically ordered distributions based on Pólya tree (Ferguson, 1974) or Bernstein polynomial (Petrone, 1999) priors, but did not show large support. Without support results for the above priors, it is not clear whether the priors rule out a large subset of the set of stochastically ordered distributions a priori.

Hoff (2003a) proposed a latent variable specification in which individual i has potential outcomes in each of K groups, with these outcomes following a partial ordering. By placing a Dirich-let process prior on the joint distribution of the potential outcomes, he induced a prior on the marginal distributions in each group having full support on the space of distributions satisfying a partial stochastic ordering. Dunson & Peddada (2008) generalized this approach to a mixture specification by proposing a restricted dependent Dirichlet process, while also developing methods for hypothesis testing. The models, theory of large support and algorithms for computation are not straightforward to modify to accommodate continuous predictors. The continuous case is much more challenging, since instead of a finite collection of unknown distributions, we are faced with an uncountable collection.

2. Stochastically ordered densities

2.1. The restricted dependent Dirichlet process mixture

Let f_𝒳 = {f_x, x ∈ 𝒳} denote a collection of conditional densities indexed by the predictor x ∈ 𝒳. To induce a prior on f_𝒳, we consider the mixture model

$$f_x\hspace{0.17em}(y;P)=\int {\tau }^{1/2}\varphi \left[{\tau }^{1/2}\left\{y-\mu (x)\right\}\right]\hspace{0.17em}d{P}_x\left\{\mu (x),\tau \right\},$$ (1)

where ϕ(·) denotes the standard normal density, μ = {(x), x ∈ 𝒳} is a real-valued stochastic process over 𝒳, τ ∈ ℜ⁺ is a scalar precision parameter and P = (P_x, x ∈ 𝒳) is a collection of predictor-dependent mixing measures.

A prior for f_𝒳 can be induced through a prior for P. One such prior is the fixed p-dependent Dirichlet process (MacEachern, 1999), which can be expressed in stick-breaking form as

$$\begin{array}{cc}{P}_x=\sum _{h=1}^{\infty }{p}_h{\delta }_{{\mathrm{\Theta }}_h(x)},& p_h=V_h\prod _{l<h}\left(1-V_l\right),\end{array}$$ (2)

where δ_a is a Dirac probability measure concentrated at a, V_h ∼ Be(1, α) and Θ_h = (μ̃_h, τ̃_h) ∼ M are mutually independent for h = 1, . . . , ∞, and M = H ⊗ L is a baseline probability measure. For this choice of prior for P = (P_x, x ∈ 𝒳), (1) becomes

$$f_x\hspace{0.17em}(y;P)=\sum _{h=1}^{\infty }{p}_h{\tilde{\tau }}_h^{1/2}\varphi \left[{\tilde{\tau }}_h^{1/2}\left\{y-{\tilde{\mu }}_h(x)\right\}\right],$$ (3)

and the conditional density f_x is marginally assigned a Dirichlet process location-scale mixture of Gaussians prior (Lo, 1984; Escobar & West, 1995). The f_x and f_x′ are not independent under this dependent Dirichlet process mixture prior but are correlated due to the common mixture weights {p_h} and scale parameters {τ̃_h} and to dependence in the kernel locations μ̃_h(x) and μ̃_h(x′). By letting τ̃_h = τ to fix the kernel bandwidth across the mixture components, the model simplifies to a location mixture. Allowing the bandwidth to vary across components has some well-known advantages in terms of sparsely characterizing unknown densities.

The above model specification closely follows that proposed in previous work. In an as-yet unpublished paper, D. Pati and D. B. Dunson recently provided conditions for large support and weak and strong posterior consistency for fixed p-dependent Dirichlet processes for unconstrained density regression. Their theory does not apply automatically to the constrained case, and it is not clear whether the prior can be specified in such a way as to restrict support to a large subset of the space of stochastically nondecreasing densities.

Assumption 1. Assume the following continuity condition holds: f_x (·) → f_x′ (·) as x → x′, for any x, x′ ∈ 𝒳.

From an applied perspective, it is appealing to focus on conditional densities satisfying the continuity condition stated in Assumption 1. This regularity condition rules out sudden jumps in f_x as x changes.

Letting (y_i, x_i) denote the data for subject i, for i = 1, . . . , n, (3) implies that

$$y_i~N\{{\mu }_i(x_i),{\tau }_i^{-1}\},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}({\mu }_i,{\tau }_i)~P,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}P=\sum _{h=1}^{\infty }{p}_h{\delta }_{({\tilde{\mu }}_h,{\tilde{\tau }}_h)}.$$ (4)

Hence, to sample y_i first generate a random function μ_i : 𝒳 → ℜ and precision τ_i ∈ ℜ⁺ from a discrete distribution, which is assigned a Dirichlet process prior. Then, draw y_i from a normal distribution with precision ${\tau }_i^{-1}$ centred on the realization of μ_i at predictor value x_i. The function μ_i can be viewed as a latent trajectory defining the mean of the normal kernel for individual i at each possible value of the predictor. With this interpretation, a natural strategy for restricting the induced conditional density to be nondecreasing with x is to restrict the μ_i s to be nondecreasing functions. To also incorporate the continuity condition, one can draw μ̃_h ∼ H independently from τ̃_h ∼ L, with H chosen as a probability measure on S_𝒳,c, the space of continuous nondecreasing 𝒳 → ℜ functions.

Theorem 1 states that restricting H to have support on S_𝒳,c is a necessary and sufficient condition for the collection of conditional densities to be stochastically nondecreasing in the index x and subject to the continuity condition with probability one.

Theorem 1. Let D_𝒳 denote the space of collections of densities indexed by x ∈ 𝒳 and subject to Assumption 1 and the nondecreasing stochastic ordering constraint. Under (2) and (3), the following statements are equivalent: (a) f_𝒳 ∈ D_𝒳 with probability one; and (b) H is a probability measure with support on S_𝒳,c.

Theorem 1 also holds for a simplified version of (3) with common unknown variance in all of the normal components.

2.2. Hypothesis testing

We also develop methods for testing equalities in conditional distributions against stochastically ordered alternatives. Letting D(x, x′) = sup_y∈ℜ |F_x (y) − F_x′ (y)| as a measure of distance between $F_x\hspace{0.17em}(y)={\int }_{-\infty }^y\hspace{0.17em}{f}_x\hspace{0.17em}(z)\hspace{0.17em}dz$ and $F_{x^{\prime }}\hspace{0.17em}(y)={\int }_{-\infty }^y\hspace{0.17em}{f}_{x^{\prime }}\hspace{0.17em}(z)dz$ for x < x′, we consider the hypotheses H₀(x, x′) : D(x, x′) < ∊ versus H₁(x, x′) : D(x, x′) ⩾ ∊, where ∊ is a small positive constant chosen so that D(x, x′) < ∊ implies that F_x and F_x′ are not significantly different. We recommend ∊ = 0.05 as a default value. As a weight of evidence in favour of H₁(x, x′), we use the Bayes factor. Fixing x at the minimum predictor value observed and varying x′, we obtain a log Bayes factor function, which is useful for identifying threshold predictor values, a common interest in biomedical studies. Global hypothesis tests of near equality in the conditional distributions across a region A = [a, b] ⊂ 𝒳 can be formulated as H₀(A) = ∩_x,x′∈A H₀(x, x′). We use the supremum of the log Bayes factor function as a weight of evidence in favour of H₁(A).

3. Prior specification and posterior computation

To complete a specification of the prior under the model proposed in § 2.1, first let L correspond to the Ga(a_τ, b_τ) distribution and choose α ∼ Ga(a_α, b_α) to allow the Dirichlet process concentration parameter to be unknown. In addition, H is chosen to induce a prior for f_𝒳 with support on a large subset of D_𝒳 by letting

$${\tilde{\mu }}_h\hspace{0.17em}(x)=\sum _{l=0}^k{\beta }_{hl}\hspace{0.17em}{b}_l\hspace{0.17em}(x)$$ (5)

for all x ∈ 𝒳, with b₀(x) = 1, ${\left\{b_l\right\}}_{l=1}^k$ monotone spline basis functions (Ramsay, 1988), and ${\left\{{\beta }_{hl}\right\}}_{l=0}^k$ basis coefficients having nonnegative constraint for l ⩾ 1. We focus in our applications on splines of degree 1, leading to piecewise linearity. It is well known that any continuous function can be approximated arbitrarily well by a piecewise linear function with sufficiently many knots.

Potentially, one can treat the number and locations of knots as unknown and adaptively determine their values by using reversible jump Markov chain Monte Carlo (Green, 1995). To facilitate efficient computation, we instead choose a large number of equally-spaced knots and then adaptively drop unnecessary basis functions by letting their coefficients have zero values (Smith & Kohn, 1996). This is accomplished through the following prior for β_h,

$$\pi \left({\beta }_h;m_0,{\nu }_0,\lambda ,{\pi }_0\right)=N({\beta }_{h0};m_0,{\nu }_0^{-1})\prod _{l=1}^k\left\{{\pi }_0{\delta }_0({\beta }_{hl})+(1-{\pi }_0)\text{Exp}({\beta }_{hl};\lambda )\right\},$$

where m₀ ∼ N(ω₀, η₀), ν₀ ∼ Ga(a_ν0, b_ν0), π₀ ∼ Be(a_π, b_π) is the prior probability that a basis function is excluded from the model, λ ∼ Ga(a_λ, b), and Exp(λ) denotes the exponential distribution with mean λ⁻¹. The exponential distribution penalizes large values of the nonzero spline coefficients.

For posterior computation, we use the exact block Gibbs sampler (Yau et al., 2011), which bypasses the need to update the infinitely-many unknowns in the model by combining the ideas of retrospective sampling (Papaspiliopoulos & Roberts, 2008) and slice sampling (Walker, 2007). This algorithm avoids using truncation of the stick-breaking process as in Ishwaran & James (2001). Let N be the number of atoms sampled at the current iteration of the Gibbs sampler, K = (K₁, . . . , K_n)^T with K_i = h if μ_i = μ̃_h and τ_i = τ̃_h, and $c_h={\sum }_{i=1}^n{1}_{\left(K_i=h\right)}$ for h = 1, . . . , ∞. The Gibbs sampler cycles through the following steps.

Algorithm 1. The Gibbs sampler.

• Step 1. Sample β_h = (β_h0, β_h1, . . . , β_hk)^T (h = 1, . . . , N).

  1. If c_h = 0 or ${\sum }_{i:K_i=h}{b}_l^2(x_i)=0$, sample β_h0 from N (m₀, ν₀), and sample β_hl from the mixture distribution π₀δ₀ + (1 − π₀)Exp(λ) for l = 1, . . . , k.

  2. If c_h > 0 and ${\sum }_{i:K_i=h}{b}_l^2(x_i)>0$, sample β_h0 from N (E_h0, W_h0), where

     $$\begin{array}{cc}{W}_{h0}={\left({\tilde{\tau }}_h{c}_h+{\nu }_0\right)}^{-1},& E_{h0}=W_{h0}\left[{\tilde{\tau }}_h\sum _{i:K_i=h}\left\{y_i-\sum _{l\ne 0}{\beta }_{hl}\hspace{0.17em}{b}_l(x_i)\right\}+{\nu }_0\hspace{0.17em}{m}_0\right].\end{array}$$

For l = 1, . . . , k, sample β_hl from π_hl δ₀ + (1 − π_hl)N₊(E_hl, W_hl), where

$$\begin{array}{l}{\pi }_{hl}={\left\{1+\frac{(1-{\pi }_0)\lambda \mathrm{\Phi }\hspace{0.17em}(W_{hl}^{-1/2}{E}_{hl})}{{\pi }_0\varphi (-W_{hl}^{-1/2}{E}_{hl})W_{hl}^{-1/2}}\right\}}^{-1},\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{W}_{hl}={\left\{{\tilde{\tau }}_h\sum _{i:K_i=h}{b}_l^2(x_i)\right\}}^{-1},\\ E_{hl}=W_{hl}\hspace{0.17em}\left[{\tilde{\tau }}_h\sum _{i:K_i=h}{b}_l\hspace{0.17em}(x_i)\hspace{0.17em}\left\{y_i-\sum _{l^{\prime }\ne l}{\beta }_{{hl}^{\prime }}\hspace{0.17em}{b}_{l^{\prime }}\hspace{0.17em}(x_i)\right\}-\lambda \right].\end{array}$$

In the above, Φ(·) denotes the cumulative distribution function associated with N(0, 1), and N₊(a, b) denotes the truncated normal distribution of N(a, b) with support (0, ∞).

• Step 2. Sample τ̃_h from $\text{Ga}(a_{\tau }+c_h/2,b_{\tau }+{\sum }_{K_i=h}{\left\{y_i-{\sum }_{l=0}^k{\beta }_{hl}\hspace{0.17em}{b}_l\hspace{0.17em}(x_i)\right\}}^2/2)$ for h = 1, . . . , N.

• Step 3. Sample V_h from $\text{Be}(1+c_h,\alpha +n-{\sum }_{j=1}^h{c}_j)$ for h = 1, . . . , N.

• Step 4. Sample U_i from Un(0, p_Ki) for i = 1, . . . , n.

• Step 5. Sample K_i, i = 1, . . . , n. Let U^* = min(U₁, . . . , U_n).

  1. If ${\sum }_{h=1}^N$ p_h > 1 − U^*, sample K_i from a multinomial distribution with

     $$\begin{array}{cc}\text{pr}(K_i=j|\cdot )\propto {\tilde{\tau }}_j^{1/2}\varphi \left[{\tilde{\tau }}_j^{1/2}\left\{y_i-\sum _{l=0}^k{\beta }_{jl}\hspace{0.17em}{b}_l\hspace{0.17em}(x_i)\right\}\right]1_{(p_j>U_i)}& \left(i=1,\dots ,n\right).\end{array}$$

  2. Otherwise, keep updating N = N + 1 and sampling (V_N, β_N, τ_N) from their priors until ${\sum }_{h=1}^N$ p_h > 1 − U^*. Then sample K_i s as in (a).

• Step 6. Sample π₀ from $\text{Be}(a_{{\pi }_0}+{\sum }_{h=1}^N{\sum }_{l=1}^k{1}_{({\beta }_{hl}=0)},\hspace{0.17em}{b}_{{\pi }_0}+{\sum }_{h=1}^N{\sum }_{l=1}^k{1}_{({\beta }_{hl}>0)})$.

• Step 7. Sample α from $\text{Ga}\left\{a_{\alpha }+N,b_{\alpha }-{\sum }_{h=1}^N\text{log}(1-V_h)\right\}$.

• Step 8. Sample λ from $\text{Ga}(a_{\lambda }+{\sum }_{h=1}^N{\sum }_{l=1}^k{1}_{({\beta }_{hl}>0)},\hspace{0.17em}{b}_{\lambda }+{\sum }_{h=1}^N{\sum }_{l=1}^k{\beta }_{hl})$.

• Step 9. Sample ν₀ from $\text{Ga}\left\{a_{{\nu }_0}+N/2,b_{{\nu }_0}+{\sum }_{h=1}^N{\left({\beta }_{h0}-m_0\right)}^2/2\right\}$.

• Step 10. Sample m₀ from $N\left\{{({\tau }_0^{-1}+N{\nu }_0)}^{-1}({\tau }_0^{-1}{\omega }_0+{\nu }_0{\sum }_{h=1}^N{\beta }_{h0}),{({\tau }_0^{-1}+N{\nu }_0)}^{-1}\right\}$.

In Algorithm 1, steps 4 and 5 allow the number of clusters N to increase as needed during the sampling process. Because all the unknowns have full conditional posterior distributions that are easy to sample from directly, the algorithm is simple to implement and computation is fast per Gibbs iteration. Many functionals, such as conditional densities, distribution functions, quantile functions and posterior hypothesis probabilities, can be estimated easily from the Markov chain Monte Carlo samples.

Nonparametric Bayes models involve infinitely-many parameters and are typically specified so that higher level parameters in the hierarchy are under-identified from a frequentist perspective and there are many values of these parameters that lead to similar values of the observed data likelihood. In this sense, the models share a close connection with parametric models that arise from redundant parameterizations (Gelman, 2004, 2006; Ghosh & Dunson, 2009). In both cases, we expect the redundancy and under-identification to lead to poor mixing for higher level unknowns that are typically not of inferential interest, while obtaining good mixing for unknowns that directly appear in the observed data likelihood. For example, we anticipate slow mixing for some of the hyperparameters sampled in Steps 6 through 10 and even for the basis coefficients sampled in Step 1. As we start with an overcomplete set of potential basis functions, we fully expect that there are many different subsets of these bases that lead to similar induced collections of conditional densities. This will intrinsically lead to slow mixing for the basis coefficients that should not degrade and may even aid mixing of the conditional density values f (y | x). Indeed, this is exactly the behaviour we have observed in comprehensive simulation and sensitivity analyses that are summarized in the Supplementary Material; rates of convergence and mixing are good for f (y | x) and functions of these conditional densities, but are poor for a number of the hyperparameters.

Prior elicitation in nonparametric models is challenging, since the impact of the hyperparameters on the induced conditional densities f (y | x) is opaque, though some insight can be obtained through sampling from the prior. This opaqueness is not specific to our proposed model and is a general feature of most nonparametric Bayes models even in simple cases. We follow the strategy of choosing a weakly informative prior (Gelman et al., 2008) that leads to good performance in a wide variety of cases, with the posterior densities for f (y | x) substantially different from the priors even in small sample sizes such as n = 25. Specifically, we suggest choosing α ∼ Ga(1, 1), λ ∼ Ga(1, 1), ν₀ ∼ Ga(1, 1), π₀ ∼ Un(0, 1), m₀ ∼ N(0, 1) and L = Ga(1, 1) after normalizing y. The Supplementary Material contains a detailed sensitivity analysis.

4. Simulation studies

Two simulation cases were investigated to evaluate the performance of the proposed method: a null case and an alternative case. For each case, we tried different sample sizes n = 25, 50, 100, 200 and 500. All x_i s were independently sampled from Un(0, 1). In the null case, y_i was generated from

$$f(y|x)=\mathrm{0·6}\hspace{0.17em}N(y;\hspace{0.17em}0,\hspace{0.17em}{\mathrm{0·3}}^2)+\mathrm{0·4}\hspace{0.17em}N(y;\hspace{0.17em}1,{\mathrm{0·2}}^2),$$

and in the alternative case from

$$f(y|x)=\mathrm{0·6}\hspace{0.17em}N\{y;\hspace{0.17em}3{\left(x-\mathrm{0·3}\right)}^2\hspace{0.17em}{1}_{(x>\mathrm{0·3})},\hspace{0.17em}{\mathrm{0·3}}^2\}+\mathrm{0·4}\hspace{0.17em}N\{y;\hspace{0.17em}5{\left(x-\mathrm{0·8}\right)}^{3/2}\hspace{0.17em}{1}_{\left(x>\mathrm{0·8}\right)},\hspace{0.17em}{\mathrm{0·2}}^2\}.$$

We generated 100 datasets for each set-up. For each data set, we ran the Gibbs sampler described in § 4. In specifying the nondecreasing piecewise linear basis functions, we chose I-splines (Ramsay, 1988) with degree equal to 1 and 19 equally spaced interior knots in the predictor space (0, 1). Markov chain Monte Carlo was started with initial parameter values sampled from the recommended priors and N = 20, and results were summarized based on 2000 iterations after deleting the first 500 iterations as a burn-in for each dataset. Using 49 equally spaced interior knots did not give noticeably different results. As shown in the Supplementary Material, we ran a smaller number of simulations using 1100 iterations with a 100 iteration burn-in and obtained essentially identical results, justifying the use of short 2500 iteration chains in these simulations.

Figure 1 shows the results of conditional density estimation at predictor values x = 0.10, 0.50, 0.75 and 0.90 with the sample sizes n = 25 and 50 in the null case shown by the panels on the left and the alternative case shown by the panels on the right respectively. In the null case, the true conditional densities are all the same, while in the alternative case, they are substantially different across 𝒳. The proposed method does a good job in estimating the conditional densities, with the performance improving with increased sample size as expected.

Fig. 1.

Estimates of the conditional densities f (y | x) at x = 0.10, 0.50, 0.75, and 0.90, respectively. The left panels correspond to the null case and the right panels correspond to the alternative case. Each plot shows the true density (solid), the average of the 100 estimated densities when sample size n = 25 (dash), and the average of the 100 estimated densities when n = 50 (dot-dash).

For global testing, we reject the global null if the supremum of the log Bayes factor function is larger than log(10) for each dataset. The choice of log(10) corresponds to a standard threshold for strong evidence against the null hypothesis (Kass & Raftery, 1995). The proportion of rejected global hypotheses is interpreted as the type 1 error of the test in the null case and the power in the alternative case. Our simulations yield type 1 errors equal to 0.05, 0.07, 0.03, 0.04 and 0.02 and powers equal to 0.79, 0.98, 1.0, 1.0 and 1.0 for sample size n = 25, 50, 100, 200, and 500, respectively.

We are also interested in local hypothesis testing of differences between F_x and F₀, with F₀ denoting the distribution of y given x = 0. We estimate a threshold x̂ = min{x : BF(0, x) > 10}, the smallest predictor value at which the Bayes factor is larger than 10. There is significant evidence in favour of the local alternative hypothesis for x > x̂. Figure 2 shows the box plots of the estimated thresholds obtained from 100 datasets in the alternative case with sample size n = 25, 50, 100, 200, and 500, respectively. The true threshold is 0.45. From Fig. 2, the centre of the box plot moves closer to the true threshold and the variation becomes smaller as sample size increases. The averages of the thresholds are 0.89, 0.74, 0.65, 0.57, and 0.46 when sample size is 25, 50, 100, 200, and 500, respectively. Hence, for small sample sizes, there is a tendency to overestimate the threshold, which is as expected.

Fig. 2.

Boxplots of thresholds x̂ identified from the local hypothesis tests for 100 datasets in the alternative case of simulation with sample size n = 25, 50, 100, 200, and 500, respectively. The dashed horizontal line shows the true value of threshold, 0.45.

5. An epidemiologic application

We apply the proposed method to an epidemiologic study relating 1,1-dichloro-2,2-bis (p-chlorophenyl)-ethylene, commonly referred to as p, p′-DDE, to risk of premature delivery. The study was based on the U.S. Collaborative Perinatal Project (Longnecker et al., 2001). Dunson & Park (2008) previously showed a relationship between DDE and the density of gestational age at delivery without incorporating the stochastic ordering constraint. As it is reasonable to assume that gestational length does not increase as dose increases, the stochastic-ordering assumption is biologically plausible. The proposed method allows for testing and identification of a threshold DDE level, which is of substantial interest from a public health perspective.

We analysed the data using a modification of (4), which adds $z_i^{\text{T}}\gamma$ to μ_i (x_i) to include a parametric adjustment for the covariates z_i, with z_i including race and standardized cholesterol and triglyceride levels for subject i. Letting z = (z₁, . . . , z_n)^T with sample size n = 2313, we chose the prior γ ∼ N{0, n(z^Tz)⁻¹}, with the prior otherwise specified as in § 3. The Gibbs sampler of § 3 can be applied with a minor modification. We used 36 I spline basis functions ${\left\{{\tilde{b}}_l\right\}}_{l=1}^{36}$ resulting from taking 35 equally spaced interior knots within (0, 180) and specifying degree equal to 1, and took b_l = 1 − b̃_l for l = 1, . . . , 36 as basis functions in (5) in order to model stochastically nonincreasing densities. We based our results on 10 000 Markov chain Monte Carlo iterations collected after discarding the first 10 000 iterations as a burn-in.

Figure 3(a) shows the estimated conditional densities of gestational age at delivery for a DDE level equal to 2.5, 24.68, 36.53, 53.75 corresponding to the minimum, the 50th, 75th, and 90th percentiles of the observed DDE values, controlling the other covariates equal to zero. These density plots show an increasingly heavy left tail as DDE increases, implying an increasing risk of premature delivery. We also obtain substantially narrower credible intervals for the density estimates compared with Dunson & Park (2008), likely due to improved efficiency attributable to inclusion of the order constraint.

Fig. 3.

Estimated conditional densities of gestational age at delivery and the log Bayes factor curve used for global hypothesis testing. (a) The estimated conditional densities of gestational age at delivery at DDE values x = 2.5 (solid), 24.68 (dot), 36.53 (dash), 53.75 (dot-dash), respectively. (b) the log Bayes factor curve log-BF (2.5, x) (solid). The three dashed horizontal lines in the right panel are log(10), log(3) and 0 for reference.

In order to assess the weight of evidence in the data in favour of stochastic ordering and to estimate a threshold dose, we apply the approach implemented in § 4. Let F₀ denote the distribution function of gestation length given x = 2.5, the minimum of the observed DDE levels, and let F_x denote the distribution function at DDE = x. Figure 3(b) plots the log Bayes factor curve for testing H₀(2.5, x) versus H₁(2.5, x) for increasing x. Treating Bayes factors greater than 10 as significant, we conclude in favour of H₁(2.5, x) for all x > 52.

In addition to density estimation and hypothesis testing, the proposed method also yields a mean function estimate, as well as various quantile curves of gestational age at delivery changing with the DDE values. The proposed mean function estimate has much narrower 95% intervals than the estimate obtained by fitting the generalized additive model fitted with the gam function in R (R Development Core Team, 2011). We also estimate different quantile curves by using the proposed approach and by running the R function rqss with the nonincreasing constraint for a nonparametric additive quantile regression model. The estimates from both methods look similar overall, but the estimates from the proposed method are much smoother. The proposed method should be more efficient in estimating different quantile curves since it estimates all of them jointly while existing methods usually estimate them separately. Matlab codes for the simulation study and data analysis are available upon request from the first author.

6. Stochastically ordered discrete distributions

When the response variable is discrete, it is useful to consider a simplification in which y_i ∼ P_xi, with P = (P_x, x ∈ 𝒳) assigned the prior in (2) simplified to let Θ_h = μ̃_h ∼ H. A restricted dependent Dirichlet process prior for P results by constraining H to have support on the space of nondecreasing functions. We show this leads to a prior with large support on the space of nondecreasing stochastically ordered distributions.

Remark 1. When y_i ∈ {0, 1, . . . , ∞} is a count, H should have support on the space of nondecreasing 𝒳 → {0, 1, . . . , ∞} functions. This can be accomplished by starting with an H^* having support on nondecreasing 𝒳 → ℜ⁺ functions. One such H^* corresponds to an exponentiated Gaussian process. Then, to obtain realizations μ̃_h ∼ H, draw ${\mu }_h^{*}~H^{*}$ and let ${\tilde{\mu }}_h(x)=\lfloor {\mu }_h^{*}(x)\rfloor$. The resulting P_xi will have nonnegative integer atoms.

Let P = (P_x, x ∈ 𝒳) denote an uncountable collection of stochastically ordered probability measures. The nondecreasing stochastic ordering constraint means P_x ≼ P_x′, for any two points x ⩽ x′ in 𝒳. Let 𝒫 denote the set of probability measures on {𝒴, 𝒝(𝒴)}, with 𝒴 a subset of the real line and 𝒝(𝒴) the Borel subsets of 𝒴. Let P_𝒳 = {P = (P_x, x ∈ 𝒳) : P_x ∈ 𝒫 for all x ∈ 𝒳}, and C_𝒳 = {P = (P_x, x ∈ 𝒳) : P_x ∈ 𝒫, P_x ≼ P_x′ for any x ⩽ x′, x, x′ ∈ 𝒳}, with P_𝒳 the set of uncountable collections of probability measures on {𝒴, 𝒝(𝒴)} indexed by x ∈ 𝒳, and C_𝒳 the subset of collections satisfying nondecreasing stochastic order.

In addition, let R_𝒳 denote the 𝒳 → 𝒴 function space and S_𝒳 the nondecreasing 𝒳 → 𝒴 function space. Define a metric d in R_𝒳 with d(s₁, s₂) = sup_x∈𝒳 |s₁(x) − s₂(x)| for any s₁, s₂ ∈ R_𝒳. This metric induces a topology on R_𝒳 and thus on S_𝒳. Let 𝒬 denote the set of probability measures on S_𝒳.

Theorem 2. Consider the prior π_P for P defined in (2), with Θ_h ∼ H and H ∈ 𝒬 a probability measure that has full support on S_𝒳. Then π _P has full support on C_𝒳 with respect to the topology of weak convergence defined in the Appendix.

From Theorem 2, the constructed prior π_P in (2) can assign a positive probability to any open set containing the true P of interest. Theorem 2 follows directly from a more general result in Theorem 3 to be presented below. In the Appendix, Lemma A1 establishes a generalized Choquet’s theorem by showing that every P ∈ C_𝒳 can be represented as a mixture over the extreme points of C_𝒳 as follows

$$P(B)=T(Q)(B)={\int }_{S𝒳}\hspace{0.17em}\{{\delta }_{S(x)}\hspace{0.17em}(B),\hspace{0.17em}x\in 𝒳\}d\hspace{0.17em}Q(s)$$ (6)

for any B ∈ 𝒝(𝒴), where Q is a mixing measure over S_𝒳, T is an integral operator mapping Q to its marginals and {δ_s(x)(·), x ∈ 𝒳} is an extreme point of C_𝒳. The mapping operator T in (6) is shown to be continuous in Lemma A2.

The integral representation (6) allows us to induce a prior π_P for P by introducing a prior π_Q for the mixing measure Q. Based on Lemmas 1 and 2, we establish the following theorem.

Theorem 3. Through (6), the prior π_P for P has full support on C_𝒳 with respect to the topology of weak convergence if and only if the prior π_Q for Q has full support on 𝒬.

The proofs of Lemmas A1 and A2 and Theorem 3 are sketched in the Appendix. Theorem 2 follows directly from Theorem 3 by taking π_Q to be DP(α H), a Dirichlet process with a precision parameter α and a base probability measure H having full support on S_𝒳, after noting that dp(α H) has a weak support on all of 𝒬 (Hoff, 2003a) with respect to the topology induced by the metric d.

Supplementary material

Supplementary material available at Biometrika online contains a sensitivity analysis, the influence of the recommended prior specification for the hyperparameters, the Markov chain Monte Carlo mixing performance in the simulation and DDE data analysis, and the estimated covariate effects in the DDE data analysis.

Appendix

Proof of Theorem 1. Under (3), the conditional cumulative distribution functions are

$$F_x\hspace{0.17em}(y)=\sum _{h=1}^{\infty }{p}_h\mathrm{\Phi }[{\tilde{\tau }}_h^{1/2}\left\{y-{\tilde{\mu }}_h(x)\right\}]$$

for any y ∈ 𝒴 and x ∈ 𝒳. Then for any two predictor values x and x′ with x ⩽ x′,

$$F_x\hspace{0.17em}(y)-F_{x^{\prime }}\hspace{0.17em}(y)=\sum _{h=1}^{\infty }{p}_h(\mathrm{\Phi }[{\tilde{\tau }}_h^{1/2}\{y-{\tilde{\mu }}_h\hspace{0.17em}(x)\}]-\mathrm{\Phi }[{\tilde{\tau }}_h^{1/2}\{y-{\tilde{\mu }}_h\hspace{0.17em}(x^{\prime })\}]).$$

First, we show that (b) implies (a). It is straightforward to see F_x (y) − F_x′ (y) ⩾ 0 for any y ∈ 𝒴 and x < x′ ∈ 𝒳 with probability 1 based on the facts that Φ is an increasing function and that μ̃_hs are nondecreasing random functions sampled from H. This suggests that f_𝒳 is subject to the nondecreasing stochastic ordering constraint. Now we show f_𝒳 under (3) are subject to the continuity condition. First, for any y₁, y₂ ∈ ℜ and b > 0, one has $\varphi (b^{1/2}\hspace{0.17em}{y}_1)-\varphi (b^{1/2}\hspace{0.17em}{y}_2)={\left(2\pi \right)}^{-1/2}\hspace{0.17em}b{y}_0\hspace{0.17em}\text{exp}(-b{y}_0^2/2)\hspace{0.17em}(y_1-y_2)$ for some y₀ between y₁ and y₂ using the mean value theorem. One further obtains ϕ (b^1/2 y₁) − ϕ (b^1/2 y₂) ⩽ (2π)^−1/2b^1/2|y₁ − y₂| since $b^{1/2}\hspace{0.17em}{y}_0\hspace{0.17em}\text{exp}(-b{y}_0^2/2)\hspace{0.17em}⩽1$ for any y₀. Fixing x′ ∈ 𝒳, for any y ∈ 𝒴 and any x ∈ 𝒳, one then has

$$\begin{array}{c}{f}_x\hspace{0.17em}(y)-f_{x^{\prime }}\hspace{0.17em}(y)=\sum _{h=1}^{\infty }{p}_h\hspace{0.17em}{\tilde{\tau }}_h^{1/2}(\varphi [{\tilde{\tau }}_h^{1/2}\hspace{0.17em}\{y-{\tilde{\mu }}_h\hspace{0.17em}(x)\}]-\varphi [{\tilde{\tau }}_h^{1/2}\hspace{0.17em}\left\{y-{\tilde{\mu }}_h(x^{\prime })\right\}])\\ ⩽{\left(2\pi \right)}^{-1/2}\sum _{h=1}^{\infty }{p}_h{\tilde{\tau }}_h\left|{\tilde{\mu }}_h(x)-{\tilde{\mu }}_h(x^{\prime })\right|.\end{array}$$

Then f_x (y) − f_x′ (y) → 0 for any y → 𝒴 as x → x′ follows directly by noting that ${\sum }_{h=1}^{\infty }{p}_h{\tilde{\tau }}_h\left|{\tilde{\mu }}_h(x)-{\tilde{\mu }}_h(x^{\prime })\right|\to 0$ as x → x′ since μ̃_hs are continuous random functions sampled from H.

Now we show (a) implying (b) by showing if (b) does not hold, then (a) does not hold. Suppose H(S_𝒳,c) < 1. Denote B_x,x′ = {s ∈ R_𝒳 : s(x) ⩽ s(x′)} for a pair of rational numbers x, x′ ∈ 𝒳 and x < x′, where R_𝒳 is the space of 𝒳 → ℜ functions. Being the space of continuous nondecreasing functions on 𝒳, S_𝒳,c can be written as a countable intersection of B_x,x′ over all possible rational pairs (x, x′) in 𝒳. Thus, R_𝒳\S_𝒳,c can be written as a countable union of $B_{x,x^{\prime }}^c$ over all possible rational pairs (x, x′) in 𝒳, where $B_{x,x^{\prime }}^c=R_{𝒳}\backslash B_{x,x^{\prime }}$. Since H(R_𝒳 \S_𝒳,c) = 1 − H(S_𝒳,c) > 0, there exist a pair of rational numbers (x₁, x₂) with x₁ < x₂ such that $H(B_{x_1,\hspace{0.17em}{x}_2}^c)>0$ by using the subadditivity of a probability measure. Taking this pair (x₁, x₂) and fixing a y ∈ 𝒴, we consider

$$C=\left\{\left(s,t\right)\in B_{x_1,x_2}^c\times {\Re }^{+}:\mathrm{\Phi }\left[t^{1/2}\left\{y-s(x_1)\right\}\right]-\mathrm{\Phi }\left[t^{1/2}\left\{y-s(x_2)\right\}\right]<0\right\}.$$

It is straightforward to show $C=B_{x_1,x_2}^c\times {\Re }^{+}$ and

$$\text{pr}\left\{\left({\mathrm{\Theta }}_1,{\tilde{\tau }}_1\right)\in C\right\}=M(C)=H\otimes L\left(B_{x_1,x_2}^c\times {\Re }^{+}\right)=H\left(B_{x_1,x_2}^c\right)L({\Re }^{+})=H\left(B_{x_1,x_2}^c\right)>0$$

due to the independence between H and L and L(ℜ⁺) = 1. Write

$$C=\underset{k=1}{\overset{\infty }{\cup }}{C}_k=\underset{k=1}{\overset{\infty }{\cup }}\left\{(s,t)\in B_{x_1,x_2}^c\times {\Re }^{+}:\mathrm{\Phi }\left[t^{1/2}\left\{y-s(x_1)\right\}\right]-\mathrm{\Phi }\left[t^{1/2}\left\{y-s(x_2)\right\}\right]<-1/k\right\}.$$

Then there exists a positive integer k₀ such that pr(C_k0) > 0 since $0<\text{pr}(C)⩽{\sum }_{k=0}^{\infty }\text{pr}(C_k)$.

Choose ∊ ∈ {0, 1/(1 + 2k₀)} and a positive integer T₀ such that T₀ > log(∊)/ log{α/(1 + α)}. Define

$$A=\left\{q=\left(q_h,h=1,\dots ,\infty \right):\sum _{h=1}^{\infty }{q}_h=1,\sum _{h=T_0+1}^{\infty }{q}_h⩽∊,\hspace{0.17em}\text{and}\hspace{0.17em}0<q_h<1\hspace{0.17em}\text{for}\hspace{0.17em}\text{any}\hspace{0.17em}h\right\}.$$

Denote $g_h=\mathrm{\Phi }\left[{\tilde{\tau }}_h^{1/2}\left\{y-{\tilde{\mu }}_h(x_1)\right\}\right]-\mathrm{\Phi }\left[{\tilde{\tau }}_h^{1/2}\left\{y-{\tilde{\mu }}_h(x_2)\right\}\right]$ for each h. Clearly, g_hs are independent and identically distributed, and pr(|g_h| ⩽ 2) = 1 for each h since Φ is a cumulative distribution function. Then on D = {p = (p_h, h = 1, . . . , ∞) ∈ A, and (μ̃_h, τ̃_h) ∈ C_k0 for h = 1, . . . , T₀},

$$\sum _{h=1}^{\infty }{p}_h{g}_h=\sum _{h=1}^{T_0}{p}_h{g}_h+\sum _{h=T_0+1}^{\infty }{p}_h{g}_h<-\frac{1}{k_0}\sum _{h=1}^{T_0}{p}_h+2∊<-\frac{1}{k_0}(1-∊)+2∊<0.$$ (A1)

By using Markov’s inequality, one has

$$\text{pr}(p\in A)=\text{pr}\left(\sum _{h=T_0+1}^{\infty }{p}_h⩽∊\right)⩾1-\frac{1}{∊}E\left(\sum _{h=T_0+1}^{\infty }{p}_h\right)=1-\frac{1}{∊}{\left(\frac{\alpha }{1+\alpha }\right)}^{{\text{T}}_0}>0.$$

Hence, one obtains pr(D) = pr(A)pr(C_k0)^T₀ > 0 by using the mutual independence of (μ̃_h, τ̃_h)s and p. This result together with (A1) indicates that F_x2 is not stochastically larger than F_x1, and thus (a) cannot hold.

Definition 1 (Weak convergence in P_𝒳 used in § 6). Define G_Z (P)(·) = ∫_𝒳 Px (·)dZ(x) for any P = (P_x, x ∈ 𝒳) ∈ P_𝒳 and any probability measure Z on 𝒝(𝒳), the Borel sets of 𝒳. Then G_Z (P) ∈ 𝒫 is a probability measure on 𝒝(𝒴). Define weak convergence in P_𝒳 as follows: a sequence ${\left\{P^{(n)}\right\}}_{n=1}^{\infty }$, with P⁽ⁿ⁾ ∈ P_𝒳 for all n, converges weakly to P ∈ P_𝒳, if and only if G_Z (P⁽ⁿ⁾) converges weakly to G_Z (P) for any probability measure Z on 𝒝(𝒳).

The above weak convergence of P⁽ⁿ⁾ to P in P_𝒳 implies pointwise weak convergence of $P_x^{(n)}$ to P_x for all x ∈ 𝒳, which can be easily seen by taking Z = δ_x. Essentially the weak convergence of P_n to P in P_𝒳 is equivalent to the uniform weak convergence of $P_x^{(n)}$ to P_x for all x ∈ 𝒳. Another equivalent definition is that d(∫ fd P, ∫fd P⁽ⁿ⁾) → 0 for any bounded continuous f on 𝒴 with d being the uniform metric defined in § 6 and ∫ fd P = (∫ fd P_x, x ∈ 𝒳) ∈ R_𝒳.

The above weak convergence induces a weak topology in P_𝒳. Under this topology, it can be shown that C_𝒳 is a weakly closed convex set and that the set of extreme points of C_𝒳 is exC_𝒳 = {(δ_s(x), x ∈ 𝒳) : s ∈ S_𝒳}. Lemma A1 generalizes Choquet’s Theorem by considering an uncountable collection of probability measures.

Lemma A1. For any probability measure Q ∈ 𝒬, T (Q) ∈ C_𝒳, where T is defined in (6). Also, for any P ∈ C_𝒳, there exists a probability measure Q ∈ 𝒬 such that Q({s ∈ S_𝒳 : s(x) ∈ ·}) = P_x (·) for all x ∈ 𝒳, where s denotes a random function over 𝒳.

Proof of Lemma A1. For a probability measure Q on S_𝒳, define Q_x (·) = ∫_S𝒳 δ_s(x)(·)d Q(s). Then Q_x is a probability measure on 𝒝(𝒴). To show T (Q) ∈ C_𝒳, one only needs to show (Q_x, x ∈ 𝒳) are subject to nondecreasing stochastic ordering constraint. For any two points x, x′ ∈ 𝒳 with x ⩽ x′, one has s(x) ⩽ s(x′) for s ∈ S_𝒳. Further one has {s ∈ S_𝒳 : s(x) > a} ⊆ {s ∈ S_𝒳 : s(x′) > a} for any a ∈ ℜ. By the definition of Q_x and Q_x′, one obtains that Q_x (a, ∞) = ∫_S𝒳 δ_s(x)(a, ∞)dQ(s) = Q[{s ∈ S_𝒳 : s(x) > a}] ⩽ Q[{s ∈ S_𝒳 : s(x′) > a}] = ∫_S𝒳 δ_s(x′)(a, ∞)d Q(s) = Q_x′ (a, ∞). This suggests Q_x ≼ Q_x′ and thus T (Q) = (Q_x, x ∈ 𝒳) ∈ C_𝒳.

Suppose P = (P_x, x ∈ 𝒳) ∈ C_𝒳. Let F_x denote the cumulative distribution function with respect to P_x and define $s_x(\omega )=F_x^{-1}(\omega )=\text{inf}\left\{u:F_x(u)⩾\omega \right\}$ for any x ∈ 𝒳 and w ∈ [0, 1]. For any x, x′ ∈ 𝒳 with x ⩽ x′, P_x ≼ P_x′ leads to {u : F_x′ (u) ⩾ ω} ⊆ {u : F_x (u) ⩾ ω} and thus s_x (ω) ⩽ s_x′ (ω) for all ω. Let Q be the canonical measure of s = (s(x), x ∈ 𝒳) on (S_𝒳, σ_S𝒳) induced by a uniform distribution on ω, where σ_S𝒳 is a σ-algebra on S_𝒳. Then Q represents P with Q_x = P_x for any x ∈ 𝒳 using similar arguments to the proof of Proposition 8 in Hoff (2003b).

Lemma A2. The integral operator T defined in (6) is continuous.

Proof of Lemma A2. The continuity of T means that if P⁽ⁿ⁾ = T (Q⁽ⁿ⁾), P = T (Q), and Q⁽ⁿ⁾ converges weakly to Q, then P⁽ⁿ⁾ converges weakly to P. By the definition of weak convergence in P_𝒳, we need to show G_Z (P⁽ⁿ⁾) converges to G_Z (P) for any probability measure Z on 𝒝(𝒳). This is equivalent to showing that ∫_𝒴 fdGZ (P⁽ⁿ⁾) → ∫_𝒴 fdGZ (P) for any bounded uniformly continuous function f on 𝒴 and any probability measure Z on 𝒝(𝒳) by noting that 𝒴 is a separable metric space (Parthasarthy, 1967; Billingsley, 1999). Denote by U(𝒴) the set of bounded uniformly continuous functions on 𝒴.

First, for any f ∈ U(𝒴) and any probability measure Z on 𝒝(𝒳), we have the following facts: G_Z (P)(·) = ∫_𝒳 Q_x (·)d Z(x) = ∫_𝒳 ∫_S𝒳 δ_s(x)(·)d Q(s)dZ(x) and

$$\begin{array}{lll}{\int }_{𝒴}\hspace{0.17em}fd{G}_Z\hspace{0.17em}(P)\hfill & =\hfill & {\int }_{𝒴}f(y){\int }_{𝒳}{\int }_{S_{𝒳}}dQ(s)dZ(x){\delta }_{s(x)}(du)\hfill \\ \hfill & =\hfill & {\int }_{S_{𝒳}}{\int }_{𝒳}{\int }_{𝒴}f(y){\delta }_{s(x)}(dy)dZ(x)dQ(s)={\int }_{S_{𝒳}}{\int }_{𝒳}f\left(s(x)\right)dZ(x)dQ(s).\hfill \end{array}$$

Let g_{(Z, f)}(s) = ∫_𝒳 f (s(x))d Z(x), a function on S_𝒳, given Z ∈ 𝒝(𝒳) and f ∈ U(𝒴). Then one has ∫_𝒴 f dG Z (P) = ∫_S𝒳 g_{(Z, f)}(s)d Q(s) and ∫_𝒴 f dG_Z (P⁽ⁿ⁾) = ∫_S𝒳 g_{(Z, f)}(s)d Q⁽ⁿ⁾(s). Since Q⁽ⁿ⁾ converges weakly to Q, one only needs to show that g_{(Z, f)} is a bounded continuous function of s on S_𝒳 for any f ∈ U(𝒴) and any measure Z on 𝒝(𝒳). This is trivial based on the uniform metric d in R_𝒳

Proof of Theorem 3. Suppose that π_Q has a full support on 𝒬. Let A be an open subset of C_𝒳, then one has π_P (A) = pr{T (Q) ∈ A} = π_Q(T⁻¹ A) > 0 based on the facts that T⁻¹ A is an open set due to the continuity of T and that π_Q has the full support on 𝒬. It is straightforward to have π_P (C_𝒳) = π_Q(𝒬) = 1 based on Lemma A1. The other part can be proven similarly.