Inflated Density Ratio and Its Variation and Generalization for Computing Marginal Likelihoods

Abstract

In the Bayesian framework, the marginal likelihood plays an important role in variable selection and model comparison. The marginal likelihood is the marginal density of the data after integrating out the parameters over the parameter space. However, this quantity is often analytically intractable due to the complexity of the model. In this paper, we first examine the properties of the inflated density ratio (IDR) method, which is a Monte Carlo method for computing the marginal likelihood using a single MC or Markov chain Monte Carlo (MCMC) sample. We then develop a variation of the IDR estimator, called the dimension reduced inflated density ratio (Dr.IDR) estimator. We further propose a more general identity and then obtain a general dimension reduced (GDr) estimator. Simulation studies are conducted to examine empirical performance of the IDR estimator as well as the Dr.IDR and GDr estimators. We further demonstrate the usefulness of the GDr estimator for computing the normalizing constants in a case study on the inequality-constrained analysis of variance.

1. Introduction

In Bayesian analysis, different models can be compared by using their marginal likelihoods (also known as normalizing constants). Defined as the integral of the likelihood function times the prior distribution over the parameter space, the marginal likelihood measures the average fit of the model to the data, and is useful in the Bayesian hypothesis testing problem. The statistical theory of hypothesis testing is associated with methods for checking whether an improvement in fit is statistically significant. The Bayes factor (BF) is one of the methods to use in this scenario. BF is the ratio of the marginal likelihoods under the two competing models. Due to the complicated form of the product of the likelihood function and the prior, which is called the posterior kernel later, the marginal likelihood is often analytically intractable. Hence, computing the marginal likelihood or the normalizing constant is of great current interest and remains an active research area.

There is a vast literature on computing marginal likelihoods or normalizing constants, including but not limited to the importance sampling (IS) [1], the harmonic means (HM) [2], the posterior density based approaches of Chib [3] and Chib and Jeliazkov [4], the path sampling (PS) [5], the inflated density ratio (IDR) [6], the latent variable approach of Chen [7], the thermodynamic integration (TI) [8], the stepping stone method (SSM) [9, 10], and the partition weighted kernel (PWK) estimator [11]. These methods can be divided into two categories. The methods such as IS, HM, IDR, Chen [7], and PWK use only a single MCMC sample while the other methods such as Chib [3], TI, and SSM use multiple MCMC samples. These methods can be shown to asymptotically converge to the quantity of interest under some mild conditions. Varying in their use of information such as Markov chain Monte Carlo (MCMC) samples or the posterior kernel, they all have pros and cons in different applications. Among them, the IDR method is developed by introducing a perturbed density such that this density has an additionally calculable mass in the posterior mode and lighter tails than the posterior kernel, and can estimate the marginal likelihood precisely based on an MCMC sample from the posterior distribution. With such a control of the posterior tails, Petris and Tardella [12] also show that IDR has a finite variance in one-dimensional problems. However, the properties of the IDR estimator still have not been fully examined and its usefulness in statistical applications has not been fully explored.

In this paper, we first reveal an interesting property of the IDR estimator, that is, its superior efficiency in two-dimensional problems. This property is further confirmed in our simulation study. To broaden the applicability of IDR, we develop a variation of IDR, namely, the dimension reduced inflated density ratio (Dr.IDR) estimator. This new estimator not only enjoys the property of dimension reduction and but also has a nice connection to the marginal posterior density estimation. We further extend the idea of Dr.IDR to develop the generalized dimension reduced (GDr) estimator, which is constructed via a new identity based on the marginal posterior density. An interesting case study is carried out to demonstrate how to compute the normalizing constant for the inequality-constrained analysis of variance model. The new estimators developed in this paper have great potential for computing marginal likelihoods or normalizing constants for high-dimensional problems and complex models such as those with an inequality-constrained parameter space.

The rest of the article is organized as follows. Section 2 presents the formulation of the IDR estimator and examines its theoretical properties. In Section 3, we develop a variation of IDR, called the dimension reduced IDR (Dr.IDR) and discuss various properties of Dr.IDR. A new identity based on the marginal posterior density is introduced and a so-called generalized dimension reduced (GDr) estimator is then constructed in Section 4. Two simulation studies to examine empirical performance of the IDR, Dr.IDR, and GDr estimators are presented in Section 5. Section 6 is a case study, which demonstrates how to compute the normalizing constant for the inequality-constrained analysis of variance model. We conclude this paper with a brief discussion in Section 7.

2. The Inflated Density Ratio (IDR) Estimator

Suppose ζ is a p-dimensional vector of parameters with the unbounded support Ω. Given that L(ζ|D) is the likelihood function of parameters ζ given the data D and π(ζ) is a proper prior distribution in the sense that ${\int }_{\Omega }\pi (\zeta )d\zeta =1$, the marginal likelihood is given by

$$c={\int }_{\Omega }L(\zeta |D)\pi (\zeta )d\zeta ={\int }_{\Omega }q(\zeta )d\zeta ,$$ (1)

where q(ζ) = L(ζ|D)π(ζ) is the posterior kernel function. The corresponding posterior distribution is written as

$$\pi (\zeta |D)=\frac{L(\zeta |D)\pi (\zeta )}{c}=\frac{q(\zeta )}{c}.$$ (2)

We note that when ${\int }_{\Omega }\pi (\zeta )d\zeta \ne 1$ or ${\int }_{\Omega }\pi (\zeta )d\zeta =\infty$ but c < ∞, c is called the normalizing constant. When the model structure is complex, the constant c is often analytically intractable. However, a Markov chain Monte Carlo (MCMC) sample can still be generated from the posterior distribution given in (2) without knowing c.

To estimate (1), Petris and Tardella [6, 12] proposed the inflated density ratio (IDR) method, which requires a single Monte Carlo (MC) or MCMC sample from (2). This method first constructs a perturbed density q_r(ζ) based on the posterior kernel q(ζ). Let ζ₀ be the “center” of the posterior distribution, which is typically the posterior mode of ζ. Define q_r(.) as

$$q_r(\zeta )=\{\begin{array}{cc}q({\zeta }_0)\hfill & \text{if}\Vert \zeta -{\zeta }_0\Vert \le r,\hfill \\ q({\zeta }_0+h(\zeta -{\zeta }_0))\hfill & \text{if}\Vert \zeta -{\zeta }_0\Vert >r,\hfill \end{array}$$ (3)

where r is a prespecified radius and h(ζ − ζ₀) = (1 − r^p/‖ζ − ζ₀‖^p)^1/p(ζ − ζ₀). It can be shown that the integral of q_r(.) over the whole support is the sum of the target c and a tractable volume k = q(ζ₀)b_r, where b_r = volume of the ball {ζ : ‖ζ − ζ₀‖ ≤ r} = π^p/2r^p/Γ(p/2+1). Thus, we have the IDR identity given by

$$c=\frac{q({\zeta }_0)b_r}{E\left[\frac{q_r(\zeta )}{q(\zeta )}|D\right]-1},$$ (4)

where $E\left[\frac{q_r(\zeta )}{q(\zeta )}\mid D\right]$ is the expectation of $\frac{q_r(\zeta )}{q(\zeta )}$ with respect to the posterior distribution in (2). Based on the IDR identity in (4), the IDR estimator for c is given by

$${\widehat{c}}_{\text{IDR}}=\frac{k}{\frac{1}{n}{\sum }_{i=1}^n\frac{q_r({\zeta }^{(i)})}{q({\zeta }^{(i)})}-1},$$ (5)

where {ζ⁽¹⁾, …, ζ⁽ⁿ⁾} is an MC/MCMC sample from the posterior distribution in (2). Figure 1 provides the visualization of q_r(.) in a univariate normal example. The area under the curve q_r(.) is k + c = 2rq(0) + 1 since ζ₀ = 0 in this case.

Figure 1:

An illustration of perturbed density in the IDR method.

Petris and Tardella [12] showed that under certain conditions, such as the log-Lipschitz condition, the variance of ${\widehat{c}}_{\mathrm{IDR}}$ is finite in one-dimensional problems. If {ζ⁽¹⁾, …, ζ⁽ⁿ⁾} is an MCMC sample from (2), under certain mild regularity conditions, such as time reversible, invariant, and irreducible conditions, it is easy to show that ${\widehat{c}}_{\mathrm{IDR}}$ is a consistent estimator of c since the denominator of (5) converges to

$$\underset{n\to \infty }{\lim }\frac{1}{n}\sum _{t=1}^n\frac{q_r({\zeta }^{(i)})}{q({\zeta }^{(i)})}-1={\int }_{\Omega }\frac{q_r(\zeta )}{q(\zeta )}\frac{q(\zeta )}{c}d\zeta -1={\int }_{\Omega }\frac{q_r(\zeta )}{c}d\zeta -1=\frac{c+k}{c}-1=\frac{k}{c},$$

as n → ∞. As pointed by Petris and Tardella [12], Arima and Tardella [13], and Wang et al. [11], the IDR method requires the full space of q(.) and a careful selection of the radius. Wang et al. [11] further pointed out that mode finding is essential and standardization of an MCMC sample with respect to the mode and the sample covariance matrix is required for IDR. Thus, the best way to use IDR is to obtain the perturbed density after the standardization as demonstrated in Section 5.2. If any parameter in the model has a bounded support, an additional transformation is needed to define the perturbed density q_r(.). This extra effort may limit its applications especially for a problem with complicated constraints (see Section 6 for an example). During our empirical investigation, we found that the IDR has an interesting property in the bivariate standard normal case.

Consider the posterior kernel function as

$$q(\zeta )=\frac{1}{2\pi }\exp \left(-\frac{{\mu }^{\prime }\mu }{2}\right),$$ (6)

where ζ = μ = (μ₁, μ₂)′ is a vector of unknown parameters with center at ζ₀ = (0, 0)^′. Under such a setting, the perturbed density is given by

$$q_r(\zeta )=\{\begin{array}{cc}\frac{1}{2\pi }\hfill & \text{if}\Vert \mu \Vert \le r,\hfill \\ \frac{1}{2\pi }\exp \left(-\frac{{\mu }^{\prime }\mu -r^2}{2}\right)\hfill & \text{if}\Vert \mu \Vert >r,\hfill \end{array}$$

so that

$$\frac{q_r(\zeta )}{q(\zeta )}=\{\begin{array}{cc}\exp \left(\frac{{\mu }^{\prime }\mu }{2}\right)\hfill & \text{if}\Vert \mu \Vert \le r,\hfill \\ \exp \left(\frac{r^2}{2}\right)\hfill & \text{if}\Vert \mu \Vert >r.\hfill \end{array}$$ (7)

We see from (7) that the ratio term within the radius is always bounded by $\exp \left(\frac{r^2}{2}\right)$ while the ratio term outside the radius is always constant, which implies that the variance or any higher-order moments of the IDR estimator are finite. We formally state this result in the following theorem.

Theorem 2.1. Given that {ζ⁽¹⁾, …, ζ⁽ⁿ⁾} is a random sample from the posterior density, any v^th moment (v > 0) of the IDR estimator is finite for the bivariate normal distribution given in (6).

Proof. Letting $g({\zeta }^{(i)})=\frac{q_r({\zeta }^{(i)})}{q({\zeta }^{(i)})}$, we have

$$g({\zeta }^{(i)})\le \exp \left(\frac{r^2}{2}\right)<\infty .$$

Thus,

$$E|\frac{1}{n}\sum _{i=1}^{n}g({\zeta }^{(i)}){|}^v\le \exp \left(\frac{\upsilon r^2}{2}\right)<\infty .$$

By the δ-method, the v^th moment of the IDR estimator is finite. □

Following Theorem 2.1, we can see that the variance of IDR is finite. This result also implies that the IDR method is potentially efficient in two-dimensional problems if the posterior distribution approximates a bivariate normal distribution. To examine the empirical property of IDR, we carry out a simulation study in Section 5.1.

3. Dimension Reduced Inflated Density Ratio (Dr.IDR) Estimator

In this section, we develop a variation of IDR called the dimension reduced inflated density ratio estimator (Dr.IDR). Suppose ζ′ = (θ′, ξ′), where θ is a p₁-dimensional vector of the parameters and ξ is a vector of the remaining p − p₁ ≡ p₂ parameters. We assume that the support of the marginal posterior density π(θ|D) = ∫ π(θ, ξ|D)dξ is R^p₁ and θ₀ is a ‘center’, such as the mode, of the marginal posterior distribution of θ. Then, we define the perturbed density as

$$q_r(\theta ,\xi )=\{\begin{array}{cc}q({\theta }_0,\xi )\hfill & \text{if}\Vert \theta -{\theta }_0\Vert \le r,\hfill \\ q({\theta }_0+h(\theta -{\theta }_0),\xi )\hfill & \text{if}\Vert \theta -{\theta }_0\Vert >r,\hfill \end{array}$$ (8)

where $h(\theta -{\theta }_0)={(1-\frac{r^{p_1}}{{\parallel \theta -{\theta }_0\parallel }^{p_1}})}^{1∕p_1}(\theta -{\theta }_0)$ and r > 0 is a prespecified radius. Similar to the IDR identity given in (4), we obtain a new identity given by

$$c=\frac{b_r\int q({\theta }_0,\xi )d\xi }{E\left[\frac{q_r(\theta ,\xi )}{q(\theta ,\xi )}|D\right]-1},$$ (9)

where $b_r=\frac{{\pi }^{p_1∕2}}{\Gamma (p_1∕2+1)}{r}^{p_1}$ and $E\left[\frac{q_r(\theta ,\xi )}{q(\theta ,\xi )}\mid D\right]$ is the expectation of $\frac{q_r(\theta ,\xi )}{q(\theta ,\xi )}$ with respect to the posterior distribution given by (2).

Let {(θ⁽ⁱ⁾, ξ⁽ⁱ⁾), i = 1, …, n} denote an MC/MCMC sample from the posterior distribution π(ζ|D). If ∫ q(θ₀, ξ)dξ is analytically available, using (9), a Dr.IDR estimate of c is given by

$${\widehat{c}}_{\text{Dr}.\text{IDR}}=\frac{b_r\int q({\theta }_0,\xi )d\xi }{\frac{1}{n}{\sum }_{i=1}^n\frac{q_r({\theta }^{(i)},{\xi }^{(i)})}{q({\theta }^{(i)},{\xi }^{(i)})}-1}.$$ (10)

Under certain Ergodic conditions, we have ${\lim }_{n\to \infty }{\widehat{c}}_{\mathrm{Dr}.\mathrm{IDR}}\stackrel{a.s.}{=}c$.

Remark 3.1: In (9) and (10), the perturbed density q_r(θ, ξ) in (8) is constructed on θ and the dimension of θ is p₁, which is smaller than p for the original vector of the parameters, ζ. When p₁ = p, (9) and (10) reduce to (4) and (5). For these reasons, ${\widehat{c}}_{\mathrm{Dr}.\mathrm{IDR}}$ is called the dimension reduced IDR (Dr.IDR) estimator.

Remark 3.2: The perturbed density in (8) can be extended as

$$q_r(\theta ,\xi )=\{\begin{array}{cc}q({\theta }_0(\xi ),\xi )\hfill & \text{if}\Vert \theta -{\theta }_0(\xi )\Vert \le r(\xi ),\hfill \\ q({\theta }_0(\xi )+h(\theta -{\theta }_0(\xi )),\xi )\hfill & \text{if}\Vert \theta -{\theta }_0(\xi )\Vert >r(\xi ),\hfill \end{array}$$ (11)

where θ₀(ξ) and r(ξ) are two functions of ξ. However, the Dr.IDR estimator in (10) needs to be modified as

$${\widehat{c}}_{\text{Dr}.\text{IDR}}=\frac{\int q({\theta }_0(\xi ),\xi )d\xi }{\frac{1}{n}{\sum }_{i=1}^n\frac{1}{b_{r(\xi )}}\left[\frac{q_r({\theta }^{(i)},{\xi }^{(i)})}{q({\theta }^{(i)},{\xi }^{(i)})}-1\right]},$$ (12)

where $b_{r(\xi )}=\frac{{\pi }^{p_1∕2}}{\Gamma (p_1∕2+1)}r{(\xi )}^{p_1}$. In (12), both the center and the radius of each perturbed circle can be determined by the conditional distribution π(θ|ξ, D) instead of the marginal posterior distribution of π(θ|D), which may lead to a more efficient estimator.

Remark 3.3: In (10), we assume that ∫ q(θ₀, ξ)dξ is analytically available. When c(θ₀) = ∫ q(θ₀, ξ)dξ is not available in the closed form, c(θ₀) is the normalizing constant of the conditional posterior distribution of π(ξ|θ₀, D). Since θ₀ is a fixed point, we can generate a MC/MCMC sample {ξ⁽ⁱ⁾, i = 1, …, n} from π(ξ|θ₀, D). Then, we can use (5) or (10) to construct an estimator for c(θ₀) and the resulting estimator of c may be termed as a Dr.IDR² estimator. But, we notice that the version of ${\widehat{c}}_{\mathrm{Dr}.\mathrm{IDR}}$ given in (12) is difficult to extend if ∫ q(θ₀(ξ), ξ)dξ is analytically intractable.

Remark 3.4: From the Dr.IDR identity in (9), we obtain

$$\int [q({\theta }_0,\xi )/c]d\xi =\frac{E\left[\frac{q_r(\theta ,\xi )}{q(\theta ,\xi )}|D\right]-1}{b_r}.$$ (13)

Since the term on the left hand side is precisely the marginal posterior density of θ evaluated at θ₀, we obtain a new marginal posterior density estimator given by

$${\widehat{\pi }}_{\text{Dr}.\text{IDR}}({\theta }_0|D)=\frac{1}{b_r}\left[\frac{1}{n}\sum _{i=1}^n\frac{q_r({\theta }^{(i)},{\xi }^{(i)})}{q({\theta }^{(i)},{\xi }^{(i)})}-1\right].$$ (14)

Write

$$w(\theta |\xi )=\frac{q_r(\theta ,\xi )-q(\theta ,\xi )}{b_{r}q({\theta }_0,\xi )}.$$ (15)

It is easy to show that ∫ w(θ|ξ)dθ = 1 and also (14) can be rewritten as

$${\widehat{\pi }}_{\text{Dr}.\text{IDR}}({\theta }_0|D)=\frac{1}{n}\sum _{i=1}^{n}w({\theta }^{(i)}|{\xi }^{(i)})\frac{q({\theta }_0,{\xi }^{(i)})}{q({\theta }^{(i)},{\xi }^{(i)})}.$$ (16)

Therefore, ${\widehat{\pi }}_{\mathrm{Dr}.\mathrm{IDR}}({\theta }_0\mid D)$ is an importance weighted marginal density estimator (IWMDE) of [14]. However, the weight function w(θ|ξ) is not the conditional posterior density π(θ|ξ, D). Thus, the IDR marginal posterior density estimator ${\widehat{\pi }}_{\mathrm{Dr}.\mathrm{IDR}}({\theta }_0\mid D)$ may not be most efficient as the “best” estimator is the conditional marginal density estimator (CMDE) with w(θ|ξ) = π(θ|ξ, D) as discussed in [14] and [15].

4. Generalization of the IDR Estimator

Observing that the marginal posterior density of θ is defined as

$$\pi (\theta |D)=\int \frac{q(\theta ,\xi )}{c}d\xi ,$$

we have the following identity

$$c=\frac{\int q(\theta ,\xi )d\xi }{\pi (\theta |D)}=\frac{c(\theta )}{\pi (\theta |D)}.$$ (17)

The above marginal posterior density based identity is a natural extension of the identity given in Chib (1995), in which the marginal likelihood is expressed as a ratio of the likelihood function times the prior and the posterior distribution of the model parameters.

We let c(θ) denote the term in the numerator of (17), i.e.,

$$c(\theta )=\int q(\theta ,\xi )d\xi .$$ (18)

Then, c(θ) is precisely the normalizing constant of the conditional posterior density of ξ given θ. We note that the identity (17) holds for all θ. Similar to [3], we take θ = θ₀ as a high marginal density point such as the posterior mean or the posterior mode of θ. Then, (17) can be rewritten as

$$c=\frac{c({\theta }_0)}{\pi ({\theta }_0|D)}.$$ (19)

In (19), the original p-dimensional estimation problem of c becomes the computation of c(θ₀) and π(θ₀|D) each with smaller dimension (p₂ and p₁). If c(θ₀) is analytically tractable, then we only need to compute π(θ₀|D). If c(θ₀) is intractable, as discussed in Remark 3.3, we first generate an MC/MCMC sample ξ⁽ⁱ⁾, i = 1, …, n} from the conditional posterior distribution π(ξ|θ₀, D) and then we can obtain an IDR estimator or a Dr.IDR estimator of c(θ₀). For instance, an IDR estimator of c(θ₀) is given by

$$\widehat{c}({\theta }_0)=\frac{b_{r}q({\theta }_0,{\xi }_0)}{\frac{1}{n}{\sum }_{i=1}^n\frac{q_r({\theta }_0,{\xi }^{(i)})}{q({\theta }_0,{\xi }^{(i)})}-1},$$ (20)

where ξ₀ is the posterior mode of ξ, $b_r=\frac{{\pi }^{p_2∕2}}{\Gamma (p_2∕2+1)}{r}^{p_2}$,

$$q_r({\theta }_0,\xi )=\{\begin{array}{cc}q({\theta }_0,{\xi }_0)\hfill & \text{if}\Vert \xi -{\xi }_0\Vert \le r,\hfill \\ q({\theta }_0,{\xi }_0+h(\xi -{\xi }_0))\hfill & \text{if}\Vert \xi -{\xi }_0\Vert >r,\hfill \end{array}$$

and $h(\xi -{\xi }_0)={(1-\frac{r^{p_2}}{{\parallel \xi -{\xi }_0\parallel }^{p_2}})}^{1∕p_2}(\xi -{\xi }_0)$ and r > 0 is a prespecified radius, which may be different than the one in (3) or (8).

For the marginal posterior density π(θ₀|D), which is the term in the denominator of (19), a Dr.IDR estimator ${\widehat{\pi }}_{\mathrm{Dr}.\mathrm{IDR}}({\theta }_0\mid D)$ given in (16) is readily available using an MC/MCMC sample {(θ⁽ⁱ⁾, ξ⁽ⁱ⁾), i = 1, …, n} from the full posterior distribution π(ζ|D). As discussed in Remark 3.4, ${\widehat{\pi }}_{\mathrm{Dr}.\mathrm{IDR}}({\theta }_0\mid D)$ may not be most efficient. A more efficient estimator can be obtained using the CMDE of [16] or the partition weighted marginal density estimator (PWMDE) of [15]. We note that the PWMDE is a special case of the IWMDE of [14], which has the potential to achieve the “optimal” CMDE in estimating π(θ₀|D). As empirically shown in [15], the PWMDE of π(θ₀|D) is more efficient when θ₀ is close to the “center” of the marginal posterior distribution π(θ|D). This is intuitively appealing since (i) the PWMDE uses the MC/MCMC sample {(θ⁽ⁱ⁾, ξ⁽ⁱ⁾), i = 1, …, n} from the full posterior distribution π(ζ|D); (ii) fewer points of the θ⁽ⁱ⁾’s are around θ₀ when θ₀ is away from the “center”; and (iii) in this case, π(θ₀|D) is more difficult to estimate and, consequently, the corresponding the PWMDE becomes less efficient. Therefore, the identity (19) works the best when θ₀ is chosen as a high marginal density point.

The new identity (19) provides us a great flexibility in computing the marginal likelihood c. This identity also sheds light on obtaining an efficient Monte Carlo estimator of c for a posterior distribution with parameter space of high-dimension. An estimator based on (19) is termed as a generalized dimension reduced (GDr) estimator. In Section 5.2, we carry out a simulation study to examine the empirical performance of the IDR, Dr.IDR, and GDr estimators.

5. Simulation Studies

5.1. Simulation Study I

In this simulation study, we empirically examine whether the result established in Theorem 2.1 implies a potential gain in estimation efficiency of an IDR estimator for computing the normalizing constant of the bivariate normal distribution compared to the univariate normal distribution or the multivariate normal distribution with more than two dimensions. Assume that the posterior kernel is given by

$$q(\zeta )={\left(\frac{1}{2\pi }\right)}^{p/2}\exp \left(-\frac{{\mu }^{\prime }\mu }{2}\right),$$ (21)

where μ is a vector of p unknown parameters. We apply the IDR estimator to compute the normalizing constant of (21) with different p’s, and evaluate its performance by comparing the means, standard errors, and root mean square errors (RMSE) of estimates in log scale. Let ${\widehat{c}}_t$ denote the IDR estimate for the t^th replicate of data and t = 1, 2, …, 1000. We have the mean and the standard error (SE) equal to $\tilde{c}={\sum }_{t=1}^{1000}\log {\widehat{c}}_t∕1000$ and $\sqrt{{\sum }_{t=1}^{1000}{(\log {\widehat{c}}_t-\tilde{c})}^2∕999}$, respectively. The RMSE is defined as

$$RMSE=\sqrt{\frac{1}{1000}\sum _{t=1}^{1000}{[\log {\widehat{c}}_t-\log c]}^2},$$

where log c is 0 in this case. The results for p = 1, 2, 3, and 5 are summarized in Table 1. We see that when r ≤ 1, the IDR estimator for p = 2 has the superior performance compared to those for p ≠ 2. Interestingly, the IDR estimator for p = 3 has a smaller SE than the one for p = 1 for every value of r considered in this simulation. However, the IDR estimator for p = 5 has the worst performance among those for p = 1, 2, and 3. These simulation results empirically confirm that the result established in Theorem 2.1 does imply that > the IDR estimator has an “optimal” performance when p = 2 under the normal distributions.

Table 1:

Performance of the IDR method in p—dimensional normal distribution with 1000 replicates: mean, standard error (SE), and root mean squared error (RMSE) of the normalizing constant estimate in log scale

		Mean	SE	RMSE			Mean	SE	RMSE
r=1.5	p=1	−0.00019	0.03222	0.03221	r=0.5	p=1	−0.00021	0.01260	0.01260
	p=2	0.00050	0.00740	0.00742		p=2	0.00003	0.00216	0.00216
	p=3	−0.00005	0.00537	0.00537		p=3	−0.00009	0.00574	0.00574
	p=5	0.00017	0.01163	0.01163		p=5	0.00014	0.03159	0.03157
r=1	p=1	−0.00031	0.01938	0.01937	r=0.01	p=1	−0.00012	0.00800	0.00800
	p=2	−0.00023	0.00450	0.00451		p=2	0.00000	0.00004	0.00004
	p=3	0.00008	0.00458	0.00457		p=3	−0.00016	0.00754	0.00754
	p=5	0.00077	0.01798	0.01799		p=5	0.00420	0.06300	0.06311

5.2. Simulation Study II

In the second simulation study, we revisit the bivariate normal example in [11]. Assume that y_j|μ, $\sum \stackrel{i.i.d.}{~}N(\mu ,\sum )$ for j = 1, …, m where μ = (μ₁, μ₂)′, and $\sum =\left(\begin{array}{cc}\hfill {\sigma }_1^2\hfill & \hfill \rho {\sigma }_1{\sigma }_2\hfill \\ \hfill \rho {\sigma }_1{\sigma }_2\hfill & \hfill {\sigma }_2^2\hfill \end{array}\right)$ are the unknown means and the variance-covariance matrix. We specify the conjugate priors

$$\sum \sim I{W}_{{\nu }_0}({\Lambda }_0^{-1}),$$

and

$$\mu |\sum \sim N({\mu }_0,\sum /{\kappa }_0),$$

respectively. Therefore, the joint posterior kernel is given by

$$\begin{gathered} q(\mu ,\sum )=f({\mathit{y}}_1,{\mathit{y}}_2,\mathrm{\dots },{\mathit{y}}_m|\mu ,\sum )\pi (\mu |\sum )\pi (\sum ) \\ ={(2\pi )}^{-m}|\sum {|}^{-(m+{\nu }_0+2)/2-1}\frac{1}{Z}\exp \left(-\frac{1}{2}\underset{j=1}{\overset{m}{\Sigma }}{({\mathit{y}}_j-\mu )}^{\prime }{\sum }^{-1}({\mathit{y}}_j-\mu )\right)\times \\ \exp \left(-\frac{{\kappa }_0}{2}{(\mu -{\mu }_0)}^{\prime }{\sum }^{-1}(\mu -{\mu }_0)\right)\exp (-\frac{1}{2}trace({\Lambda }_0{\sum }^{-1})), \end{gathered}$$

with Z = 2π2^ν₀Γ₂(ν₀/2)|Λ₀|^−ν₀/2/κ₀. Also, the marginal likelihood has the closed form

$$c=\frac{1}{{\pi }^{md/2}}\frac{{\Gamma }_d({\nu }_m/2)}{{\Gamma }_d({\nu }_0/2)}\frac{|{\Lambda }_0{|}^{{\nu }_0/2}}{|{\Lambda }_m{|}^{{\nu }_m/2}}{(\frac{{\kappa }_0}{{\kappa }_m})}^{d/2},$$

where ${\Lambda }_m={\Lambda }_0+S^2+\frac{{\kappa }_{0}m}{{\kappa }_0+m}({\mu }_0-\stackrel{‒}{\mathit{y}}){({\mu }_0-\stackrel{‒}{\mathit{y}})}^{\prime }$, $S^2={\sum }_{j=1}^m({\mathit{y}}_j-\stackrel{‒}{\mathit{y}}){({\mathit{y}}_j-\stackrel{‒}{\mathit{y}})}^{\prime }$, κ_m = κ₀ + m, and ν_m = ν₀ + m. A random sample y with m = 200 is generated from a bivariate normal distribution with μ = (0, 0)′ and $\sum =\left(\begin{array}{cc}\hfill 1\hfill & \hfill 0.7\hfill \\ \hfill 0.7\hfill & \hfill 1\hfill \end{array}\right)$. The corresponding sample mean $\stackrel{‒}{\mathit{y}}$ is (−0.029, 0.040)′, and the sample variance-covariance matrix S is $\left(\begin{array}{cc}\hfill 201.987\hfill & \hfill 143.330\hfill \\ \hfill 143.330\hfill & \hfill 192.365\hfill \end{array}\right)$. Under this setting and the pre-specified hyper-parameters μ₀ = (0, 0)′, k₀ = 0.01, ν₀ = 3, and ${\Lambda }_0=\left(\begin{array}{cc}\hfill 1\hfill & \hfill 0.7\hfill \\ \hfill 0.7\hfill & \hfill 1\hfill \end{array}\right)$, the marginal likelihood in log scale is −507.278.

To implement the IDR estimator, we consider two “center” values of (−0.0281, 0.0401, 0.0097, 1.8471, −0.0389)’ and (0.0437, 0.1101, 0.1093, 1.9891, 0.0615)’ for the transformed parameter vector of ${({\mu }_1,{\mu }_2,\log {\sigma }_1^2,\log \{(\rho +1)∕(1-\rho )\},\log {\sigma }_2^2)}^{\prime }$, where (−0.0281, 0.0401, 0.0097, 1.8471, −0.0389)′ is the posterior mean.

To implement the Dr.IDR estimator, we consider two blocks as θ = (μ₁, μ₂)′ and ξ = (σ₁, σ₂, ρ)′. First, since θ|ξ, $D~N({\mu }_c=(m\stackrel{‒}{\mathit{y}}+{\nu }_0{\mu }_0)∕(m+{\nu }_0),{\sum }_c=\sum ∕(m+{\nu }_0))$, we consider the standardization ${\theta }^{\ast }={\sum }_c^{-1∕2}(\mu -{\mu }_c)$. With such a standardization, we then have $\int \tilde{q}(0,\xi )d\xi =\int q({\mu }_c,\sum ){\mid {\sum }_c\mid }^{1∕2}d\sum =c∕(2\pi )$ and $\frac{{\tilde{q}}_r({\theta }^{\ast },\xi )}{\tilde{q}({\theta }^{\ast },\xi )}$

$$\begin{array}{l}=\{\begin{array}{cc}\exp \left(\frac{1}{2}{\theta }^{{\ast }^{\prime }}{\theta }^{\ast }\right)\hfill & \text{if}\Vert {\theta }^{\ast }\Vert \le r,\hfill \\ \frac{\exp \left(-\frac{1}{2}{\left[{\left(1-\frac{r^2}{\left|\right|{\theta }^{\ast }|{|}^2}\right)}^{1/2}{\theta }^{\ast }\right]}^{\prime }\left[{\left(1-\frac{r^2}{\left|\right|{\theta }^{\ast }|{|}^2}\right)}^{1/2}{\theta }^{\ast }\right]\right)}{\exp \left(-\frac{1}{2}{\theta }^{\ast }{}^{\prime }{\theta }^{\ast }\right)}=\exp \left(\frac{1}{2}{r}^2\right)\hfill & \text{if}\Vert {\theta }^{\ast }\Vert >r,\hfill \end{array}\hfill \end{array}$$

where $\tilde{q}(.)$ is the posterior kernel of (θ*, ξ) and ${\tilde{q}}_r(.)$ is the perturbed density of $\tilde{q}(.)$. For the Dr.IDR estimator, we also consider ${\theta }^{\ast }={\sum }_c^{-1∕2}(\mu -{\mu }_c-{(0.0709,0.0709)}^{\prime })$, where 0.0709 is the average of the posterior standard deviations of μ₁ and μ₂, for a sensitivity analysis.

For the GDr estimator, we consider two blocks as θ = (σ₁, σ₂, ρ)′ and ξ = (μ₁, μ₂)′ instead. Note that the posterior mean for θ is (1.015, 0.967, 0.727)′ and the posterior mode for θ is (0.985, 0.939, 0.727)′. We first choose θ₀ = (1, 1, 0.7)′ (also θ₀ = (0.5, 0.5, 0)′ for a sensitivity analysis) and estimated c(θ₀) by the IDR approach. By using the same standardization ${\xi }^{\ast }={\sum }_c^{-1∕2}(\mu -{\mu }_c)$, (20) becomes

$$\widehat{c}({\theta }_0)=\frac{\pi r^2\overline{q}({\theta }_0,0)}{\frac{1}{n}{\sum }_{i=1}^n\frac{\overline{q_r}({\theta }_0,{\xi }^{\ast (i)})}{\overline{q}({\theta }_0,{\xi }^{\ast (i)})}-1},$$

where $\stackrel{‒}{q}({\theta }_0,0)=q(1,1,0.7,{\mu }_c){\mid {\sum }_c\mid }^{1∕2}$ with the corresponding perturbed density $\overline{q_r}(.)$, and

$$\frac{\overline{q_r}({\theta }_0,{\xi }^{\ast })}{\overline{q}({\theta }_0,{\xi }^{\ast })}=\{\begin{array}{cc}\exp \left(\frac{1}{2}{\xi }^{{\ast }^{\prime }}{\xi }^{\ast }\right)\hfill & \text{if}\Vert {\xi }^{\ast }\Vert \le r,\hfill \\ \exp \left(\frac{1}{2}{r}^2\right)\hfill & \text{if}\Vert {\xi }^{\ast }\Vert >r.\hfill \end{array}$$

Secondly, we use the CMDE to estimate π(θ₀|D) since π(θ|ξ, D) is an inverse Wishart distribution with v_m + 1 degrees of freedom and scale matrix ${\Lambda }_0+{\sum }_{j=1}^m({\mathit{y}}_j-\xi ){({\mathit{y}}_j-\xi )}^{\prime }+{\kappa }_0(\xi -{\mu }_0){(\xi -{\mu }_0)}^{\prime }$. Specifically $\widehat{\pi }({\theta }_0\mid D)=\frac{1}{n}{\sum }_{i=1}^n\pi ({\theta }_0\mid {\xi }^{(i)},D)$. We note that for the GDr estimator, {${\theta }^{(i)}={({\sigma }_1^{(i)},{\sigma }_2^{(i)},{\rho }^{(i)})}^{\prime }$, ξ⁽ⁱ⁾ = μ⁽ⁱ⁾, i = 1, …, n} is a MC sample from the posterior distribution with the kernel q(μ, ∑), ${\sum }^{(i)}=\left(\begin{array}{cc}\hfill {({\sigma }_1^{(i)})}^2\hfill & \hfill {\rho }^{(i)}{\sigma }_1^{(i)}{\sigma }_2^{(i)}\hfill \\ \hfill {\rho }^{(i)}{\sigma }_1^{(i)}{\sigma }_2^{(i)}\hfill & \hfill {({\sigma }_2^{(i)})}^2\hfill \end{array}\right)$, ${\sum }_c^{(i)}={\sum }^{(i)}∕(m+{\nu }_0)$, and ${\xi }^{\ast (i)}={\left({\sum }_c^{(i)}\right)}^{-1∕2}({\mu }^{(i)}-{\mu }_c)$ for i = 1, …, n.

Considering that the GDr estimator requires an extra MCMC sample, we set the size (n = 5, 000) of each MCMC sample for the GDr estimator as half of the MCMC sample size (n = 10, 000) for the IDR and Dr.IDR estimators in each replicate. Table 2 summarizes the results of IDR, Dr.IDR, and GDr estimators based on 1,000 replicates when r = 0.001, 0.5, 1, 1.5. It is not surprising that the IDR estimator is sensitive to the specification of the “center” value as the values of SE and RMSE in columns 6 and 7 are much larger than those corresponding values in columns 3 and 4. As expected, both the Dr.IDR and GDr estimators outperform the IDR estimator. The Dr.IDR estimator has the best performance when the posterior mean is used in the standardization since in this case it enjoys the dimensional reduction and the closed form of the conditional marginal likelihood, which is estimated in the GDr estimator. The GDr estimator yields the results comparable to the Dr.IDR estimator. It is interesting to see that unlike the IDR estimator, both the Dr.IDR and GDr estimators are quite robust in the choices of the “center” values of μ_c for Dr.IDR and θ₀ for GDr. Certainly, better choices of μ_c and θ₀ do yield slightly smaller values of SE and RMSE as expected.

Table 2:

Simulation Results of the IDR, Dr.IDR, and GDr estimators for a bivariate normal example with 1000 replicates: mean, standard error (SE), and root mean squared error (RMSE) of the marginal likelihood estimate in the log scale

	Mean	SE	RMSE	Mean	SE	RMSE
	IDR with a “center” value of ${({\mu }_1,{\mu }_2,\log {\sigma }_1^2,\log \{(\rho +1)∕(1-\rho )\},\log {\sigma }_2^2)}^{\prime }$
	(−0.0281, 0.0401, 0.0097, 1.8471, −0.0389)’			(0.0437, 0.1101, 0.1093, 1.9891, 0.0615)’
r=1.5	−509.124	0.141	1.852	−511.609	0.310	4.342
r=1.0	−508.910	0.043	1.633	−511.118	0.166	3.844
r=0.5	−508.774	0.052	1.497	−510.597	0.250	3.329
r=0.001	−508.711	0.137	1.440	−511.592	0.373	4.331
	Dr.IDR with
	${\sum }_c^{-0.5}(\mu -{\mu }_c)$			${\sum }_c^{-0.5}(\mu -{\mu }_c-{(0.0709,0.0709)}^{\prime })$
r=1.5	−507.278	0.007	0.007	−507.279	0.019	0.019
r=1.0	−507.278	0.004	0.004	−507.278	0.016	0.016
r=0.5	−507.278	0.002	0.002	−507.278	0.015	0.015
r=0.001	−507.278	0.000	0.000	−507.276	0.030	0.030
	GDr with (σ10, σ20, ρ0)′
	(1, 1, 0.7)’			(0.5,0.5,0.0)’
r=1.5	−507.278	0.010	0.010	−507.279	0.019	0.019
r=1.0	−507.277	0.006	0.006	−507.279	0.017	0.017
r=0.5	−507.277	0.003	0.003	−507.279	0.016	0.016
r=0.001	−507.278	0.001	0.001	−507.278	0.016	0.016

Remark 5.1: From Table 1 in Simulation Study I, we see that the IDR estimator has the smallest SE and RMSE when r = 0.01 for p =1, 2, r = 1 for p = 3, and r = 1.5 for p = 5. The results shown in Table 2 in Simulation Study II indicate that a smaller but not too small value of r generally yields a smaller SE or RMSE for IDR, Dr.IDR, as well as GDr. In practice, as suggested in [13], we first calculate the IDR, Dr.IDR, and GDr estimates for a grid of different values of r, then use the overlapping batch statistics approach of [17] to compute the Monte Carlo errors of these estimates, and finally choose an “optimal” r* as the r for which the Monte Carlo errors are minimum.

Remark 5.2: In this simulation study, the two blocks θ = (μ₁, μ₂)^′ and ξ = (σ₁, σ₂, ρ)′ yielded an efficient Dr.IDR estimator as shown in Table 2. Another natural combination of these parameters would be θ = (σ₁,σ₂,ρ)′ and ξ = (μ₁, μ₂)′. Under this combination, the dimension of the reduced parameter space corresponding to θ is 3 while an additional transformation (log σ₁, log σ₂, log{(ρ + 1)/(1 − ρ)})′ is also needed. After this transformation, the full conditional distribution given ξ is not available analytically. Furthermore, the closed-form expression of θ₀(ξ) in (11) for the perturbed density is also no longer available. Thus, this combination of the parameters leads to a less efficient and more computationally extensive implementation of Dr.IDR. Thus, an appropriate combination of the parameters should be carefully selected in order to implement Dr.IDR efficiently.

6. A Case Study: Inequality-Constrained Analysis of Variance

In this section, we consider the inequality-constrained analysis of variance model to demonstrate how to compute the normalizing constant using the GDr estimator. We use the data in Hoijtink et al. [18] for studying amnesia in patients with dissociative identity disorder (DID). We call this data set as the DID data thereafter. We model memory performance score y_kj for the k^th subject in group j as an independent observation from a normal distribution with mean μ_j and variance σ² for k = 1, 2, …, m_j and j = 1, …, J. In the DID data, J = 4, m₁ = 19, m₂ = 25, m₃ = 25, and m₄ = 25. The sample means of the memory performance scores are 3.105, 13.28, 1.88, and 4.56, respectively, for these four groups. Let $D=\{m,\mathit{y}\}=\{m={\sum }_{j=1}^J{m}_j,y_{11},\dots ,y_{m_{1}1},\dots ,y_{1J},\dots ,y_{m_{J}J}\}$ denote the observed data. Then, the likelihood function is given by

$$L(\mu ,{\sigma }^2\mid D)={(2\pi {\sigma }^2)}^{-\frac{m}{2}}\exp \left\{-\frac{1}{2{\sigma }^2}\sum _{j=1}^J\sum _{k=1}^{m_j}{(y_{k_j}-{\mu }_j)}^2\right\},$$

where μ = (μ₁, …, μ_J)′. Let Ω denote the parameter space. Following Chen and Kim [19] and Wang et al. [15], we specify the joint prior of (μ, σ²) as

$$\begin{gathered} \pi (\mu ,{\sigma }^2|a_0)\propto {\pi }^{\ast }(\mu ,{\sigma }^2|a_0)=\left[{(2\pi {\sigma }^2)}^{-\frac{J}{2}}\prod _{j=1}^J{(a_0{m}_j)}^{1/2}\exp \left\{-\frac{a_0{m}_j}{2{\sigma }^2}{\mu }_j^2\right\}\right] \\ \times \frac{b_{02}^{b_{01}}}{\Gamma (b_{01})}{({\sigma }^2)}^{-b_{01}-1}\exp \left(-\frac{b_{02}}{{\sigma }^2}\right), \end{gathered}$$ (22)

for (μ, σ²) ∈ Ω where a₀ > 0 is a scalar parameter and b₀₁ > 0 and b₀₂ > 0 are prespecified hyperparameters.

Following Chen and Kim [19] and Wang et al. [15], we specify b₀₁ = b₀₂ = 0.0001 and a₀ = 0.01. We also consider the constrained parameter space: Ω = {μ₂ > (μ₁, μ₄) > μ₃, σ² > 0}. Under this setting, the posterior kernel is given by

$$q(\mu ,{\sigma }^2)=L(\mu ,{\sigma }^2|D){\pi }^{\ast }(\mu ,{\sigma }^2|a_0),$$ (23)

where π*(μ, σ²|a₀) is defined in (22). We are interested in computing

$$c={\int }_{\Omega }q(\mu ,{\sigma }^2)d\mu d{\sigma }^2.$$ (24)

Under the constrained parameter space, c in (24) is a normalizing constant but it is not the marginal likelihood since ${\int }_{\Omega }{\pi }^{\ast }(\mu ,{\sigma }^2\mid a_0)d\mu d{\sigma }^2\ne 1$. Also, the closed form expression of c in (24) is not available. However, for the unconstrained parameter space Ω_U = {(μ, σ²) : −∞ < μ_j < ∞, j = 1, …, 4, σ² > 0}, we have

$$\begin{gathered} c_U={{\int }_{{\Omega }_U}q(\mu ,{\sigma }^2)d\mu d{\sigma }^2=(2\pi )}^{-\frac{m}{2}}\frac{b_{02}^{b_{01}}\Gamma (b_{01}+\frac{m}{2})}{\Gamma (b_{01})}{\left(\frac{a_0}{1+a_0}\right)}^{\frac{J}{2}} \\ \times {\left[b_{02}+\frac{1}{2}\underset{j=1}{\overset{J}{\Sigma }}\left\{\underset{k=1}{\overset{m_j}{\Sigma }}{(y_{kj}-\overline{y}{}_j)}^2+\frac{a_0{m}_j}{1+a_0}{({\overline{y}}_j)}^2\right\}\right]}^{-({\alpha }_0+\frac{m}{2})}, \end{gathered}$$ (25)

where ${\stackrel{‒}{y}}_j=\frac{1}{m_j}{\sum }_{k=1}^{m_j}{\mathit{yk}}_j$ for j = 1, …, J. Since ${\int }_{{\Omega }_U}{\pi }^{\ast }(\mu ,{\sigma }^2\mid a_0)d\mu d{\sigma }^2=1$, c_U is the marginal likelihood. Let 1{A} denote the indicator function such that 1{A} = 1 if A is true and 0 if A is false. Observe that

$$\begin{gathered} c={\int }_{{\Omega }_U}q(\mu ,{\sigma }^2)1\{(\mu ,{\sigma }^2)\in \Omega \}d\mu d{\sigma }^2=c_U{\int }_{{\Omega }_U}1\{(\mu ,{\sigma }^2)\in \Omega \}\frac{q(\mu ,{\sigma }^2)}{c_U}d\mu d{\sigma }^2 \\ =c_U{E}_U\left[1\{(\mu ,{\sigma }^2)\in \Omega \}|D\right], \end{gathered}$$ (26)

where the posterior expectation is taken with respect to the posterior distribution $\frac{q(\mu ,{\sigma }^2)}{c_U}$. From (26), we see that the posterior probability $P_U((\mu ,{\sigma }^2)\in \Omega \mid D)=E_U\left[1\{(\mu ,{\sigma }^2)\in \Omega \}\mid D\right]=\frac{c}{c_U}$.

Next, we present the detailed formulation of the GDr estimator of c and then use the identity given in (26) to empirically validate the accuracy of the GDr estimator since in this case, the “true” value of c is unknown. We set θ = (μ₁, μ₄, σ²)′ and ξ = (μ₂, μ₃). Let ζ = (θ′, ξ′)′. Let ${\theta }_0={({\mu }_{10},{\mu }_{40},{\sigma }_0^2)}^{\prime }$ denote the posterior mean of θ under the posterior distribution π(ζ|D) with the constrained parameter space. Using (19), we have

$$c=\frac{c({\mu }_{10},{\mu }_{40},{\sigma }_0^2)}{\pi ({\mu }_{10},{\mu }_{40},{\sigma }_0^2|D)},$$ (27)

where $c({\mu }_{10},{\mu }_{40},{\sigma }_0^2)={\int }_{\Omega }q({\mu }_{10},{\mu }_2,{\mu }_3,{\mu }_{40},{\sigma }_0^2)d{\mu }_{2}d{\mu }_3$. After some lengthy algebra, we obtain an analytic expression of $c({\mu }_{10},{\mu }_{40},{\sigma }_0^2)$ as follows

$$\begin{gathered} c({\mu }_{10},{\mu }_{40},{\sigma }_0^2)={(2\pi )}^{-\frac{m+J-2}{2}}\left[\underset{j=1}{\overset{J}{\Pi }}{(a_0{m}_j)}^{1/2}\right]\frac{b_{02}^{b_{01}}}{\Gamma (b_{01})}{({\sigma }_0^2)}^{-\left(\frac{m+J-2}{2}+b_{01}+1\right)} \\ \times {[{(a_0+1)}^2{m}_2{m}_3]}^{-1} \\ \times \exp \left(-\frac{1}{{\sigma }_0^2}\left[b_{02}+\frac{1}{2}\underset{j=1}{\overset{J}{\Sigma }}\left\{\underset{k=1}{\overset{m_j}{\Sigma }}{({\mathit{yk}}_j-{\overline{y}}_j)}^2+\frac{a_0}{1+a_0}{m}_j{\overline{y}}_j^2\right\}\right]\right) \\ \times \exp \left[-\frac{a_0+1}{2{\sigma }_0^2}\left\{m_1{\left({\mu }_{10}-\frac{{\overline{y}}_1}{a_0+1}\right)}^2+m_4{\left({\mu }_{40}-\frac{{\overline{y}}_4}{a_0+1}\right)}^2\right\}\right] \\ \times \left(1-\Phi \left[{\left\{\frac{(a_0+1)m_2}{{\sigma }_0^2}\right\}}^{\frac{1}{2}}\left({\mu }_{10}\vee {\mu }_{40}-\frac{{\overline{y}}_2}{a_0+1}\right)\right]\right) \\ \times \Phi \left[{\left\{\frac{(a_0+1)m_3}{{\sigma }_0^2}\right\}}^{\frac{1}{2}}\left({\mu }_{10}\wedge {\mu }_{40}-\frac{{\overline{y}}_3}{a_0+1}\right)\right], \end{gathered}$$

where μ₁₀ ∨ μ₄₀ = max{μ₁₀, μ₄₀}, μ₁₀ ∧ μ₄₀ = min{μ₁₀, μ₄₀}, and Φ(·) is the standard normal N(0, 1) cumulative distribution function. For the term in the denominator of (27), we write

$$\pi ({\mu }_{10},{\mu }_{40},{\sigma }_0^2|D)=\pi ({\mu }_{10},{\mu }_{40}|D)\pi ({\sigma }_0^2|{\mu }_{10},{\mu }_{40},D).$$

We generate one MCMC sample $\{({\mu }_{2,1}^{(i)},{\mu }_{3,1}^{(i)},{\sigma }_1^{2(i)}),i=1,\dots ,n_1\}$ from the posterior π(ζ|D) with the constrained parameters and another MCMC sample $\{({\mu }_{2,2}^{(i)},{\mu }_{3,2}^{(i)}),i=1,\dots ,n_2\}$ from the conditional posterior distribution π(μ₂, μ₃|μ₁₀, μ₄₀, D). Then, the CMDE of π(μ₁₀, μ₄₀|D) is given by

$$\widehat{\pi }({\mu }_{10},{\mu }_{40}|D)=\frac{1}{n_1}\sum _{i=1}^{n_1}\pi ({\mu }_{10}|{\mu }_{2,1}^{(i)},{\mu }_{3,1}^{(i)},{\sigma }_1^{2(i)},D)\pi ({\mu }_{40}|{\mu }_{2,1}^{(i)},{\mu }_{3,1}^{(i)},{\sigma }_1^{2(i)},D),$$

where

$$\begin{gathered} \pi ({\mu }_{10}|{\mu }_2,{\mu }_3,{\sigma }^2,D) \\ =\frac{{\left\{\frac{(a_0+1)m_1}{2\pi {\sigma }^2}\right\}}^{\frac{1}{2}}\exp \left\{-\frac{(a_0+1)m_1}{2{\sigma }^2}{\left({\mu }_{10}-\frac{{\overline{y}}_1}{a_0+1}\right)}^2\right\}}{\Phi \left[{\left\{\frac{(a_0+1)m_1}{{\sigma }^2}\right\}}^{\frac{1}{2}}\left({\mu }_2-\frac{{\overline{y}}_1}{a_0+1}\right)\right]-\Phi \left[{\left\{\frac{(a_0+1)m_1}{{\sigma }^2}\right\}}^{\frac{1}{2}}\left({\mu }_3-\frac{{\overline{y}}_1}{a_0+1}\right)\right]}, \\ \pi ({\mu }_{40}|{\mu }_2,{\mu }_3,{\sigma }^2,D) \\ =\frac{{\left\{\frac{(a_0+1)m_4}{2\pi {\sigma }^2}\right\}}^{\frac{1}{2}}\exp \left\{-\frac{(a_0+1)m_4}{2{\sigma }^2}{\left({\mu }_{40}-\frac{{\overline{y}}_4}{a_0+1}\right)}^2\right\}}{\Phi \left[{\left\{\frac{(a_0+1)m_4}{{\sigma }^2}\right\}}^{\frac{1}{2}}\left({\mu }_2-\frac{{\overline{y}}_4}{a_0+1}\right)\right]-\Phi \left[{\left\{\frac{(a_0+1)m_4}{{\sigma }^2}\right\}}^{\frac{1}{2}}\left({\mu }_3-\frac{{\overline{y}}_4}{a_0+1}\right)\right]}. \end{gathered}$$

The CMDE of $\pi ({\sigma }_0^2\mid {\mu }_{10},{\mu }_{40},D)$ is also available, which is given by

$$\widehat{\pi }({\sigma }_0^2|{\mu }_{10},{\mu }_{40},D)=\frac{1}{n_2}\sum _{i=1}^{n_2}\pi ({\sigma }_0^2|{\mu }_{10},{\mu }_{2,2}^{(i)},{\mu }_{3,2}^{(i)},{\mu }_{40},D),$$

where

$$\begin{gathered} \pi ({\sigma }_0^2|{\mu }_1,{\mu }_2,{\mu }_3,{\mu }_4,D) \\ =\frac{{\left[b_{02}+\frac{1}{2}{\sum }_{j=1}^J\left\{S_j^2+\frac{a_0}{a_0+1}{m}_j{\overline{y}}_j^2+(a_0+1)m_j{\left({\mu }_j-\frac{{\overline{y}}_j}{a_0+1}\right)}^2\right\}\right]}^{\frac{m+J}{2}+b_{01}}}{{({\sigma }_0^2)}^{\frac{m+J}{2}+b_{01}+1}\Gamma \left(\frac{m+J}{2}+b_{01}\right)} \\ \times \exp \left(-\frac{1}{{\sigma }_0^2}\left[b_{02}+\frac{1}{2}\underset{j=1}{\overset{J}{\Sigma }}\left\{S_j^2+\frac{a_0}{a_0+1}m{}_j\overline{y}{}_j^2+(a_0+1)m_j{\left({\mu }_j-\frac{{\overline{y}}_j}{a_0+1}\right)}^2\right\}\right]\right), \\ \text{and}{S}_j^2={\sum }_{k=1}^{m_j}{({\mathit{yk}}_j-{\stackrel{‒}{y}}_j)}^2. \end{gathered}$$

For the DID data, we first generate an MCMC sample of size 50,000 from each of the posterior distributions with or without constraints to compute the posterior means, the posterior standard deviations (SDs), and the 95% highest posterior density (HPD) intervals of μ and σ². The results are shown in Table 3. We see from this table that both sets of the posterior estimates are very close to each other, implying that the hypothesis of μ₂ > (μ₁, μ₄) > μ₃ is likely true. Notice that these four means are from the four groups consisting of DID, mimic normal, symptom simulated, and true amnesic subjects. Next, we compute log c_U as the closed form expression given in (25) and use the dropping-needle method discussed in Hoijtink et al. [18] to calculate $E_U\left[1\{(\mu ,{\sigma }^2)\in \Omega \}\mid D\right]$ using an MCMC sample {(μ⁽ⁱ⁾, σ²⁽ⁱ⁾), i = 1, …, n} from the posterior distribution without constraints on μ. We further generate two MCMC samples to estimate log c. Using (27), we have $\log \widehat{c}=\log c({\mu }_{10},{\mu }_{40},{\sigma }_0^2)-\log \widehat{\pi }({\mu }_{10},{\mu }_{40}\mid D)-\log \widehat{\pi }({\sigma }_0^2\mid {\mu }_{10},{\mu }_{40},D)$. The results of these Monte Carlo estimates are reported in Table 3. We see from Table 3 that the estimate ${\widehat{E}}_U[1\{(\mu ,{\sigma }^2)\in \Omega \}\mid D]$ based on the dropping-needle method is very close to $\widehat{c}∕C_U$ based on the GDr estimate for various MC sample sizes and ${\widehat{E}}_U\left[1\{(\mu ,{\sigma }^2)\in \Omega \}\mid D\right]\approx 0.988$. The high posterior probability of {ζ ∈ Ω} implies that the constraints on μ are most likely to hold and also provides a further explanation of why the two sets of the posterior estimates of μ and σ² in Table 3 are very similar. We note that since the GDr estimate requires two MCMC samples, we choose n₁ = n₂ = n/2 to make fair comparison between two different methods. These empirical results indicate that the “true” value of log(c) is very likely to be −205.645.

Table 3:

Posterior Estimates of μ and σ² and MC Estimates of the Posterior Probability and the Normalizing Constant for the DID Data

Parameter	Unconstrained Model			Constrained Model
	Posterior Mean	SD	95% HPD Interval	Posterior Mean	SD	95% HPD Interval
μ1	3.073	0.405	(2.271, 3.856)	3.085	0.393	(2.326, 3.865)
μ2	13.148	0.351	(12.465, 13.847)	13.149	0.354	(12.467, 13.863)
μ3	1.862	0.351	(1.170, 2.545)	1.856	0.349	(1.182, 2.550)
μ4	4.516	0.354	(3.836, 5.229)	4.516	0.351	(3.825, 5.197)
σ2	3.147	0.469	(2.282, 4.083)	3.145	0.471	(2.283, 4.085)
Monte Carlo Estimates of EU [1{(μ, σ2) ∈ Ω}|D] and c
log cU	n	${\widehat{E}}_U[1\{(\mu ,{\sigma }^2)\in \Omega \}\mid D]$		n1 = n2	$\log \widehat{c}$	$\widehat{c}∕c_U$
−205.63261	50,000	0.98810		25,000	−205.64522	0.98747
	200,000	0.98781		100,000	−205.64520	0.98748
	500,000	0.98776		250,000	−205.64470	0.98799

Finally, we note that instead of setting θ = (μ₁, μ₄, σ²)′ and ξ = (μ₂, μ₃), we can also take θ = (μ₂, μ₃, σ²)′ and ξ = (μ₁, μ₄). Let ${\theta }_0={({\mu }_{20},{\mu }_{30},{\sigma }_0^2)}^{\prime }$ denote the posterior mean of θ under the posterior distribution π(ξ|D) with the constrained parameter space. Then we have

$$c=\frac{c({\mu }_{20},{\mu }_{30},{\sigma }_0^2)}{\pi ({\mu }_{20},{\mu }_{30},{\sigma }_0^2|D)},$$ (28)

where $c({\mu }_{20},{\mu }_{30},{\sigma }_0^2)={\int }_{\Omega }q({\mu }_1,{\mu }_{20},{\mu }_{30},{\mu }_4,{\sigma }_0^2)d{\mu }_{1}d{\mu }_4$. Similar to (27), a closed form expression of $c({\mu }_{20},{\mu }_{30},{\sigma }_0^2)$ is available. For the term in the denominator of (28), we write

$$\pi ({\mu }_{20},{\mu }_{30},{\sigma }_0^2|D)=\pi ({\mu }_{20},{\mu }_{30}|D)\pi ({\sigma }_0^2|{\mu }_{20},{\mu }_{30},D).$$

Again, the CMDEs are available for estimating π(μ₂₀, μ₃₀|D) and $\pi ({\sigma }_0^2\mid {\mu }_{20},{\mu }_{30},D)$. This formulation is as efficient as the one given in (27).

7. Discussion

In this paper, we first examine properties of the IDR estimator and we find that the IDR estimator is most efficient in two dimensions when the distribution is approximately normal. We then develop an extension of the IDR estimator, called the Dr.IDR estimator. The Dr.IDR estimator is more attractive than the IDR estimator since it allows for dimension reduction and it also has a nice connection to marginal posterior density estimation. The GDr estimator is constructed using the identity based on the marginal posterior density. This new identity, which is given in (19), is a natural extension of the identity of Chib [3] based on the full posterior density. Both the Dr.IDR and GDr estimators are potentially useful in computing marginal likelihoods or normalizing constants for the models with high-dimensional parameters or the complex structure such as constraints on the model parameters and the autocorrelation structure in a time-series model or the spatial structure in a spatial-temporal model. Computing the normalizing constant in (24) for the DID data is quite interesting. We use two different approaches for computing this constant and the results are quite comparable. There are other methods, such as the stepping stone approaches of Xie et al. [9] and Fan et al. [10] and the PWK method of Wang et al. [11], that can be used to compute this normalizing constant. These additional comparisons and further investigation of applicability of the Dr.IDR and GDr estimators in high-dimensional problems are interesting future research projects, which are currently under investigation.