Bayesian Variable Selection and Computation for Generalized Linear Models with Conjugate Priors

Abstract

In this paper, we consider theoretical and computational connections between six popular methods for variable subset selection in generalized linear models (GLM’s). Under the conjugate priors developed by Chen and Ibrahim (2003) for the generalized linear model, we obtain closed form analytic relationships between the Bayes factor (posterior model probability), the Conditional Predictive Ordinate (CPO), the L measure, the Deviance Information Criterion (DIC), the Aikiake Information Criterion (AIC), and the Bayesian Information Criterion (BIC) in the case of the linear model. Moreover, we examine computational relationships in the model space for these Bayesian methods for an arbitrary GLM under conjugate priors as well as examine the performance of the conjugate priors of Chen and Ibrahim (2003) in Bayesian variable selection. Specifically, we show that once Markov chain Monte Carlo (MCMC) samples are obtained from the full model, the four Bayesian criteria can be simultaneously computed for all possible subset models in the model space. We illustrate our new methodology with a simulation study and a real dataset.

1 Introduction

Bayesian variable selection is still one of the most theoretically and computationally challenging problems encountered in practice due to issues regarding i) prior elicitation, ii) analytic evaluation of the model selection criterion, and iii) numerical computation of the criterion for all possible models in the model space. These issues have been discussed by many authors for various linear and generalized linear models including George and McCulloch (1993), Laud and Ibrahim (1995), George et al. (1996), Raftery (1996), Smith and Kohn (1996), George and McCulloch (1997), Raftery et al. (1997), Brown et al. (1998), Brown et al. (2002), Clyde (1999), Chen et al. (1999), Dellaportas and Forster (1999), Ibrahim et al. (1999), Chipman et al. (1998), Chipman et al. (2001), Chipman et al. (2003), George (2000), George and Foster (2000), Ibrahim et al. (2000), Ntzoufras et al. (2003), and Chen et al. (2003). Clyde and George (2004) present an excellent review article on Bayesian model selection and uncertainty, and give an excellent exposition of the theoretical and computational issues involved in Bayesian variable selection and Bayesian model uncertainty in general. An entire monograph devoted to Bayesian model selection is given by Lahiri (2001).

One of the important unresolved issues in Bayesian model selection and Bayesian variable selection in particular is what the analytic or empirical connections are between the various methods. For example, it is not clear what the relationship is between BIC and DIC, or DIC and the L measure, and whether one is a monotonic function of the other, and whether one can compute BIC from DIC or vice versa. A related question is that if one has MCMC samples from the full model, how can those samples be used to obtain all four Bayesian criteria mentioned above. To answer these questions, we investigate the following in this paper: (i) for the normal linear model with conjugate priors, we obtain analytic relationships between the Bayes factor, CPO, the L measure, DIC, AIC, and BIC, and (ii) for the class of GLM’s we show via the development of several theorems and identities how one can compute all of these Bayesian criteria simultaneously using only an MCMC sample from the full model.

The relationships obtained in (i) for the linear model shed light on the behavior and connections between these criteria for GLM’s. The development of (ii) above is important and useful since it establishes the computational relationships in the model space for each of the four Bayesian criteria and shows that for variable subset selection in GLM’s using the conjugate priors of Chen and Ibrahim (2003), we can compute the four Bayesian criteria for all possible 2^p subset models using only an MCMC sample from the full model with p covariates. Another important issue we examine in this paper is the performance of the conjugate priors proposed by Chen and Ibrahim (2003) in Bayesian variable subset selection. We demonstrate that these priors perform quite well in this context, and they are easy to specify and computationally feasible.

The rest of this paper is organized as follows. Section 2 gives formulas for each of the criteria under the conjugate priors of Chen and Ibrahim (2003) for GLM’s and Section 3 develops the theoretical connections between the six criteria for the normal linear model. Section 4 establishes the computational connections in the model space for the four Bayesian criteria and several key identities and theorems that are needed. Section 5 presents a detailed simulation study examining various properties of the six criteria, and Section 6 presents a real data example. We conclude the article with brief remarks in Section 7. All proofs are given in the Appendix.

2 The Method

2.1 Model and Notation

Suppose that {(x_i, y_i), i = 1, 2, …, n} are independent observations, where y_i is the response variable, and x_i = (1, x_i1, …, x_ik)′ is a (k + 1) × 1 random vector of covariates. Let ℳ denote the model space. We enumerate the models in ℳ by m = 1, 2,…,, where is the dimension of ℳ and model denotes the full model. Also, let β⁽⁾ = (β₀, β₁, …, β_k)′ denote the regression coefficients for the full model including an intercept, and let ${\mathbf{x}}_i^{(m)}$ and β^(m) denote k_m × 1 vectors of covariates and regression coefficients for model m with an intercept, and a specific choice of k_m − 1 covariates. We write ${\mathbf{x}}_i=({\mathbf{x}}_i^{(m{)}^{\prime }},{\mathbf{x}}_i^{(-m{)}^{\prime }}{)}^{\prime }$, and β⁽⁾ = (β^(m)′, β^(−m)′)′, where ${\mathbf{x}}_i^{(-m)}$ is x_i with ${\mathbf{x}}_i^{(m)}$ deleted and β^(−m) is β⁽⁾ with β^(m) deleted.

Under model m, the generalized linear model (GLM) is assumed for [ $y_i\mid {\mathbf{x}}_i^{(m)}$], which has the conditional density given by

$$f(y_i\mid {\mathbf{x}}_i^{(m)},{\mathit{\beta }}^{(m)},\tau )=exp\left[a_i^{-1}(\tau )\{y_i{\theta }_i^{(m)}-b({\theta }_i^{(m)})\}+c(y_i,\tau )\right],i=1,2,\dots ,n,$$ (1)

where ${\theta }_i^{(m)}=\theta ({\eta }_i^{(m)})$ is the canonical parameter, ${\eta }_i^{(m)}={\mathbf{x}}_i^{(m{)}^{\prime }}{\mathit{\beta }}^{(m)}$, and τ is a dispersion parameter. The functions a, b and c determine a particular family in the class. The functions a_i(τ) are commonly of the form $a_i(\tau )={\tau }^{-1}{w}_i^{-1}$, where the w_i’s are known weights. For ease of exposition, we assume throughout that τ = 1 and w_i = 1, as, for example, in logistic and Poisson regression. The methods proposed here can be easily extended to the case when τ is unknown. Under this assumption, (1) can be rewritten as

$$f(y_i\mid {\mathbf{x}}_i^{(m)},{\mathit{\beta }}^{(m)})=exp\left\{y_i{\theta }_i^{(m)}-b({\theta }_i^{(m)})+c(y_i)\right\},i=1,2,\dots ,n.$$ (2)

2.2 Prior and Posterior

In the context of Bayesian variable selection, a prior distribution for β^(m) needs to be specified for each model in the model space ℳ. To this end, we consider a conjugate prior for the GLM proposed by Chen and Ibrahim (2003). Under model m, the conjugate prior is of the form

$$\begin{array}{l}\pi ({\mathit{\beta }}^{(m)}\mid {\mathbf{y}}_0,a_0,m)\\ \propto \prod _{i=1}^{n}exp\left[a_0\{y_{0i}{\theta }_i^{(m)}-b({\theta }_i^{(m)})\}\right]=exp\left[a_0\{{\mathbf{y}}_0^{\prime }{\mathit{\theta }}^{(m)}-J^{\prime }\mathit{b}({\mathit{\theta }}^{(m)})\}\right],\end{array}$$ (3)

where a₀ > 0 is a scalar prior parameter, y₀ = (y₀₁, …, y_0n)′ is an n × 1 vector of prior parameters, J is an n×1 vector of ones, and $\mathit{b}({\mathit{\theta }}^{(m)})=(b({\theta }_1^{(m)}),\dots ,b({\theta }_n^{(m)}){)}^{\prime }$ is an n×1 vector of the $b({\theta }_i^{(m)})$’s. As discussed in Chen and Ibrahim (2003), y_0i can be viewed as a prior prediction for the marginal mean of y_i at x_i. Thus, in eliciting y₀, the user must focus on a prediction (or guess) for E(y), which narrows the possibilities for choosing y₀. Moreover, the specification of all y_0i equal has an appealing interpretation. A prior specification with y₀₁ = … = y_0n implies a prior in which the prior modes of the slopes in the regression model are the same, but the prior modes of intercepts in the regression model vary. For example, a prior with y_0i = 0.5 will have the same modes of slopes but a different mode of intercept than a prior with y_0i = 0.1. This is intuitively appealing since in this case the prior prediction on y_0i does not depend on the i^th subject’s specific information. Mathematically, this result was established in Chen and Ibrahim (2003). The details are as follows. Suppose we drop model index m. Let μ₀ be any prespecified p × 1 vector, where p = k + 1. Suppose we take

$${\mathbf{y}}_0=\stackrel{.}{\mathit{b}}(\mathit{\theta })=\stackrel{.}{\mathit{b}}(\theta (X{\mathit{\mu }}_0)),$$

where ḃ(θ) is the gradient vector of b(θ). Then, the conjugate prior yields a prior mode of β equal to μ₀. Now we can see that μ₀ = (β₀, 0, …, 0)′ yields y₀₁ = y₀₂ = … = y_0n = ḃ(θ(β₀)). On the other hand, as under some mild conditions, the prior mode is unique, and, hence, the specification of y₀ = y₀1 leads to the prior mode μ₀ = (β₀, 0, …, 0)′, where β₀ satisfies ḃ(θ(β₀)) = y₀. For instance, under normal linear regression, we can show that the prior mode μ₀ of β is given by

$${\mathit{\mu }}_0={(X^{\prime }X)}^{-1}{X}^{\prime }{\mathbf{y}}_0.$$

If we specify y₀ = y₀1, we have

$${\mathit{\mu }}_0=(y_0,0,0,\dots ,0{)}^{\prime },$$

which implies that all the slopes are 0 while the intercept is equal to y₀. This attractive feature allows us to do sensitivity analyses by varying the intercepts in the prior. The parameter a₀ in (3) can be generally viewed as a precision parameter that quantifies the strength of our prior belief in y₀.

In the context of Bayesian variable selection, (3) specifies the priors for all models in ℳ in an automatic and systematic fashion. Although various theoretical properties of (3) were examined in Chen and Ibrahim (2003) in a great detail, it is not clear how well this type of the prior performs in the context of Bayesian variable selection.

Now, under model m, the posterior distribution of β^(m) with the conjugate prior (3) is given by

$$\begin{array}{l}\pi ({\mathit{\beta }}^{(m)}\mid D,m)\propto exp\left\{{\mathbf{y}}^{\prime }{\mathit{\theta }}^{(m)}-J^{\prime }\mathit{b}({\mathit{\theta }}^{(m)})\right\}\pi ({\mathit{\beta }}^{(m)}\mid {\mathbf{y}}_0,a_0,m)\\ \propto exp\left\{(\mathbf{y}+a_0{\mathbf{y}}_0{)}^{\prime }{\mathit{\theta }}^{(m)}-(1+a_0)J^{\prime }\mathit{b}({\mathit{\theta }}^{(m)})\right\},\end{array}$$ (4)

where D = {(y_i, x_i), i = 1, 2, …, n} denotes the observed data. From (4), we can see that under the conjugate prior, the resulting posterior has a very attractive form. Furthermore, when a₀ → 0, the posterior π(Σ^(m)|D, m) in (4) reduces to

$$\pi ({\mathit{\beta }}^{(m)}\mid D,m)\propto exp\left\{{\mathbf{y}}^{\prime }{\mathit{\theta }}^{(m)}-J^{\prime }\mathit{b}({\mathit{\theta }}^{(m)})\right\},$$

which is the posterior distribution based on an improper uniform prior for β^(m).

2.3 Variable Selection Criteria

In this section, we consider four Bayesian model assessment criteria, namely, Conditional Predictive Ordinate (CPO) statistic (Geisser (1993); Gelfand et al. (1992); and Gelfand and Dey (1994)), L measure (Ibrahim and Laud (1994); Laud and Ibrahim (1995); Gelfand and Ghosh (1998); Ibrahim et al. (2001a); and Chen et al. (2004)), Deviance Information Criterion (DIC) (Spiegelhalter et al. (2002)), and marginal likelihood (Bayes factor).

The CPO, L measure, and DIC are criterion based methods which can be attractive in the sense that they are well defined under improper priors as long as the posterior distribution is proper, and thus have an advantage over the marginal likelihood or Bayes factor approach in this sense. Because of this reason, these three criterion based methods can be directly compared to AIC (Akaike (1973)) and BIC (Schwarz (1978)). On the other hand, the marginal likelihood or the Bayes factor is well calibrated and relatively easy to interpret, but generally sensitive to vague proper priors. In the context of variable selection, it is not clear how these methods perform under the conjugate prior given in (3) for the GLM.

Under model m, for the i^th observation, we define the CPO statistic as follows:

$${\text{CPO}}_i=f(y_i\mid {\mathbf{x}}_i,D^{(-i)})=\int f(y_i\mid {\mathbf{x}}_i^{(m)},{\mathit{\beta }}^{(m)})\pi ({\mathit{\beta }}^{(m)}\mid D^{(-i)},m)d{\mathit{\beta }}^{(m)},$$

where D⁽⁻ⁱ⁾ is D with the i^th observation deleted, and π(β|D^(−i),m) is the posterior distribution based on the data D⁽⁻ⁱ⁾. Due to the construction of the conjugate prior (3), it is more natural to define

$$\pi ({\mathit{\beta }}^{(m)}\mid D^{(-i)},m)\propto \prod _{j\ne i}exp\left\{(y_j+a_0{y}_{0j}){\theta }_j^{(m)}-(1+a_0)b({\theta }_j^{(m)})\right\}.$$

After some messy algebra, we can show that CPO_i takes the following form:

$$\begin{array}{l}{\text{CPO}}_i=f(y_i\mid {\mathbf{x}}_i,D^{(-i)})\\ =\frac{\int \frac{1}{exp[a_0\{y_{0i}{\theta }_i^{(m)}-b({\theta }_i^{(m)})\}]}\pi ({\mathit{\beta }}^{(m)}\mid D,m)d{\mathit{\beta }}^{(m)}}{\int \frac{1}{f(y_i\mid {\mathbf{x}}_i^{(m)},{\mathit{\beta }}^{(m)})exp[a_0\{y_{0i}{\theta }_i^{(m)}-b({\theta }_i^{(m)})\}]}\pi ({\mathit{\beta }}^{(m)}\mid D,m)d{\mathit{\beta }}^{(m)}},\end{array}$$ (5)

where $f(y_i\mid {\mathbf{x}}_i^{(m)},{\mathit{\beta }}^{(m)})$ is the density function given in (2). Also, we notice that the CPO defined in (5) is slightly different from the usual CPO (Geisser (1993) and Gelfand et al. (1992)), which is of the form

$${\left\{\int \frac{1}{f(y_i\mid {\mathbf{x}}_i^{(m)},{\mathit{\beta }}^{(m)})}\pi ({\mathit{\beta }}^{(m)}\mid D,m)d{\mathit{\beta }}^{(m)}\right\}}^{-1}.$$

However, these two forms will be identical as a₀ → 0. As suggested in Ibrahim et al. (2001b), a natural summary statistic of the CPO_i’s is the logarithm of the Pseudo-marginal likelihood (LPML) defined as

$${\text{LPML}}_m=\sum _{i=1}^{n}log({\text{CPO}}_i).$$

We will use LPML_m as a criterion-based measure for variable selection.

The L measure criterion is another useful tool for model comparison and variable selection. The L measure is constructed from the posterior predictive distribution of the data. For the entire class of GLM’s in (2), under model m, the L measure is defined as:

$$\begin{array}{l}{L}_m(\nu )=\sum _{i=1}^n\left[E\{b^{″}({\theta }_i^{(m)})\mid D,m\}+\mathbf{Var}\{b^{\prime }({\theta }_i^{(m)})\mid D,m\}\right]\\ +\nu \sum _{i=1}^m{\left[E\{b^{\prime }({\theta }_i^{(m)})\mid D,m\}-y_i\right]}^2,\end{array}$$ (6)

where b′(.) and b″(.) are the mean and variance functions of the GLM in (2), and all expectations and variances are taken with respect to the posterior distribution π(β^(m)|D, m) in (4). We note that for the GLM in (1), we need to modify L_m(ν) in (6) accordingly, and in this case, the L measure takes the form

$$\begin{array}{l}{L}_m(\nu )=\sum _{i=1}^n\left[E\{a_i(\tau )b^{″}({\theta }_i^{(m)})\mid D,m\}+\mathbf{Var}\{b^{\prime }({\theta }_i^{(m)})\mid D,m\}\right]\\ +\nu \sum _{i=1}^m{\left[E\{b^{\prime }({\theta }_i^{(m)})\mid D,m\}-y_i\right]}^2.\end{array}$$ (7)

The DIC criterion, proposed by Spiegelhalter et al. (2002), is given by

$${\text{DIC}}_m=D({\overline{\mathit{\beta }}}^{(m)})+2p_D^{(m)},$$ (8)

where

$$p_D^{(m)}=\overline{D({\mathit{\beta }}^{(m)})}-D({\overline{\mathit{\beta }}}^{(m)}),$$

β̄^(m) = E[β^(m)|D, m], and $\overline{D({\mathit{\beta }}^{(m)})}=E[D({\mathit{\beta }}^{(m)})\mid D,m]$. For the GLM in (2), under model m,

$$D({\mathit{\beta }}^{(m)})=-2\sum _{i=1}^n\left\{y_i{\theta }_i^{(m)}-b({\theta }_i^{(m)})\right\}.$$ (9)

Similar to (6), under the GLM in (1), D(β^(m)) needs to be modified accordingly.

In the spirit of marginal likelihoods, after ignoring the constants shared by all variable subset models in model space ℳ for the GLM in (2), for the purpose of variable subset selection it suffices to compute the posterior normalizing constant

$$C_m(D)=\int exp\left\{(\mathbf{y}+a_0{\mathbf{y}}_0{)}^{\prime }{\mathit{\theta }}^{(m)}-(1+a_0)J^{\prime }\mathit{b}({\mathit{\theta }}^{(m)})\right\}d{\mathit{\beta }}^{(m)}$$ (10)

and the prior normalizing constant

$$C_{0m}({\mathbf{y}}_0)=\int exp\left[a_0\{{\mathbf{y}}_0^{\prime }{\mathit{\theta }}^{(m)}-J^{\prime }\mathit{b}({\mathit{\theta }}^{(m)})\}\right]d{\mathit{\beta }}^{(m)}.$$ (11)

Similar to the modification of (6) yielding (7), under the GLM in (1), D(β^(m)) in (9), C_m(D) in (10), and C_0m(y₀) in (11) need to be modified accordingly. In the context of variable selection, we select a variable subset model which yields the largest LPML_m under the CPO, the smallest L_m(ν) under the L measure, the smallest DIC_m under the DIC, and the largest C_m(D)/C_0m(y₀) or log[C_m(D)/C_0m(y₀)] under the marginal likelihood.

3 Analytic Connections Between Variable Selection Criteria For the Normal Linear Regression Model

In this section, we consider the normal linear regression models given by

$$f(y_i\mid x_i^{(m)},{\mathit{\beta }}^{(m)},\tau )=\frac{{\tau }^{1/2}}{\sqrt{2\pi }}exp\left\{-\frac{\tau }{2}{(y_i-{\mathbf{x}}_i^{(m{)}^{\prime }}{\mathit{\beta }}^{(m)})}^2\right\}.$$ (12)

Let $X_m=({\mathbf{x}}_1^{(m)},{\mathbf{x}}_2^{(m)},\dots ,{\mathbf{x}}_n^{(m)}{)}^{\prime }$, which is the design matrix for the normal linear regression under model m. Assume X_m is of full rank k_m throughout. We focus only on the τ known case as analytical connections are more difficult to establish when τ is unknown. For the model in (12) with a known τ, the conjugate prior for β^(m) in (3) reduces to

$$[{\mathit{\beta }}^{(m)}\mid {\mathbf{y}}_0,a_0,m]\sim N_{k_m}\left({(X_m^{\prime }{X}_m)}^{-1}{X}_m^{\prime }{\mathbf{y}}_0,\frac{1}{\tau a_0}{(X_m^{\prime }{X}_m)}^{-1}\right),$$ (13)

and the posterior distribution for β^(m) is given by

$$[{\mathit{\beta }}^{(m)}\mid D,m]\sim N_{k_m}\left({(X_m^{\prime }{X}_m)}^{-1}{X}_m^{\prime }\frac{\mathbf{y}+a_0{\mathbf{y}}_0}{1+a_0},\frac{1}{\tau (1+a_0)}{(X_m^{\prime }{X}_m)}^{-1}\right).$$

For (12), AIC and BIC under model m are given by

$${\text{AIC}}_m=-2logL({\widehat{\mathit{\beta }}}^{(m)}\mid D)+2k_m=-nlog\left(\frac{\tau }{2\pi }\right)+\tau {\text{SSE}}_m+2k_m,$$ (14)

where β̂^(m) is the maximum likelihood estimate of β^(m) and

$${\text{SSE}}_m={\mathbf{y}}^{\prime }\{I-X_m{(X_m^{\prime }{X}_m)}^{-1}{X}_m^{\prime }\}\mathbf{y}$$

is the usual sum of squared errors, and

$${\text{BIC}}_m=-2logL({\widehat{\mathit{\beta }}}^{(m)}\mid D)+\{log(n)\}{k}_m=-nlog\left(\frac{\tau }{2\pi }\right)+\tau {\text{SSE}}_m+\{log(n)\}{k}_m.$$ (15)

After some algebra, we can show that after putting back all normalizing constants, the logarithm of the marginal likelihood under model m is given by

$$\begin{array}{l}log\{C_m(D)/C_{0m}({\mathbf{y}}_0)\}\\ =\frac{n}{2}log\left(\frac{\tau }{2\pi }\right)-\frac{\tau }{2}{\mathbf{y}}^{\prime }\mathbf{y}+\frac{\tau (1+a_0)}{2}{\left(\frac{\mathbf{y}+a_0{\mathbf{y}}_0}{1+a_0}\right)}^{\prime }{X}_m{(X_m^{\prime }{X}_m)}^{-1}{X}_m^{\prime }\left(\frac{\mathbf{y}+a_0{\mathbf{y}}_0}{1+a_0}\right)\\ -\frac{\tau a_0}{2}\{{\mathbf{y}}_0^{\prime }{X}_m{(X_m^{\prime }{X}_m)}^{-1}{X}_m^{\prime }{\mathbf{y}}_0\}+\left(\frac{1}{2}log\frac{a_0}{1+a_0}\right)k_m.\end{array}$$ (16)

When y₀ = 0, the conjugate prior in (13) reduces to Zellner’s g-prior (Zellner (1986)). For this special case, (16) becomes

$$\begin{array}{l}log[C_m(D)/C_{0m}(\mathbf{0})]\\ =\frac{n}{2}log\left(\frac{\tau }{2\pi }\right)-\frac{\tau a_0}{2(1+a_0)}{\mathbf{y}}^{\prime }\mathbf{y}-\frac{\tau }{2(1+a_0)}{\text{SSE}}_m+\left(\frac{1}{2}log\frac{a_0}{1+a_0}\right)k_m.\end{array}$$ (17)

Thus, we have

$$\begin{array}{l}{\mathcal{M}}_m(a_0)\equiv -2(1+a_0)\left[log\{C_m(D)/C_{0m}(\mathbf{0})\}+\frac{\tau a_0}{2(1+a_0)}{\mathbf{y}}^{\prime }\mathbf{y}\right]+a_{0}nlog\left(\frac{\tau }{2\pi }\right)\\ =-nlog\left(\frac{\tau }{2\pi }\right)+\tau {\text{SSE}}_m+\left\{(1+a_0)log\frac{1+a_0}{a_0}\right\}{k}_m.\end{array}$$ (18)

For purposes of variable selection, it suffices to compare ℳ_m(a₀) and we then choose a model with the smallest ℳ_m(a₀). From (18), we can see that

$${\mathcal{M}}_m(a_0)=\{\begin{array}{ll}{\text{AIC}}_m\hfill & \text{if}(1+a_0)log\frac{1+a_0}{a_0}=2,\hfill \\ {\text{BIC}}_m\hfill & \text{if}(1+a_0)log\frac{1+a_0}{a_0}=logn.\hfill \end{array}$$ (19)

For (12), we use (7) to compute L_m(ν). In particular, we have a_i(τ) = 1/τ, $E[a_i(\tau )b^{″}({\theta }_i^{(m)})\mid D,m]=\frac{1}{\tau }$,

$$\begin{array}{l}\mathbf{Var}\{b^{\prime }({\theta }_i^{(m)})\mid D,m\}=\mathbf{Var}\{{\mathbf{x}}_i^{(m{)}^{\prime }}{\mathit{\beta }}^{(m)}\mid D,m\}\\ ={\mathbf{x}}_i^{(m{)}^{\prime }}\mathbf{Var}({\mathit{\beta }}^{(m)}\mid D,m){\mathbf{x}}_i^{(m)}=\frac{1}{\tau (1+a_0)}{\mathbf{x}}_i^{(m{)}^{\prime }}{(X_m^{\prime }{X}_m)}^{-1}{\mathbf{x}}_i^{(m)},\end{array}$$

and $E\{b^{\prime }({\theta }_i^{(m)})\mid D,m\}=E\{{\mathbf{x}}_i^{(m{)}^{\prime }}{\mathit{\beta }}^{(m)}\mid D,m\}={\mathbf{x}}_i^{(m{)}^{\prime }}{(X_m^{\prime }{X}_m)}^{-1}{X}_m^{\prime }\frac{\mathbf{y}+a_0{\mathbf{y}}_0}{1+a_0}$. Thus, we obtain

$$\begin{array}{l}{L}_m(\nu )=\frac{n}{\tau }+\frac{1}{\tau (1+a_0)}\sum _{i=1}^n{\mathbf{x}}_i^{(m{)}^{\prime }}{(X_m^{\prime }{X}_m)}^{-1}{\mathbf{x}}_i^{(m)}\\ +\nu \sum _{i=1}^n{\left\{y_i-{\mathbf{x}}_i^{(m{)}^{\prime }}{(X_m^{\prime }{X}_m)}^{-1}{X}_m^{\prime }\frac{\mathbf{y}+a_0{\mathbf{y}}_0}{1+a_0}\right\}}^2\\ =\frac{n}{\tau }+\frac{1}{\tau (1+a_0)}{k}_m+\nu [{\left\{\mathbf{y}-X_m{(X_m^{\prime }{X}_m)}^{-1}{X}_m^{\prime }\frac{\mathbf{y}+a_0{\mathbf{y}}_0}{1+a_0}\right\}}^{\prime }\\ \times \left\{\mathbf{y}-X_m{(X_m^{\prime }{X}_m)}^{-1}{X}_m^{\prime }\frac{\mathbf{y}+a_0{\mathbf{y}}_0}{1+a_0}\right\}].\end{array}$$ (20)

When y₀ = 0, (20) reduces to

$$L_m(\nu )=\frac{n}{\tau }+\frac{1}{\tau (1+a_0)}{k}_m+\frac{\nu a_0^2}{{(1+a_0)}^2}{\mathbf{y}}^{\prime }\mathbf{y}+\frac{\nu (1+2a_0)}{{(1+a_0)}^2}{\text{SSE}}_m.$$ (21)

Write

$${\stackrel{\sim }{L}}_m(\nu ,a_0)=\frac{\tau {(1+a_0)}^2}{\nu (1+2a_0)}\left\{L_m(\nu )-\frac{n}{\tau }-\frac{\nu a_0^2}{{(1+a_0)}^2}{\mathbf{y}}^{\prime }\mathbf{y}\right\}-nlog\left(\frac{\tau }{2\pi }\right).$$ (22)

Using (21) and (22), we obtain

$${\stackrel{\sim }{L}}_m(\nu ,a_0)=-nlog\left(\frac{\tau }{2\pi }\right)+\tau {\text{SSE}}_m+\frac{1+a_0}{\nu (1+2a_0)}{k}_m,$$

and hence

$${\stackrel{\sim }{L}}_m(\nu ,a_0)=\{\begin{array}{ll}{\text{AIC}}_m\hfill & \text{if}\frac{1+a_0}{\nu (1+2a_0)}=2,\hfill \\ {\text{BIC}}_m\hfill & \text{if}\frac{1+a_0}{\nu (1+2a_0)}=logn.\hfill \end{array}$$

Note that in the context of variable selection, a model with the smallest L_m(ν) is the same model that has the smallest L̃_m(ν, a₀). Thus, in this sense, the L measure can be equivalent to AIC or BIC by appropriately tuning (ν, a₀). It is interesting to mention that in order to achieve L̃_m(ν, a₀) = AIC_m or L̃_m(ν, a₀) = BIC_m, ν must be small, and hence when ν = 1, the L measure always has a smaller dimensional penalty than both AIC and BIC. Unlike the marginal likelihood, a₀ plays a minimum role in controlling dimensional penalty in the L measure.

When y₀ = 0, the posterior mean of β^(m) is given by ${\overline{\mathit{\beta }}}^{(m)}=\frac{1}{1+a_0}{(X_m^{\prime }{X}_m)}^{-1}{X}_m^{\prime }\mathbf{y}$. Thus, we have $D({\mathit{\beta }}^{(m)})=-nlog\left(\frac{\tau }{2\pi }\right)+\tau (\mathbf{y}-X_m{\mathit{\beta }}^{(m)}{)}^{\prime }(\mathbf{y}-X_m{\mathit{\beta }}^{(m)})$,

$$\begin{array}{l}\overline{D({\mathit{\beta }}^{(m)})}\\ =E[D({\mathit{\beta }}^{(m)})\mid D,m]=-nlog\left(\frac{\tau }{2\pi }\right)+\tau E[\{\mathbf{y}-X_m{\overline{\mathit{\beta }}}^{(m)}-X_m({\mathit{\beta }}^{(m)}-{\overline{\mathit{\beta }}}^{(m)}){\}}^{\prime }\\ \times \{\mathbf{y}-X_m{\overline{\mathit{\beta }}}^{(m)}-X_m({\mathit{\beta }}^{(m)}-{\overline{\mathit{\beta }}}^{(m)})\}\mid D,m]\\ =-nlog\left(\frac{\tau }{2\pi }\right)+\frac{1}{1+a_0}{k}_m+\frac{\tau a_0^2}{{(1+a_0)}^2}{\mathbf{y}}^{\prime }\mathbf{y}+\frac{\tau (1+2a_0)}{{(1+a_0)}^2}{\text{SSE}}_m,\end{array}$$ (23)

and

$$\begin{array}{l}D({\overline{\mathit{\beta }}}^{(m)})=-nlog\left(\frac{\tau }{2\pi }\right)+\tau (\mathbf{y}-X_m{\overline{\mathit{\beta }}}^{(m)}{)}^{\prime }(\mathbf{y}-X_m{\overline{\mathit{\beta }}}^{(m)})\\ =-nlog\left(\frac{\tau }{2\pi }\right)+\frac{\tau a_0^2}{{(1+a_0)}^2}{\mathbf{y}}^{\prime }\mathit{y}+\frac{\tau (1+2a_0)}{{(1+a_0)}^2}{\text{SSE}}_m.\end{array}$$ (24)

Combining (23) and (24) gives

$$p_D^{(m)}=\overline{D({\mathit{\beta }}^{(m)})}-D({\overline{\mathit{\beta }}}^{(m)})=\frac{1}{1+a_0}{k}_m.$$ (25)

Thus, the DIC_m for (12) is given by

$${\text{DIC}}_m=-nlog\left(\frac{\tau }{2\pi }\right)+\frac{\tau a_0^2}{{(1+a_0)}^2}{\mathbf{y}}^{\prime }\mathbf{y}+\frac{\tau (1+2a_0)}{{(1+a_0)}^2}{\text{SSE}}_m+\frac{2}{1+a_0}{k}_m.$$ (26)

Write

$${\text{DIC}}_m^{\ast }(a_0)=\frac{{(1+a_0)}^2}{1+2a_0}\left\{{\text{DIC}}_m-\frac{\tau a_0^2}{{(1+a_0)}^2}{\mathbf{y}}^{\prime }\mathbf{y}\right\}+\frac{n{a}_0^2}{1+2a_0}log\left(\frac{\tau }{2\pi }\right).$$

We have

$${\text{DIC}}_m^{\ast }(a_0)=-nlog\left(\frac{\tau }{2\pi }\right)+\tau {\text{SSE}}_m+\frac{2(1+a_0)}{1+2a_0}{k}_m.$$ (27)

Therefore, when a₀ = 0, ${\text{DIC}}_m^{\ast }(0)={\text{DIC}}_m={\text{AIC}}_m$, and when a₀ > 0, $\frac{2(1+a_0)}{1+2a_0}<2$, which implies that ${\text{DIC}}_m^{\ast }(a_0)$ has a smaller dimensional penalty than both AIC and BIC.

Similarly to DIC, we consider only y₀ = 0. From (5), we have

$${\text{LPML}}_m=\sum _{i=1}^{n}log({\text{CPO}}_i)=\sum _{i=1}^{n}log({\text{CPO}}_{1i})-\sum _{i=1}^{n}log({\text{CPO}}_{2i}),$$ (28)

where ${\text{CPO}}_{1i}=\int exp\left\{\frac{a_0\tau }{2}{\mathit{\beta }}^{(m{)}^{\prime }}{\mathbf{x}}_i^{(m)}{\mathbf{x}}_i^{(m{)}^{\prime }}{\mathit{\beta }}^{(m)}\right\}\pi ({\mathit{\beta }}^{(m)}\mid D,m)d{\mathit{\beta }}^{(m)}$ and

$$\begin{array}{l}{\text{CPO}}_{2i}={\left(\frac{\tau }{2\pi }\right)}^{-1/2}exp\left\{\frac{\tau }{2}{y}_i^2\right\}\int exp[\frac{\tau (1+a_0)}{2}\\ \times \left\{{\mathit{\beta }}^{(m{)}^{\prime }}{\mathbf{x}}_i^{(m)}{\mathbf{x}}_i^{(m{)}^{\prime }}{\mathit{\beta }}^{(m)}-\frac{2}{1+a_0}{\mathit{\beta }}^{(m{)}^{\prime }}{\mathbf{x}}_i^{(m)}{y}_i\right\}]\pi ({\mathit{\beta }}^{(m)}\mid D,m)d{\mathit{\beta }}^{(m)}\end{array}$$

for i = 1, 2, …, n. After some messy algebra, we obtain

$$\begin{array}{l}{\text{CPO}}_{1i}={\left\{1-\frac{a_0}{1+a_0}{\mathbf{x}}_i^{(m{)}^{\prime }}{(X_m^{\prime }{X}_m)}^{-1}{\mathbf{x}}_i^{(m)}\right\}}^{-1/2}\\ \times exp\left\{\frac{\tau a_0}{2{(1+a_0)}^2}\frac{{\mathbf{y}}^{\prime }{X}_m{(X_m^{\prime }{X}_m)}^{-1}{\mathbf{x}}_i^{(m)}{\mathbf{x}}_i^{(m{)}^{\prime }}{(X_m^{\prime }{X}_m)}^{-1}{X}_m^{\prime }\mathbf{y}}{1-\frac{a_0}{1+a_0}{\mathbf{x}}_i^{(m{)}^{\prime }}{(X_m^{\prime }{X}_m)}^{-1}{\mathbf{x}}_i^{(m)}}\right\}\end{array}$$

and

$$\begin{array}{l}{\text{CPO}}_{2i}={\left(\frac{\tau }{2\pi }\right)}^{-1/2}{\left\{1-{\mathbf{x}}_i^{(m{)}^{\prime }}{(X_m^{\prime }{X}_m)}^{-1}{\mathbf{x}}_i^{(m)}\right\}}^{-1/2}exp\left(\frac{\tau }{2}{y}_i^2\right)\\ \times exp\left[\frac{\tau }{2(1+a_0)}\left\{y_i{\mathbf{x}}_i^{(m{)}^{\prime }}{(X_m^{\prime }{X}_m)}^{-1}{\mathbf{x}}_i^{(m)}{y}_i-2{\mathbf{y}}^{\prime }{X}_m{(X_m^{\prime }{X}_m)}^{-1}{\mathbf{x}}_i^{(m)}{y}_i\right\}\right]\\ \times exp\left[\frac{\tau (X_m^{\prime }\mathbf{y}-{\mathbf{x}}_i^{(m)}{y}_i{)}^{\prime }{(X_m^{\prime }{X}_m)}^{-1}{\mathbf{x}}_i^{(m)}{\mathbf{x}}_i^{(m{)}^{\prime }}{(X_m^{\prime }{X}_m)}^{-1}(X_m^{\prime }\mathbf{y}-{\mathbf{x}}_i^{(m)}{y}_i)}{2(1+a_0)\{1-{\mathbf{x}}_i^{(m{)}^{\prime }}{(X_m^{\prime }{X}_m)}^{-1}{\mathbf{x}}_i^{(m)}\}}\right].\end{array}$$

Let ${\widehat{\mathit{\beta }}}^{(m)}={(X_m^{\prime }{X}_m)}^{-1}{X}_m^{\prime }\mathbf{y},{\widehat{y}}_i^{(m)}={\mathbf{x}}_i^{(m{)}^{\prime }}{\widehat{\mathit{\beta }}}^{(m)}$, and $h_{ii}^{(m)}={\mathbf{x}}_i^{(m{)}^{\prime }}{(X_m^{\prime }{X}_m)}^{-1}{\mathbf{x}}_i^{(m)}$. Plugging CPO_1i and CPO_2i into (28) yields

$$\begin{array}{l}{\text{LPML}}_m=\frac{n}{2}log(\frac{\tau }{2\pi })-\frac{\tau }{2}\sum _{i=1}^n{y}_i^2+\frac{1}{2}\sum _{i=1}^n\left\{log(1-h_{ii}^{(m)})-log\left(1-\frac{a_0}{1+a_0}{h}_{ii}^{(m)}\right)\right\}\\ -\frac{\tau }{2(1+a_0)}\sum _{i=1}^n\left\{h_{ii}^{(m)}{y}_i^2-2y_i{\widehat{y}}_i^{(m)}\right\}+\frac{\tau a_0}{2{(1+a_0)}^2}\sum _{i=1}^n\left\{\frac{{\widehat{y}}_i^{{(m)}^2}}{1-\frac{a_0}{1+a_0}{h}_{ii}^m}\right\}\\ -\frac{\tau }{2(1+a_0)}\sum _{i=1}^n\frac{{({\widehat{y}}_i^{(m)}-h_{ii}^{(m)}{y}_i)}^2}{1-h_{ii}^{(m)}}.\end{array}$$ (29)

Using Taylor expansion and after some algebra, LPML_m in (29) can be rewritten as

$${\text{LPML}}_m=\frac{n}{2}log\left(\frac{\tau }{2\pi }\right)-\frac{\tau a_0^2}{2{(1+a_0)}^2}{\mathbf{y}}^{\prime }\mathbf{y}-\frac{\tau (1+2a_0)}{2{(1+a_0)}^2}{\text{SSE}}_m-\frac{k_m}{2(1+a_0)}+R_m^{\ast },$$ (30)

where

$$\begin{array}{l}{R}_m^{\ast }=-\frac{\tau }{2(1+a_0)}\sum _{i=1}^n\frac{{(y_i-{\widehat{y}}_i^{(m)})}^2{h}_{ii}^{(m)}}{1-h_{ii}^{(m)}}+\frac{\tau a_0}{2{(1+a_0)}^2}\sum _{i=1}^n\frac{a_0}{1+a_0}\frac{h_{ii}^{(m)}{\widehat{y}}_i^{{(m)}^2}}{1-\frac{a_0}{1+a_0}{h}_{ii}^{(m)}}\\ +\frac{1}{2}\sum _{i=1}^n\sum _{j=2}^{\infty }\left\{1-{\left(\frac{a_0}{1+a_0}\right)}^j\right\}\frac{{(-1)}^j{h}_{ii}^{{(m)}^j}}{j}.\end{array}$$

Write

$${\text{LPML}}_m^{\ast }=-\frac{2{(1+a_0)}^2}{1+2a_0}\left\{{\text{LPML}}_m+\frac{\tau a_0^2}{2{(1+a_0)}^2}{\mathbf{y}}^{\prime }\mathbf{y}\right\}+\frac{n{a}_0^2}{1+2a_0}log\left(\frac{\tau }{2\pi }\right).$$ (31)

Using (30) and (31), we obtain

$${\text{LPML}}_m^{\ast }=-nlog\left(\frac{\tau }{2\pi }\right)+\tau {\text{SSE}}_m+\frac{1+a_0}{(1+2a_0)}{k}_m+R_m,$$

where $R_m=-\frac{2{(1+a_0)}^2}{1+2a_0}{R}_m^{\ast }$. We choose a model with the smallest ${\text{LPML}}_m^{\ast }$. Note that the remainder term R_m is small when all $h_{ii}^{(m)}$’s are small. From (14), (15), and (27), we see that when R_m is small and does not vary much in the model space ℳ, LPML has a smaller dimensional penalty than DIC, AIC and BIC. In addition, when a₀ = 0, LPML_m in (30) is consistent with the one derived by Gelfand and Dey (1994) based on the asymptotic approximation.

Finally, we note that the quantities defined in (18), (22), (27) and (31) are linear transformations of those defined by (17), (21), (26) and (30), respectively. In these linear transformations, the relevant coefficients are independent of m. Thus, for the purposes of variable subset selection, these linearly transformed quantities act exactly like those original forms. With (18), (22), (27) and (31), we can much more clearly see the analytical connections to AIC and BIC. We also note that George and Foster (2000) provided some similar connections between model selection probabilities and various model selection criteria for this setup.

4 Computational Development: Theory and Implementation

For the purpose of variable selection, we need to compute LPML_m, L_m(ν), DIC_m, C_m(D) and C_0m(y₀) for the Bayesian variable selection criteria described in the previous section for m = 1, 2, …, . Due to the complexity and generality of the GLM in (2), the analytical evaluation of these measures does not appear possible. Thus, a Monte Carlo (MC) based method is required for each of those measures under consideration. However, the MC methods currently available in the Bayesian computational literature require a Markov chain Monte Carlo (MCMC) sample from the posterior distribution π(β^(m)|D, m) in (4) under each variable subset model m. When the number of the models in ℳ is large, sampling from the posterior distribution under each variable subset model can be expensive. Thus, the computation of these four measures for all submodels becomes a difficult and challenging task. Therefore, the development of an efficient Monte Carlo method for variable selection for the GLM is very essential.

After examining (5), (6), and (8), we observe that there is a common feature in computing LPML_m, L_m(ν), and DIC_m, i.e., all of these three measures require to compute

$$g_m=E\{g({\mathit{\beta }}^{(m)})\mid D,m\},$$

for various functions g, where the expectation is taken with respect to the joint posterior distribution in (4) under model m. Specifically, the functions required in these calculations include

1. $g({\mathit{\beta }}^{(m)})=exp[-a_0\{y_{0i}{\theta }_i^{(m)}-b({\theta }_i^{(m)})\}]$ and $g({\mathit{\beta }}^{(m)})={\left(f(y_i\mid {\mathbf{x}}_i^{(m)},{\mathit{\beta }}^{(m)})exp[a_0\{y_{0i}{\theta }_i^{(m)}-b({\theta }_i^{(m)})\}]\right)}^{-1}$ for LPML_m;

2. $g({\mathit{\beta }}^{(m)})=b^{\prime }({\theta }_i^{(m)}),g({\mathit{\beta }}^{(m)})={\{b^{\prime }({\theta }_i^{(m)})\}}^2$, and $g({\mathit{\beta }}^{(m)})=b^{″}({\theta }_i^{(m)})$ for L_m(ν);

3. g(β^(m)) = β^(m) and g(β^(m)) = D(β^(m)) for DIC_m.

Write

$$L({\mathit{\beta }}^{(m)}\mid D,m)=exp\left\{(\mathbf{y}+a_0{\mathbf{y}}_0{)}^{\prime }{\mathit{\theta }}^{(m)}-(1+a_0)J^{\prime }\mathit{b}({\mathit{\theta }}^{(m)})\right\}$$

under model m and let L(β|D) = L(β⁽⁾|D, ), C(D) = C (D), and C₀(y₀) = C₀ (y₀) under the full model. Here, we abuse the notation a little bit as L(β^(m)|D, m) is not a likelihood function in the usual sense. Then, for a given function g, mathematically, we have

$$g_m=E[g({\mathit{\beta }}^{(m)})\mid D,m]=\int g({\mathit{\beta }}^{(m)})\frac{L({\mathit{\beta }}^{(m)}\mid D,m)}{C_m(D)}d{\mathit{\beta }}^{(m)},$$

where C_m(D) is defined in (10). Now, we present a useful identity for g_m, which is formally stated in the following theorem.

Theorem 5

For any given function g, such that E[|g(β^(m))| |D, m] < ∞, we have

$$g_m=\frac{C(D)}{C_m(D)}E\left\{g({\mathit{\beta }}^{(m)})\frac{L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid D)}\mid D\right\},$$ (32)

where the expectation is taken with respect to the joint posterior distribution in (4) under the full model. Here, w(β^(−m)| β^(m)) is a completely known conditional density, whose support is contained in, or equal to, the support of the conditional density of β^{(− m)} given β^(m) with respect to the joint posterior distribution in (4) under the full model.

Observing that when g ≡ 1, we have

$$1=\frac{C(D)}{C_m(D)}E\left\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid D)}\mid D\right\},$$

which leads to

$$\frac{C_m(D)}{C(D)}=E\left\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid D)}\mid D\right\}$$ (33)

and

$$g_m=\frac{E\left\{\frac{g({\mathit{\beta }}^{(m)})L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid D)}\mid D\right\}}{E\left\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid D)}\mid D\right\}}.$$ (34)

It is interesting to mention that the identity (33) is a by-product of this derivation and this identity can be used to compute the posterior normalizing constant under model m. The identities (33) and (34) play an important role in developing a novel Monte Carlo method for computing LPML_m, L_m(ν), DIC_m, and C_m(D) simultaneously using a single MCMC sample from the joint posterior distribution under the full model. Towards this goal, we let {β_s = (β^(m)′_s, β^(−m)′_s), s = 1, 2, …, S} denote a MCMC sample from the joint posterior distribution (4) under the full model, where S is the MCMC sample size. Then, an estimate of g_m is given by

$${\widehat{g}}_m=\frac{{\sum }_{s=1}^S\frac{g({\mathit{\beta }}_s^{(m)})L({\mathit{\beta }}_s^{(m)}\mid D,m)w({\mathit{\beta }}_s^{(-m)}\mid {\mathit{\beta }}_s^{(m)})}{L({\mathit{\beta }}_s\mid D)}}{{\sum }_{s=1}^S\frac{L({\mathit{\beta }}_s^{(m)}\mid D,m)w({\mathit{\beta }}_s^{(-m)}\mid {\mathit{\beta }}_s^{(m)})}{L({\mathit{\beta }}_s\mid D)}}.$$ (35)

Under certain regularity conditions, such as ergodicity, we have

$$\underset{S\to \infty }{lim}{\widehat{g}}_m=g_m,$$

which indicates that ĝ_m is consistent.

Letting

$$A_S=\frac{1}{S}\sum _{s=1}^S\frac{g({\mathit{\beta }}_s^{(m)})L({\mathit{\beta }}_s^{(m)}\mid D,m)w({\mathit{\beta }}_s^{(-m)}\mid {\mathit{\beta }}_s^{(m)})}{L({\mathit{\beta }}_s\mid D)}$$ (36)

and

$$B_S=\frac{1}{S}\sum _{s=1}^S\frac{L({\mathit{\beta }}_s^{(m)}\mid D,m)w({\mathit{\beta }}_s^{(-m)}\mid {\mathit{\beta }}_s^{(m)})}{L({\mathit{\beta }}_s\mid D)},$$ (37)

we have

$$\underset{S\to \infty }{lim}{A}_S=\frac{C_m(D)}{C(D)}{g}_m\equiv A,$$ (38)

and

$$\underset{S\to \infty }{lim}{B}_S=\frac{C_m(D)}{C(D)}\equiv B.$$ (39)

From (38) and (39), we obtain

$$g_m=\frac{C(D)}{C_m(D)}A=\frac{A}{B}.$$ (40)

Using (36)–(40), we have

$${\widehat{g}}_m-g_m=\frac{A_S}{B_S}-g_m=\frac{A_S}{B_S}-\frac{A}{B}=\frac{A_S-\frac{A}{B}{B}_S}{B_S}=A\frac{\frac{A_S}{A}-\frac{B_S}{B}}{B_S}=g_m\frac{\frac{A_S}{A}-\frac{B_S}{B}}{\frac{B_S}{B}}.$$ (41)

In (41), ${lim}_{S\to \infty }\frac{B_S}{B}=1$ and

$$\underset{S\to \infty }{lim}\left(\frac{A_S}{A}-\frac{B_S}{B}\right)=0$$ (42)

In addition, we have

$$\begin{array}{l}\frac{A_S}{A}-\frac{B_S}{B}=\frac{1}{S}\sum _{s=1}^S[\frac{1}{A}\left\{\frac{g({\mathit{\beta }}_s^{(m)})L({\mathit{\beta }}_s^{(m)}\mid D,m)w({\mathit{\beta }}_s^{(-m)}\mid {\mathit{\beta }}_s^{(m)})}{L({\mathit{\beta }}_s\mid D)}\right\}\\ -\frac{1}{B}\left\{\frac{L({\mathit{\beta }}_s^{(m)}\mid D,m)w({\mathit{\beta }}_s^{(-m)}\mid {\mathit{\beta }}_s^{(m)})}{L({\mathit{\beta }}_s\mid D)}\right\}].\end{array}$$

We are then led to the following theorem.

Theorem 6

Let {β_s, s = 1, 2, …, S} be a random sample. Assume A ≠ 0,

$$\begin{array}{l}{V}_w(g_m)=E[\{\frac{g({\mathit{\beta }}^{(m)})L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{AL(\mathit{\beta }\mid D)}\\ {-\frac{L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{BL(\mathit{\beta }\mid D)}\}}^2\mid D]<\infty \end{array}$$ (43)

and

$$E[{\{g({\mathit{\beta }}^{(m)})\}}^2\mid D]<\infty ,$$ (44)

where the expectation is taken with respect to the joint posterior distribution in (4) under the full model. Then we have

$$\underset{S\to \infty }{lim}\left[S\times E\left\{{\left(\frac{{\widehat{g}}_m-g_m}{g_m}\right)}^2\right\}\right]=V_w(g_m),$$ (45)

where V_w(g_m) is defined by (43) and

$$\sqrt{S}({\widehat{g}}_m-g_m)\stackrel{\mathcal{D}}{\to }N(0,g_m^2{V}_w(g_m)).$$

The proof of Theorem 6 directly follows from the proof of Theorem 3.1 of Chen and Shao (1997). Thus, the detail is omitted for brevity. From (45), we notice that $E{\left[\frac{{\widehat{g}}_m-g_m}{g_m}\right]}^2$ is the relative mean-square error and Theorem 6 implies that when S is large,

$$E{\left(\frac{{\widehat{g}}_m-g_m}{g_m}\right)}^2\approx \frac{1}{S}{V}_w(g_m).$$

Remark 4.1

As discussed in Chen et al. (2000), the simulation standard error of ĝ_m can be approximated by

$$se({\widehat{g}}_m)=\frac{\mid {\widehat{g}}_m\mid }{\widehat{A}}\sqrt{\frac{1}{S}\sum _{s=1}^S{\left[\{g({\mathit{\beta }}_s^{(m)})-{\widehat{g}}_m\}\frac{L({\mathit{\beta }}_s^{(m)}\mid D,m)w({\mathit{\beta }}_s^{(-m)}\mid {\mathit{\beta }}_s^{(m)})}{L({\mathit{\beta }}_s\mid D)}\right]}^2},$$

where  = A_S.

Remark 4.2

From (34), it is quite natural that one may think a more efficient way to obtain a MC estimate of g_m is by generating two MC samples from the posterior distribution so that one sample is used for computing $E\{\frac{g({\mathit{\beta }}^{(m)})L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid D)}\mid D\}$ while the second sample is used for computing $E\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid D)}\mid D\}$. In this remark, we show that the use of two MC samples in obtaining the MC estimate of g_m may not necessarily be more efficient than the use of just one MC sample. In addition, generating two MC samples requires more computing time. Specifically, suppose that {β_1;s, s = 1, 2, …, S₁} and {β_2;s, s = 1, 2, …, S₂} are two independent random samples from the joint posterior distribution (4) under the full model. Then g_m can be estimated by

$${\widehat{g}}_m^{\ast }=\frac{\frac{1}{S_1}{\sum }_{s=1}^{S_1}\frac{g({\mathit{\beta }}_{1,s}^{(m)})L({\mathit{\beta }}_{1,s}^{(m)}\mid D,m)w({\mathit{\beta }}_{1,s}^{(-m)}\mid {\mathit{\beta }}_{1,s}^{(m)})}{L({\mathit{\beta }}_{1,s}\mid D)}}{\frac{1}{S_2}{\sum }_{s=1}^{S_2}\frac{L({\mathit{\beta }}_{2,s}^{(m)}\mid D,m)w({\mathit{\beta }}_{2,s}^{(-m)}\mid {\mathit{\beta }}_{2,s}^{(m)})}{L({\mathit{\beta }}_{2,s}\mid D)}}.$$ (46)

By the δ-Method, we have

$$\begin{array}{l}E{\left(\frac{{\widehat{g}}_m^{\ast }-g_m}{g_m}\right)}^2=\frac{\mathbf{Var}\left\{\frac{1}{S_1}{\sum }_{s=1}^{S_1}\frac{g({\mathit{\beta }}_{1,s}^{(m)})L({\mathit{\beta }}_{1,s}^{(m)}\mid D,m)w({\mathit{\beta }}_{1,s}^{(-m)}\mid {\mathit{\beta }}_{1,s}^{(m)})}{L({\mathit{\beta }}_{1,s}\mid D)}\right\}}{A^2}\\ +\frac{\mathbf{Var}\left\{\frac{1}{S_2}{\sum }_{s=1}^{S_2}\frac{L({\mathit{\beta }}_{2,s}^{(m)}\mid D,m)w({\mathit{\beta }}_{2,s}^{(-m)}\mid {\mathit{\beta }}_{2,s}^{(m)})}{L({\mathit{\beta }}_{2,s}\mid D)}\right\}}{B^2}+O\left\{\frac{1}{{(S_1+S_2)}^2}\right\}\\ =\frac{1}{S_1{A}^2}\mathbf{Var}\left\{\frac{g({\mathit{\beta }}^{(m)})L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid D)}\right\}\\ +\frac{1}{S_2{B}^2}\mathbf{Var}\left\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid D)}\right\}+O\left\{\frac{1}{{(S_1+S_2)}^2}\right\},\end{array}$$

where the expectation and variance are taken with respect to the joint posterior distribution (4) under the full model.

Assuming that S₁ = S₂ = S, we have

$$\begin{array}{l}\underset{S\to \infty }{lim}\left\{S\times E{\left(\frac{{\widehat{g}}_m^{\ast }-g_m}{g_m}\right)}^2\right\}=\frac{1}{A^2}\mathbf{Var}\left\{\frac{g({\mathit{\beta }}^{(m)})L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid D)}\right\}\\ +\frac{1}{B^2}\mathbf{Var}\left\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid D)}\right\}.\end{array}$$ (47)

Thus, if

$$E\left\{\frac{g({\mathit{\beta }}^{(m)})L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{AL(\mathit{\beta }\mid D)}\times \frac{L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{BL(\mathit{\beta }\mid D)}\right\}\ge 0,$$ (48)

we have

$$\underset{S\to \infty }{lim}\left\{S\times E{\left(\frac{{\widehat{g}}_m^{\ast }-g_m}{g_m}\right)}^2\right\}\ge \underset{S\to \infty }{lim}\left\{S\times E{\left(\frac{{\widehat{g}}_m-g_m}{g_m}\right)}^2\right\}.$$

It is easy to see that when g(β^(m)) ≥ 0 or g(β^(m)) ≤ 0, (48) automatically holds. Therefore, for many cases, it is unnecessary to use two MC samples instead of one MC sample in obtaining the MC estimate of g_m.

Note that the estimate ĝ_m depends on w(β^(−m)|β^(m)). It is reasonable to argue that the best choice of w should yield the smallest asymptotic variance of the estimate ĝ_m among all possible w’s. The following theorem precisely addresses this optimality issue.

Theorem 7

Let

$$w_{\mathit{opt}}=\pi ({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)},D)$$ (49)

be the conditional posterior density of β^(−m) given β^(m) under the full model, then we have

$$V_{w_{\mathit{opt}}}(g_m)\le V_w(g_m)$$ (50)

for all w’s, where V_w(g_m) is defined by (43).

Remark 4.3

Note that (50) holds for any function g that satisfies the condition given in (44). Thus, for various functions g involved in LPML_m, L_m(ν) and DIC_m, the best choice of w is the same w_opt given in (49).

Remark 4.4

When we use ${\widehat{g}}_m^{\ast }$ in (46), we can also show that w_opt = π(β^(−m) | β^(m), D) yields the smallest asymptotic relative mean-square error of ${\widehat{g}}_m^{\ast }$, for example, the one given by (47).

Remark 4.5

For computing CPO_i in (5) under model m, we do not need to compute $\frac{C(D)}{C_m(D)}$ in (32). In fact, it is easy to see that

$${\text{CPO}}_i^{(m)}=\frac{E\left\{g_1({\mathit{\beta }}^{(m)})\frac{L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid D)}\mid D\right\}}{E\left\{g_2({\mathit{\beta }}^{(m)})\frac{L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid D)}\mid D\right\}},$$

where $g_1({\mathit{\beta }}^{(m)})=exp[-a_0(y_{0i}{\theta }_i^{(m)}-b({\theta }_i^{(m)}))]$ and $g_2({\mathit{\beta }}^{(m)})={\left\{f(y_i\mid {\mathbf{x}}_i^{(m)},{\mathit{\beta }}^{(m)})exp[a_0(y_{0i}{\theta }_i^{(m)}-b({\theta }_i^{(m)}))]\right\}}^{-1}$. Thus, given a MCMC sample { ${\mathit{\beta }}_s=({{\mathit{\beta }}^{(m)}}_s^{\prime },{{\mathit{\beta }}^{(-m)}}_s^{\prime })$, s = 1, 2, …, S} from the joint posterior distribution (4), a MC estimate of CPO_i is given as follows:

$${\widehat{\text{CPO}}}_i^{(m)}=\frac{{\sum }_{s=1}^S\frac{g_1({\mathit{\beta }}_s^{(m)})L({\mathit{\beta }}_s^{(m)}\mid D,m)w({\mathit{\beta }}_s^{(-m)}\mid {\mathit{\beta }}_s^{(m)})}{L({\mathit{\beta }}_s\mid D)}}{{\sum }_{s=1}^S\frac{g_2({\mathit{\beta }}_s^{(m)})L({\mathit{\beta }}_s^{(m)}\mid D,m)w({\mathit{\beta }}_s^{(-m)}\mid {\mathit{\beta }}_s^{(m)})}{L({\mathit{\beta }}_s\mid D)}}.$$

Following the proof of Theorem 7, we can easily show that the optimal choice of w for ${\widehat{\text{CPO}}}_i^{(m)}$ is still the same w_opt given in (49).

Remark 4.6

To compute LPML, L(ν) and DIC under the full model, we can simply take β⁽⁾ = β and w(β⁽⁻⁾ |β⁽⁾) = 1. Then, for various functions g, given a MCMC sample {β_s, s = 1, 2, …, S} (35) reduces to

$$\widehat{g}=\frac{1}{S}\sum _{s=1}^{S}g({\mathit{\beta }}_s),$$

where {β_s, s = 1, 2, …, S} is a MCMC sample from the posterior distribution (4) under the full model.

Remark 4.7

As shown in Theorem 7, the optimal choice of w is w_opt = π(β^(−m) | β^(m), D). However, for the GLM in (2), w_opt is not available in closed form. Fortunately, for the GLM, a good w(β^(−m)|β^(m)), which is close to the optimal choice, can be constructed based on the asymptotic approximation to the joint posterior proposed by Chen (1985). Let β̂ denote the posterior mode of β under the full model, i.e.,

$$\widehat{\mathit{\beta }}=\underset{\mathit{\beta }}{argmax}logL(\mathit{\beta }\mid D)=\underset{\mathit{\beta }}{argmax}\{(\mathbf{y}+a_0{\mathbf{y}}_0{)}^{\prime }\mathit{\theta }-(1+a_0)J^{\prime }\mathit{b}(\mathit{\theta })\}.$$

Also let

$$\widehat{\mathrm{\Sigma }}={\left\{-\frac{{\partial }^{2}logL(\mathit{\beta }\mid D)}{\partial \mathit{\beta }\partial {\mathit{\beta }}^{\prime }}{\mid }_{\mathit{\beta }=\widehat{\mathit{\beta }}}\right\}}^{-1}.$$

Then, the joint posterior π(β|D) under the full model can be approximated by

$$\widehat{\pi }(\mathit{\beta }\mid \widehat{\mathit{\beta }},D)={(2\pi )}^{-\frac{k+1}{2}}\mid \widehat{\mathrm{\sum }}{\mid }^{-\frac{1}{2}}exp\left\{-\frac{1}{2}(\mathit{\beta }-\widehat{\mathit{\beta }}{)}^{\prime }{\widehat{\mathrm{\sum }}}^{-1}(\mathit{\beta }-\widehat{\mathit{\beta }})\right\}.$$ (51)

Using (51), we simply take w(β^(−m) | β^(m)) = π̂(β^(−m) | β^(m), β̂, D), which is the conditional distribution of β^(−m) given β^(m) with respect to the (k + 1)-dimensional multivariate normal distribution in (51).

Remark 4.8

As a by-product, C_m(D)/C(D) is ready to compute via the identity (33). It can also be shown that

$$\frac{C_{0m}({\mathbf{y}}_0)}{C_0({\mathbf{y}}_0)}=E\left\{\frac{L({\mathit{\beta }}^{(m)}\mid {\mathbf{y}}_0,a_0,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid {\mathbf{y}}_0,a_0)}\mid {\mathbf{y}}_0,a_0\right\},$$ (52)

where $L({\mathit{\beta }}^{(m)}\mid {\mathbf{y}}_0,a_0,m)=exp\left[a_0\{{\mathbf{y}}_0^{\prime }{\mathit{\theta }}^{(m)}-J^{\prime }\mathit{b}({\mathit{\theta }}^{(m)})\}\right]$ and the expectation is taken with respect to the prior distribution in (3) under the full model. After examining the construction of the conjugate prior and the form of the GLM in (2), we can also show that

$$B_m=\frac{C_m(D)/C(D)}{C_{0m}({\mathbf{y}}_0)/C_0({\mathbf{y}}_0)}=\frac{\pi ({\mathit{\beta }}^{(-m)}=\mathbf{0}\mid D)}{\pi ({\mathit{\beta }}^{(-m)}=\mathbf{0}\mid {\mathbf{y}}_0,a_0)},$$ (53)

where π(β^(−m) = 0|D) and π(β^(−m) = 0|y₀, a₀) are the marginal posterior density and the marginal prior density of β^(−m) evaluated at β^(−m) = 0 under the full model. Furthermore, B_m in (53) is the Bayes factor for comparing model m to the full model. Thus, to compute B_m, we need to generate two MCMC samples, one from the posterior distribution and another one from the prior distribution of β under the full model, and then use (33) and (52).

Finally, we note that we derive w_opt under the independence assumption. We expect that this optimal choice will work well even when a dependent MCMC sample is used. Some related empirical studies have been reported and discussed in Meng and Wong (1996), Diciccio et al. (1997) and Meng and Schilling (2002). They suggested that the optimal or near-optimal procedures constructed under the independence assumption can work remarkably well in general, providing orders of magnitude improvement over other methods with similar computational effort. Alternatively, suppose we systematically take a 1-in-b subsample of size S from the Markov chain that is generated from the joint posterior distribution in (4). Then, following from Guha et al. (2004), we can show that (45) holds under some mild regularity conditions such as geometrical ergodicity and a sufficiently large b. Thus, if we take a MCMC sample in such a way, this MCMC sample can be treated as “a random sample.”

5 A Simulation Study

In Section 3, we have established theoretical connections among AIC, BIC and the four Bayesian criteria in the normal linear regression setting. However, it does not appear possible that there are any analytic connections between AIC or BIC and the four Bayesian criteria for Poisson regression. For this reason, we present a simulation study for Poisson regression to empirically examine whether there exist any connections among these criteria and to examine the performance of conjugate priors in the context of variable selection. Suppose y_i|θ_i are independent Poisson observations with mean $e^{{\mathbf{x}}_i^{\prime }\mathit{\beta }}$, where ${\mathbf{x}}_i^{\prime }$ is a 1 × p vector, i = 1, 2, …, n. The conjugate prior takes the form

$$\pi (\mathit{\beta }\mid a_0,{\mathbf{y}}_0)\propto exp\left\{\sum _{i=1}^n{a}_0(y_{0i}{\mathbf{x}}_i^{\prime }\mathit{\beta }-exp\{{\mathbf{x}}_i^{\prime }\mathit{\beta }\})\right\},$$ (54)

where y_0i is the i^th component of y₀. In the simulation, we assume that x_i0 = 1, x_ij ~ N (0, 1) independently for j = 1, 2, 3 and i = 1, 2, …, n. In (54), we take y_0i = 1 for i = 1, 2, …, n, which yields a prior mode of β to be 0, as shown in Chen and Ibrahim (2003). Further we use β = (−0.3, 0.3, 0, 0)′, β = (−0.3, 0.3, 0.2, 0)′, and β= (−0.3, 0.3, 0.2, −0.15)′ which correspond to the true models (x₁), (x₁, x₂), and (x₁, x₂, x₃) (full model), respectively. We also use the sample size of n = 500.

Under the simulation design, we independently generated N = 500 datasets. For each simulated dataset, we fit 2³ = 8 models. To compute the posterior model probabilities based on the conjugate priors, we implemented the Monte Carlo algorithm proposed in Section 4 with a Monte Carlo sample size of S = 20, 000. For all of these 8 models, we computed BF, DIC, L measure, LPML, AIC, and BIC.

Tables 1 and 2 show results for the various methods. Our model performance evaluation criterion is a 0-1 loss function, the loss being 0 if the true model is selected and 1 otherwise. In Table 1, we see that BIC performs better than AIC in the number of times the true model is selected as best when the true model is a smaller model. For example, when (x₁) is the true model, AIC correctly identifies this model as best 361 times out of 500 and BIC correctly identifies this model as best 490 times. Table 2 compares the performance of the four other criteria under several values of a₀ from the conjugate prior as well as several values of ν for the L measure. We see from the table that, in general, for small values of a₀, which imply a noninformative prior, the Bayes factor results are quite consistent with DIC, the L measure, and LPML for small models being the true models, whereas when the full model is the true model, the Bayes factor tends to do worse for small a₀ compared to large a₀. In general, as a₀ increases, the performance of DIC, LPML, and the Bayes factor becomes worse, whereas for the L measure, it is fairly robust over several values of a₀. The L measure seems to perform best under moderate values of ν, such as ν = 0.5.

Table 1.

Frequencies for Ranking the True Model as Best Using AIC and BIC Based on n = 500 and N = 500 Datasets

True Model	AIC	BIC
(x1)	361	490
(x1, x2)	425	446
(x1, x2, x3)	474	316

Table 2.

Frequencies for Ranking the True Model as Best Using BF, DIC, CPO and L measure for Various a₀ Based on n = 500 and N = 500 Datasets

					L Measure (ν)
True Model	a0	LPML	DIC	BF	0.1	0.3	0.5	0.7	0.9
(x1)	0.001	395	361	492	398	396	359	318	276
	0.01	396	357	466	396	396	357	319	275
	0.1	377	332	386	408	396	352	304	268
	0.5	342	308	311	424	381	335	279	243
	1	320	299	288	424	372	321	264	222
(x1, x2)	0.001	425	425	436	164	347	390	380	356
	0.01	423	425	470	157	352	390	383	355
	0.1	417	417	443	195	370	399	372	353
	0.5	398	405	405	254	400	402	362	339
	1	382	394	391	269	410	390	359	329
(x1, x2, x3)	0.001	475	474	291	88	371	456	475	480
	0.01	475	474	388	94	375	458	475	482
	0.1	479	475	460	125	402	466	480	488
	0.5	485	479	479	176	436	478	486	489
	1	486	481	481	214	453	483	487	490

6 A Real Data Example

Due to lack of analytic connections between AIC or BIC and the four Bayesian criteria for logistic regression, we consider the Chapman data from Los Angeles Heart Study of men (n = 200) presented in Dixon and Massey (1983) to empirically examine whether there exist any connections among these criteria.

In our analysis, we consider a coronary incident as a binary response variable (y), which takes the values 0 and 1, where a 1 denotes that an incident had occurred in the previous ten years and a 0 indicates otherwise. We consider five prognostic factors: age (Ag), systolic blood pressure in millimeters of mercury (S), diastolic blood pressure in millimeters of mercury (D), Cholesterol in milligrams per DL (Ch), and BMI = (703.07Weight)/(Height²).

Let x₁, x₂, x₃, x₄, and x₅ denote Ag, S, D, Ch, and BMIH. For the Chapman data, we fit a logistic regression model

$$\text{logit}\{P(y=1\mid \mathbf{x})\}=log\left\{\frac{P(y=1\mid \mathbf{x})}{1-P(y=1\mid \mathbf{x})}\right\}={\mathbf{x}}^{\prime }\mathit{\beta }.$$ (55)

The conjugate prior in (3) corresponding to the model (55) takes the form

$$\pi (\mathit{\beta }\mid a_0,{\mathbf{y}}_0)\propto exp\left(\sum _{i=1}^n{a}_0\left[y_{0i}{\mathbf{x}}_i^{\prime }\mathit{\beta }-log\{1+exp({\mathbf{x}}_i^{\prime }\mathit{\beta })\}\right]\right),$$ (56)

where y_i0 = 0.5, i = 1, 2, …, n, to ensure the prior mode of β to be 0. We wish to compare the following 32 models: Intercept only, (x₁), …, (x₅), (x₁, x₂), …, (x₁, x₂, x₃, x₄, x₅). We note that the notation (x₁, x₂, x₃, x₄, x₅), for example, implies that ${\mathbf{x}}_i^{\prime }\mathit{\beta }={\beta }_0+{\beta }_1{\text{Ag}}_i+{\beta }_2{\text{S}}_i+{\beta }_3{\text{D}}_i+{\beta }_4{\text{Ch}}_i+{\beta }_5\text{B}{\text{MI}}_i$ in (55). Thus, “Intercept only” is the model with zero predictors while (x₁, x₂, x₃, x₄, x₅) is the full model with the largest model dimension. We also note that an intercept is included in every model. Further we denote that M₁ = (Int), M₂ = (Int, Ag), M₃ = (Int, S), M₄ = (Int, D), M₅ = (Int, Ch), M₆ =(Int, BMI), M₇ = (Int, Ag, S), M₈ =(Int, Ag, D), M₉ =(Int, Ag, Ch), M₁₀ =(Int, Ag, BMI), M₁₁ =(Int, S, D), M₁₂ =(Int, S, Ch), M₁₃ =(Int, S, BMI), M₁₄ =(Int, D, Ch), M₁₅ =(Int, D, BMI), M₁₆ =(Int, Ch, BMI), M₁₇ =(Int, Ag, S, D), M₁₈ =(Int, Ag, S, Ch), M₁₉ =(Int, Ag, S, BMI), M₂₀ =(Int, Ag, D, Ch), M₂₁ =(Int, Ag, D, BMI), M₂₂ =(Int, Ag, Ch, BMI), M₂₃ =(Int, S, D, Ch), M₂₄ =(Int, S, D, BMI), M₂₅ =(Int, S, Ch, BMI), M₂₆ = (Int, D, Ch, BMI), M₂₇ =(Int, Ag, S, D, Ch), M₂₈ =(Int, Ag, S, D, BMI), M₂₉ = (Int, Ag, S, Ch, BMI), M₃₀ =(Int, Ag, D, Ch, BMI), M₃₁ = (Int, S, D, Ch, BMI), and M₃₂ = (Int, Ag, S, D, Ch, BMI).

To compute the posterior model probability (PMP), DIC, LPML, and L measure under various conjugate priors, we implemented the Monte Carlo algorithm proposed in Section 4 with a Monte Carlo sample size of S = 20, 000. We see from Table 3 that M₂₂ is selected as the best model by AIC and the fourth model by BIC, whereas M₁₀ is selected as the second best model by both criteria. Table 4 shows the results of the L measure, posterior model probability (PMP), LPML, and DIC for several values of a₀, as well as several values of ν for the L measure. Table 3 reveals a similar story as the simulation study. Model M₂₂ is selected as either the top model or second best model for most values of a₀ for DIC and PMP, as well as for the L measure under small values of ν. Under larger values of ν the L measure as well a LPML appear to favor model M₃₂. Finally, for small values of a₀, LPML and PMP appear to favor a smaller model, namely M₂. Thus, from these analyses, models {M₂, M₂₂, M₃₂} appear to be the most promising based on all of these model selection criteria. Table 5 shows the top five models selected for each of the four variable selection criteria (PMP, DIC, L measure, LPML). Again we see a remarkable consistency between the four criteria, in which the ordering of the top models is similar for the four criteria for small, moderate, and large values of a₀, and for a wide range of ν values for the L measure.

Table 3.

The Top Model Based on AIC and BIC for Chapman Data

AIC		BIC
Mk	Values	Mk	Values
M22	142.75	M2	153.34
M10	143.73	M10	153.63
M29	144.69	M9	155.83
M30	144.75	M22	155.94
M19	145.57	M16	155.99

Table 4.

The Best Model Based on Posterior Model Probability (PMP), DIC, LPML, and L Measure for Chapman Data

	a0 = 0.001		a0 = 0.01		a0 = 0.1		a0 = 0.5		a0= 1.0
Criterion	Mk	Values	Mk	Values	Mk	Values	Mk	Values	Mk	Values
PMP	M2	0.57	M2	0.25	M22	0.14	M22	0.07	M22	0.06
DIC	M22	142.83	M22	142.67	M22	144.74	M22	165.65	M22	186.77
LPML	M2	−73.38	M2	−73.30	M32	−73.79	M32	−83.10	M30	−93.29
L(ν = 0.1)	M22	21.47	M22	21.98	M22	26.96	M22	38.92	M30	45.21
L(ν = 0.25)	M22	24.79	M22	25.29	M22	30.23	M22	42.56	M30	49.39
L(ν = 0.5)	M32	30.20	M32	30.73	M32	35.66	M29	48.59	M30	56.36
L(ν = 0.75)	M32	35.24	M32	35.76	M32	40.77	M32	54.52	M30	63.33
L(ν = 0.9)	M32	38.26	M32	38.78	M32	43.83	M32	58.06	M30	67.51

Table 5.

The Top Five Models Based on PMP, DIC, LPML, and L Measure for Chapman Data

	a0 = 0.001		a0 = 0.01		a0 = 0.1		a0 = 0.5		a0 = 1.0
Criterion	Mk	Values	Mk	Values	Mk	Values	Mk	Values	Mk	Values
PMP	M2	0.57	M2	0.25	M22	0.14	M22	0.07	M22	0.06
	M1	0.11	M10	0.23	M10	0.14	M10	0.07	M10	0.05
	M5	0.07	M9	0.08	M9	0.06	M29	0.05	M19	0.04
	M10	0.07	M22	0.08	M2	0.06	M19	0.05	M29	0.04
	M6	0.06	M16	0.07	M16	0.06	M30	0.05	M21	0.04
DIC	M22	142.83	M22	142.67	M22	144.74	M22	165.65	M22	186.77
	M10	143.79	M10	143.70	M10	145.70	M10	166.02	M10	186.88
	M30	144.85	M29	144.74	M29	146.43	M29	166.42	M30	187.10
	M29	144.96	M30	144.78	M30	146.48	M19	166.87	M21	187.38
	M21	145.63	M21	145.59	M21	147.29	M30	166.90	M20	187.75
LPML	M2	−73.38	M2	−73.30	M32	−73.79	M32	−83.10	M30	−93.29
	M5	−73.56	M5	−73.50	M2	−74.11	M29	−83.32	M32	−93.35
	M4	−73.58	M6	−73.55	M7	−74.43	M10	−83.34	M21	−93.41
	M6	−73.73	M4	−73.64	M8	−74.48	M19	−83.55	M10	−93.41
	M32	−73.91	M3	−73.65	M9	−74.68	M21	−83.56	M20	−93.55
L ν = 0.1	M22	21.47	M22	21.98	M22	26.96	M22	38.92	M30	45.21
	M30	22.04	M25	22.63	M25	27.33	M25	38.99	M22	45.23
	M26	22.13	M30	22.64	M30	27.41	M29	39.02	M26	45.26
	M32	22.15	M29	22.65	M26	27.45	M19	39.16	M20	45.28
	M10	22.21	M10	22.67	M29	27.47	M10	39.18	M21	45.31
L ν = 0.25	M22	24.79	M22	25.29	M22	30.23	M22	42.56	M30	49.39
	M30	25.17	M32	25.69	M32	30.56	M29	42.61	M22	49.45
	M32	25.17	M30	25.77	M30	30.56	M25	42.67	M20	49.49
	M10	25.43	M29	25.78	M29	30.62	M32	42.73	M26	49.51
	M20	25.47	M10	25.89	M25	30.65	M19	42.79	M21	49.52
L ν = 0.5	M32	30.20	M32	30.73	M32	35.66	M29	48.59	M30	56.36
	M22	30.31	M22	30.80	M22	35.70	M32	48.62	M32	56.47
	M30	30.38	M30	30.98	M30	35.81	M22	48.64	M22	56.50
	M20	30.77	M29	31.00	M29	35.88	M30	48.78	M20	56.50
	M10	30.80	M10	31.26	M10	36.10	M25	48.79	M21	56.55
L ν = 0.75	M32	35.24	M32	35.76	M32	40.77	M32	54.52	M30	63.33
	M30	35.59	M30	36.20	M30	41.06	M29	54.56	M32	63.41
	M22	35.84	M29	36.22	M29	41.14	M22	54.71	M20	63.52
	M29	36.04	M22	36.31	M22	41.16	M30	54.78	M22	63.54
	M20	36.08	M10	36.63	M10	41.48	M19	54.88	M21	63.57
L ν = 0.9	M32	38.26	M32	38.78	M32	43.83	M32	58.06	M30	67.51
	M30	38.72	M30	39.33	M30	44.21	M29	58.15	M32	67.58
	M22	39.16	M29	39.35	M29	44.29	M22	58.36	M20	67.73
	M29	39.17	M22	39.61	M22	44.44	M30	58.37	M22	67.77
	M20	39.26	M27	39.83	M10	44.70	M19	58.51	M21	67.79

Table 6 shows the posterior means (Estimates), the posterior standard errors (SEs), and 95% HPD intervals for the β_j‘s under model M₂₂ (Ag, Ch, BMI) and model M₃₂ (Ag, S, D, Ch, BMI) when a₀ = 0.01. Table 6 also shows the corresponding maximum likelihood estimates (MLEs), the standard errors, and p-values. We see from Table 6 that the posterior estimates are very close to the MLEs, which is intuitively appealing, as a fairly noninformative (a₀ = 0.01) is used. We also see from this table that under these two “best” models, age and BMI are only two prognostic factors for the coronary incident, which are significant at the 5% significance level.

Table 6.

Estimates of the β under Model (Ag, Ch, BMI) and Model (Ag, S, D, Ch, BMI) for the Chapman Data when a₀ = 0.01

		Maximum Likelihood Estimates			Posterior Estimates
Model	Variable	Estimate	SE	p-value	Estimate	SE	95% HPD Interval
M22	Intercept	−2.252	0.275	< .0001	−2.265	0.272	(−2.805, −1.748)
	Ag	0.556	0.245	0.0230	0.554	0.242	(0.087, 1.032)
	Ch	0.405	0.233	0.0816	0.402	0.234	(−0.064, 0.854)
	BMI	0.470	0.204	0.0211	0.465	0.207	(0.069, 0.882)
M32	Intercept	−2.248	0.274	< .0001	−2.292	0.273	(−2.828, −1.766)
	Ag	0.527	0.270	0.0507	0.531	0.270	(0.012, 1.067)
	S	0.106	0.336	0.7523	0.097	0.344	(−0.583, 0.757)
	D	−0.077	0.383	0.8417	−0.069	0.383	(−0.806, 0.687)
	Ch	0.404	0.235	0.0857	0.402	0.240	(−0.074, 0.866)
	BMI	0.474	0.226	0.0361	0.473	0.230	(0.028, 0.930)

To examine performance of the proposed Monte Carlo method in Section 4, we first computed various model selection criteria under a sub-model using a MCMC sample from the full model. We then computed the same quantities using a MCMC sample directly from the posterior distribution under the same sub-model. For illustrative purposes, we considered a single variable sub-model M₂ = (Int, Ag) using the conjugate prior (56) with a₀ = 0.01. Using a MCMC sample size of S = 20, 000, the Monte Carlo estimates (simulation standard errors) of DIC, LPML, L(ν = 0.1), L(ν = 0.5), and L(ν = 0.9) under model M₂ are 146.68 (0.08), −73.30 (0.04), 23.91 (0.05), 32.44 (0.06), and 40.96 (0.06), respectively, using the proposed Monte Carlo method via (35). With the same MC sample size, these quantities are 146.67 (0.02), −73.29 (0.01), 23.90 (0.02), 32.42 (0.02), and 40.95 (0.02), respectively, using the MC sample directly from the posterior distribution under model M₂. All simulation standard errors were computed using the overlapping batch statistics (OBS) method of Schmeiser et al. (1990). As expected, the simulation standard errors using the MC sample from the full model are slightly larger than those computed using the MC sample directly from model M₂. However, these two sets of the MC estimates are very close. This empirically demonstrates that the proposed MC method works quite well. Finally, we compared the computational times between the proposed Monte Carlo method and the exhaustive alternative. With 2,000 “burn-in” iterations and S = 20, 000, the computational times of the proposed Monte Carlo method for 32 DIC’s, LPML’s, and L(ν)’s are 71.28, 100.11, and 76.36 seconds, respectively, on a Dell WS Xeon dual 2.4GHZ CPU Linux workstation. Using the same number of “burn-in” iterations, the same MC sample size, and the same computer, the computational times of the exhaustive alternative Monte Carlo method for 32 DIC’s, LPML’s, and L(ν)’s are 324.05, 357.97, and 322.13 seconds, respectively. Thus, it becomes apparent that the proposed Monte Carlo method leads to a substantial computational saving over the exhaustive alternative.

7 Concluding Remarks

We have examined and established theoretical and computational relationships between six commonly used methods for variable subset selection. These connections were facilitated from the class of conjugate priors of Chen and Ibrahim (2003). We saw that under this class of priors the four Bayesian criteria were quite similar in terms of model choice especially under small values of a₀, and the results were fairly robust under a wide choice of a₀ values. Further work remains to be done. In particular, it is of interest to obtain analytic connections between these criteria for specific GLM’s, such as the logistic and Poisson regression models, as well as theoretically examine the small sample and large sample behavior of these methods. In Section 4, the theory and algorithm are developed for computing the four Bayesian criteria which are defined for the GLM in (2). With some straightforward modification, these theory and algorithm can be applied for computing the four Bayesian criteria that are defined for the general GLM in (1).

We note some philosophical issues about model selection that are worth noting. In this paper, we have evaluated the performance of all criteria based on how well they can pick up the true sampling model. However, there are other ways of defining the “Bayesian model.” Many advocate that a Bayesian model is specified by the sampling density and the prior, not only by the sampling density. When one only evaluates the success of a criterion based on how well it picks up the sampling model, then a comparison between AIC (or BIC) and DIC is not meaningful when DIC is computed using an informative prior. Since AIC is equivalent to DIC based on a noninformative prior, a comparison of AIC (or BIC) to DIC is simply not meaningful when using informative priors. In general, one should avoid such comparisons, and only comparable criteria should be compared. For example, it is meaningful to compare AIC, BIC, DIC, LPML, the L-measure, and the Bayes factor based on noninformative priors. It is meaningful to compare DIC, the L-measure, LPML, and the Bayes factor based on informative priors. Finally, we note that most criteria for model assessment, especially the information criteria, are based on a well-defined utility function. If a utility function is chosen, a comparison to a criterion based on a different utility function is not justified. For example, the Bayes factor and BIC are prior predictive criteria aiming at the explanation of the data given the prior, whereas DIC (AIC as a special case) and LPML are posterior predictive criteria aiming at the explanation of replicate (unseen) data given the posterior. Thus, one must use caution in comparing these criteria in terms in picking up the true sampling model.

Appendix: Proofs of Theorems

Proof of Theorem 5

Since ∫ w(β^(−m)|β^(m))dβ^(−m) = 1 and β = (β^(m)′ β^(−m)′)′, we have

$$\begin{array}{l}{g}_m=\int g({\mathit{\beta }}^{(m)})\frac{L({\mathit{\beta }}^{(m)}\mid D,m)}{C_m(D)}d{\mathit{\beta }}^{(m)}\\ =\int \int g({\mathit{\beta }}^{(m)})\frac{L({\mathit{\beta }}^{(m)}\mid D,m)}{C_m(D)}w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})d{\mathit{\beta }}^{(-m)}d{\mathit{\beta }}^{(m)}\\ =\frac{C(D)}{C_m(D)}\int g({\mathit{\beta }}^{(m)})\frac{L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid D)}\frac{L(\mathit{\beta }\mid D)}{C(D)}d\mathit{\beta }\\ =\frac{C(D)}{C_m(D)}E\left\{\frac{g({\mathit{\beta }}^{(m)})L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid D)}\mid D\right\},\end{array}$$

which completes the proof.

Proof of Theorem 7

From (43), we have

$$V_w(g_m)=E\left[{\left\{\frac{g({\mathit{\beta }}^{(m)})}{A}-\frac{1}{B}\right\}}^2{\left\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{L(\mathit{\beta }\mid D)}\right\}}^2\mid D\right].$$ (A.1)

Plugging w_opt into (A.1), we have

$$\begin{array}{l}{V}_{w_{\mathit{opt}}}(g_m)\\ =\int {\left\{\frac{g({\mathit{\beta }}^{(m)})}{A}-\frac{1}{B}\right\}}^2{\left\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)}{C(D)}\right\}}^2\frac{\pi {({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)},D)}^2}{\pi (\mathit{\beta }\mid D)}d\mathit{\beta }\\ =\int {\left\{\frac{g({\mathit{\beta }}^{(m)})}{A}-\frac{1}{B}\right\}}^2{\left\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)}{C(D)}\right\}}^2\frac{\pi ({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)},D)\frac{\pi ({\mathit{\beta }}^{(-m)},{\mathit{\beta }}^{(m)}\mid D)}{\pi ({\mathit{\beta }}^{(m)}\mid D)}}{\pi (\mathit{\beta }\mid D)}d\mathit{\beta }\\ =\int {\left\{\frac{g({\mathit{\beta }}^{(m)})}{A}-\frac{1}{B}\right\}}^2{\left\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)}{C(D)}\right\}}^2\frac{\pi ({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)},D)}{\pi ({\mathit{\beta }}^{(m)}\mid D)}d\mathit{\beta }\\ =\int {\left\{\frac{g({\mathit{\beta }}^{(m)})}{A}-\frac{1}{B}\right\}}^2{\left\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)}{C(D)}\right\}}^2\frac{d{\mathit{\beta }}^{(m)}}{\pi ({\mathit{\beta }}^{(m)}\mid D)}\int \pi ({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)},D)d{\mathit{\beta }}^{(-m)}\\ \int {\left\{\frac{g({\mathit{\beta }}^{(m)})}{A}-\frac{1}{B}\right\}}^2{\left\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)}{\pi ({\mathit{\beta }}^{(m)}\mid D)C(D)}\right\}}^2\pi ({\mathit{\beta }}^{(m)}\mid D)d{\mathit{\beta }}^{(m)},\end{array}$$ (A.2)

where π(β^(m) | D) denotes the marginal posterior distribution of β^(m) under the full model. Thus, it suffices to show

$$\begin{array}{l}\int {\left\{\frac{g({\mathit{\beta }}^{(m)})}{A}-\frac{1}{B}\right\}}^2{\left\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)}{\pi ({\mathit{\beta }}^{(m)}\mid D)}\right\}}^2\pi ({\mathit{\beta }}^{(m)}\mid D)d{\mathit{\beta }}^{(m)}\\ \le \int {\left\{\frac{g({\mathit{\beta }}^{(m)})}{A}-\frac{1}{B}\right\}}^2{\left\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{\pi (\mathit{\beta }\mid D)}\right\}}^2\pi (\mathit{\beta }\mid D)d\mathit{\beta }.\end{array}$$ (A.3)

By the Cauchy-Schwarz inequality, we have

$$\begin{array}{l}1={\left\{\int w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})d{\mathit{\beta }}^{(-m)}\right\}}^2\\ ={\left\{\int \frac{w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{\sqrt{\pi ({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)},D)}}\sqrt{\pi ({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)},D)}d{\mathit{\beta }}^{(-m)}\right\}}^2\\ \le \int \frac{w^2({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{\pi ({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)},D)}d{\mathit{\beta }}^{(-m)}\int \pi ({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)},D)d{\mathit{\beta }}^{(-m)}\\ =\int \frac{w^2({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{\pi ({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)},D)}d{\mathit{\beta }}^{(-m)}.\end{array}$$ (A.4)

Using (A.4), the left-hand side of (A.3) becomes

$$\begin{array}{l}\int {\left\{\frac{g({\mathit{\beta }}^{(m)})}{A}-\frac{1}{B}\right\}}^2{\left\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)}{\pi ({\mathit{\beta }}^{(m)}\mid D)}\right\}}^2\pi ({\mathit{\beta }}^{(m)}\mid D)d{\mathit{\beta }}^{(m)}\\ \le \int {\left\{\frac{g({\mathit{\beta }}^{(m)})}{A}-\frac{1}{B}\right\}}^2{\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{\pi ({\mathit{\beta }}^{(m)}\mid D)})}^2\frac{\pi ({\mathit{\beta }}^{(m)}\mid D)}{\pi ({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)},D)}d\mathit{\beta }\\ =\int {\left\{\frac{g({\mathit{\beta }}^{(m)})}{A}-\frac{1}{B}\right\}}^2{\{\frac{L({\mathit{\beta }}^{(m)}\mid D,m)w({\mathit{\beta }}^{(-m)}\mid {\mathit{\beta }}^{(m)})}{\pi (\mathit{\beta }\mid D)})}^2\pi (\mathit{\beta }\mid D)d\mathit{\beta },\end{array}$$

which exactly matches the right-hand side of (A.3).