Bayes variable selection in semiparametric linear models

Abstract

There is a rich literature on Bayesian variable selection for parametric models. Our focus is on generalizing methods and asymptotic theory established for mixtures of g-priors to semiparametric linear regression models having unknown residual densities. Using a Dirichlet process location mixture for the residual density, we propose a semiparametric g-prior which incorporates an unknown matrix of cluster allocation indicators. For this class of priors, posterior computation can proceed via a straightforward stochastic search variable selection algorithm. In addition, Bayes factor and variable selection consistency is shown to result under a class of proper priors on g even when the number of candidate predictors p is allowed to increase much faster than sample size n, while making sparsity assumptions on the true model size.

1. INTRODUCTION

Bayesian variable selection is widely applied, with O’Hara and Sillanpää providing a recent review (2009). There is a rich literature proposing variable selection methods and studying asymptotic properties for parametric models, while our focus is variable selection in semiparametric linear regression models of the form:

$${\mathbf{Y}}^n={\mathbf{X}}_{\mathit{\gamma }}{\mathit{\beta }}_{\mathit{\gamma }}+\mathit{\epsilon },{\epsilon }_i~f,$$ (1)

where Yⁿ is n × 1, γ = {γ_j, j = 1, …, p} ∈ Γ, γ_j = 1 if the jth candidate predictor is included in the model with γ_j = 0 otherwise, Γ is the set of all possible subsets that are given non-zero prior probability, $p_{\mathit{\gamma }}={\sum }_{j=1}^p{\gamma }_j$ is the size of model γ, β_γ is the p_γ × 1 vector of regression coefficients, X_γ is the n × p_γ design matrix containing the predictors in model γ, and f is an unknown residual density. Our focus is on avoiding parametric assumptions on f, while accommodating high-dimensional settings in which the number of candidate predictors p can be much larger than the sample size n but Γ is restricted to sparse models having p_γ < n.

There has been limited consideration of variable selection in semiparametric Bayesian models, with essentially no results on asymptotic properties. In particular, it would be appealing to provide a computationally efficient procedure for Bayesian variable selection based on (1) for which it can be shown that the posterior probability on the true model converges to one as n → ∞ even when the number of candidate predictors increases much faster than n. In order for the asymptotic analysis to reflect the high dimensionality, it is important to allow p to grow with n. There has been some consideration of increasing p asymptotics in Bayesian parametric models. Castillo and van der Vaart (2012) study concentration of the posterior distribution in the normal means problem. Armagan et al. (2013) provide conditions for consistency in high-dimensional normal linear regression with shrinkage priors on the coefficients. Jiang (2007) studies convergence rates of the predictive distribution resulting from Bayesian model averaging in generalized linear models with high-dimensional predictors. These approaches do not consider consistency of model selection or semiparametric settings.

This article proposes a practical, useful and general methodology for Bayesian variable selection in semiparametric linear models (1), while providing basic theoretical support by showing Bayes factor and variable selection consistency. We also extend our approach and theory to increasing model dimensions involving p ≫ n candidate predictors while making sparsity assumptions on the true model. Our approach relies on placing a Dirichlet process (DP, Ferguson, 1972) location mixture of Gaussians (Lo, 1984) prior on the residual density f, inducing clustering of subjects. We introduce a prior on the coefficients β_γ specific to each model γ, which generalizes mixtures of g-priors (Zellner and Siow, 1980; Liang et al., 2008) to include cluster allocation indices induced through the Dirichlet process. The formulation leads to a straightforward implementation via a stochastic search variable selection (SSVS) algorithm (George and McCulloch, 1997).

Section 2 develops the proposed framework. Section 3 considers asymptotic properties. Section 4 contains simulation results. Section 5 applies the approach to a type 2 diabetes data example, and the proofs of Theorems are contained in the Appendix.

2. MIXTURES OF SEMIPARAMETRIC g-PRIORS

2.1 Model Formulation

In this section, we propose a new class of priors for Bayesian variable selection in linear regression models with an unknown residual density characterized via a Dirichlet process (DP) location mixture of Gaussians. In particular, let

$$\begin{array}{l}{y}_i={\mathbf{x}}_{\mathit{\gamma },i}^{\prime }{\mathit{\beta }}_{\mathit{\gamma }}+{\epsilon }_i,{\epsilon }_i~f,i=1,\dots ,n,\\ f(·)=\int N(·;\alpha ,{\tau }^{-1})dP(\alpha ),P~DP(m{P}_0),P_0=N(0,{\tau }^{-1}),\end{array}$$ (2)

where x_γ,i is the ith row of X_γ and does not include an intercept as we do not restrict f to have zero mean, and f is a density with respect to Lebesgue measure on ℜ. We address uncertainty in subset selection by placing a prior on γ, while the prior on β_γ characterizes prior knowledge of the size of the coefficients for the selected predictors.

The DP mixture prior on the density f induces clustering of the n subjects into k groups/subclusters, where k is random and each group has a distinct intercept in the linear regression model. Let A denote an n × k allocation matrix, with A_ij = 1 if the ith subject is allocated to the jth cluster and 0 otherwise. The jth column of A then sums to n_j, the number of subjects allocated to subcluster j, with ${\sum }_{j=1}^k{n}_j=n$. Following Kyung, Gill and Casella (2009), conditionally on the allocation matrix A, (2) can be represented as a linear model with random intercepts

$${\mathbf{Y}}^n=\mathbf{A}\mathit{\eta }+{\mathbf{X}}_{\mathit{\gamma }}{\mathit{\beta }}_{\mathit{\gamma }}+\mathit{\epsilon },\mathit{\eta }\sim N(0,{\tau }^{-1}{\mathbf{I}}_k),\mathit{\epsilon }~N(0,{\tau }^{-1}{\mathbf{I}}_n),$$ (3)

where A is random with a certain prior probability given by the coefficients in the summation of the likelihood expression (8) and the response and predictors are centered prior to analysis. In the special case in which A = 1_n, the model reduces to a linear regression model with a common intercept η and Gaussian residuals. In this case, the conditional posterior for η given A = 1_n is $N\left(\frac{1}{1+n}{\sum }_{i=1}^n(y_i-{\mathbf{x}}_{\mathit{\gamma },i}{\mathit{\beta }}_{\mathit{\gamma }}),\sqrt{\frac{1}{\tau (1+n)}}\right)$, which has realizations increasingly concentrated at zero as n increases.

We would like the prior on the regression coefficients to retain the essential elements of Zellner’s g-prior (Zellner, 1986), while being suitably adapted to the semiparametric case. To this effect, we propose a mixture of semi-parametric g-priors constructed to scale the covariance matrix in Zellner’s g-prior to reflect the clustering phenomenon as follows:

$$\pi ({\mathit{\beta }}_{\mathit{\gamma }})=N(0,g{\tau }^{-1}{({\mathbf{X}}_{\mathit{\gamma }}^{\prime }{\mathbf{\sum }}_A^{-1}{\mathbf{X}}_{\mathit{\gamma }})}^{-1}),{\mathbf{\sum }}_A={\mathbf{I}}_n+{\mathbf{AA}}^{\prime },g~\pi (g).$$ (4)

Prior (4) inherits advantages of previous mixtures of g-priors including computational efficiency in computing marginal likelihoods (conditional on A) and robustness to mis-specification of g. The prior can be interpreted as having arisen from the analysis of a conceptual sample generated using a scaled design matrix ${\mathbf{\sum }}_A^{-1/2}{\mathbf{X}}_{\mathit{\gamma }}$, reflecting the clustering phenomenon due to the DP kernel mixture prior. Moreover, the proposed prior leads to Bayes factor and variable selection consistency in semi-parametric linear models (2) as we will show.

Note that $({\mathbf{X}}_{\mathit{\gamma }}^{\prime }{\mathbf{\sum }}_A^{-1}{\mathbf{X}}_{\mathit{\gamma }})=({\mathbf{X}}_{\mathit{\gamma }}^{\prime }{\mathbf{X}}_{\mathit{\gamma }})-{\mathbf{X}}_{\mathit{\gamma }}^{\prime }\mathbf{A}{({\mathbf{I}}_k+{\mathbf{A}}^{\prime }\mathbf{A})}^{-1}{\mathbf{A}}^{\prime }{\mathbf{X}}_{\mathit{\gamma }}<({\mathbf{X}}_{\mathit{\gamma }}^{\prime }{\mathbf{X}}_{\mathit{\gamma }})$, so ${({\mathbf{X}}_{\mathit{\gamma }}^{\prime }{\mathbf{\sum }}_A^{-1}{\mathbf{X}}_{\mathit{\gamma }})}^{-1}>{({\mathbf{X}}_{\mathit{\gamma }}^{\prime }{\mathbf{X}}_{\mathit{\gamma }})}^{-1}$, implying that the prior variance of β_γ conditional on (g, τ) is higher for the semi-parametric g–prior as compared to the traditional g–prior for any allocation matrix A. To assess the influence of A on the prior for β_γ, we did simulations which revealed that for fixed (n, p), var(β_γl) increases but the cov(β_γl, β_γl′) decreases as the number of underlying subclusters in the data increase (l′, l = 1, …, p, l′ ≠ l). This suggests that as the number of groups in A increase, the components of β_γ are likely to be more dispersed with decreasing association between each other.

2.2 Bayes Factor in Semiparametric Linear Models

In studying asymptotic properties of our proposed approach, we follow standard practice in Bayesian model selection, and assume that the data Yⁿ = (y₁, …, y_n)′ arise from one of the models in the list under consideration. This true model is denoted as defined in equation (5). For pairwise comparison, we evaluate the evidence in favor of compared to an alternative model using the Bayes factor, where

$$\begin{array}{l}{\mathcal{M}}_1:{\mathbf{Y}}^n={\mathbf{X}}_{{\mathit{\gamma }}_1}{\mathit{\beta }}_{{\mathit{\gamma }}_1}+{\mathit{\epsilon }}_1,{\epsilon }_{1i}~f\\ {\mathcal{M}}_2:{\mathbf{Y}}^n={\mathbf{X}}_{{\mathit{\gamma }}_2}{\mathit{\beta }}_{{\mathit{\gamma }}_2}+{\mathit{\epsilon }}_2,{\epsilon }_{2i}~f\\ f(·)=\int N(·;\alpha ,{\tau }^{-1})dP(\alpha ),P~DP(m{P}_0),P_0=N(0,{\tau }^{-1})\\ {\mathit{\beta }}_{{\mathit{\gamma }}_j}~\pi ({\mathit{\beta }}_{{\mathit{\gamma }}_j}),j=1,2,\pi ({\tau }^{-1})\propto 1/{\tau }^{-1},g~\pi (g),\end{array}$$ (5)

where γ_j ∈ Γ indexes models of dimension p_j and π(β_γj) is defined in (4), j = 1, 2. Our prior specification philosophy is similar to the one adopted by Guo and Speckman (2009) for normal linear models, in that we assign proper priors on all elements of both β_γ1, β_γ2 conditional on (g, τ⁻¹), and an improper prior on τ⁻¹ for a more objective assessment. However unlike Guo and Speckman (2009), our focus is on Bayesian variable selection in semi-parametric linear models.

Note that the conditional likelihood of the response after marginalizing out η in (3) is L(Yⁿ|A, β_γ, τ⁻¹) = N (X_γβ_γ, τ⁻¹Σ_A) (Kyung et. al., 2009). Thus conditional on A and under the DP mixture of Gaussians prior on f, in (5) reduces to the normal linear model:

$${\mathbf{\sum }}_A^{-1/2}{\mathbf{Y}}^n={\mathbf{Z}}_A={\stackrel{\sim }{\mathbf{X}}}_{A,{\mathit{\gamma }}_j}{\mathit{\beta }}_{{\mathit{\gamma }}_j}+\mathit{\epsilon },\mathit{\epsilon }~N(0,{\tau }^{-1}{\mathbf{I}}_n),\pi ({\mathit{\beta }}_{{\mathit{\gamma }}_j})=N(0,g{\tau }^{-1}{({\stackrel{\sim }{\mathbf{X}}}_{A,{\mathit{\gamma }}_j}^{\prime }{\stackrel{\sim }{\mathbf{X}}}_{A,{\mathit{\gamma }}_j})}^{-1}),$$ (6)

where ${\stackrel{\sim }{\mathbf{X}}}_{A,{\mathit{\gamma }}_j}={\mathbf{\sum }}_A^{-1/2}{\mathbf{X}}_{{\mathit{\gamma }}_j}$. Under a mixture of semi-parametric g-priors, we can directly use expression (17) in Guo and Speckman (2009) to obtain (conditional on A) for j = 1, 2

$$L({\mathbf{Z}}_A\mid {\mathcal{M}}_j)\equiv L({\mathbf{Y}}^n\mid \mathbf{A},{\mathcal{M}}_j)\propto {({\mathbf{Z}}_A^{\prime }{\mathbf{Z}}_A)}^{-n/2}{\int }_0^{\infty }{(1+g)}^{-p_j/2}{\left[1-\frac{g}{1+g}\frac{{\mathbf{Z}}_A^{\prime }{\stackrel{\sim }{\mathbf{H}}}_{A,j}{\mathbf{Z}}_A}{{\mathbf{Z}}_A^{\prime }{\mathbf{Z}}_A}\right]}^{-n/2}\pi (dg),$$ (7)

where ${\stackrel{\sim }{\mathbf{H}}}_{A,j}={\stackrel{\sim }{\mathbf{X}}}_{A,{\mathit{\gamma }}_j}{({\stackrel{\sim }{\mathbf{X}}}_{A,{\mathit{\gamma }}_j}^{\prime }{\stackrel{\sim }{\mathbf{X}}}_{A,{\gamma }_j})}^{-1}{\stackrel{\sim }{\mathbf{X}}}_{A,{\mathit{\gamma }}_j}^{\prime }$.

Also, marginalizing over all possible subcluster allocations for a given sample size n, the following marginal likelihood can be obtained under a DP prior on f (Kyung et. al., 2009):

$$L({\mathbf{Y}}^n\mid {\mathcal{M}}_j)=\frac{\mathrm{\Gamma }(m)}{\mathrm{\Gamma }(m+n)}\sum _{k=1}^n{m}^k\sum _{\mathbf{A}\in {\mathcal{A}}_k}\prod _{i=1}^k\mathrm{\Gamma }(n_i)L({\mathbf{Y}}^n\mid \mathbf{A},{\mathcal{M}}_j)=\sum _{{\mathbf{A}}_l\in {\mathcal{C}}_n}{w}_{l}L({\mathbf{Y}}^n\mid {\mathbf{A}}_l,{\mathcal{M}}_j),$$ (8)

where is the collection of all possible n×k matrices corresponding to different allocations of n subjects into k subclusters, and is the collection of all possible allocation matrices for a sample size n with w_l = 1. In the limiting case as n → ∞, we have as the class of limiting allocation matrices. Further using (7), the Bayes factor in favor of conditional on the allocation matrix A is given by

$$B{F}_{21,A}^n=\frac{L({\mathbf{Z}}_A\mid {\mathcal{M}}_2)}{L({\mathbf{Z}}_A\mid {\mathcal{M}}_1)}=\frac{{\int }_0^{\infty }{(1+g)}^{-p_2/2}{\left[1-\frac{g}{1-g}{\stackrel{\sim }{R}}_{A,2}^2\right]}^{-n/2}\pi (dg)}{{\int }_0^{\infty }{(1+g)}^{-p_1/2}{\left[1-\frac{g}{1+g}{\stackrel{\sim }{R}}_{A,1}^2\right]}^{-n/2}\pi (dg)},$$ (9)

where ${\stackrel{\sim }{R}}_{A,j}^2={\mathbf{Z}}_A^{\prime }{\stackrel{\sim }{\mathbf{H}}}_{A,j}{\mathbf{Z}}_A/{\mathbf{Z}}_A^{\prime }{\mathbf{Z}}_A$, (j = 1, 2). Finally using (8), the unconditional Bayes factor in favor of marginalizing out A is

$$B{F}_{21}^n=\frac{L({\mathbf{Y}}^n\mid {\mathcal{M}}_2)}{L({\mathbf{Y}}^n\mid {\mathcal{M}}_1)}=\frac{{\sum }_{{\mathbf{A}}_l\in {\mathcal{C}}_n}{w}_{l}L({\mathbf{Z}}_{{\mathbf{A}}_l}\mid {\mathcal{M}}_2)}{{\sum }_{{\mathbf{A}}_l\in {\mathcal{C}}_n}{w}_{l}L({\mathbf{Z}}_{{\mathbf{A}}_l}\mid {\mathcal{M}}_1)}.$$ (10)

2.3 Posterior Computation

We propose a MCMC algorithm for posterior computation for model (2), which combines a stochastic search variable selection algorithm or SSVS (George and McCulloch, 1997) with recently proposed methods for efficient computation in DP mixture models. In particular, we utilize the slice sampler of Walker (2007) incorporating the modification of Yau et al.(2011). Using Sethuraman’s (1994) stick-breaking representation, let

$$P=\sum _{j=1}^{\infty }{w}_j{\eta }_j,{\eta }_j~N(0,{\tau }^{-1}),w_j={\nu }_j\prod _{l<j}(1-{\nu }_l),{\nu }_l~\mathit{Beta}(1,m).$$ (11)

The slice sampler of Walker (2007) relies on augmentation with uniform latent variables, which allows us to move from an infinite summation for P in (11) to a finite sum given the uniform latent variable. In particular,

$$f_{w,\eta }(y\mid u)\propto \sum _{j\in B_w(u)}N(y\mid {\eta }_j),B_w(u)=\{j:w_j>u\}\text{is}\mathrm{a}\text{finite}\text{set},u~U(0,1).$$

For the DP precision parameter, we specify the hyperprior m ~ Ga(a_m, b_m) for greater flexibility. We specify a Ga(a_τ, b_τ) prior on τ and Be(a₁, b₁) prior on Pr(γ_l = 1) for implementing SSVS, l = 1, …, p. We choose π(g) as the hyper-g prior with a = 4 and use the fact that $\frac{g}{1+g}~Be(1,1)$ to sample g using a griddy Gibbs approach employing equally spaced quantiles. Inverting the n × n matrix Σ_A in the mixtures of semiparametric g-prior in (4) does not add much to the computational burden even for large n, as we can use the closed form expression ${\mathbf{\sum }}_A^{-1}={\mathbf{I}}_n-\mathbf{A}{({\mathbf{I}}_k+{\mathbf{A}}^{\prime }\mathbf{A})}^{-1}{\mathbf{A}}^{\prime }$, where k grows at a rate log(n) (Antoniak, 1974) and is small to moderate in most practical applications. We outline the posterior computation steps in Appendix I.

3. ASYMPTOTIC PROPERTIES

In this section we establish asymptotic properties for the proposed approach using γ₁ to index the true model defined in (5) and γ₂ to index an arbitrary model being compared to , with ⊂ denoting nesting of in . Before proceeding, we introduce some regularity conditions essential for the development of asymptotic theory.

• (A1′) ${lim}_{n\to \infty }\frac{{\mathit{\beta }}_{{\mathit{\gamma }}_1}^{\prime }({\mathbf{X}}_{{\mathit{\gamma }}_1}^{\prime }{\mathbf{X}}_{{\mathit{\gamma }}_1}){\mathit{\beta }}_{{\mathit{\gamma }}_1}}{n}\to b_1>0$.
• (A2′)For ⊈ , ${lim}_{n\to \infty }\frac{{\mathit{\beta }}_{{\mathit{\gamma }}_1}^{\prime }{\mathbf{X}}_{{\mathit{\gamma }}_1}^{\prime }{\mathbf{H}}_2{\mathbf{X}}_{{\mathit{\gamma }}_1}{\mathit{\beta }}_{{\mathit{\gamma }}_1}}{n}\to b_2\in [0,b_1]$, with ${\mathbf{H}}_2={\mathbf{X}}_{{\gamma }_2}{({\mathbf{X}}_{{\mathit{\gamma }}_2}^{\prime }{\mathbf{X}}_{{\mathit{\gamma }}_2})}^{-1}{\mathbf{X}}_{{\mathit{\gamma }}_2}^{\prime }$.
• (A1)For p₁ = O(n^a₁), 0 ≤ a₁ < 1, ${lim}_{n\to \infty }\frac{{\mathit{\beta }}_{{\mathit{\gamma }}_1}^{\prime }({\mathbf{X}}_{{\mathit{\gamma }}_1}^{\prime }{\mathbf{\sum }}_A^{-1}{\mathbf{X}}_{{\mathit{\gamma }}_1}){\mathit{\beta }}_{{\mathit{\gamma }}_1}}{n}\to b_{A,1}>0$.
• (A2)For ⊈ , ${lim}_{n\to \infty }\frac{{\mathit{\beta }}_{{\mathit{\gamma }}_1}^{\prime }{\stackrel{\sim }{\mathbf{X}}}_{A,{\mathit{\gamma }}_1}^{\prime }{\stackrel{\sim }{\mathbf{H}}}_{A,2}{\stackrel{\sim }{\mathbf{X}}}_{A,{\mathit{\gamma }}_1}{\mathit{\beta }}_{{\mathit{\gamma }}_1}}{n}\to b_{A,2}$, where b_A,2 ∈ [0, b_A,1) for fixed p₁, p₂, and b_A,2 ∈ (0, b_A,1) for p_j = O(n^a_j) (j = 1, 2, 0 ≤ a₁ < a₂ < 1).

  (A1), (A2) depend on the allocation matrix A, which is a n × k binary matrix that for large n tends to have k ≪ n, and be very sparse containing mostly zeros with sparsity increasing with column index. We also assume the following for the class of proper priors π(g) on g:

• (A3)There exists a constant k ≥ 0 such that ${\int }_{a_n}^{c_0{a}_n}\pi (dg)\approx n^{-k}$ for any constant c₀ > 1 and any sequence a_n ≈ n. Here a_n ≈ b_n implies that lim_n→∞ a_n/b_n > 0.
• (A4)There exists a constant k_u such that k−(p₂−p₁)/2 < k_u ≤ k and ${\int }_0^{\infty }{(1+g)}^{k_u}\pi (dg)\approx 1$.

We state (A1′), (A2′) as the standard assumptions for establishing Bayes factor consistency in normal linear models, on which our assumptions (A1), (A2) are based. We develop asymptotic theory for semiparametric linear models (5) based on assumptions (A1)–(A4). We note that (A1) is stronger compared to (A1′), since (A1) implies (A1′) as ${\mathbf{\sum }}_A^{-1}={\mathbf{I}}_n-\mathbf{A}{({\mathbf{I}}_k+{\mathbf{A}}^{\prime }\mathbf{A})}^{-1}{\mathbf{A}}^{\prime }$. Further, in the extreme case when A = I_n, we have $\frac{{\mathbf{X}}_{{\mathit{\gamma }}_1}^{\prime }{\mathbf{\sum }}_A^{-1}{\mathbf{X}}_{{\mathit{\gamma }}_1}}{n}=\frac{1}{2}\frac{{\mathbf{X}}_{{\mathit{\gamma }}_1}^{\prime }{\mathbf{X}}_{{\mathit{\gamma }}_1}}{n}$, so that (A1′) implies (A1). Again when A =1_n, ${\mathbf{X}}_{{\mathit{\gamma }}_1}^{\prime }{\mathbf{\sum }}_A^{-1}{\mathbf{X}}_{{\mathit{\gamma }}_1}\approx {\mathbf{X}}_{{\mathit{\gamma }}_1}^{\prime }{\mathbf{X}}_{{\mathit{\gamma }}_1}-n{\overline{\mathbf{X}}}_{{\mathit{\gamma }}_1}^{\prime }{\overline{\mathbf{X}}}_{{\mathit{\gamma }}_1}$ for large n, for ${\overline{\mathbf{X}}}_{{\mathit{\gamma }}_1}={\mathbf{1}}_n^{\prime }{\mathbf{X}}_{{\mathit{\gamma }}_1}/n$. Hence $\frac{{\mathit{\beta }}_{{\mathit{\gamma }}_1}^{\prime }({\mathbf{X}}_{{\mathit{\gamma }}_1}^{\prime }{\mathbf{\sum }}_A^{-1}{\mathbf{X}}_{{\mathit{\gamma }}_1}^{\prime }){\mathit{\beta }}_{{\mathit{\gamma }}_1}}{n}\approx \frac{{\mathit{\beta }}_{{\mathit{\gamma }}_1}^{\prime }({\mathbf{X}}_{{\mathit{\gamma }}_1}^{c^{\prime }}{\mathbf{X}}_{{\mathit{\gamma }}_1}^c){\mathit{\beta }}_{{\mathit{\gamma }}_1}}{n}$, where ${\mathbf{X}}_{{\mathit{\gamma }}_1}^c$ is the centered design matrix. When ${lim}_{n\to \infty }\frac{{\mathit{\beta }}_{{\mathit{\gamma }}_1}^{\prime }({\mathbf{X}}_{{\mathit{\gamma }}_1}^{c^{\prime }}{\mathbf{X}}_{{\mathit{\gamma }}_1}^c){\mathit{\beta }}_{{\mathit{\gamma }}_1}}{n}>0$, (A1′) implies (A1).

Assumption (A2) can be interpreted as a positive ‘limiting distance’ between the two models corresponding to design matrices X_γ1 and X_γ2 in (3) conditional on A, after marginalizing out η, i.e. ${\mathrm{\Delta }}_{21,A}={lim}_{n\to \infty }\frac{{\mathit{\beta }}_{{\mathit{\gamma }}_1}^{\prime }{\stackrel{\sim }{\mathbf{X}}}_{A,{\mathit{\gamma }}_1}^{\prime }({\mathbf{I}}_n-{\stackrel{\sim }{\mathbf{H}}}_{A,2}){\stackrel{\sim }{\mathbf{X}}}_{A,{\mathit{\gamma }}_1}{\mathit{\beta }}_{{\mathit{\gamma }}_1}}{n{\tau }^{-1}}=\frac{b_{A,1}-b_{A,2}}{{\tau }^{-1}}\in (0,\infty )$. Such a ‘limiting distance’ (Δ_21,A) can be considered as a natural extension of the definition of distance between two normal linear models in Casella et. al. (2009) and Moreno et. al. (2010) to models with random intercept as in (3).

Assumptions (A3), (A4) define a class of proper priors for g described in Guo and Speckman (2009). This class includes $\text{hyper}-g(\frac{a-2}{2}{(1+g)}^{-a/2})$ and $\text{hyper}-g/n(\frac{a-2}{2n}{(1+g/n)}^{-a/2})$ priors with 2 < a ≤ 4 (Liang et. al. 2008), Zellner-Siow priors (Zellner and Siow, 1980) as well as beta-prime priors (Maruyama and George, 2008). It is clear that these assumptions on π(g) are satisfied by quite a few standard priors are hence are quite reasonable.

The following lemma gives the limits of quantities such as ${\stackrel{\sim }{R}}_{A,j}^2={\mathbf{Z}}_A^{\prime }{\stackrel{\sim }{\mathbf{H}}}_{A,j}{\mathbf{Z}}_A/{\mathbf{Z}}_A^{\prime }{\mathbf{Z}}_A$, which would be useful for establishing asymptotic properties. The proof follows directly using Lemmas 1, 2 of Guo and Speckman (2009) and from (6) which essentially states that under the DP mixture of Gaussians prior on f for in (5) and conditional on allocation matrix A, ${\mathbf{Z}}_A={\mathbf{\sum }}_A^{-1/2}{\mathbf{Y}}^n~N({\stackrel{\sim }{\mathbf{X}}}_{A,{\mathit{\gamma }}_j}{\mathit{\beta }}_{{\mathit{\gamma }}_j},{\tau }^{-1}{\mathbf{I}}_n)$, j = 1, 2.

Lemma 1

Let assumptions (A1), (A2) hold.

1. If ⊂ , conditional on A, ${\stackrel{\sim }{R}}_{A,1}^2\stackrel{a.s.}{\to }\frac{b_{A,1}}{{\tau }^{-1}+b_{A,1}},{\stackrel{\sim }{R}}_{A,2}^2\stackrel{a.s.}{\to }\frac{b_{A,1}}{{\tau }^{-1}+b_{A,1}}$, under

2. If ⊈ , conditional on A, ${\stackrel{\sim }{R}}_{A,1}^2\stackrel{a.s.}{\to }\frac{b_{A,1}}{{\tau }^{-1}+b_{A,1}},{\stackrel{\sim }{R}}_{A,2}^2\stackrel{a.s.}{\to }\frac{b_{A,2}}{{\tau }^{-1}+b_{A,1}}$, under

As shown by the following result, the proposed approach leads to Bayes factor consistency when comparing fixed dimensional models as well as models growing at the rate O(n^t), 0 < t < 1, when the truth is sparse.

Theorem I

Let assumptions (A1), (A2) hold.

1. Suppose p₁ and p₂ are fixed. If ⊂ , then under and assumptions (A3), (A4), $B{F}_{21}^n\stackrel{P}{\to }0$ as n→ ∞ and if p₂ − p₁ > 2 + 2(k − k_u), $B{F}_{21}^n\stackrel{a.s.}{\to }0$ as n → ∞. Further, if ⊈ , then under and assumption (A3), $B{F}_{21}^n\stackrel{a.s.}{\to }0$ as n → ∞.

2. Suppose p_j is growing at the rate O(n^a_j), j=1,2, with 0 ≤ a₁ < a₂ < 1. Then under and assumption (A3), $B{F}_{21}^n\stackrel{a.s.}{\to }0$ as n → ∞.

REMARK 1

Although we omit the proof here, Theorem I can be modified to accommodate the case of improper priors on g (i.e. $\pi (g)\propto \frac{1}{1+g}$). In such a case, assumptions (A3), (A4) are excluded and we require p₂ − p₁ ≥ 3 for a.s. convergence in (I) for ⊂ .

The next result establishes model selection consistency for the proposed approach, even in cases when the cardinality of the model space increases with n. In particular, we consider cases when the number of candidate predictors p_n is growing at the rate O(n^a), a > 0, but the prior on the model space assigns zero probability to models growing at a rate equal to or faster than n. When a ≥ 1, the prior support consists of models constructed using O(n^t) (0 ≤ t < 1) sized subsets of p_n = O(n^a) candidate predictors.

To elaborate, let the support of the prior on the model space be = ∪ , where is the set of all (non-null) models γ such that there exists a sample size n₀ < ∈ for which γ_j = 0 for all j > p_n0, and is the set of all models with dimensions growing at a rate strictly less than n, ${\mathcal{M}}_I=\{\mathit{\gamma }:{\sum }_{j=1}^{p_n}{\gamma }_j=O(n^t),0<t<1\}$. Letting p₀ = max{j: γ ∈ , γ_j = 1}, we can discard predictors having a higher index than p₀ for all γ ∈ and treat as finite dimensional having 2^p₀ − 1 elements (excluding the null model). Let γ_jl denote the lth model having dimension p_j. Consider the following sequence of priors which penalizes models with increasing dimensions, thus encouraging sparsity:

$${\pi }^n({\mathit{\gamma }}_{jl})\propto {\pi }^{p_j}{(1-\pi )}^{p_0-p_j}I[{\mathit{\gamma }}_{jl}\in {\mathcal{M}}_F]+{\left(\begin{array}{c}{p}_n\\ p_j\end{array}\right)}^{-1}I[{\mathit{\gamma }}_{jl}\in {\mathcal{M}}_I],\pi ~Be(a_1,b_1).$$ (12)

When the truth is sparse such that ∈ , we have the following result.

Theorem II

Suppose assumptions (A1)–(A4) hold. For fixed p and under $P({\mathcal{M}}_1\mid {\mathbf{Y}}^n)\stackrel{P}{\to }1$ for any prior on Γ with π( ) > 0. When p_n = O(n^a) (a > 0) and $P({\mathcal{M}}_1\mid {\mathbf{Y}}^n)\stackrel{P}{\to }1$ under , for πⁿ(γ_jl) defined as in (12).

4. SIMULATION STUDY

We present the results of two simulation studies comparing our method (SLM) with the normal linear model (NLM) having ${\mathit{\beta }}_{\mathit{\gamma }}~N(0,g{\tau }^{-1}{({\mathbf{X}}_{\mathit{\gamma }}^{\prime }{\mathbf{\sum }}_{A={\mathbf{1}}_n}^1{\mathbf{X}}_{\mathit{\gamma }})}^{-1})$ (designed to assign comparable prior information when the residual is Gaussian), the lasso (Tibshirani, 1996) and elastic net (Zou and Hastie, 2005), as well as robust variable selection methods including an MM-type regression estimator (Yohai, 1987; Koller and Stahel, 2011), and a median regression model with SSVS for variable selection (Yu et al., 2013). The data is generated as follows:

$$\begin{array}{ll}\mathbf{Case}\mathbf{I}:\hfill & y_i={\mathbf{x}}_i{\mathit{\beta }}_T+{\epsilon }_i,{\epsilon }_i~0.5N(2.5,1)+0.5N(-2.5,1),\hfill \\ \mathbf{Case}\mathbf{II}:\hfill & y_i=1+{\mathbf{x}}_i{\mathit{\beta }}_T+{\epsilon }_i,{\epsilon }_i~N(0,1),i-1,\dots ,n,\hfill \end{array}$$

where x_i is a ten dimensional predictor (p=10), with x_ij, j = 1, …, 10 generated independently from U(−1,1), and β_T = (3, 2, −1, 0, 1.5, 1, 0, −4, −1.5, 0).

We used Ga(0.1, 1) prior on the DP precision parameter and Be(0.1, 1) prior on P(γ_j = 1), j=1,…,p, which corresponds to a weakly informative prior favoring parsimony. We update g using the griddy Gibbs approach having 1000 equally spaced quantiles for $\frac{g}{1+g}~Be(1,1)$ corresponding to a = 4 in the hyper-g prior. For both SLM and NLM, we ran 50,000 iterations with a burn in of 5,000. We implemented the lasso (L1) and elastic net (EL) using the GLMNET package in R with default settings, while the MM-type estimator (LMR) was implemented using ‘lmrob’ function in ‘robustbase’ package in R and the median regression with SSVS (QR) was implemented using function ‘SSVSquantreg’ in ‘MCMCpack’ package in R, with a Be(0.1, 1) prior on the prior inclusion probability for predictors. All results are summarized across 20 replicates. The computation time for SLM per iteration was marginally slower than NLM. The mixing for the fixed effects was good under both the methods. The results for SLM do not appear to be sensitive to the hyper-parameters in π(m), but are mildly sensitive to hyper-parameters in π(g) for n = 100.

We study the marginal inclusion probabilities (MIP) under SLM and NLM over varying sample sizes in Figures 1 and 2. These plots suggest a faster rate of increase of the MIP for the important predictors under SLM as compared to NLM when the true residuals are non-Gaussian, and a very similar rate of increase under both methods when the true residuals are Gaussian (thus justifying the prior choice for NLM). In contrast, the exclusion probabilities for the unimportant predictors converge to one slowly under both the methods, reflecting the well known tendency for slower accumulation of evidence in favor of the true null.

Figure 1.

Marginal Inclusion Probabilities (MIP) over varying sample sizes: Truth generated from bimodal residual. Solid lines - Semi-parametric Linear Model, dashed lines - Normal Linear Model.

Figure 2.

Marginal Inclusion Probabilities (MIP) over varying sample sizes: Truth generated from Gaussian residual. Solid lines - Semi-parametric Linear Model, dashed lines - Normal Linear Model.

Tables 1 and 2 present some summaries for n = 100 for Case I. The MIPs in Table I suggests correct variable selection decision by SLM, but poor performance by NLM which fails to exclude any of the unimportant predictors under median probability model. Further, L1, EL and QR seem to favor an overly complex model by choosing a superset of important predictors. In terms of estimation of the fixed effects, SLM has the highest degree of accuracy as reflected by the smallest mean square error ( $\frac{{\Vert \widehat{\mathit{\beta }}-{\mathit{\beta }}_T\Vert }_2}{p}$) in Table 2, where β_T is the vector of true regression coefficients. In addition, the replicate average mean square error for out of sample prediction for a test sample size of 25 (Table 2) is smallest under the SLM, followed by lasso and elastic net. NLM is seen to be clearly inadequate for prediction purposes as indicated by the extremely high out of sample predictive MSE. Thus in conclusion, when the true residual is non-Gaussian, the SLM has the best performance compared to competitors, whereas NLM performs poorly in general.

Table 1.

Fixed effects estimates and marginal inclusion probabilities (MIP) for fixed effects for Case I when n=100.

βT	MIPSLM	βSLM	MIPNLM	βSLM	βL1	βEL	βLMR	βQR
3	1.00	2.88(2.34, 3.41)	1.00	2.83(1.86, 3.81)	3.08	3.08	3.15	2.92
2	0.99	1.89(1.34, 2.44)	0.98	1.95(0.96, 2.91)	2.06	2.06	2.11	1.84
−1	0.93	−0.91(−1.46, −0.36)	0.75	−0.78(−1.75,0.03)	−0.98	−0.98	−0.87	−0.78
0	0.45	−0.01(−0.44,0.44)	0.53	0.006(−0.82, 0.81)	0.01	0.009	−0.003	−0.02
1.5	0.98	1.43(0.89,1.98)	0.90	1.35(0.35, 2.35)	1.54	1.54	1.57	1.29
1	0.90	0.79(0.28, 1.35)	0.68	0.54(−0.26, 1.48)	0.74	0.74	0.66	0.42
0	0.43	−0.005(−0.44, 0.42)	0.53	−0.05(−0.85, 0.73)	−0.04	−0.04	−0.09	−0.06
−4	1.00	−3.89(−4.43, −3.33)	1.00	−3.75(−4.74, −2.74)	−4.05	−4.04	−4.14	−3.95
−1.5	0.99	−1.54(−2.08, −0.98)	0.92	−1.43(−2.41, −0.41)	−1.57	−1.57	−1.54	−1.30
0	0.42	0.008(−0.43, 0.43)	0.54	−0.12(−0.93, 0.64)	−0.12	−0.12	−0.06	−0.14

SLM: Semi-parametric linear model, NLM: Normal linear model, L1: Lasso, EL: Elastic Net, LMR: MM-type estimator, QR: Median regression with SSVS.

Table 2.

Summaries for Case I when n=100.

Measure	SLM	NLM	L1	EL	LMR	QR
MSE around βT	0.07	0.21	0.24	0.24	0.40	0.50
MSE for out of sample prediction	7.70	16.44	8.33	8.32	8.83	9.11

SLM: Semi-parametric linear model, NLM: Normal linear model, L1: Lasso, EL: Elastic Net, LMR: MM-type estimator, QR: Median regression with SSVS. MSE: mean square error and β_T is the vector of true regression coefficients.

5. APPLICATION TO DIABETES DATA

The prevalence of diabetes in the United States is expected to more than double to 48 million people by 2050 (Mokdad et. al., 2001). Previous medical studies have suggested that Diabetes Mellitus type II (DM II) or adult onset diabetes could be associated with high levels of total cholesterol (Brunham et. al., 2007) and obesity (often characterized by BMI and waist to hip ratio) (Schmidt et. al., 1992), as well as hypertension (indicated by a high systolic or diastolic blood pressure or both) which is twice as prevalent in diabetics compared to non-diabetic individuals (Epstein and Sowers, 1992).

We develop a comprehensive variable selection strategy for indicators of DM II in African-Americans based on data obtained from Department of Biostatistics, Vanderbilt University website. Our primary focus is to discover important indicators of DM II by modeling the continuous outcome glycosylated hemoglobin (> 7mg/dL indicates a positive diagnosis of diabetes) based on predictors such as total cholesterol (TC), stabilized glucose (SG), high density lipoprotein (HDL), age, gender, body mass index (BMI) indicator (overweight and obese with normal as baseline), systolic and diastolic blood pressure (SBP and DBP), waist to hip ratio (WHR) and postprandial time indicator (PPT) (1 if blood was drawn within 2 hours of a meal, 0 otherwise). We note that lower levels of HDL have been known to be associated with insulin resistance syndrome, often considered a precursor of DM II with a conversion rate around 30%. We also expect PPT to be a significant indicator as blood sugar levels are high up to 2 hours after a meal.

After excluding the records containing missing values, the data consisted of 365 subjects which was split into multiple training and test samples of sizes 330 and 35 respectively. The replicate averaged fixed effects estimates (multiplied by 100) for the SLM, NLM, L1, EL, LMR and QR are presented in Table 3, and the marginal inclusion probabilities (MIP) for the SLM, NLM and QR are summarized in Table 4. We also evaluate the out of sample predictive performance for each training-test split using predictive MSE in Table 5, and additionally provide the mean coverage (COV) and width (CIW) of 95% pointwise credible intervals for the predicted responses under SLM and NLM. The same values of hyper-parameters were used as in section 5. For each replicate, we randomized the initial starting points and made 100,000 runs for SLM (burn in = 20,000) and 50,000 runs for NLM (burn in = 5,000).

Table 3.

Fixed effects (times 100) for type-II diabetes example.

Predictor	β̂SLM	β̂NLM	β̂L1	β̂EL	β̂LMR	β̂QR
TC	0.55(0.11,0.73)	0.74(0.25,1.20)	0.75	0.75	0.29	0.01
SG	2.11(1.75,2.48)	2.82(2.5,3.15)	2.83	2.82	2.99	3.23
HDL	−0.50(−1.4,0.015)	−0.36(−1.61,0)	−1.02	−1.02	−0.42	0
Age	0.34(−0.06,1.3)	0.98(0,2.35)	1.19	1.19	0.57	0.04
Gender	−3.72(−30.12,4.39)	−1.53(−25.46,3.22)	−19.66	−19.81	−7.87	−0.86
BMI(overwt)	1.55(−9.43,24.03)	2.04(−3.33,29.53)	4.33	4.27	15.12	1.84
BMI(obese)	−0.74(−20.33,13.44)	−0.91(−21.93,6.14)	−14.88	−15.03	8.16	0.62
SBP	0.53(0,1.35)	0.03(−0.13,0.65)	0.25	0.25	0.56	0.009
DBP	−0.03(−0.99,0.69)	0(−0.45,0.45)	0.018	0.017	−0.55	0.002
WHR	224.27(67.72,381.88)	3.16(−44.74,91.4)	90.47	91.53	90.79	129.23
PPT	21.42(1.89,57.49)	33.04(0,80.39)	47.31	47.32	37.55	18.99

Table 4.

Marginal Inclusion Probabilities for SLM, NLM, QR in type-II diabetes data.

Predictor	TC	SG	HDL	Age	Gender	BMI(overwt)	BMI(obese)	SBP	DBP	WHR	PPT
MIPSLM	0.97	1.00	0.64	0.43	0.17	0.15	0.22	0.72	0.23	0.93	0.64
MIPNLM	0.98	1.00	0.39	0.67	0.12	0.13	0.11	0.14	0.10	0.13	0.68
MIPQR	0.02	1.00	0.002	0.03	0.08	0.10	0.08	0.01	0.004	0.71	0.42

Table 5.

Out of Sample Prediction.

Replicate	S 1	S 2	S 3	S 4	S 5	S 6	S 7	S 8
MSESLM	1.25	1.24	1.55	1.21	1.45	1.47	3.44	1.23
MSENLM	1.23	1.33	1.74	1.29	1.14	1.46	3.43	1.52
MSEL1	1.28	1.45	2.49	2.34	1.13	1.45	3.47	1.75
MSEEL	1.29	1.47	2.51	2.36	1.14	1.45	3.48	1.75
MSELMR	2.23	1.21	2.15	1.02	1.09	1.36	4.06	1.69
MSEQR	1.82	1.91	2.64	1.15	1.64	2.68	3.98	2.44
CovSLM	100.00	97.14	100.00	97.14	100.00	100.00	91.42	100.00
CovNLM	97.12	97.14	94.28	97.14	100.00	97.14	91.42	100.00
CIWNLM	5.92	5.41	5.84	5.94	5.93	5.91	5.59	5.90
CIWSLM	6.93	6.16	6.80	6.81	6.84	6.86	6.13	6.77

MSE: out of sample predictive mean square error, Cov: 95% credible interval coverage, CIW: 95% credible interval width.

It is interesting to note from Table 4 that the variable selection decisions under SLM (using median probability model) are quite different compared to the NLM. In particular, while both the models successfully identify total cholesterol, stabilized glucose and postprandial time as important predictors, it is only the SLM which identifies systolic blood pressure (MIP = 0.72), HDL (MIP = 0.64) and waist to hip ratio (MIP = 0.93) as important indicators, compared to NLM which assigns MIP = 0.14, 0.39 and 0.13 to these three predictors respectively. Age is identified as an important predictor under NLM (MIP = 0.67), but not under SLM (MIP=0.43). For both the methods, the MIPs for BMI (overweight and obese) were low, which could potentially be attributed to adjusting for the other obesity factors such as waist to hip ratio. From Tables 3 and 4, we also see that the lasso, elastic net and the MM-type estimator select an overly complex model by excluding minimal number of predictors, while the quantile regression with SSVS fails to include several important predictors and selects a highly parsimonious and inadequate model.

Variable selection in this application is clearly influenced by the assumptions on the residual density, with the nonparametric residual density providing a more realistic characterization that should lead to a more accurate selection of the important predictors. Figure 3 shows an estimate of the residual density obtained from the SLM analysis, suggesting a uni-modal right skewed density with a heavy right tail. The SLM results suggest that a mixture of two Gaussians provides an adequate characterization of this density. The computation time for SLM is only marginally slower than NLM, and in addition SLM exhibits good mixing for most of the fixed effects (Table 6). These results are robust to SSVS starting points, and consistency in the results across training-test splits also indirectly suggests adequate computational efficiency of SSVS.

Figure 3.

Residual density for Type II Diabetes study under Semi-parametric Linear Model.

Table 6.

Auto-correlations across lags for fixed effects in type-II diabetes data.

Predictor	Lag 1		Lag 5		Lag 10		Lag 25		Lag 50
	SLM	NLM	SLM	NLM	SLM	NLM	SLM	NLM	SLM	NLM
TC	0.22	0.18	0.113	0.194	0.073	0.159	0.032	0.111	0.013	0.059
SG	0.59	0.06	0.386	0.038	0.285	0.022	0.14	0.009	0.06	0.016
HDL	0.19	0.02	0.081	0.012	0.041	0.013	0.01	0.021	0.0005	−0.006
Age	0.21	0.04	0.072	0.009	0.053	−0.0001	0.025	0.006	0.007	−0.014
Gender	0.06	−0.007	0.030	0.0003	0.013	−0.006	0.009	−0.014	0.005	0.019
BMI(overwt)	0.02	−0.002	0.01	−0.006	0.006	0.013	−0.006	0.009	0.0014	0.018
BMI(obese)	0.02	0.002	0.017	0.004	0.004	0.018	0.007	−0.003	0.000	0.000
SBP	0.29	0.0711	0.137	0.019	0.096	0.007	0.047	0.03	0.014	0.022
DBP	0.07	0.0239	0.021	0.019	0.019	0.031	0.009	−0.003	0.004	−0.012
WHR	0.44	0.0642	0.353	0.043	0.321	0.061	0.251	0.06	0.186	−0.003
PPT	0.22	0.0600	0.118	0.047	0.068	0.045	0.015	0.004	−0.002	0.019

In terms of out of sample predictive MSE (Table 5), the relative performance between SLM, NLM, L1 and EL vary across training-test splits so that none of the models can be said to dominate the others, while LMR and QR produce relatively inferior prediction results. Overall, the NLM has narrower 95% pointwise credible intervals compared to SLM, often resulting in poorer coverage for out of sample predictions. In conclusion, SLM succeeds in choosing the most reasonable model for DM II, consistent with previous medical evidence, and compares favorably with other competitors for prediction purposes.

APPENDIX A: PROOF OF RESULTS

Proof of Theorem I

Using similar methods as in the proof of Theorem 2 in Guo and Speckman (2009), it can be shown that conditional on A and assumptions (A3) and (A4), the upper and lower bounds of $L({\mathbf{Y}}^n\mid \mathbf{A},M_1)={\int }_0^{\infty }{(1+g)}^{-p_1/2}{\left[1-\frac{g}{1+g}{\stackrel{\sim }{R}}_{A,1}^2\right]}^{-n/2}\pi (dg)$ are

$$\begin{array}{l}L({\mathbf{Y}}^n\mid \mathbf{A},M_1)\le {\left(\frac{p_1+2k_u}{n-p_1-2k_u}\right)}^{p_1/2+k_u}{\left(\frac{1-{\stackrel{\sim }{R}}_{A,1}^2}{{\stackrel{\sim }{R}}_{A,1}^2}\right)}^{p_1/2+k_u}{\left(\frac{n}{n-p_1-2k_u}\right)}^{-n/2}{\left(1-{\stackrel{\sim }{R}}_{A,1}^2\right)}^{-n/2}\\ \approx {\left(\frac{p_1+2k_u}{n-p_1-2k_u}\right)}^{p_1/2+k_u}{\left(\frac{1-{\stackrel{\sim }{R}}_{A,1}^2}{{\stackrel{\sim }{R}}_{A,1}^2}\right)}^{p_1/2+k_u}{\left(1-{\stackrel{\sim }{R}}_{A,1}^2\right)}^{-n/2}=U_{A,1}(n),\end{array}$$

and $L({\mathbf{Y}}^n\mid \mathbf{A},M_1)\ge n^{-p_1/2-k}{\left(1-{\stackrel{\sim }{R}}_{A,1}^2\right)}^{-n/2}=L_{\mathbf{A},1}(n)$. Similarly, $L_{\mathbf{A},2}(n)\le L({\mathbf{Y}}^n\mid \mathbf{A},M_2)={\int }_0^{\infty }{(1+g)}^{-p_2/2}{\left[1-\frac{g}{1+g}{\stackrel{\sim }{R}}_{A,2}^2\right]}^{-n/2}\pi (dg)\le U_{A,2}(n)$. Therefore, ${\text{BF}}_{21,A}^n\le \frac{U_{A,2}(n)}{L_{\mathbf{A},1}(n)}$

$$={\left(\frac{p_2+2k_u}{n-p_2-2k_u}\right)}^{p_2/2+k_u}{\left(\frac{1-{\stackrel{\sim }{R}}_{A,2}^2}{{\stackrel{\sim }{R}}_{A,2}^2}\right)}^{p_2/2+k_u}{\left(1-{\stackrel{\sim }{R}}_{A,2}^2\right)}^{-n/2}/\left(n^{-p_1/2-k}{(1-{\stackrel{\sim }{R}}_{A,1}^2)}^{-n/2}\right).$$ (13)

Case (I): For fixed p_j (j = 1, 2) and large n, ${\text{BF}}_{21,A}^n\le \zeta (\mathbf{A},n)=n^{\frac{p_1-p_2}{2}+k-k_u}{\left(\frac{1-{\stackrel{\sim }{R}}_{A,2}^2}{1-{\stackrel{\sim }{R}}_{A,1}^2}\right)}^{-n/2}$, ignoring terms independent of n. Using the results in proof of Theorems 2, 3 in Guo and Speckman (2009), we can show that $n\zeta (\mathbf{A},n)\stackrel{a.s.}{\to }0$ when ⊈ . Again for ⊂ , using results in the aforementioned proofs, we have ${\left(\frac{1-{\stackrel{\sim }{R}}_{A,2}^2}{1-{\stackrel{\sim }{R}}_{A,1}^2}\right)}^{-n/2}=O_P(1)$. Further for ⊂ when p₂ − p₁ > 2 + 2(k − k_u), we have $n^{\delta }\zeta (\mathbf{A},n)\stackrel{a.s.}{\to }0$, where δ > 0 is such that i ≤ 2(k − k_u) + 2δ < i + 1 when i ≤ 2(k − k_u) < i + 1. This implies that for large enough n,

$$\begin{array}{l}\text{for}{\mathcal{M}}_1\subset {\mathcal{M}}_2:\zeta (\mathbf{A},n)=n^{-\frac{p_2-p_1}{2}+(k-k_u)}\text{in}\text{probability},\\ \text{for}{\mathcal{M}}_1\subset {\mathcal{M}}_2:\zeta (\mathbf{A},n)\le n^{-\delta }\text{almost}\text{surely}\text{when}{p}_2-p_1>2+2(k-k_u),\\ \text{for}{\mathcal{M}}_1⊈{\mathcal{M}}_2:\zeta (\mathbf{A},n)\le n^{-1}\text{almost}\text{surely}.\end{array}$$ (14)

Then for large enough n, we have,

$$\begin{array}{l}{\text{BF}}_{21,A}^n\le \zeta (\mathbf{A},n)\iff L({\mathbf{Y}}^n\mid \mathbf{A},{\mathcal{M}}_2)\le \zeta (\mathbf{A},n)L({\mathbf{Y}}^n\mid \mathbf{A},{\mathcal{M}}_1)\\ \Rightarrow L({\mathbf{Y}}^n\mid {\mathcal{M}}_2)\le \sum _{{\mathbf{A}}_l\in {\mathcal{C}}_n}{w}_l\zeta ({\mathbf{A}}_l,n)L({\mathbf{Y}}^n\mid {\mathbf{A}}_l,{\mathcal{M}}_1)\le {\zeta }^{\ast }(n)L({\mathbf{Y}}^n\mid {\mathcal{M}}_1),\end{array}$$ (15)

where ζ^*(n) is the LHS in equations (14) which is independent of A, and ζ^*(n) → 0 as n → ∞ (using (A4)). Dividing both sides of (15) by L(Yⁿ| ), we have ${\text{BF}}_{21}^n\le {\zeta }^{\ast }(n)\to 0$ as n → ∞.

Case (II): For increasing dimensions p₁ = O(n^a₁), p₂ = O(n^a₂) with 0 ≤ a₁ < a₂ < 1, we will only assume (A3) for g ~ π(g) so that k_u = 0. We have using (13)

$$B{F}_{21,A}^n\le \zeta (\mathbf{A},n)n^{p_1/2-(1-a_2)p_2/2+k}{\left(\frac{1-{\stackrel{\sim }{R}}_{A,2}^2}{{\stackrel{\sim }{R}}_{A,2}^2}\right)}^{p_2/2}{\left(\frac{1-{\stackrel{\sim }{R}}_{A,2}^2}{1-{\stackrel{\sim }{R}}_{A,1}^2}\right)}^{-n/2}.$$ (16)

Let us consider the following cases under 0 ≤ a₁ < a₂ < 1.

Case C1: ⊂ . We have $Q_j=\tau ({\mathbf{Z}}_A^{\prime }{\mathbf{Z}}_A-{\mathbf{Z}}_A^{\prime }{\stackrel{\sim }{\mathbf{H}}}_{A,j}{\mathbf{Z}}_A)~{\chi }_{n-p_j}^2(0)$, j=1,2, and $Q_1-Q_2=\tau \left({\mathbf{Z}}_A^{\prime }({\stackrel{\sim }{\mathbf{H}}}_{A,2}-{\stackrel{\sim }{\mathbf{H}}}_{A,1}){\mathbf{Z}}_A\right)~{\chi }_{p_2-p_1}^2(0)$. Using Lemma 1 of Guo et. al. (2009),

$$\frac{1-{\stackrel{\sim }{R}}_{A,1}^2}{1-{\stackrel{\sim }{R}}_{A,2}^2}=\frac{{\mathbf{Z}}_A^{\prime }{\mathbf{Z}}_A-{\mathbf{Z}}_A^{\prime }{\stackrel{\sim }{\mathbf{H}}}_{A,1}{\mathbf{Z}}_A}{{\mathbf{Z}}_A^{\prime }{\mathbf{Z}}_A-{\mathbf{Z}}_A^{\prime }{\stackrel{\sim }{\mathbf{H}}}_{A,2}{\mathbf{Z}}_A}=\frac{Q_1}{Q_2}=1+\frac{(Q_1-Q_2)/(p_2-p_1)}{Q_2/(n-p_2)}\frac{p_2-p_1}{n-p_2}\stackrel{a.s.}{\to }1.$$

Moreover $\left(\frac{1-{\stackrel{\sim }{R}}_{A,2}^2}{{\stackrel{\sim }{R}}_{A,2}^2}\right)\stackrel{a.s.}{\to }\left(\frac{{\tau }^{-1}}{b_{A,1}}\right)$ under . Then for large n,

$$\zeta (\mathbf{A},n)\approx n^{p_1/2-(1-a_2)p_2/2+k}{\left(\frac{{\tau }^{-1}}{b_{A,1}}\right)}^{p_2/2}=n^{p_1/2-(1-a^{\ast }-a_2)p_2/2+k}{\left(\frac{{\tau }^{-1}}{n^{a^{\ast }}{b}_{A,1}}\right)}^{p_2/2}\mathrm{a}.\mathrm{s}.,$$

where a^* > 0 is such that 0 < 1 − a^* − a₂ < 1. This implies that $n^{K^{\ast }}\zeta (\mathbf{A},n)\stackrel{a.s.}{\to }0$ under for any constant K^* > 0. Case C2: ⊈ . Using Lemma 1,

$$\frac{1-{\stackrel{\sim }{R}}_{A,2}^2}{{\stackrel{\sim }{R}}_{A,2}^2}\stackrel{a.s.}{\to }\frac{{\tau }^{-1}+b_{A,1}-b_{A,2}}{b_{A,2}}>1,\frac{1-{\stackrel{\sim }{R}}_{A,1}^2}{1-{\stackrel{\sim }{R}}_{A,2}^2}\stackrel{a.s.}{\to }\frac{{\tau }^{-1}}{{\tau }^{-1}+b_{A,1}-b_{A,2}}<1,\text{under}{\mathcal{M}}_1.$$

For fixed τ⁻¹ and b_A,2 > 0 (under (A2)), ${\left(\frac{1-{\stackrel{\sim }{R}}_{A,2}^2}{{\stackrel{\sim }{R}}_{A,2}^2}\right)}^{p_2/2}{\left(\frac{1-{\stackrel{\sim }{R}}_{A,2}^2}{1-{\stackrel{\sim }{R}}_{A,1}^2}\right)}^{-n/2}\stackrel{a.s.}{\to }0$. This implies that in the limiting case when n → ∞, we have

$$\begin{array}{l}\text{for}{\mathcal{M}}_1\subset {\mathcal{M}}_2:\zeta (\mathbf{A},n)\le n^{-K^{\ast }}\text{almost}\text{surely},\\ \text{for}{\mathcal{M}}_1\not\subset {\mathcal{M}}_2:\zeta (\mathbf{A},n)\le n^{p_1/2-(1-a_2)p_2/2+k}\le n^{-K^{\ast }}\text{almost}\text{surely},\end{array}$$ (17)

where K^* > 0 is a constant. Denoting the upper bounds as ζ^*(n), it is clear that ζ^*(n) is independent of A and ζ^*(n) → 0 as n → ∞ when 0 ≤ a₁ < a₂ < 1. Using similar arguments as in equation (15) of Case (I), we have ${\text{BF}}_{21}^n\le n^{-K^{\ast }}$ and consistency follows.

Proof of Theorem II

Given the assumptions (A1)–(A4), Bayes factor consistency holds under the different cases elaborated in Theorem I. For fixed p, the proof follows trivially using Bayes factor consistency. For increasing p_n = O(n^a) (a > 0), our prior is ${\pi }^n({\mathit{\gamma }}_{jl})\propto {\pi }^{p_j}{(1-\pi )}^{p_0-p_j}I[{\mathit{\gamma }}_{j_l}\in {\mathcal{M}}_F]+N_j^{-1}I[{\mathit{\gamma }}_{j_l}\in {\mathcal{M}}_I]$, where π ~ Be(a₁, b₁) and $N_j=\left(\begin{array}{l}{p}_n\hfill \\ p_j\hfill \end{array}\right)$. Let W_γ denote the prior weight for γ ∈ after marginalizing out π under the Be(a₁, b₁) prior (W₁ being the weight for ). Let ${\text{BF}}_{\mathit{\gamma }1}^n=$ Bayes factor between models γ and , let D = {p_γ : γ ∈ } and denote = {γ ∈ : dim(γ) = p_j}. Note that under (A1)–(A4) and ∈ , ${\text{BF}}_{\mathit{\gamma }1}^n\stackrel{P}{\to }0$ for all $\mathit{\gamma }\in \left({\mathcal{M}}_F\cap {\mathcal{M}}_1^c\right)\cup {\mathcal{M}}_I$, using Theorem I. Also,

$$\begin{array}{l}P({\mathcal{M}}_1\mid {\mathbf{Y}}^n)={[1+W_1^{-1}\sum _{\mathit{\gamma }\in {\mathcal{M}}_F\cap {\mathcal{M}}_1^c}{W}_{\gamma }B{F}_{\mathit{\gamma }1}^n+W_1^{-1}\sum _{\mathit{\gamma }\in {\mathcal{M}}_I}{N}_j^{-1}B{F}_{\mathit{\gamma }1}^n]}^{-1}\\ ={[1+W_1^{-1}\sum _{\mathit{\gamma }\in {\mathcal{M}}_F\cap {\mathcal{M}}_1^c}{W}_{\gamma }B{F}_{\mathit{\gamma }1}^n+W_1^{-1}\sum _{p_j\in D}\sum _{{\mathit{\gamma }}_{jl}\in {\mathcal{H}}_j}{N}_j^{-1}B{F}_{{\mathit{\gamma }}_{jl}1}^n]}^{-1}\\ \ge {[1+{\epsilon }_0+W_1^{-1}\sum _{p_j\in D}\sum _{{\mathit{\gamma }}_{jl}\in {\mathcal{H}}_j}{N}_j^{-1}B{F}_{{\mathit{\gamma }}_{jl}1}^n]}^{-1},\end{array}$$

where $W_1^{-1}{\sum }_{\mathit{\gamma }\in {\mathcal{M}}_F\cap {\mathcal{M}}_1^c}{W}_{\gamma }B{F}_{\mathit{\gamma }1}^n\le {\epsilon }_0$ for large enough n, and ε₀ → 0 as n → ∞ since all the individual terms in the finite summation → 0 using Theorem I. Further using (17), the upper bound of $B{F}_{{\mathit{\gamma }}_{jl}1}^n$ for any γ_jl ∈ is given by ζ^*(n) = n^−K* when n is large, where K^* > 0 is a constant. Noting that the cardinality of ${\mathcal{H}}_j\le N_j=\left(\begin{array}{l}{p}_n\hfill \\ p_j\hfill \end{array}\right)$, we have for large n,

$$\sum _{{\gamma }_{jl}\in {\mathcal{H}}_j}{N}_j^{-1}B{F}_{{\mathit{\gamma }}_{jl}1}^n\le n^{-K^{\ast }}\Rightarrow P({\mathcal{M}}_1\mid {\mathbf{Y}}^n)\ge {[1+{\epsilon }_0+W_1^{-1}\sum _{p_j\in D}{n}^{-K^{\ast }}]}^{-1},\text{under}{\mathcal{M}}_1.$$

Now note that W₁ is fixed and the cardinality of D < κ₀n for some constant κ₀ > 0. Thus it is clear that $W_1^{-1}{\sum }_{p_j\in D}{n}^{-K^{\ast }}\to 0$ as n → ∞ for large K^*. The rest is straightforward.

APPENDIX B: COMPUTATIONAL STEPS FOR MCMC

The posterior computation steps are:

• Step 1.1

  Update the ν’s after marginalizing out the augmented uniform variable using π(ν_h|−) = Be(1 + n_h, Σ_j>h n_j + m), h=1, …, M, where M is the total number of clusters satisfying ${\sum }_{h=1}^M{w}_h>1-min(u_1,\dots ,u_n)$, with w_h = ν_h Π_l<h(1 − ν_l).

• Step 1.2

  Update u_i, i=1, …, n, from its full conditional as described in Walker (2007).

• Step 2Update the cluster membership of different subjects using f(y_i|u_i, A_ih = 1) ∝ N(y_i|η_h, x_γ,i, β_γ, τ⁻¹)I(h ∈ B_w(u_i)), h=1, …, M, with B_w(u_i) defined as in section 2.3.
• Step 3Update the Dirichlet process atom η_l for the l-th cluster using $\pi ({\eta }_l\mid -)=N\left(\frac{{\sum }_{i:A_{il}=1}(y_i={\mathbf{x}}_{\mathit{\gamma },i}{\mathit{\beta }}_{\mathit{\gamma }})}{1+n_l},\sqrt{\frac{1}{\tau (1+n_l)}}\right)$, where $n_l={\sum }_{i=1}^n{\mathbf{A}}_{il}$ is the cardinality of the l-th cluster, l=1, …, M.
• Step 4Update the DP precision using $\pi (m\mid -)=Ga(a_m+M,b_m-{\sum }_{l=1}^{M}log(1-{\nu }_l))$.
• Step 5Letting $p_{\gamma }={\sum }_{j=1}^p{\gamma }_j$, update precision τ using $\pi (\tau \mid -)=Ga\left(a_{\tau }+\frac{n+p_{\mathit{\gamma }}}{2},b_{\tau }+\frac{1}{2}\left\{{({\mathbf{Y}}^n-{\mathbf{X}}_{\mathit{\gamma }}{\mathit{\beta }}_{\mathit{\gamma }})}^{\prime }{\mathbf{\sum }}_A^{-1}({\mathbf{Y}}^n-{\mathbf{X}}_{\mathit{\gamma }}{\mathit{\beta }}_{\mathit{\gamma }})+\frac{1}{g}{\mathit{\beta }}_{\mathit{\gamma }}^{\prime }({\mathbf{X}}_{\mathit{\gamma }}^{\prime }{\mathbf{\sum }}_A^{-1}{\mathbf{X}}_{\mathit{\gamma }}){\mathit{\beta }}_{\mathit{\gamma }}\right\}\right)$.
• Step 6Using the hyper-g prior and the fact that $\frac{g}{1+g}~Be(1,1)$ for a = 4, we can adopt the griddy Gibbs approach (Ritter and Tanner, 1992) to update g.
• Step 7Update the prior inclusion probability π = Pr(γ_j = 1) using f(π|−) = Be(a₁ + p_γ, b₁ + p − p_γ).
• Step 8
  Update γ_j’s one at a time by computing their posterior inclusion probabilities after marginalizing out β_γ and conditional on inclusion indicators for the remaining predictors as well as g, τ and A. Denote γ(j) and γ(−j) as the vector of current variable inclusion indicators with γ_j fixed at 1 and 0 respectively, and let p_γ(j) and p_γ(−j) denote the corresponding vector sums. We can sample γ_j from the Bernoulli conditional posterior distribution with probabilities Pr(γ_j = 1|−) = p_j1/(p_j1 + p_j0) and Pr(γ_j = 0|−) = p_j0/(p_j1 + p_j0), where

  $$\begin{array}{l}{p}_{j1}=\pi {(1+g)}^{-p_{\mathit{\gamma }(j)}/2}exp\left\{\frac{\tau }{2}\left(\frac{g}{1+g}\right)\left({({\mathbf{Y}}^n)}^T{\mathbf{\sum }}_A^{-1}{\mathbf{X}}_{\mathit{\gamma }(j)}{({\mathbf{X}}_{\mathit{\gamma }(j)}^{\prime }{\mathbf{\sum }}_A^{-1}{\mathbf{X}}_{\mathit{\gamma }(j)})}^{-1}{\mathbf{X}}_{\mathit{\gamma }(j)}^{\prime }{\mathbf{\sum }}_A^{-1}{\mathbf{Y}}^n\right)\right\},\\ p_{j0}=(1-\pi ){(1+g)}^{-p_{\mathit{\gamma }(-j)}/2}exp\left\{\frac{\tau }{2}\left(\frac{g}{1+g}\right)\left({({\mathbf{Y}}^n)}^T{\mathbf{\sum }}_A^{-1}{\mathbf{X}}_{\mathit{\gamma }(-j)}{({\mathbf{X}}_{\mathit{\gamma }(-j)}^{\prime }{\mathbf{\sum }}_A^{-1}{\mathbf{X}}_{\mathit{\gamma }(-j)})}^{-1}{\mathbf{X}}_{\mathit{\gamma }(-j)}^{\prime }{\mathbf{\sum }}_A^{-1}{\mathbf{Y}}^n\right)\right\}.\end{array}$$
• Step 9Set {β_j : γ_j = 0} = 0 and update β_γ = {β_j : γ_j = 1} using π(β_γ|−) = N(β_γ; E, V), where $\mathbf{V}={\left(\frac{\tau }{g}({\mathbf{X}}_{\mathit{\gamma }}^{\prime }{\mathbf{\sum }}_A^{-1}{\mathbf{X}}_{\mathit{\gamma }})+\tau ({\mathbf{X}}_{\mathit{\gamma }}^{\prime }{\mathbf{X}}_{\mathit{\gamma }})\right)}^{-1}$ and $\mathbf{E}=\mathbf{V}\left(\tau {\mathbf{X}}_{\mathit{\gamma }}^{\prime }({\mathbf{Y}}^n-\mathbf{A}\mathit{\eta })\right)$.