Bayesian Criterion Based Variable Selection

Abstract

Bayesian approaches for criterion based selection include the marginal likelihood based highest posterior model (HPM) and the deviance information criterion (DIC). The DIC is popular in practice as it can often be estimated from sampling based methods with relative ease and DIC is readily available in various Bayesian software. We find that sensitivity of DIC based selection can be high, in the range of 90 – 100%. However, correct selection by DIC can be in the range of 0 – 2%. These performances persist consistently with increase in sample size. We establish that both marginal likelihood and DIC asymptotically disfavor under-fitted models, explaining the high sensitivities of both criteria. However, mis-selection probability of DIC remains bounded below by a positive constant in linear models with $g$-priors whereas mis-selection probability by marginal likelihood converges to 0 under certain conditions. A consequence of our results is that not only the DIC cannot asymptotically differentiate between the data-generating and an over-fitted model, but, in fact, it cannot asymptotically differentiate between two over-fitted models as well. We illustrate these results in multiple simulation studies and in a biomarker selection problem on cancer cachexia of non-small cell lung cancer patients. We further study performances of HPM and DIC in generalized linear model as practitioners often choose to use DIC that is readily available in software in such non-conjugate settings.

1. Introduction

Variable selection and the broader problem of model selection remain among the most theoretically and computationally challenging problems, and at the same time, some of the most frequent questions encountered in practice. Our focus in this article is on criterion based variable selection, or what is known as “wrapper” method in the feature selection literature (Saeys et al., 2007), which assesses subsets of predictor variables based on their usefulness according to a criterion. Variable selection is then done by optimizing this criterion either by complete enumeration or by deterministic or stochastic search over the model space. These methods thus differ from “embedded” methods such as lasso, which performs variable selection in the process of fitting the model.

In 2011, a panel of international experts participated in a formal consensus process to develop a framework for definition and classification of cancer cachexia (Fearon et al., 2011). This panel defined cancer cachexia as a “multifactorial syndrome defined by an ongoing loss of skeletal muscle mass (with or without loss of fat mass) that cannot be fully reversed by conventional nutritional support and leads to progressive functional impairment”. A series of recent studies have examined cancer cachexia and its prognostic value in cancer outcome (Bonomi et al., 2017; Martin et al., 2014; Gielda et al., 2011). Patel et al. (2016) explored the relation between cancer efficacy outcome and cancer cachexia in advanced non-small cell lung cancer (NSCLC) patients and concluded that “weight gain during treatment may be an early indicator of clinical benefit” and “monitoring weight change may provide important information regarding survival outcomes in NSCLC”.

Neutrophil and lymphocyte are the two most abundant types of white blood cells. The neutrophil-to-lymphocyte ratio (NLR) is increasingly being used as a marker of subclinical inflammation. Multiple studies have explored the association of increased baseline NLR with poor clinical outcomes for several types of cancers, In Derman, Macklis, Azeem, Sayidine, Basu, Batus, Esmail, Borgia, Bonomi, & Fidler (2017), we established associations among neutrophil-to-lymphocyte ratio, cancer cachexia and survival from NSCLC. We were involved in a follow-up cancer cachexia study which included advanced/metastatic, stage IIIB or IV, NSCLC patients treated first-line with platinum doublet chemotherapy. More than one-third of the patients gained weight on treatment from baseline to 6 weeks. Additionally, the study considered targeted biomarkers for NSCLC and measured levels of 33 such markers by Luminex^® Multiplex Assays at baseline. One objective of the study was to recommend a selected panel of biomarkers that can prognosticate decreased levels of neutrophil-to-lymphocyte ratio (NLR).

In the framework of criterion based selection, a leading Bayesian selection criterion is the marginal likelihood. Bayesian model comparison has traditionally been performed using the Bayes factor (Kass & Raftery, 1995), calculated as the ratio of marginal likelihoods of two comparator models. The Bayes factor is associated with Schwarz’s Bayesian information criterion BIC, as difference in BIC provides an approximation to the log (Bayes factor) under appropriate prior assumptions (Kass & Raftery, 1995). The estimation of the marginal likelihood from Markov chain sampling approaches is studied in Kass & Raftery (1995), Chib (1995), Chib & Jeliazkov (2001, 2005); Chan & Jeliazkov (2009) and many others. These proposed approaches require additional computational work that can be complex, intensive and are often not readily available in software packages.

A widely popular Bayesian selection criterion is the Deviance Information Criterion or DIC (Spiegelhalter et al., 2002, 2014). The DIC can be easily monitored in various implementations of the BUGS software. For models with latent structures such as hierarchical or random effects, Ariyo et al. (2019, 2020); Merkle et al. (2019); Quintero & Lesaffre (2018) recommend use of marginalized DIC over (conditional) DIC. In spite of its wide acceptance, criticisms of DIC include that it may not be invariant to reparametrization, it uses a plug-in predictive approach rather than a proper Bayesian predictive distribution and it does not focus on identifying the “true” model.

The penalty in DIC is analogous to the 2× (number of parameters) penalty in AIC, which has been shown to be asymptotically equivalent to leave-one-out cross-validation. Another Bayesian selection criterion based on one-deleted cross validation is the conditional predictive ordinate (CPO), originally introduced in Geisser (1980), whose implementation in sampling based approaches is discussed in Gelfand et al. (1992). The CPOs are combined into the Log Pseudo Marginal Likelihood (LPML) criterion, originally proposed in Geisser & Eddy (1979). One well known drawback of cross-validation methods, and hence LPML, is that their computation burden increases with increase in sample size. The other worrisome fact is the asymptotic inconsistency of one-deleted cross-validation in differentiating over-fitted models which has been studied in non-Bayesian literature (see Shao 1993 and references therein). Other Bayesian predictive criteria are reviewed in Gelman et al. (2014); Vehtari et al. (2017) and cross-validation equivalence of marginal likelihood is discussed in Fong & Holmes (2020).

Chen, Huang, Ibrahim, & Kim (2008) provide review, comparison and computational expressions of Bayes factor, DIC, LPML and other criteria in the setting of generalized linear models. Earlier works on Bayesian criterion based methods include Ibrahim, Chen, & Sinha (2001) and Chen, Dey, & Ibrahim (2004). Review of marginal likelihood, its properties and its various applications can be found in Kass & Raftery (1995) and in a series of works of Chib (1995), Chib & Jeliazkov (2001, 2005). Criteria based on posterior predictive approaches are considered in Laud & Ibrahim (1995), Meyer & Laud (2002) Ibrahim, Chen, & Sinha (2001), and Daniels, Chatterjee, & Wang (2012)

In the cancer cachexis study of advanced stage metastatic NSCLC patients and the question of selecting a panel of biomarkers that can prognosticate decreased levels of neutrophil-to-lymphocyte ratio (NLR) the individual markers were found to have varying levels of marginal correlation with NLR, the highest absolute correlation being 0.27. The biomarkers also had varying degrees of collinearity with the largest pairwise absolute correlation among two markers being 0.62. The selection of a prognosticating biomarker panel in this study is one of the motivating applications that initiated the research work in this article. We find that Bayesian variable selections by horseshoe prior (Carvalho et al., 2010), and nonlocal prior (Shin et al., 2018) select two different models (with 2 and 3 biomarkers respectively) which are completely distinct (have no common markers between them). In contrast, the non-Bayesian sure independent screening (Fan & Lv, 2008) approach selects a model with 20 biomarkers. The differences among these selections as well as selections from marginal likelihood and DIC based approaches in this scientific problem are discussed in detail in section 4.2.

The rest of the article is organized as follows. We begin with an illustrative example in section 2. In section 3, we construct the framework of under and over-fitted models for studying the properties of the Bayesian criteria. We obtain mis-selection probability expressions for DIC and marginal likelihood, and establish that mis-selection probability of marginal likelihood converges to 0 under certain conditions but show that the same does not hold for the DIC. In section 4, we compare the performances of DIC and marginal likelihood based selections with horseshoe prior (Carvalho et al., 2010), nonlocal prior (Shin et al., 2018) and the non-Bayesian sure independent screening (Fan & Lv, 2008) in the presence and absence of multicollinearity among predictors. We find that DIC based selection provides high sensitivity, in the range of 90 – 100%, however the probability of correct selection by DIC is in between 0 – 2% and the false discovery rate is between 25 – 49%. This poor performance of DIC persists consistently in simulation studies ranging from $n=50$ to $n=\mathrm{10,000}$ and for various signal-to-noise ratios. Correct selection by marginal likelihood, in contrast, improves from 40% for $n=50$ to 95% for $n=\mathrm{10,000}$. Theorem 1 establishes that both marginal likelihood and DIC asymptotically disfavor under-fitted models, thus explaining the high sensitivity of both of these criteria found in the simulation studies. On the other hand Theorem 3 establishes that mis-selection probability of DIC remains away from zero for every sample size whereas mis-selection probability by marginal likelihood converges to 0 under certain conditions.

The motivating scientific problem of selecting a prognosticating biomarker panel for cancer cachexia based on baseline level biomarkers measured by Luminex^® Multiplex Assays for metastatic advanced stage non-small cell lung cancer patients is considered in detail in section 4.2. In section 6, we explore beyond the setting of linear models where we examine the performance of these criteria for variable selection in generalized linear models in the presence of multicollienarity. We find poor performance of the DIC in generalized linear model to be disconcerning as practitioners often choose to use DIC that is readily available in software in such non-conjugate settings. Finally we conclude with a brief discussion in Section 7.

2. An illustrative example

Shao (1993) considered variable selection in a dataset from Gunst & Mason (1980) to compare selection based on one-deleted and many-deleted cross-validations. We consider the same setting as Shao (1993) where the model space consists of all possible models that always include the first covariate $x_1\equiv 1$. We evaluate DIC and marginal likelihood by complete enumeration of all models in the model space and report the models with the smallest DIC, highest marginal likelihood or highest posterior model (HPM). We also report the highest LPML model for comparison. As noted before, the LPML criterion combines the conditional predictive ordinates, each of which is obtained as predictive density evaluated at current observation given the remaining observations (one deleted cross-validation), see Geisser & Eddy (1979); Gelfand et al. (1992); Chen et al. (2008). $g$-prior is assumed on the regression coefficients in the analysis models. We consider three different values of the constant multiplier of the $g$-prior (see (3.1)), $a_0=1/n$, and a lower and a higher value.

We simulate data from data-generating model $ℳ\left(D\right):\mathit{y}~N\left(\mathit{X}\mathit{\beta }\left(D\right),{\tau }^{-1}I\right)$ and consider four different scenarios: (i) $ℳ\left(D\right)=\left\{x_1,x_4\right\}$, (ii) $ℳ\left(D\right)=\left\{x_1,x_4,x_5\right\}$, (iii) $ℳ\left(D\right)=\left\{x_1,x_2,x_4,x_5\right\}$, and (iv) $ℳ\left(D\right)=\left\{x_1,x_2,x_3,x_4,x_5\right\}$, the full model. We report relative frequencies of the optimally selected models by DIC, marginal likelihood and LPML over repeated simulations in Table 1.

Table 1:

Relative frequency of optimally selected models by DIC, marginal likelihood (HPM) and LPML. $ℳ\left(D\right)$ is the data-generating model and Category II are models which includes the variables in $ℳ\left(D\right)$ (but can be larger). $a_0$ refers to the constant multiplier of the g-prior.

Model	Category	${\mathit{a}}_0=0.001$			${\mathit{a}}_0=\frac{1}{\mathit{n}}$			${\mathit{a}}_0=0.1$
		DIC	HPM	LPML	DIC	HPM	LPML	DIC	HPM	LPML
$\text{n}=40$
1,4	$M\left(D\right)$	0.575	0.973	0.584	0.621	0.876	0.602	0.572	0.727	0.350
1,2,4	II	0.104	0.006	0.114	0.109	0.038	0.123	0.117	0.075	0.113
1,3,4	II	0.131	0.010	0.132	0.092	0.029	0.103	0.105	0.073	0.070
1,4,5	II	0.108	0.011	0.097	0.106	0.045	0.096	0.129	0.085	0.287
1,2,3,4	II	0.026		0.024	0.024	0.005	0.030	0.025	0.013	0.042
1,2,4,5	II	0.026		0.022	0.018	0.005	0.014	0.025	0.014	0.053
1,3,4,5	II	0.018		0.018	0.023	0.002	0.028	0.018	0.009	0.068
1,2,3,4,5	II	0.009		0.009	0.007		0.004	0.009	0.004	0.017
1,4,5	$M\left(D\right)$	0.707	0.986	0.714	0.722	0.910	0.699	0.678	0.813	0.424
1,2,4,5	II	0.124	0.006	0.121	0.121	0.048	0.129	0.118	0.075	0.114
1,3,4,5	II	0.134	0.007	0.136	0.120	0.039	0.130	0.145	0.092	0.270
1,2,3,4,5	II	0.035	0.001	0.029	0.037	0.003	0.042	0.059	0.020	0.192
1,2,4,5	$M\left(D\right)$	0.822	0.993	0.834	0.832	0.954	0.818	0.826	0.897	0.149
1,2,3,4,5	II	0.178	0.007	0.166	0.168	0.046	0.182	0.174	0.103	0.851
1,2,3,4,5	$ℳ\left(𝒟\right)$	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
$\text{n}=400$
1,4	$ℳ\left(𝒟\right)$	(0.605)	0.973	(0.605)	(0.625)	0.0.955	0.0.620	(0.574)	0.716	(0.228)
1,2,4	II	0.110	0.011	0.118	0.107	0.014	0.106	0.124	0.094	0.082
1,3,4	II	0.104	0.008	0.102	0.091	0.013	0.096	0.110	0.084	0.057
1,4,5	II	0.108	0.007	0.105	0.114	0.016	0.112	0.106	0.078	0.396
1,2,3,4	II	0.024	0.001	0.023	0.029	0.001	0.029	0.035	0.011	0.037
1,2,4,5	II	0.016		0.014	0.015	0.001	0.018	0.021	0.009	0.081
1,3,4,5	II	0.023		0.021	0.012		0.012	0.020	0.007	0.090
1,2,3,4,5	II	0.010		0.012	0.007		0.007	0.010	0.001	0.029
1,4,5	$ℳ\left(𝒟\right)$	0.710	0.985	0.712	0.712	0.970	0.722	0.689	0.802	0.427
1,2,4,5	II	0.129	0.008	0.124	0.143	0.014	0.135	0.128	0.076	0.108
1,3,4,5	II	0.129	0.007	0.129	0.106	0.016	0.104	0.145	0.106	0.285
1,2,3,4,5	II	0.032		0.035	0.039		0.039	0.038	0.016	0.180
1,2,4,5	$ℳ\left(𝒟\right)$	0.819	0.986	0.820	0.834	0.988	0.839	0.805	0.881	0.012
1,2,3,4,5	II	0.181	0.014	0.180	0.166	0.012	0.161	0.195	0.119	0.988
1,2,3,4,5	$ℳ\left(𝒟\right)$	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000

We first notice that across all the scenarios considered in Table 1, remarkably, the optimally selected models, either by DIC, marginal likelihood or LPML, are never sub-models of the respective data-generating models $ℳ\left(D\right)$. As we will see in the following section 3, this finding is consistent with Theorem 1.

Secondly, we notice that the HPM approach performs superiorly in selecting the data-generating model uniformly across Table 1. This is especially evident when the data-generating model $ℳ\left(D\right)=\left\{x_1,x_4\right\}$ is relatively sparse. The DIC and LPML criteria here select the data-generating model only 62% and 60% of times when $a_0=1/n$ whereas, in comparison, the marginal likelihood criterion selects 88% of times. The performances of all three criteria, but especially of DIC and LPML improve when the data-generating model is less sparse.

To study the performances of these three criteria with increased sample size, we consider the setting of n = 400 by replicating the original $\mathit{X}$ matrix 10 times. We notice that surprisingly there is almost no improvements in the performance of the DIC and LPML when the sample size is increased from $n=40$ to $n=400$. We further notice that the performance of HPM remains mostly unchanged from $n=40$ to $n=400$ as well when $a_0$ remains constant in $n$ (0.001 or 0.1). However, for the choice of $a_0=1/n$, there is a significant increase in correct selection by HPM for $n=400$ in Table 1. There is almost no improvement in the performances of DIC and LPML even in the case of $a_0=1/n$. As we will see in the following section 3, these findings are consistent with Theorem 3.

3. Bayesian Variable Selection

3.1. Selection Criteria

We are interested in exploring the association between outcome $Y$ and $p$ predictors or features $x_1,\dots ,x_p$, and suppose that we have observations $\left\{\left(y_i,x_{i1},\dots ,x_{ip}\right),i=1,\dots ,n\right\}=\left(\mathit{y},{\mathit{x}}_1,\dots ,{\mathit{x}}_p\right)$. Let $ℳ\subseteq \left\{\text{all}\text{subsets}\text{of}\right\{\mathrm{1,2},\dots ,p\left\}\right\}$ denote the model space under consideration in the variable selection. A member of $ℳ$ is $ℳ\left(\alpha \right)$ where $\alpha$ is a subset of $\{\mathrm{1,2},\dots ,p\}$ and ${\mathit{X}}_n\left(\alpha \right)$ is the matrix composed of columns of $\left\{{\mathit{x}}_1,\dots ,{\mathit{x}}_p\right\}$ with indices in $\alpha$. In model $ℳ\left(\alpha \right)$, the association of the stochastic outcome ${\mathit{Y}}_n$ with ${\mathit{X}}_n\left(\alpha \right)$ is expressed in terms of a quantity ${\xi }_{{\mathit{Y}}_n\mid {\mathit{X}}_n\left(\alpha \right)}$ associated with the probability distribution $F_{{\mathit{Y}}_n\mid {\mathit{X}}_n\left(\alpha \right)}$ via the regression model ${\xi }_{{\mathit{Y}}_n\mid {\mathit{X}}_n\left(\alpha \right)}={\mathit{X}}_n\left(\alpha \right)\mathit{\beta }\left(\alpha \right)$. In normal linear model, ${\xi }_{{\mathit{Y}}_n\mid {\mathit{X}}_n\left(\alpha \right)}=E\left({\mathit{Y}}_n\mid {\mathit{X}}_n\left(\alpha \right)\right)$ and $F_{{\mathit{Y}}_n\mid {\mathit{X}}_{n\left(\alpha \right)}}$ is normal, resulting in model $ℳ\left(\alpha \right):{\mathit{Y}}_n~N\left({\mathit{X}}_n\left(\alpha \right)\mathit{\beta }\left(\alpha \right),{\tau }^{-1}{I}_n\right)$ where $I_n$ is the identity matrix. We assume all relevant ${\mathit{X}}_n\left(\alpha \right)$ matrices are of full column rank.

The marginal likelihood has traditionally been used as a Bayesian model selection criterion. For model $ℳ\left(\alpha \right)$, the marginal likelihood is given by

$$p\left({\mathit{y}}_n\mid ℳ\left(\alpha \right)\right)=\int f\left(\mathit{y}∣\mathit{\beta }\left(\alpha \right),\tau ,ℳ\left(\alpha \right),\tau \right)\pi \left(\mathit{\beta }\left(\alpha \right),\tau ∣ℳ\left(\alpha \right)\right)d\mathit{\beta }\left(\alpha \right)d\tau ,$$

where $f\left({\mathit{y}}_n\mid \mathit{\beta }\left(\alpha \right),\tau ,ℳ\left(\alpha \right)\right)$ and $\pi \left(\mathit{\beta }\right(\alpha ),\tau \mid ℳ(\alpha \left)\right)$ are the likelihood and the prior under model $ℳ\left(\alpha \right)$. The Bayes factor for two comparator models is obtained as ratio of their marginal likelihoods. It has a large literature (Kass & Raftery, 1995; Chib, 1995; Chib & Jeliazkov, 2001, 2005; Chan & Jeliazkov, 2009) and is associated with Schwarz’s Bayesian information criterion BIC. The posterior probability of model $ℳ\left(\alpha \right)$ in the model space $ℳ$ is given by

$$P\left(ℳ\left(\alpha \right)\mid {\mathit{y}}_n\right)=\frac{p\left({\mathit{y}}_n\mid ℳ\left(\alpha \right)\right)P\left(ℳ\right(\alpha \left)\right)}{\sum _{{\alpha }^{\prime }}p\left({\mathit{y}}_n\mid ℳ\left({\alpha }^{\prime }\right)P\left(ℳ\left({\alpha }^{\prime }\right)\right)\right.}\propto p\left({\mathit{y}}_n\mid ℳ\left(\alpha \right)\right)P\left(ℳ\left(\alpha \right)\right),$$

where $P\left(ℳ\right(\alpha \left)\right)$ is the prior probability of model $ℳ\left(\alpha \right)$ in the model space. The highest posterior model (HPM) is argmax $\left\{P\left(ℳ\left(\alpha \right)\mid {\mathit{y}}_n\right):\alpha \in ℳ\right\}$ and the HPM approach for model selection has an extensive literature from both theoretical and computational aspects.

The Deviance Information Criterion (Spiegelhalter et al., 2002, 2014), (Chen, Huang, Ibrahim, & Kim, 2008) combines goodness of fit of a model with a penalty for model complexity and is defined as deviance + 2× (effective number of parameters), evaluated at a posterior point estimate of the parameter, In particular, $\text{DIC}=D\left(\tilde{\theta }\right)+2p_{\text{D}}$, where $D\left(\theta \right)=-2\text{log}f\left({\mathit{y}}_n\mid \theta \right),f\left({\mathit{y}}_n\mid \theta \right)$ is the likelihood function of the model and $\tilde{\theta }$ is an estimate of the model parameter $\theta$. In the above expression $p_{\text{D}}$ is termed as the effective number of parameters and is defined as $p_{\text{D}}=\overline{\text{D}\left(\theta \right)}-D\left(\tilde{\theta }\right)$, where $\overline{\text{D}\left(\theta \right)}={\text{E}}_{\theta \mid {\mathit{y}}_n}\left\{-2\text{log}f\left({\mathit{y}}_n\mid \theta \right)\right\}+2\text{log}h\left({\mathit{y}}_n\right)$ is considered to be a measure of model fit, and $h\left({\mathit{y}}_n\right)$ is some fully specified standardizing term that is a function of the data alone. A model with a smaller value of DIC is preferred by this criterion.

The first term of DIC, $D\left(\hat{\theta }\right)$ is evaluated at a posterior point estimate $\hat{\theta }$. In models with latent structures, such as mixed effects, hierarchical, latent variables or missing data models, the unobservables also include the latent variables. In these settings, what should be considered as likelihood and $\theta$ can result in different definitions of DIC (Celeux et al., 2006). A series of recent works (Quintero & Lesaffre, 2018; Merkle et al., 2019; Ariyo et al., 2019, 2020) have shown superior performances of marginalized criteria (such as marginalized DIC) over conditional criteria in such settings. We consider linear models without any latent structures in this article.

In Bayesian variable selection, the g-prior (Zellner, 1986) has a long history. In the setting of the normal linear model $ℳ\left(\alpha \right):{\mathit{Y}}_n~N\left({\mathit{X}}_n\left(\alpha \right)\mathit{\beta }\left(\alpha \right),{\tau }^{-1}{I}_n\right)$, we consider the following $g$-prior on the regression parameter $\beta \left(\alpha \right)$ and $\text{Gamma}(a,b)$ prior on the precision parameter $\tau$

$$\mathit{\beta }\left(\alpha \right)\left|\tau ~N\left(0,\frac{1}{\tau a_0}{\left({\mathit{X}}_n\left(\alpha {)}^{\prime }{\mathit{X}}_n\right(\alpha )\right)}^{-1}\right)\right.\text{and}\tau ~\text{Gamma}(a,b).$$ (3.1)

The $g$-prior is popular and widely studied (Chen, Huang, Ibrahim, & Kim, 2008) in the context of Bayesian variable selection in linear models. The choice of $a_0$ is important for variable selection and different choices have been suggested (George & Foster, 2000). When ${\mathit{Y}}_n\left|\mathit{\beta }\left(\alpha \right),\tau ~N_n\left({\mathit{X}}_n\left(\alpha \right)\mathit{\beta }\left(\alpha \right),\frac{1}{\tau }I\right),\mathit{\beta }\left(\alpha \right)\right|\tau ~N\left(0,\frac{1}{\tau a_0}{\left({\mathit{X}}_n\left(\alpha {)}^{\prime }{\mathit{X}}_n\right(\alpha )\right)}^{-1}\right)$ and $\tau ~\text{Gamma}(a,b)$ as in (3.1), the marginal distribution of ${\mathit{Y}}_n$ is a multivariate t-distribution and the log-marginal likelihood of the model $ℳ\left(\alpha \right)$ is given by

$$\begin{array}{r}\text{logML}\left(ℳ\right(\alpha \left)\right)=-\frac{n}{2}\text{log}\left(2\pi \right)+\frac{d\left(\alpha \right)}{2}\text{log}\left(\frac{a_0}{a_0+1}\right)+\text{log}\left(\frac{\Gamma (a+n/2)}{\Gamma a}\right)-a\text{log}\left(b\right)-\left(a+\frac{n}{2}\right)\text{log}\left(b+\frac{a_0}{2\left(1+a_0\right)}{\mathbf{Y}}_n^{\prime }{\mathbf{Y}}_n+\frac{1}{2\left(1+a_0\right)}\text{SSE}\left(\alpha \right)\right).\\ \end{array}$$ (3.2)

Here $\text{SSE}\left(\alpha \right)={\mathit{Y}}_n^{\prime }(I-H(\alpha \left)\right){\mathit{Y}}_n,H\left(\alpha \right)={\mathit{X}}_n\left(\alpha \right){\left({\mathit{X}}_n\left(\alpha {)}^{\prime }{\mathit{X}}_n\right(\alpha )\right)}^{-1}{\mathit{X}}_n\left(\alpha \right)$ is the hat matrix, and $d\left(\alpha \right)$ is the model dimension of $ℳ\left(\alpha \right)$. Under prior specifications in (3.1), the DIC of model $ℳ\left(\alpha \right)$ is obtained

$$\text{DIC}\left(ℳ\left(\alpha \right)\right)=\text{D}\left(\stackrel{-}{\mathit{\beta }}\left(\alpha \right),\bar{\tau }\left(\alpha \right)\right)+2p_{\text{D}}\left(\alpha \right),$$

where $\overline{\mathit{\beta }}(\alpha )=E\left[\mathit{\beta }\mid {\mathit{y}}_n\right]$ and $\overline{𝒯}(\alpha )=E\left\{\tau (\alpha )\mid {\mathit{y}}_n\right\}=\frac{a+n/2}{b+\frac{1}{2}SSE(\alpha )+\frac{1}{2}\frac{a_0}{1+a_0}{\mathit{y}}_n^{\prime }H(\alpha ){\mathit{y}}_n}$ are posterior estimates of $\mathit{\beta }\left(\alpha \right)$ and $\tau$. We have (see Spiegelhalter et al. 2002) $p_{\text{D}}\left(\alpha \right)=\stackrel{-}{\text{D}\left(\mathit{\beta }\right(\alpha ),\tau (\alpha \left)\right)}-D\left(\stackrel{-}{\mathit{\beta }}\right(\alpha ),\bar{\tau }(\alpha \left)\right)={\text{E}}_{\tau \mid {\mathit{y}}_n}\left[E_{\beta \mid {\mathit{y}}_n,\tau }\left[-2\text{log}f\left({\mathit{y}}_n\mid \theta \right)\right]\right]-D\left(\stackrel{-}{\mathit{\beta }}\right(\alpha ),\bar{\tau }(\alpha \left)\right)=\frac{d\left(\alpha \right)}{1+a_0}-n\left\{\psi \right(a+n/2)-\text{log}(a+n/2\left)\right\}$ where $\psi (.)$ is the digamma function. It follows that

$$\begin{array}{r}\text{DIC}\left(ℳ\right(\alpha \left)\right)=-n\text{ln}\left(\frac{\bar{\tau }\left(\alpha \right)}{2\pi }\right)+\frac{a_0^2\bar{\tau }\left(\alpha \right)}{{\left(1+a_0\right)}^2}{\mathit{Y}}_n^{\prime }{\mathit{Y}}_n+\frac{\left(1+2a_0\right)\bar{\tau }\left(\alpha \right)}{{\left(1+a_0\right)}^2}\text{SSE}\left(\alpha \right)+2\frac{d\left(\alpha \right)}{1+a_0}-2n\left\{\psi \right(a+n/2)-\text{log}(a+n/2\left)\right\}.\\ \end{array}$$ (3.3)

Both DIC and marginal likelihood have extensive literature. For marginal likelihood and Bayes factor in general setting, see Kass & Raftery (1995); Chib (1995); Chib & Jeliazkov (2001, 2005); Chan & Jeliazkov (2009) and references therein. In normal linear model with $g$-prior, Fernandez et al. (2001a) show that posterior probability of data generating model converges to 1. In similar setting, George & Foster (2000) show (see also Fernandez et al. 2001b) that the HPM approach can be equivalent to either AIC or BIC or other approaches depending on the choice of the constant of the $g$-prior. Liang et al. (2008) study Bayesian variable selection under mixtures of $g$-priors. Besides the setting of $g$-priors, Casella & Moreno (2006); Casella et al. (2009) establish consistency of Bayes factor under intrinsic priors whereas Moreno et al. (2010) study consistency property with the rate of growth of the model dimension. Chib et al. (2018) study marginal likelihood in the exponentially tilted empirical likelihood framework whereas recent work by Fong & Holmes (2020) establish that marginal likelihood is formally equivalent to a type of exhaustive cross-validation. Johnson & Rossell (2010, 2012) propose non-local prior to regularize imbalance in point null hypothesis testing and study Bayes factor, see also Shin et al. (2018).

DIC is introduced in Spiegelhalter et al. (2002) and its expression depends on the likelihood and the posterior. It thus can be obtained when the likelihood and posterior are tractable, such as in the setting of improper priors. The terms in DIC can often be estimated from Markov chain samples and DIC is readily available in popular Bayesian software. DIC is evaluated at a plug-in posterior estimate $\tilde{\theta }$ and has been criticized as not fully Bayesian (Gelman et al., 2014) and being not a consistent model selection procedure (Moreno & Vâzquez-Polo, 2014), There is substantial recent interest on DIC based on observed-data likelihood as opposed to conditional likelihood (Celeux et al., 2006; Quintero & Lesaffre, 2018; Merkle et al., 2019; Ariyo et al., 2019, 2020) in the setting of models with latent structure. Chan & Grant (2016) develop fast computation for observed-data DIC whereas Li et al. (2020) establish relation to observed-data AIC as well as propose DIC for misspecified models.

We study properties of marginal likelihood and DIC under category I and Category II models, which are introduced below.

3.2. Category I and II models

In the following, we assume there is a data-generating model within the model space, denoted by $ℳ\left({\alpha }_D\right)$, or in short, simply as $ℳ\left(D\right)$. We consider a partition $\left\{{ℳ}_1,{ℳ}_2\right\}$ of $ℳ\setminus \left\{ℳ\right(𝒟\left)\right\}$, where ${ℳ}_1=\left\{ℳ\right(\alpha )\left.{:\alpha }_D⊈\alpha \right\}$ and ${ℳ}_2=\left\{ℳ\left(\alpha \right):{\alpha }_D\subset \alpha \right\}.{ℳ}_1$ contains all models that do not contain at least one component of the data-generating model whereas, ${ℳ}_2$ contains all super-models of the data-generating model. We refer to ${ℳ}_1$ and ${ℳ}_2$ as the class of category I and category II models respectively, and similar definitions can be found in Shao (1993).

Under the data generating model $ℳ\left(D\right)\in ℳ,{\mathit{Y}}_n~N\left({\mu }_n,{\tau }^{-1}{I}_n\right)$ where ${\mu }_n={\mathit{X}}_n\left(D\right)\beta \left(D\right)$. Many results in this article are derived under the following assumptions: (A1) ${\text{lim}}_{n\to \infty }{n}^{-1}\left({\mathit{X}}_n^{\prime }{\mathit{X}}_n\right)=V$, where $V$ is finite and positive definite matrix, (A2) ${\text{lim}}_{n\to \infty }{\text{max}}_{1\le i\le n}{h}_{ii}\left(\alpha \right)\to 0,h_{ii}\left(\alpha \right)$ is the $i$th diagonal element of $H_n\left(\alpha \right)$, and (A3) ${\text{lim}}_{n\to \infty }{n}^{-1}{\mu }_n^{\prime }{\mu }_n={\mu }^{\prime }\mu$. Assumptions (A1), (A2), and (A3) are standard for deriving asymptotic results in normal linear models. We additionally assume (A4): for any model $ℳ\left({\alpha }^{\prime }\right)\in {ℳ}_1,{\text{lim}}_{n\to \infty }{n}^{-1}{\mu }_n^{\prime }\left(I-H\left({\alpha }^{\prime }\right){\mu }_n\right)=b\left({\alpha }^{\prime }\right)$, similar to Fernandez et al. (2001a). (A1) ensures the covariates are bounded and (A3) ensures the mean vector of the data generating model is finite whereas (A4) is needed for identifiability.

Our first set of results establishes that even the worst model in ${ℳ}_2\cup \left\{ℳ\right(D\left)\right\}$ is asymptotically preferred by marginal likelihood and DIC than the best category I model in ${ℳ}_1$. All proofs are available in the supplementary material.

Theorem 1. If ${ℳ}_1$ and ${ℳ}_2\cup \left\{ℳ\right(D\left)\right\}$ are not empty and for the choice of $a_0=1/n$

1. ${\text{lim}}_{n\to \infty }{P}_{{\mathit{Y}}_n}\left\{{\text{min}}_{ℳ\left(\alpha \right)\in {ℳ}_2\cup \left\{ℳ\right(D\left)\right\},ℳ\left({\alpha }^{\prime }\right)\in {ℳ}_1}\right.\mathit{\text{Bayes}}\mathit{\text{factor}}\left.\left(ℳ\left(\alpha \right),ℳ\left({\alpha }^{\prime }\right)\right)\ge 1\right\}=1$.

2. ${\text{lim}}_{n\to \infty }{P}_{{\mathit{Y}}_n}\left\{{\text{min}}_{ℳ\left(\alpha \right)\in {ℳ}_1}DIC\left(ℳ\right(\alpha \left)\right)\ge {\text{max}}_{ℳ\left(\alpha \right)\in {ℳ}_2\cup \left\{ℳ\right(D\left)\right\}}DIC\left(ℳ\right(\alpha \left)\right)\right\}=1$.

Theorem 1 establishes that selection by these criteria concentrates on the class of $\left\{\mathit{\text{category}}\text{II}\right\}\cup ℳ\left(D\right)\}$ models. In our numerical studies in sections 2 and 4, we find that both marginal likelihood and DIC based selections yield high sensitivities (see Table 2) corroborating this result. Scientific problems increasingly involve a large number of features, and the typical scientific postulate is that the underlying data generating model $ℳ\left(D\right)$ is sparse, involving only a few of the features. In such cases, the class of category II models is large and it is important to perform appropriate variable selection. Our next set of results consider the mis-selection probability within this class.

Table 2:

$ℳ\left(D\right)$ is the proportion of times the data generating model is selected. FDR is the false discovery rate (=FP/(TP+FP)) and SEN is the sensitivity (=TP/(TP+FN)), both averaged over replications. Here TP (True Positive) and FN (False Negative) are number of $\beta$ coefficients selected respectively as important and not important by the analysis approach out of the non-zero coefficients in the data generating model. Similarly FP (False Positive) is the number of coefficients selected as important out of the zero coefficients in the data generating model. Results are based on 100 replications.

	DIC			HPM			HorseShoe			BayesS5			SIS
	$ℳ\left(D\right)$	FDR	SEN	$ℳ\left(D\right)$	FDR	SEN	$ℳ\left(D\right)$	FDR	SEN	$ℳ\left(D\right)$	FDR	SEN	$ℳ\left(D\right)$	FDR	SEN
	$\text{n}=50,\text{p}=25,\beta \left(D\right)=(1.5,-\mathrm{1.5,1.5},-\mathrm{1.5,1.5})$
$\rho =0$	0.00	0.46	1.00	0.67	0.07	1.00	0.18	0.02	0.68	0.72	0.05	1.00	0.21	0.34	1.00
$\rho =0.5$	0.01	0.49	0.92	0.40	0.11	0.88	0.00	0.01	0.27	0.42	0.11	0.87	0.02	0.53	0.84
AR	0.01	0.46	0.99	0.59	0.09	0.95	0.00	0.01	0.16	0.58	0.09	0.93	0.16	0.43	0.99
	$\text{n}=100,\text{p}=25,\beta \left(D\right)=(1.5,-\mathrm{1.5,1.5},-\mathrm{1.5,1.5})$
$\rho =0$	0.01	0.47	1.00	0.79	0.05	1.00	0.83	0.02	0.99	0.79	0.04	1.00	0.57	0.20	1.00
$\rho =0.5$	0.00	0.49	0.97	0.72	0.06	0.98	0.01	0.00	0.65	0.60	0.10	0.97	0.14	0.46	0.90
AR	0.00	0.48	1.00	0.83	0.03	1.00	0.35	0.01	0.70	0.68	0.06	1.00	0.50	0.21	1.00
	$\text{n}=10000,\text{p}=25,\beta \left(D\right)=(1.5,-\mathrm{1.5,1.5},-\mathrm{1.5,1.5})$
$\rho =0$	0.00	0.44	1.00	0.96	0.01	1.00	0.75	0.04	1.00	0.98	0.00	1.00	0.92	0.02	1.00
$\rho =0.5$	0.00	0.43	1.00	0.95	0.01	1.00	0.76	0.04	1.00	0.90	0.02	1.00	0.81	0.04	1.00
AR	0.00	0.42	1.00	0.95	0.01	1.00	0.77	0.04	1.00	0.97	0.00	1.00	0.87	0.03	1.00
	$\text{n}=100,\text{p}=25,\beta \left(D\right)=(0.7,-\mathrm{0.7,0.7},-\mathrm{1,1},-\mathrm{1,1},-\mathrm{1.5,1.5},-1.5)$
$\rho =0$	0.02	0.26	1.00	0.68	0.03	0.99	0.05	0.01	0.72	0.73	0.02	0.98	0.20	0.20	1.00
$\rho =0.5$	0.02	0.26	0.94	0.06	0.03	0.86	0.00	0.01	0.51	0.02	0.01	0.80	0.09	0.21	0.95
AR	0.01	0.25	0.99	0.46	0.02	0.90	0.00	0.01	0.29	0.43	0.03	0.88	0.09	0.24	0.99
	$\text{n}=200,\text{p}=25,\beta \left(D\right)=(0.7,-\mathrm{0.7,0.7},-\mathrm{1,1},-\mathrm{1,1},-\mathrm{1.5,1.5},-1.5)$
$\rho =0$	0.02	0.25	1.00	0.83	0.02	1.00	0.55	0.00	0.95	0.82	0.02	1.00	0.49	0.10	1.00
$\rho =0.5$	0.02	0.28	0.98	0.38	0.03	0.95	0.00	0.01	0.74	0.43	0.02	0.94	0.28	0.15	0.98
AR	0.02	0.26	1.00	0.75	0.02	1.00	0.08	0.00	0.74	0.70	0.03	1.00	0.27	0.15	1.00

Theorem 2. Consider any model $ℳ\left(\alpha \right)\in {ℳ}_2$.

1. The probability of mis-selection by marginal likelihood is given by

   $$P_{{\mathit{Y}}_n}\left\{\mathit{\text{Bayes}}\mathit{\text{factor}}\left(ℳ\left(\alpha \right),ℳ\left({\alpha }_D\right)\right)\ge 1\right\}\le 1-F_{{\chi }_{\eta }^2}\left[\left(1+a_0\right)\left(2n+a\right)\left\{{\left(1+a_0^{-1}\right)}^{\frac{\eta }{\left(n+2a\right)}}-1\right\}\right]+o\left(1\right).$$ (3.4)

   where $\eta =d\left(\alpha \right)-d\left({\alpha }_D\right)$ and $F_{{\chi }_{\eta }^2}\left(x\right)$ is the cdf of ${\chi }^2$ distribution with $\eta$ degrees of freedom as before.
2. The probability of mis-selection by DIC is given by

   $$P_{{\mathit{Y}}_n}\left\{DIC\left(ℳ\left(\alpha \right)\right)\le DIC\left(ℳ\left(D\right)\right)\right\}\le 1-F_{{\chi }_{\eta }^2}\left(\frac{2\left(1+a_0\right)}{1+2a_0}\eta \right)+o\left(1\right).$$ (3.5)

Remark 1: Note that the expressions in (3.4) and (3.5) are free of $\mu \left(D\right)$ (or the limiting value ${\mu }^{\prime }\mu$ as in A3) and $\tau \left(D\right)$ of the data generating model and thus can be readily evaluated without knowing the data-generating model.

Remark 2: It is noteworthy that the upper bound in (3.5) is independent of $n$ and thus provides a constant bound for all $n$ (when $a_0$ does not depend on $n$).

Theorem 2 is illustrated in Figure 1 in which the marginal likelihood and DIC of the data generating model $ℳ\left(D\right)$ and different $ℳ\left(\alpha \right)$ models are evaluated for different sample sizes. We used $a_0=1/n$ for the $g$-prior on $\mathit{\beta }\left(\alpha \right)$ and Gamma(1, 1) prior on $\tau$. The figure shows the empirical estimates of the left hand side expressions in Theorem 2 based on 2,500 repeated data simulations (red) and the upper bounds of mis-selection probabilities (blue) from the right hand sides of Theorem 2. We note that the mis-selection probabilities of Bayes factor are relatively low across all three panels and decreases to zero with increased sample size. The corresponding probability for DIC is relatively high and stays bounded from below. Both of these results are established theoretically in the following.

Figure 1:

Red ● are empirical estimates of $P_{{\mathit{Y}}_n}\left[\text{DIC}\right(ℳ\left(\alpha \right))\le \text{DIC}(ℳ\left(𝒟\right)\left)\right]$ (left panel) and $P_{{\mathit{Y}}_n}[$ Bayes factor $\left(ℳ\right(\alpha ),ℳ(D\left)\right)\ge 1]$ (right panel) based on 2, 500 repeated data simulations from data generating model $ℳ\left(D\right)=\left\{x_3,x_4\right\}$. Blue ● are ${\chi }^2$ tail probabilities from Theorem 2. (a) Left-subpanel $ℳ\left(\alpha \right)=\left\{x_1,x_3,x_4\right\}$, (b) middle-subpanel $ℳ\left(\alpha \right)=\left\{x_2,x_3,x_4\right\}$ and (c) right-subpanel $ℳ\left(\alpha \right)=\left\{x_1,x_2,x_3,x_4\right\}$.

Usual recommendation in Bayesian variable selection is to consider $a_0\left(n\right)$ decreasing in sample size $n$. The following Theorem establishes that the mis-selection probability of DIC asymptotically continues to be positive for any such sequence.

Theorem 3. (a) For $ℳ\left(\alpha \right)\in {ℳ}_2,a_0=a_0\left(n\right)\to 0$, and $a_0(n{)}^{-1}=O(n)$ as $n\to \infty$,

$$\underset{n\to \infty }{\text{lim}}{P}_{{\mathit{Y}}_n}\left\{\mathit{\text{Bayes}}\mathit{\text{factor}}\right(ℳ\left(\alpha \right),ℳ\left(D\right))\ge 1\}=0.$$

(b) Consider any sequence $\left\{a_0\left(n\right)\right\}$ converging to 0 as $n\to \infty$. For $ℳ\left(\alpha \right)\in {ℳ}_2$

$$\underset{n\to \infty }{\text{lim}}{P}_{{\mathit{Y}}_n}\left\{DIC\right(ℳ\left(\alpha \right))\le DIC(ℳ\left(D\right)\left)\right\}>0.$$

Note that the above results in Theorem 3 are comparative results between models. Fernandez et al. (2001a) provide a result on consistency of the data generating model $ℳ\left(D\right)$ under a non-informative $\pi \left(\tau \right)\propto 1/\tau$ and $a_0\left(n\right)\to 0$.

The implication of the next result is that given data ${\mathit{y}}_n$ and a fixed $a_0<\frac{1}{\text{exp}\left(\frac{4a}{n}+2\right)-1}$, if we mis-select model $ℳ\left(\alpha \right)$ instead of the data generating model $ℳ\left(D\right)$ using the Bayes factor criterion, then we will also make the same mis-selection using the DIC criterion. The previous results in this article are stated in terms of probability under the data-generating sampling model. In contrast, the following result is, in fact, conditional on data ${\mathit{y}}_n$.

Theorem 4. If $a_0<\frac{1}{\text{exp}\left(\frac{4a}{n}+2\right)-1}$, then for the data-generating model $ℳ\left(D\right)$ and a model $ℳ\left(\alpha \right)\in {ℳ}_2$,

$$\left\{{\mathit{y}}_n:\mathit{\text{Bayes}}\mathit{\text{factor}}\left(ℳ\left(\alpha \right),ℳ\left(D\right)\mid {\mathit{y}}_n\right)\ge 1\right\}\subseteq \left\{{\mathit{y}}_n:\text{DIC}\left(ℳ\left(\alpha \right)\mid {\mathit{y}}_n\right)\le \text{DIC}\left(ℳ\left(D\right)\mid {\mathit{y}}_n\right)\right\}.$$

We briefly sketch parallel results on the performances of DIC and marginal likelihood when $\tau$ is known in section 5.

4. Numerical Studies

We present a numerical study and the cancer cachexia scientific study to further illustrate our results and the performances of DIC and marginal Likelihood.

4.1. HPM, DIC, Horseshoe, Nonlocal, SIS

We compare the empirical properties of marginal likelihood based HPM (Chib, 1995; Chib & Jeliazkov, 2001, 2005) and minimum DIC (Chen, Huang, Ibrahim, & Kim, 2008) approaches with several Bayesian and non-Bayesian variable selection methods. The horseshoe prior (Carvalho et al. 2010, Bhadra et al. 2016) has a well-established literature for sparse signals. We consider the implementation of the horseshoe prior in the R package Horseshoe (van der Pas et al., 2016) and the variable selection function in this package in its default setting. We further consider the nonlocal prior based selection (Johnson & Rossell, 2012) whose theoretical and numerical performances are recently considered in Shin et al. (2018) and its stochastic search implementation in the R package BayesS5 (Shin & Tian, 2017) in its default setting. We further compare these Bayesian approaches with the non-Bayesian (iterative) sure independent screening method whose sure screening and consistency are established in Fan & Lv (2008); Fan et al. (2011) and which is implemented in the R package SIS (Saldana & Feng, 2018).

We use SNR to denote the signal-to noise ratio in the data generating model (Meier et al., 2009; Fan et al., 2011; Dicker, 2014),

$$\text{SNR}=\frac{\text{Var}\left(X\right(D\left)\beta \right(D\left)\right)}{\text{Var}\left(\epsilon \right)}=\tau \left(D\right)\beta (D{)}^{\prime }{\Sigma }_{X\left(D\right)}\beta \left(D\right),$$

where ${\Sigma }_{X\left(D\right)}$ is the covariance matrix of the randomly generated $X\left(D\right)$ in the data generating model.

We consider the following simulation models

1. $\left(p=25,p_{\mathit{\text{Datagen}}}=5\right.$, uncorrelated x’s, SNR = 5): We simulate data according to the Gaussian linear model $\mathit{Y}~N\left(2\cdot 1+\mathit{X}\left(D\right)\beta \left(D\right),{\tau }^{-1}I\right)$ where 1 is a column of 1’s. We take the data generating model $ℳ\left(D\right)$ to be $\{\mathrm{1,2},\mathrm{3,4},5\},\beta \left(D\right)=(1.5,-\mathrm{1.5,1.5},-\mathrm{1.5,1.5})$ and $\tau =(1.5{)}^{-2}$. Each row of $X$ is independently generated from $N_p\left(0,{\Sigma }_X\right)$ where ${\Sigma }_X=I$ is taken to be isotropic. We consider three sample sizes $n=\mathrm{50,100}$ and a large $n=10000$ to explore selection consistency.

2. ($p=25,p_{\mathit{\text{Datagen}}}=5$, correlated x’s, SNR ≈ 2.58): Same as above but the rows of $X$ are generated from $N_p\left(0,{\Sigma }_X\right)$ so that $var\left(x_i\right)=1,cor\left(x_4,x_j\right)={\rho }^{1/2},j\ne 4$, and $cor\left(x_i,x_j\right)=\rho$ for all other $i\ne j$ (Fan & Lv, 2008). We take $\rho =0.5$.

3. $\left(p=25,p_{\mathit{\text{Datagen}}}=5\right.$, autoregressive correlated X’s, SNR ≈ 2.13): Same as above but the rows of $X$ are generated from $N_p\left(0,{\Sigma }_X\right)$ so that $var\left(x_i\right)=1$, and $cor\left(x_i,x_j\right)={\rho }^{|i-j|}$ for $i\ne j$. We take $\rho =0.5$

4. $\left(p=25,p_{\mathit{\text{Datagen}}}=10\right.$, uncorrelated x’s, SNR ≈ 5.43): Same as 1 but we take $\beta \left(D\right)=(0.7,-\mathrm{0.7,0.7},-\mathrm{1,1},-\mathrm{1,1},-\mathrm{1.5,1.5},-1.5)$ to consider a mixture of weak and moderate coefficients whereas keeping the SNR value similar to 1.

5. $\left(p=25,p_{\mathit{\text{Datagen}}}=10\right.$, correlated x’s, SNR ≈ 3.8) Same as 2 but with $\beta \left(D\right)$ same as 4.

6. $\left(p=25,p_{\mathit{\text{Datagen}}}=10\right.$, autoregressive correlated x’s, SNR ≈ 2.13): Same as 3 but with $\beta \left(D\right)$ same as 4.

For each scenario, we generate 100 replicated data sets and apply the different variable selection methods to each. We use $a_0=\text{max}{\left(n,p^2\right)}^{-1}$ in the g-prior (George & Foster, 2000). We further compare the complete model space enumeration based HPM selection with the MCMC method of HPM selection implemented in the R package BAS (https://cran.r-project.org/package=BAS) and the two approaches selected identical models in all the replicated cases we consider. The computation of the complete model space enumeration based HPM selection for $p=25$ and 100 replicated datasets is performed in a high performance computing cluster (HPC) with 60 nodes each equipped with 2 Intel X5650 2.66 GHz6-core processors and 72 GB RAM and took about 20 hours.

We notice in Table 2 that the HPM, DIC, BayesS5, SIS methods (except for Horseshoe) report high sensitivity values (often = 1) for almost all the cases; this suggests that these methods mostly select models from category II, that is, super-models of the data-generating model. For HPM and DIC, the high sensitivity values corroborates with Theorem 1.

The performance of the DIC is poor uniformly across Table 2 in selecting the data-generating model $ℳ\left(D\right)$. More importantly, the performance remains constantly poor even with increase in sample size $n$, and even for $n=\mathrm{10,000}$, corroborating with Theorem 3. The reported average FDR values in Table 2 suggest that DIC often chooses models with quite a few additional x-variables.

The HPM based selection using the $g$-prior perform superiorly across Table 2 and often performed similar to the nonlocal prior based HPM selection in BayesS5. We also note the performance improvement of HPM in selecting the data-generating model $ℳ\left(D\right)$ with increasing sample size $n$, consistent with Theorem 3.

All methods perform weakly for the case of non-autoregressive $\rho =0.5$, especially for small sample size and more in the case when $\beta \left(D\right)$ has a mixture of weak and moderate coefficients. The performances of HPM and BayesS5 based HPM, however, improve with increase in sample size. The Horseshoe approach generally perform weakly in Table 2 in presence of multicollineraity, but this could be partially due to our choice of default settings in the Horseshoe package. The SIS approach, surprisingly, does not perform strongly.

We note that Chen, Huang, Ibrahim, & Kim (2008) provide a simulation study comparing selection accuracy of DIC, HPM and other approaches in generalized linear models. Our results for $p=25$ are different from the good performances they observed for the case of $p=3$

4.2. Cancer Cachexia

We have described cancer cachexia in section 1 where we mention that prognostic value of cancer cachexia in cancer outcome is of increased interest. Fearon et al. (2011) described the formal consensus development of a framework for definition and classification of cancer cachexia by an international expert panel. Cancer cachexia is a “multifactorial syndrome defined by an ongoing loss of skeletal muscle mass (with or without loss of fat mass) that cannot be fully reversed by conventional nutritional support and leads to progressive functional impairment”. There is a substantial literature on cancer cachexia and its prognosic value. In a study on advanced non-small cell lung cancer (NSCLC) patients Patel et al. (2016) concluded that “weight gain during treatment may be an early indicator of clinical benefit”.

Neutrophils and lymphocytes are the two most abundant leukocytes, representing about 62% and 30% respectively of white blood cells. The neutrophil-to-lymphocyte ratio (NLR) is increasingly being used as a marker of subclinical inflammation. Multiple studies have explored the association of increased baseline NLR with poor clinical outcomes for several types of cancers, In Derman, Macklis, Azeem, Sayidine, Basu, Batus, Esmail, Borgia, Bonomi, & Fidler (2017), we established associations among longitudinal neutrophil-to-lymphocyte ratio, cancer cachexia and survival from NSCLC. We were involved in a follow-up study which involved $n=107$ advanced/metastatic, stage IIIB or IV, NSCLC patients treated first-line with platinum doublet chemotherapy at Rush University Medical Center in Chicago. The median age at baseline was 66 years and 52% of the patients were females. NSCLC histology was primarily adenocarcinoma (65%) and squamous cell carcinoma (18%). The median weight at baseline and at 6 weeks after treatment were 158.8 lbs and 156.9 lbs respectively 37% of patients gained weight on treatment from baseline to 6 weeks. In addition to their neutrophil and lymphocuyte counts, the levels of 33 biomarkers were measured by Luminex^® Multiplex Assays for the patients at baseline. These biomarkers are well known in the literature for their association with NSCLC. In our analysis, we consider the neutrophil to lymphocyte ratio (NLR) as prognostic for efficacy outcome and consider the association of the baseline level biomarkers with NLR.

The biomarkers in decreasing order of marginal correlation with neutrophil-to-lymohocyte ratio are Adipisin (a protease that stimulates glucose transport) Visfatin (also known as NAMPT, Nicotinamide phosphoribosyltransferase), MIP-1-$\beta$ (Macrophage inflammatory protein 1-$\beta$, also known as CCL4), GM-CSF (Granulocyte macrophage colony-stimulating factor), Adiponectin, Ghrelin etc. as listed in Table 3a. We note that these biomarkers are correlated among themselves as well, resulting in the potential of significant multicollinearity in the variable selection process.

Table 3:

(a): Correlation matrix of Neutrophil-to-Lymphocyte ratio (NLR) and 6 biomarkers which have highest marginal correlation with NLR. (b) Selected model for Weight Change and $p=33$ biomarkers. Reported log-Bayes factor is in reference to the HPM model

(a)
Biomarkers	NLR	Adipisin	Visfatin	MIP-1-$\mathit{\beta }$	GM-CSF	Adiponectin
Adipisin	−0.27
Visfatin	0.23	−0.11
MIP-1-$\beta$	−0.16	−0.02	0.04
GM-CSF	−0.15	−0.09	0.07	0.24
Adiponectin	0.14	0.04	0.06	−0.05	0.00
Ghrelin	0.13	0.00	0.26	−0.07	0.10	0.26

(b)
Method	Selected model	log Bayes factor
HPM	Adipisin, Visfatin	reference
DIC	Adipisin, Visfatin, MIP-1-$\beta$, GM-CSF, IL-21, IL-23, IL-8	−3.49
Nonlocal	IL-1-$\beta$, IL-21	−6.04
Horseshoe	Adipisin, IL-23, Adiponectin	−2.30
Median Prob Model	Adipisin, Visfatin	0
SIS	20 biomarkers	−28.90

Table 3b reports the results of different variable selection methods applied to these data. We applied the HPM and min-DIC based optimal model choice for $a_0=n^{-1}=0.009$. We also report nonlocal prior based variable selection (Shin et al., 2018) and model selected based on horseshoe prior(Carvalho et al., 2010; van der Pas et al., 2016) (both in their default settings). We further report the median probability model (MPM) whose optimality with respect to predictive loss function in the setting of orthogonal design matrix is established in Barbieri et al. (2004). Finally we report the model selected by the non-Bayesian sure independent screening (SIS (Fan & Lv, 2008)) method.

We note substantial differences among the models selected by the different variable selection methods. For example, the model selected by the nonlocal prior approach is distinctly different and does not include the variables listed in Table 3a which have high marginal correlations with the outcome. The model selected by the SIS method includes many more variables compared to the other methods. The model selected by the min-DIC approach is a super-model of the models selected by HPM or Median probability approaches and includes many additional variables. This finding is consistent with our results in section 3.2 and our simulation studies where we found that the DIC has high sensitivity but also has high false discovery rate in selecting additional variables. The HPM approach on the other hand, was found to have high sensitivity as well as low false discovery rate in those studies.

5. The case of known variance

For completeness, we briefly sketch here a set of parallel results when $\tau$ is known. The expressions of the marginal likelihood and the DIC under $g$-prior on $\beta \left(\alpha \right)$ are given by

$$\text{logML}\left(ℳ\right(\alpha \left)\right)=\frac{n}{2}\text{log}\left(\frac{\tau }{2\pi }\right)-\frac{\tau a_0}{2\left(1+a_0\right)}{\mathit{Y}}_n^{\prime }{\mathit{Y}}_n-\frac{\tau \text{SSE}\left(\alpha \right)}{2\left(1+a_0\right)}+\frac{1}{2}\text{log}\left(\frac{a_0}{1+a_0}\right)d\left(\alpha \right),\text{and}$$ (5.1)

$$\text{DIC}\left(\alpha \right)=-n\text{log}\left(\frac{\tau }{2\pi }\right)+\frac{a_0^2\tau }{{\left(1+a_0\right)}^2}{\mathit{Y}}_n^{\prime }{\mathit{Y}}_n+\frac{\left(1+2a_0\right)\tau }{{\left(1+a_0\right)}^2}\text{SSE}\left(\alpha \right)+\frac{2}{1+a_0}d\left(\alpha \right),$$ (5.2)

for the choice of the posterior mean $\stackrel{-}{\mathit{\beta }}\left(\alpha \right)$ as the posterior estimate $\tilde{\mathit{\beta }}\left(\alpha \right)$ (Chen, Huang, Ibrahim, & Kim, 2008). The following result parallels Theorem 1.

Theorem 5. If ${ℳ}_1$ and ${ℳ}_2\cup \left\{ℳ\right(D\left)\right\}$ are not empty,

1. $\underset{n\to \infty }{\text{lim}}{P}_{{\mathit{Y}}_n}\left\{\underset{ℳ\left(\alpha \right)\in {ℳ}_2\cup \left\{ℳ\right(D\left)\right\},ℳ\left({\alpha }^{\prime }\right)\in {ℳ}_1}{\text{min}}\right.\mathit{\text{Bayes}}\mathit{\text{factor}}\left.\left(ℳ\left(\alpha \right),ℳ\left({\alpha }^{\prime }\right)\right)\ge 1\right\}=1$.

2. $\underset{n\to \infty }{\text{lim}}{P}_{{\mathit{Y}}_n}\left\{\underset{ℳ\left({\alpha }^{\prime }\right)\in {ℳ}_1}{\text{min}}DIC\left(ℳ\right(\alpha \left)\right)\ge \underset{ℳ\left(\alpha \right)\in {ℳ}_2\cup \left\{ℳ\right(D\left)\right\}}{\text{max}}DIC\left(ℳ\right(\alpha \left)\right)\right\}=1$.

The next set of results consider mis-selection probabilities and parallel Theorem 3

Theorem 6. For $ℳ\left(\alpha \right)\in {ℳ}_2$, and $a_0=a_0\left(n\right)\to 0$ as $n\to \infty$,

$$\underset{n\to \infty }{\text{lim}}{P}_{{\mathit{Y}}_n}\left\{\mathit{\text{Bayes}}\mathit{\text{factor}}\right(ℳ\left(\alpha \right),ℳ\left(D\right))\ge 1\}=0.$$

In contrast, we have the following result for DIC.

Theorem 7. For $ℳ\left(\alpha \right)\in {ℳ}_2$, any $a_0>0$ and any $n$,

$$P_{{\mathit{Y}}_n}\left\{DIC\right(ℳ\left(\alpha \right))\le DIC(ℳ\left(D\right)\left)\right\}\ge 1-F_{{\chi }_{\eta }^2}\left(2\eta \right).$$

These results are illustrated in Figure 2 in which the marginal likelihood, DIC and LPML of the data generating model $ℳ\left(D\right)$ and different $ℳ\left(\alpha \right)$ models were evaluated for 2,500 repeated data simulations for different sample sizes, and for the choice of $a_0=1/n$. The figure shows the empirical estimates of mis-selection probabilities based on repeated data simulations (red) and evaluations based on analytical expressions (blue). We note that the mis-selection probabilities of Bayes factor are relatively low across all three panels and decrease to zero with increasing sample size as established in Theorem 6. In comparison, the mis-selection probabilities for both DIC and LPML are relatively high and stay relatively flat. The fact that mis-selection probability of DIC is bounded from below is established in Theorem 7.

Figure 2:

Red ● are empirical estimates of (a) $P_{{\mathit{Y}}_n}\left[\text{DIC}\right(ℳ\left(\alpha \right))\le \text{DIC}(ℳ\left(𝒟\right)\left)\right]$ (left panel) and (b) $P_{Y_n}[$ Bayes factor $\left(ℳ\right(\alpha ),ℳ(D\left)\right)\ge 1]$ (right panel) based on 2,500 repeated data simulations from data generating model $ℳ\left(D\right)=\left\{x_3,x_4\right\}$. Blue ● are corresponding probabilities calculated from analytic expressions. (i) Left-subpanel $ℳ\left(\alpha \right)=\left\{x_1,x_3,x_4\right\}$, (ii) middle-subpanel $ℳ\left(\alpha \right)=\left\{x_2,x_3,x_4\right\}$ and (iii) right-subpanel $ℳ\left(\alpha \right)=\left\{x_1,x_2,x_3,x_4\right\}$.

Remark: In fact, for any $ℳ\left(\alpha \right)$ in $ℳ\left(D\right)\cup {ℳ}_2$, it can be shown that ${\text{lim}}_{n\to \infty }\frac{1}{n}\text{DIC}\left(\alpha \right)\stackrel{p}{\to }1+\frac{\tau a_0^2}{{\left(1+a_0\right)}^2}{\mu }^{\prime }\mu -\text{log}\left(\frac{\tau }{2\pi }\right)$. Since the right hand side is identical for every $ℳ\left(\alpha \right)$, DIC cannot differentiate among the models in ${ℳ}_2$ in this sense. This result parallels a result in Shao (1993) who showed that predictive squared error criterion under one-deleted cross-validation converges to an identical constant for every category II model.

We conclude this section with a result which shows that mis-selection by marginal likelihood for a given data ${\mathit{y}}_n$, in fact, deterministically implies mis-selection by DIC. The reverse however is not established, suggesting that the set of ${\mathit{y}}_n$ resulting in mis-selection by DIC is potentially much bigger.

Theorem 8. For any $a_0>0$,

$$\left\{{\mathit{y}}_n:\text{Bayes}\mathit{\text{factor}}\left(ℳ\left(\alpha \right),ℳ\left(D\right)\mid {\mathit{y}}_n\right)\ge 1\right\}\subseteq \left\{{\mathit{y}}_n:\mathit{\text{DIC}}\left(ℳ\left(\alpha \right)\mid {\mathit{y}}_n\right)\le \mathit{\text{DIC}}\left(ℳ\left(D\right)\mid {\mathit{y}}_n\right)\right\}.$$

Theorem 8 is illustrated in Figure 3 in which the data-generating model $ℳ\left(D\right)=\left\{x_3,x_4\right\}$ and $ℳ\left(\alpha \right)=\left\{x_1,x_3,x_4\right\}$ of the Gunst and Mason data (see Section 2). We evaluated the marginal likelihood and DIC of these two models for 2,500 repeated data simulations. The first quadrant of Figure 3 is empty which is the important finding of Theorem 8. The fourth quadrant shows the cases log Bayes factor $\left(ℳ\right(\alpha ),ℳ(D\left)\right)\ge 0$, and $\text{DIC}\left(ℳ\right(\alpha \left)\right)-\text{DIC}\left(ℳ\right(D\left)\right)\le 0$ and the third quadrant highlights the cases where $\text{DIC}\left(ℳ\right(\alpha \left)\right)-\text{DIC}\left(ℳ\right(D\left)\right)\le 0$ but log Bayes factor $\left(ℳ\right(\alpha ),ℳ(D\left)\right)\nleqq 0$.

Figure 3:

log Bayes Factor $\left(ℳ\right(\alpha ),ℳ(D\left)\right)$ in the horizontal axis vs $\text{DIC}\left(ℳ\right(\alpha \left)\right)-\text{DIC}\left(ℳ\right(D\left)\right)$ in the vertical axis. $ℳ\left(\alpha \right)=\left\{x_1,x_3,x_4\right\}$ and $ℳ\left(D\right)=\left\{x_3,x_4\right\}$. Plot based on 1000 repeated data simulations.

6. Generalized linear model

In previous sections, we study properties of Bayesian selection criteria in linear models. It is of interest to examine how these selection criteria compare in nonlinear, models. In this subsection, we provide empirical investigation of the performance of Bayesian selection criteria in generalized linear models (GLMs) with binary outcome.

GLMs assume distributions within the exponential family (McCullagh and Nelder 1989), with density

$$p\left({\mathit{Y}}_n\right)=\prod _{i=1}^n\text{exp}\left\{\frac{1}{a\left(\varphi \right)}\left(Y_i{\theta }_i-b\left({\theta }_i\right)\right)+c\left(Y_i,\varphi \right)\right\}$$ (6.1)

where $a(\cdot ),b(\cdot )$ and $c(\cdot )$ are specific functions and $\varphi$ is a scaling parameter. In model $ℳ\left(\alpha \right)$, the association of the stochastic outcome ${\mathit{Y}}_n$ with ${\mathit{X}}_n\left(\alpha \right)$ is expressed in terms of $E\left[{\mathit{Y}}_n\mid {\mathit{X}}_n\left(\alpha \right)\right]$ via the regression model

$$g\left(E\left[{\mathit{Y}}_n\mid {\mathit{X}}_n\left(\alpha \right)\right]\right)={\mathit{X}}_n\left(\alpha \right)\mathit{\beta }\left(\alpha \right).$$ (6.2)

where $g(\cdot )$ is the link function of the GLM.

Chen, Huang, Ibrahim, & Kim (2008) provide review and comparison of Bayes factor, DIC, LPML and other criteria in the setting of generalized linear models with conjugate power prior. Other works on Bayesian criterion based methods in generalized linear models include Meyer & Laud (2002) and Chen, Dey, & Ibrahim (2004). There is a substantial literature on g-priors for GLMS and several variants have been suggested, see Li & Clyde (2018) for a recent review. Unlike the case of normal linear model, the information matrix in model $ℳ\left(\alpha \right)$ for GLM depends on $\mathit{\beta }\left(\alpha \right)$. The alternative g-priors vary depending on whether the expected or observed information is used in the prior and whether it is considered at $\mathit{\beta }\left(\alpha \right)=0$ or at the maximum likelihood estimate $\widehat{\mathit{\beta }\left(\alpha \right)}$ for model $ℳ\left(\alpha \right)$. We consider a g-prior proposed in Li & Clyde (2018) based on its geometric interpretability and computational efficiency and is given by

$$\mathit{\beta }\left(\alpha \right)~N\left(0,\frac{1}{a_0}{ℐ}_n\left(\widehat{\mathit{\beta }}\right(\alpha \left)\right)\right)$$

Here ${ℐ}_n\left(\widehat{\mathit{\beta }}\right(\alpha \left)\right)$ is the observed information matrix for $\mathit{\beta }\left(\alpha \right)$ excluding the intercept.

6.1. Study 1:

We consider the Pima Indian dataset on onset of Diabetes Mellitus (Smith et al., 1988). The data are collected from the Pima Indian population near Phoenix, Arizona who has been under study since 1965 by the National Institute of Diabetes and Digestive and Kidney Diseases due to high incidence rate of diabetes. The outcome variable $y_i,i\le 200$ is an indicator of Diabetes Mellitus onset, diagnosed according to World Health Organization Criteria, among females of ≥ 21 years of age. We consider five covariates: number of times pregnant $x_2$, plasma glucose concentration at 2 Hours in an oral glucose tolerance test $x_3$, body mass index $x_4$, diabetes pedigree function $x_5$ and age in years $x_6$. This dataset has been extensively studied. We also consider a randomly selected subsample of $n=100$ to examine performance under a smaller sample size.

In this study, we simulate binary data vector $\mathit{y}$ from logistic regression models based on observed $\mathit{x}$ values and the results reported in Table 5 are based on 100 repeated data simulations. The model space $ℳ$ consists of all possible models which always includes the first covariate $x_1\equiv 1$. For each data simulation, we report the models with the optimal DIC (smallest) and marginal likelihood (highest) by complete enumeration of all models in the space $ℳ$. For comparison, we also report the highest LPML model. The relative frequency of the optimally selected models by DIC, marginal likelihood and LPML over the repeated simulations are reported in Table 5. We estimate the DIC and LPML for each model by Markov chain sampling and evaluate approximate marginal likelihood based on an integrated Laplace approximation considered in Li & Clyde (2018) which yields a closed form expression.

Table 5:

Relative frequency of optimally selected models by DIC, marginal likelihood (ML) and LPML for Pima Indian data

	$ℳ\left(D\right)=\{\mathrm{1,3},5\}$			$ℳ\left(D\right)=\{\mathrm{1,3},\mathrm{5,6}\}$			$ℳ\left(D\right)=\{\mathrm{1,3},\mathrm{4,5},6\}$			$ℳ\left(D\right)=\{\mathrm{1,2},\mathrm{3,4},\mathrm{5,6}\}$
	DIC	HPM	LPML	DIC	HPM	LPML	DIC	HPM	LPML	DIC	HPM	LPML
Subsample of $n=100$
category I				0.05	0.02	0.01	0.08	(0.24)	0.17	0.03	0.03	0.18
$ℳ\left(D\right)$	0.43	0.86	0.49	0.69	0.90	0.81	0.55	0.76	0.48	0.97	0.97	0.82
category II	0.57	0.14	0.51	0.26	0.08	0.18	0.37		0.35
$n=200$
category I								0.01			0.02	0.01
$ℳ\left(D\right)$	0.46	0.91	0.48	0.71	0.97	0.73	0.65	0.94	0.60	1.00	0.98	0.99
category II	0.54	0.09	0.52	0.29	0.03	0.23	0.35	0.05	0.40

As in section 2, we consider four different scenarios: (i) the data-generating model $ℳ\left(D\right)=\left\{x_1,x_3,x_5\right\}$, (ii) $ℳ\left(D\right)=\left\{x_1,x_3,x_5,x_6\right\}$, (iii) $ℳ\left(D\right)=\left\{x_1,x_3,x_4,x_5,x_6\right\}$, and (iv) $ℳ\left(D\right)=\left\{x_1,x_2,x_3,x_4,x_5,x_6\right\}$, the full model. We first note that, in contrast to the case of linear models in section 2, the optimally selected models belong to category I ${ℳ}_1$ for a non-negligible proportion of times in case of smaller $n=100$. This is especially evident when $ℳ\left(D\right)=\left\{x_1,x_3,x_4,x_5,x_6\right\}$ for all three, DIC, HPM and LPML, criteria. For the case of larger $n=200$, category I models are substantially less frequently selected.

The superior performance of the marginal likelihood in selecting the data-generating model in the setting of this generalized linear model is evident across Table 5. Paralleling the case of linear model, DIC and LPML performs poorly when the data-generating model $ℳ\left(D\right)=\left\{x_1,x_3,x_4\right\}$ is relatively sparse whereas their performances improve as $ℳ\left(D\right)$ becomes less sparse. Also paralleling the liner model case, there is little improvement on the performance of the DIC with increase in sample size $n$, whereas the performance of HPM improves in most of the cases.

6.2. Study 2:

The design for generating covariate $\mathit{X}$ in this study is similar to section 4.1. Each row of $\mathit{X}$ is independently generated from $N_p\left(0,{\Sigma }_X\right)$ where (i) ${\Sigma }_X=I$ is taken to be isotropic for the uncorrelated case, (ii) $cor\left(x_i,x_j\right)=\rho ,i\ne j$ for the equi-correlated case and (iii) $cor\left(x_i,x_j\right)={\rho }^{|i-j|},i\ne j$ for the autoregressive case. The binary outcome $\mathit{y}$ is then generated from a logistic regression model using a subset $\mathit{X}\left(D\right)$ of the $\mathit{X}$ matrix. We consider $p=12$ and two cases of $n=200$ and $n=800$. We further consider the cases of $\left(a\right)p_{\mathit{\text{datagen}}}=4$ when the data-generating model is relatively sparse and $\left(b\right)p_{\mathit{\text{datagen}}}=10$.

The results are given in Table 6 and we first note that both DIC and HPM report average sensitivity of 1 in each of the cases, thus implying neither criteria select a category I model as optimal in any of the 100 replications and in any of the cases. In general, the performance of the DIC is quite poor in selecting the data-generating model and the performance does not improve even with a four times increase in sample size. The HPM, on the other hand, performs superiorly in selecting the data generating model in this setting of generalized linear model and the performance improves with increase in sample size. As we noted before, Chen, Huang, Ibrahim, & Kim (2008) provided a simulation study comparing selection accuracy of DIC, HPM and other approaches in Poisson regression model for the case of $p=3$. Our results for larger $p$’s in Tables 5 and 6 are different from the good performances they observed for the case of $p=3$

Table 6:

Relative frequency of selecting data generating model $ℳ\left(D\right)$, false discovery rate (FDR), and sensitivity (SEN) in logistic regression. Results based on averaged over 100 replications.

	DIC			HPM			DIC			HPM
	$ℳ\left(D\right)$	FDR	SEN	$ℳ\left(D\right)$	FDR	SEN	$ℳ\left(D\right)$	FDR	SEN	$ℳ\left(D\right)$	FDR	SEN
	$n=200,p_{\mathit{\text{datagen}}}=4$						$n=200,p_{\mathit{\text{datagen}}}=10$
$\rho =0$	0.00	0.36	1.00	0.76	0.05	1.00	0.26	0.07	1.00	0.94	0.01	1.00
$\rho =0.5$	0.01	0.32	1.00	0.86	0.03	1.00	0.15	0.08	1.00	0.98	0.00	1.00
AR	0.00	0.34	1.00	0.87	0.03	1.00	0.06	0.10	1.00	0.84	0.02	1.00
	$n=800,p_{\mathit{\text{datagen}}}=4$						$n=800,p_{\mathit{\text{datagen}}}=10$
$\rho =0$	0.01	0.34	1.00	0.97	0.01	1.00	0.16	0.08	1.00	0.95	0.00	1.00
$\rho =0.5$	0.02	0.35	1.00	0.96	0.01	1.00	0.09	0.08	1.00	0.98	0.00	1.00
AR	0.02	0.32	1.00	0.95	0.01	1.00	0.17	0.08	1.00	0.94	0.01	1.00

7. Conclusion

In our theoretical investigations and especially in the numerical studies, we find that the marginal likelihood based HPM approach performs superiorly overall in selecting the data-generating model. Some undesirable properties of the DIC are well-known but we are surprised by the degree of poor performance of the DIC in identifying the data generating model in the numerical examples. This is especially disconcerting in the generalized linear model in section 6 as practitioners often choose to use DIC that is readily available in software as opposed to the complication of marginal likelihood computation in this non-conjugate setting.

In models with latent structures, such as mixed effects, hierarchical or latent variable models, a series of works considered marginalized DIC based on observed-data likelihood as opposed to conditional likelihood (Celeux et al., 2006; Chan & Grant, 2016; Quintero & Lesaffre, 2018; Merkle et al., 2019; Ariyo et al., 2019, 2020; Li et al., 2020). Comparative performance of Bayesian variable selection criteria in these settings is an interesting topic of further research.

We additionally study performances of WAIC (Watanabe-Akaike information criterion, Watanabe & Opper (2010); Gelman et al. (2014)) as well as performances of DIC and HPM under a non-conjugate double-exponential prior in the setting of section 2 following recommendations of anonymous reviewers. We find that HPM performs superior to WAIC in selecting the data generating model. The difference in performances between HPM and DIC approaches persists almost in the same scale under the double-exponential prior.

In high-dimensional variable selection with many $x$-variables, criterion based variable selection approaches face the challenge of a large model space whose enumeration can be infeasible. Recent works on intelligent and stochastic search strategies on the model space (Shin et al., 2018; Maity, 2016) have made significant progress on the use of criterion based selection in such settings.