A Bayesian Approach for Graph-constrained Estimation for High-dimensional Regression

Abstract

Many different biological processes are represented by network graphs such as regulatory networks, metabolic pathways, and protein-protein interaction networks. Since genes that are linked on the networks usually have biologically similar functions, the linked genes form molecular modules to affect the clinical phenotypes/outcomes. Similarly, in large-scale genetic association studies, many SNPs are in high linkage disequilibrium (LD), which can also be summarized as a LD graph. In order to incorporate the graph information into regression analysis with high dimensional genomic data as predictors, we introduce a Bayesian approach for graph-constrained estimation (Bayesian GRACE) and regularization, which controls the amount of regularization for sparsity and smoothness of the regression coefficients. The Bayesian estimation with their posterior distributions can provide credible intervals for the estimates of the regression coefficients along with standard errors. The deviance information criterion (DIC) is applied for model assessment and tuning parameter selection. The performance of the proposed Bayesian approach is evaluated through simulation studies and is compared with Bayesian Lasso and Bayesian Elastic-net procedures. We demonstrate our method in an analysis of data from a case-control genome-wide association study of neuroblastoma using a weighted LD graph.

1 Introduction

Partially motivated by genomic applications, regularization methods for variable selection and coefficient estimation for linear regression models have been extensively studied in recent years, including the popular procedures such as Lasso [1] and SCAD [2] and their various extensions. The fused Lasso [3] imposes the L₁ penalty on the absolute differences of the regression coefficients in order to account for some smoothness of the regression coefficients when the covariates are naturally ordered. For grouped variables, Yuan and Lin [4] proposed the group Lasso, where the penalty function is intermediate between the L₁ penalty and L₂ penalty. Zou and Hastie [5] proposed the Elastic-net as a stabilized version of the Lasso. Particularly, it is useful for the analysis of genomic data because it deals with groups of highly correlated variables.

One limitation of all these regularized regression approaches is that they do not provide valid variance estimates automatically. Methods based on data splitting have recently been proposed to obtain valid p-values for high-dimensional regression [6]. However, these methods requires large sample sizes for data splitting. Alternatively, the Lasso estimates could be interpreted as the Bayes posterior mode under independent Laplace priors for the predictors. Park and Casella [7] have recently developed a Bayesian approach using the Gibbs sampler for the Lasso estimates with the Laplace prior in the hierarchical model. They showed that the posterior median estimates for regression coefficients in their model are similar to the ordinary Lasso estimates. Such a Bayesian Lasso can provide interval estimates, i.e., Bayesian credible intervals, along with their standard errors for the regression coefficient estimates. This is a great advantage over the ordinary Lasso, which does not provide any confidence assessment of the point estimates of the regression coefficients. Yi and Xu [8] have recently applied the Bayesian Lasso to the problem of mapping multiple quantitative trait loci (QTL). Since the traditional linear regression model is used to relate genotypes to phenotypes, the Bayesian Lasso can fit the model to estimate all possible genetic effects associated with all molecular markers across the entire genome.

Besides the QTL mapping problems, gene expression data are often studied in a linear regression framework with phenotypes as responses and numerical measurements on genes as predictors. Often external information about biological graphs such as regulatory networks, metabolic pathways and protein-protein interaction networks is available and provides additional information about how genes are related on the networks. Similarly, in large-scale genetic association studies, many SNPs are in high linkage disequilibrium (LD), which can also be summarized as a LD graph [9]. In order to use this extra graph information in analysis of genomics data, a graph-constrained estimation (GRACE) procedure [10, 11] was proposed for fitting linear regression models and variable selection. It uses a Laplacian matrix-based penalty to incorporate a priori information of graph structure into regression analysis. The cyclic coordinate descent algorithm [12] is used to provide efficient algorithm for GRACE estimates. However, GRACE has the same limitation as other regularization methods that do not provide variance estimates or confidence intervals of the estimates.

In this paper, extending the idea of Bayesian Lasso [7], we develop a Bayesian formulation for the graph-constrained estimation for high-dimensional linear and probit regression models, which provides posterior estimates of the regression coefficients and valid standard errors. We call this procedure the Bayesian GRACE. We develop an efficient Markov chain Monte Carlo (MCMC) procedure for sampling the posterior distributions of the parameters. We use the deviance information criterion (DIC) [13] to choose the tuning parameters and to assess the model. Variable selection problem is resolved through Bayesian credible intervals for coefficients that do not include zero. The performance of the proposed approach is evaluated and compared with the Bayesian Lasso and Bayesian Elastic-net through simulated data sets. We also demonstrate our method in an analysis of data from a case-control genome-wide association study (GWAS) of neuroblastoma incorporating with a weighted LD graph defined as in [9].

The rest of the paper is organized as follows. We first briefly review the graph-constrained estimation for linear models [10, 11]. We then introduce a Bayesian hierarchical model with graph constraints on coefficients and present an efficient Gibbs sampling approach to obtain the posterior distributions of the regression coefficients. We also present a similar formulation for the probit models for binary outcomes. We present simulation studies and application to a real data set. Finally, we give a brief discussion and conclusions.

2 Graph-constrained Regularization for Linear Models

We first briefly describe the graph-constrained estimation procedure [10, 11]. Consider a standard linear regression model with n observations and p predictors:

$$Y=\mathit{X}\mathit{\beta }+\mathit{\epsilon },$$

where Y = (y₁, …, y_n)^⊤ ∈ ℝⁿ, X = (X₁, …, X_p) ∈ ℝ^n×p, X_j = (x_1j, …, x_nj)^⊤, β = (β₁, …, β_p)^⊤, and ε ~ N_n(0, σ²I_n). We assume that the predictors and the response are centered so that

$$\sum _{i=1}^n{y}_i=0,\sum _{i=1}^n{x}_{ij}=0,\text{and}\sum _{i=1}^n{x}_{ij}^2=n$$

for j = 1, …, p.

Assume that the p predictors are nodes of a graph, which correspond to some prior known genetic networks or weighted LD graph. Denote the graph as G = (V, E), where V is the set of vertices that correspond to the p predictors, and E = (u ~ v) is the set of edges indicating that the predictors u and v are linked on the graph and there is an edge between u and v. Then, the Laplacian matrix represents linked or unlinked genes on the graph. The normalized weighted Laplacian matrix L = (l_uv)_p×p is defined as

$$l_{uv}=\{\begin{array}{ll}1-w(u,u)/d_u\hfill & \text{if}u=v\text{and}{d}_u\ne 0\hfill \\ -w(u,v)/\sqrt{d_u{d}_v}\hfill & \text{if}u\text{and}v\text{are}\text{adjacent}\hfill \\ 0\hfill & \text{otherwise},\hfill \end{array}$$

where d_u = Σ_u~v w(u, v) is the degree of the vertex u. Note that u is an isolated vertex if d_u = 0. Let w(u, u) = 0. It is well-known that the matrix L is semi-positive definite with 0 as the smallest eigenvalue and 2 as the largest eigenvalue [14]. In order to incorporate the graph information into regression analysis, Li and Li [10] proposed an improper Gaussian Markov random field priors on β, i.e.,

$$\pi (\mathit{\beta }\mid {\sigma }^2)\propto exp\left(-\frac{1}{2{\sigma }^2}{\mathit{\beta }}^{\top }L\mathit{\beta }\right).$$

Note that

$${\mathit{\beta }}^{\top }L\mathit{\beta }=\sum _{u~v}{\left(\frac{{\beta }_u}{\sqrt{d_u}}-\frac{{\beta }_v}{\sqrt{d_v}}\right)}^2,$$

which can be regarded as a smoothness measurement of the regression coefficients with respect to the graph structure. Li and Li [11] proposed the following graph-constrained regularization procedure for variable selection and for parameter estimation,

$$\widehat{\mathit{\beta }}=arg\underset{\mathit{\beta }}{min}\left[{(Y-\mathit{X}\mathit{\beta })}^{\top }(Y-\mathit{X}\mathit{\beta })+{\lambda }_1\sum _{j=1}^p\mid {\beta }_j\mid +{\lambda }_2{\mathit{\beta }}^{\top }L\mathit{\beta }\right],$$ (1)

where the tuning parameters λ₁ and λ₂ control the amount of regularization for sparsity and smoothness, respectively. When λ₂ = 0, the graph-constrained penalty reduces to the ordinary Lasso penalty, and when L = I, the penalty becomes the Elastic-net penalty [5].

The graph-constrained estimation as presented in equation (1) adjusts the coefficients in order to account for different degrees of the vertices on the graph, allowing the genes with more connections such as the hub genes to have larger coefficients so that small changes of expressions of such genes can lead to large changes in the response. Biologically, the genes that are linked on the networks often have similar functions and have correlated expression levels so they are expected to have smoothed regression coefficients. Friedman et al. [12] presented a cyclic coordinate descent algorithm for solving Lasso and Elastic-net regularization. Li and Li [11] presented a similar coordinate descent algorithm to the problem (1) and employed a cross-validation method for selecting the tuning parameters.

3 A Bayesian Approach to Graph-constrained Estimation and Regularization

In this section, we present a Bayesian hierarchical model for the graph-constrained estimation (Bayesian GRACE) and present a Gibbs sampling procedure to obtain the posterior distributions of the regression coefficients. Our development is similar to Bayesian Lasso [7]. We consider both high-dimensional linear and probit regression models.

3.1 A hierarchical graph-constrained estimation

In the penalized linear regression with the graph-constrained penalty, the conditional prior of β, given σ² can be expressed as

$$\pi (\mathit{\beta }\mid {\sigma }^2)\propto exp\left(-\frac{{\lambda }_1}{\sqrt{{\sigma }^2}}\sum _{j=1}^p\mid {\beta }_j\mid -\frac{{\lambda }_2}{2{\sigma }^2}{\mathit{\beta }}^{\top }L\mathit{\beta }\right),$$ (2)

where L is the Laplacian matrix defined for the graph. Similar to the original Lasso, this prior can be attained from the basic identity of the Laplace distribution,

$$\frac{a}{2}{e}^{-a\mid z\mid }={\int }_0^{\propto }\frac{1}{\sqrt{2\pi s}}{e}^{z^2/(2s)}\frac{a^2}{2}{e}^{-a^{2}s/2}ds,a>0,$$

i.e., a scaled mixture of normals with an exponential mixing density is equivalent to the Laplace distribution. Specifically, the Bayesian formulation of the graph-constrained estimation is then given by the following hierarchical model

$$\begin{array}{l}Y\mid \mathit{X},\mathit{\beta },{\sigma }^2~N_n(\mathit{X}\mathit{\beta },{\sigma }^2{I}_n),\\ \mathit{\beta }\mid L,{\sigma }^2,{\tau }_1^2,\dots ,{\tau }_p^2,{\lambda }_2~N_p(0,{\sigma }^2\mathrm{\sum }),\mathrm{\sum }={(D^{-1}+{\lambda }_{2}L)}^{-1}\\ {\tau }_j^2\mid {\lambda }_1^2~exp({\lambda }_1^2/2),j=1,\dots ,p,\\ {\sigma }^2~\pi ({\sigma }^2)d{\sigma }^2\end{array}$$

where $D=\text{diag}({\tau }_1^2,\dots ,{\tau }_p^2)$. Any inverse Gamma prior for σ² would maintain conjugacy, including the improper prior density π(σ²) = 1/σ². After integrating out ${\tau }_1^2,\dots ,{\tau }_p^2$ and using the scaled mixture of normals to define the Laplace distribution, the conditional prior on β has the desired form (2). Park and Casella [7] has pointed out that conditioning on σ² is important because it guarantees a unimodal full posterior. Lack of unimodality slows convergence of the Gibbs sampler and makes point estimates less meaningful.

Using the improper prior density π(σ²) = 1/σ², the full conditional posterior distributions of β, ${\tau }_1^2,\dots ,{\tau }_p^2$, and σ² with fixed ${\tau }_1^2$ and λ₂ are then

$$\begin{array}{l}\mathit{\beta }\mid \cdots ~N_p({({\mathit{X}}^{\top }\mathit{X}+{\mathrm{\sum }}^{-1})}^{-1}{\mathit{X}}^{\top }Y,{\sigma }^2{({\mathit{X}}^{\top }\mathit{X}+{\mathrm{\sum }}^{-1})}^{-1}),\\ {\tau }_j^{-2}\mid \cdots ~\text{Inverse}\text{Gaussian}\left(\sqrt{{\tau }_1^2{\sigma }^2{\beta }_j^{-2}},{\lambda }_1^2\right),j=1,\dots ,p,\\ {\sigma }^2\mid \cdots ~\text{Inverse}\text{Gamma}\left(\frac{n+p}{2},\frac{1}{2}{(Y-\mathit{X}\mathit{\beta })}^{\top }(Y-\mathit{X}\mathit{\beta })+\frac{1}{2}{\mathit{\beta }}^{\top }{\mathrm{\sum }}^{-1}\mathit{\beta }\right).\end{array}$$ (3)

These full conditional distributions form the basis for an efficient Gibbs sampler, so they are easy to sample. The posterior distribution of the univariate β_j also can be obtained as

$${\beta }_k\mid {\beta }_{-k},\cdots ~N\left(\frac{{\sum }_{i=1}^n{x}_{ik}(y_i-{\sum }_{j\ne k}{x}_{ij}{\beta }_j)-{\lambda }_2{\sum }_{j\ne k}{\beta }_j{l}_{kj}}{n+{\lambda }_2{l}_{kk}+{\tau }_k^{-2}},\frac{{\sigma }^2}{n+{\lambda }_2{l}_{kk}+{\tau }_k^{-2}}\right),$$ (4)

after some algebra. Updating each β_j becomes simple and fast. It is clear from this conditional distribution that larger λ₁ leads to larger ${\tau }_k^{-2}$ and therefore bigger regularization and bigger shrinkage of β_k. On the other hand, larger value of λ₂ leads to stronger influence of the values of the regression coefficients of the neighboring genes.

It is also interesting to notice that the mean of the conditional distribution for β_k given by equation (4) is closely related to the soft-thresholding update formula based on the cyclic coordinate descent algorithm [11], which is given by

$${\beta }_k\leftarrow \frac{S(\frac{1}{n}{\sum }_{i=1}^n{x}_{ik}(y_i-{\sum }_{j\ne k}{x}_{ij}{\beta }_j))-{\lambda }_2{\sum }_{j\ne k}{\beta }_j{l}_{kj},{\lambda }_1}{1+{\lambda }_2},$$

where S(z, γ) is the soft-thresholding operator with value

$$\mathit{sign}(z){(\mid z\mid -\gamma )}_{+}=\{\begin{array}{ll}z-\gamma \hfill & \text{if}z>0\text{and}\gamma <\mid z\mid \hfill \\ z+\gamma \hfill & \text{if}z<0\text{and}\gamma <\mid z\mid \hfill \\ 0\hfill & \text{otherwise}.\hfill \end{array}$$

This link explains both the similarity and the key difference between the regularization method [10] and the Bayesian GRACE proposed here: the estimates for the coefficients of the relevant variables should be similar, however, the Bayesian GRACE procedure can lead to very small estimates of β but not exactly zero and therefore does not automatically select the variables. The problems of selecting the tuning parameters and variables are discussed in Section 3.3.

3.2 A Bayesian approach for graph-constrained estimation for binary response

Binary data such as absence/presence of some disease and positive/negative of a treatment are often considered as response variables in many biomedical studies. The logistic regression model is most commonly used for binary response, but including Y directly to the hierarchical model is not feasible because Y follows a Bernoulli distribution. Albert and Chib [15] suggested a standard Gaussian link function and a latent variable framework to estimate posterior distribution for β using the Gibbs sampler.

Let us denote Xβ = (g₁, …, g_n)^⊤, and suppose that we have latent variables Z = (z₁, z₂, …, z_n)^⊤ from independent normal distribution with a mean of g_i and unit variance. If we assume that

$$y_i\mid \mathit{X},\mathit{\beta }~\mathit{Bernoulli}(p_i),p_i=P(z_i>0)=\mathrm{\Phi }(g_i),$$

then it can be easily shown that

$$f(z_i\mid Y,\mathit{X},\mathit{\beta })=\{\begin{array}{ll}\frac{\phi (z_i-g_i)}{\mathrm{\Phi }(g_i)}{1}_{(z_i>0)},\hfill & \text{if}{y}_i=1\hfill \\ \frac{\phi (z_i-g_i)}{1-\mathrm{\Phi }(g_i)}{1}_{(z_i\le 0)},\hfill & \text{if}{y}_i=0,\hfill \end{array}$$

where φ (·) is a standard normal density function, Φ (·) is a standard normal distribution function, and 1_(·) is an indicator function. The conditional distribution of z_i, given Y, X, and β is then a truncated normal distribution.

To complete a Gibbs sampler, these latent variables are sampled from

$$z_i\mid Y,\mathit{X},\mathit{\beta }~{[N^{-}(g_i,1)]}^{y_i}{[N^{+}(g_i,1)]}^{1-y_i},$$

where N⁻ is a truncated normal distribution at the left by 0 and N⁺ is a truncated normal distribution at the right by 0.

We can still easily update β, ${\tau }_1^2,\dots ,{\tau }_p^2$, and σ² from

$$\begin{array}{l}\mathit{\beta }\mid \cdots ~N_p\left({({\mathit{X}}^{\top }\mathit{X}+{\mathrm{\sum }}^{-1})}^{-1}{\mathit{X}}^{\top }\stackrel{\sim }{Z},{\sigma }^2{({\mathit{X}}^{\top }\mathit{X}+{\mathrm{\sum }}^{-1})}^{-1}\right),\\ {\tau }_j^{-2}\mid \cdots ~\text{Inverse}\text{Gaussian}\left(\sqrt{{\lambda }_1^2{\sigma }^2{\beta }_j^{-2}},{\lambda }_1^2\right),j=1,\dots ,p\\ {\sigma }^2\mid \cdots ~\text{Inverse}\text{Gamma}\left(\frac{p}{2},\frac{{\mathit{\beta }}^{\top }{\mathrm{\sum }}^{-1}\mathit{\beta }}{2}\right),\end{array}$$

where Z̃ = Z − z̄1_n.

3.3 Variable and tuning parameters selection

Bayesian estimation based on the result of a Gibbs sampler consists of posterior means or medians and the standard errors through posterior distribution of coefficients, but these posterior estimates cannot be exactly zero. Therefore, the Bayesian GRACE procedure itself cannot directly lead to variable selection. One strategy is to set to zero any coefficient estimates whose Bayesian credible intervals contain zero. Since we have estimates of the standard errors of these point estimates, we can assess how sure that such coefficients are actually zero. This is a great advantage over the GRACE procedures [11].

Since the tuning parameters λ₁ and λ₂ influence on the posterior estimates, with larger λ₁ values leading to more shrinkage of the estimates toward zero, these tuning parameters have to be determined first before selecting the relevant variables. Typically, cross-validation is used for choosing these tuning parameters, but it can be computationally very expensive. An alternative is to use the BIC or AIC, however, it is not clear how one should define the degrees of freedom or the effective number of the parameters. Park and Casella [7] suggested marginal maximum likelihood for the parameters, using a Monte Carlo EM algorithm [16]. However, we observed that in our setting, this approach does not work well when p is large.

We instead use the DIC to select the tuning parameters. The DIC has been the criterion of choice for Bayesian model selection and model comparison since it was proposed by Spiegelhalter et al. [13]. The DIC is a generalization of the AIC, but is intended for use with MCMC output and for hierarchical linear models. The DIC has been recently used in genetics in the context of evaluation hierarchical models of gene-environment interactions [17, 18], and mapping for multiple QTLs [19].

Let us assume, in general, that the distribution of the data y depends on a multidimensional parameter vector θ, and L(θ|y) is a likelihood function of θ. The statistical deviance is defined as

$$D_{\theta }(y)=-2logL(\theta \mid y)$$

by Spiegelhalter et al. [13]. The DIC consists of two components, a term that measures goodness of fit and a penalty term for increasing model complexity, i.e.,

$$\text{DIC}=E[D_{\theta }(y)\mid y]+p_D.$$

The second component p_D measures the complexity of the model by the effective number of parameters, which is defined as the difference between the posterior mean of the deviance and the deviance evaluated at the posterior estimates, i.e.,

$$p_D=E[D_{\theta }(y)\mid y]-D_{\overline{\theta }}(y)$$

where θ̄ is the posterior estimates such as posterior means, medians, and modes.

Specifically, the posterior distribution of β in (3) forms a p-dimensional multivariate normal distribution, so the likelihood is

$$L(\theta \mid \text{data})=\frac{{\mid {\mathit{X}}^{\top }\mathit{X}+{\mathrm{\sum }}^{-1}\mid }^{1/2}}{{(2\pi {\sigma }^2)}^{p/2}}exp\left(-\frac{1}{2{\sigma }^2}{[\mathit{\beta }-\mathit{\mu }]}^{\top }[{\mathit{X}}^{\top }\mathit{X}+{\mathrm{\sum }}^{-1}][\mathit{\beta }-\mathit{\mu }]\right),$$

where $\theta =(\mathit{\beta },{\tau }_1^2,\dots ,{\tau }_p^2,{\sigma }^2)$, μ = (X^⊤ X+ Σ⁻¹)⁻¹X^⊤Y, and the observed data is (X, Y, L, λ₂).

Let us denote t-th MCMC sample of θ by ${\theta }^{[t]}={(\mathit{\beta },{\tau }_1^2,\dots ,{\tau }_p^2,{\sigma }^2)}^{[t]}$, t = 1, …, K, then the posterior mean deviance is

$$E[D_{\theta }(\text{data})]=\frac{1}{K}\sum _{t=1}^K-2logL({\theta }^{[t]}\mid \text{data}),$$

and the deviance at the posterior estimates of θ, denoted by θ̄, is

$$D_{\overline{\theta }}(\text{data})=-2logL(\overline{\theta }\mid \text{data}).$$

The DIC is finally computed as

$$\text{DIC}({\lambda }_1,{\lambda }_2)=2E[D_{\theta }(\text{data})]-D_{\overline{\theta }}(\text{data}).$$

Since the data (X, Y, L) is fixed, the DIC depends only on the tuning parameters (λ₁, λ₂). The computation of the covariance matrix of β involves the parameter λ₂, and the posterior distribution of ( ${\tau }_1^{-2},\dots ,{\tau }_p^{-2}$) are determined by the magnitude of λ₁. The larger λ₁ is, the more sparse the model is. To determine the optimal tuning parameters, we can first start with different sets of (λ₁, λ₂) and obtain MCMC output results for each set. We then compute DIC based on each output and choose the set of (λ₁, λ₂) that has the smallest DIC.

4 Simulation Results

To demonstrate our proposed Bayesian GRACE and to compare with the results with GRACE, we simulated a network graph that mimics gene regulation modules. Suppose that the graph consists of 50 unconnected regulatory modules with 50 transcription factors (TFs) and each regulating 10 different genes, so we have a total of 550 variables or genes. Among these modules and genes, we further assume that the first four TFs and their regulated genes, i.e., a total of 44 variables, are associated with the response based on the following model,

$$Y=\sum _{j=1}^{44}{\beta }_j{X}_j+\epsilon ,$$ (5)

where ε is randomly generated from the normal distribution of a mean of 0 and a variance of $\frac{1}{4}{\sum }_j{\beta }_j^2$. For each TF, the X_j value is simulated from a standard normal distribution with a sample size of n = 1000. The 10 regulated genes for each TF are then simulated from a conditional normal distribution with a correlation of 0.1, 0.5, and 0.9 between the genes and its regulator, respectively. To assess the prediction performance of the methods, we also generated additional 500 test samples 50 times to compute the prediction mean squared errors. We considered two different models for the regression coefficients for the relevant genes.

For the first model, we assume that the true coefficients are specified as

$$\text{Model}1:{\beta }_j=\{\begin{array}{cc}(2,\frac{2}{\sqrt{10}},\dots ,\frac{2}{\sqrt{10}}),& 1\le j\le 11\\ (-2,\frac{-2}{\sqrt{10}},\dots ,\frac{-2}{\sqrt{10}}),& 12\le j\le 22\\ (4,\frac{4}{\sqrt{10}},\dots ,\frac{4}{\sqrt{10}}),& 23\le j\le 33\\ (-4,\frac{-4}{\sqrt{10}},\dots ,\frac{-4}{\sqrt{10}}),& 34\le j\le 44\\ 0,& j\ge 45\end{array}$$

This is also the model used in [11]. For each of the simulated data sets with different network correlations, we fitted a linear model using the Bayesian Lasso (λ₂ = 0), Bayesian Elastic-net (L = I), and Bayesian GRACE, respectively. For each model, we ran the Gibbs sampling in (3) and obtained 5000 MCMC outputs of ${\theta }^{[t]}={(\mathit{\beta },{\tau }_1^2,\dots ,{\tau }_p^2,{\sigma }^2)}^{[t]}$, t = 1, …, 6000, after the first 1000 burn-ins. All models were repeatedly fitted with 5 different shrinkage parameters λ₁ = 0.1, 1, 10, 50, 100. Since the Bayesian Elastic-net and Bayesian GRACE have the second tuning parameter for smoothness, we ran them with λ₂ = 1, 10, 10², 10³, 10⁴ for each fixed λ₁. For each given λ₁, the DICs were then computed to choose the λ₂. The Bayesian Elastic-net has the minimum DIC with λ₂ = 1 and the Bayesian GRACE achieved the smallest DIC when λ₂ = 10⁴ was used. Table 1 shows the DIC results of three Bayesian procedures for the three simulated data sets with different correlation structures. It appears that the Bayesian GRACE have much smaller DIC than the other two procedures over all different shrinkage and correlation parameters.

Table 1.

Deviance information criterion (DIC) for the Bayesian Lasso (B.Lasso), the Bayesian Elastic-net (B.Elastic-net), and the Bayesian GRACE (B.GRACE) for three simulated data sets based on Model 1 with different correlation structures. For each λ₁, λ₂ was searched over a range of values to obtain the minimum DICs.

Correlation	Method	λ1 = 0.1	λ1 = 1	λ1 = 10	λ1 = 50	λ1 = 100
0.1	B.Lasso	−317.76	−332.06	−421.78	−680.56	−920.04
	B.Elastic-net	−313.68	−328.30	−425.44	−680.31	−919.79
	B.GRACE	−1347.92	−1352.36	−1384.57	−1421.52	−1388.90
0.5	B.Lasso	−249.94	−267.41	−373.32	−680.28	−912.33
	B.Elastic-net	−246.34	−264.36	−374.24	−682.02	−912.07
	B.GRACE	−1411.03	−1414.42	−1439.82	−1465.14	−1423.96
0.9	B.Lasso	501.17	446.17	141.86	−517.24	−854.50
	B.Elastic-net	505.23	450.84	142.38	−515.67	−851.36
	B.GRACE	−1395.35	−1398.83	−1424.41	−1452.28	−1411.22

We next compute the prediction mean squared errors (PMSEs) based on 500 test samples for each model and repeated this 50 times. The average of PMSEs are shown in Figure 1. The results of the Bayesian Lasso and Bayesian Elastic-net are very similar for all three data sets. Both methods achieved the smallest prediction errors at λ₁ = 50. In contrast, the Bayesian GRACE has smaller prediction errors than the other two methods, regardless of the shrinkage parameter λ₁. Since the Bayesian GRACE has the smallest DIC at λ₁ = 50, λ₁ = 50 was chosen for all three Bayesian estimation procedures.

Figure 1.

The average of prediction mean squared errors of the Bayesian Lasso, the Bayesian Elastic-net, and the Bayesian GRACE for three simulated data sets based on Model 1 with different correlations between the TFs and the genes they regulate.

We also examine the estimates of the regression coefficients. The scatter plots for the squared errors of the posterior median estimates, (β̂_j − β_j)², j = 1, …, 550, are presented in Figure 2 for the three simulated data sets. It appears that most of posterior median estimates of the zero regression coefficients are close to zero for all three procedures. It is clear that the squared errors of median estimates from the Bayesian GRACE for the 44 relevant variables are much smaller than those from Bayesian Lasso or Bayesian Elastic net, especially when the network correlation is strong.

Figure 2.

The squared errors of posterior median estimates for the regression coefficients by Bayesian Lasso, Bayesian Elastic-net, and Bayesian GRACE for three simulated data sets from Model 1 with a correlation of 0.1, 0.5, and 0.9 between the TFs and the genes that they regulates, respectively. The x-axis represents the variable index, where the first 44 variables are relevant.

We finally examine how well the Bayesian credible intervals can be used for variable selection. We computed 99% and 95% Bayesian credible intervals for the coefficient estimates and selected the variables whose credible intervals did not include zero. Accordingly, we set the median estimate to zero if its credible region contains zero. Since only the first 44 variables are associated with the response, we can evaluate and compare the performance of each method in terms of estimation and selection. Sensitivity and specificity of the selection show how well the methods choose the relevant variables and the mean squared error (MSE) of the median estimates of only selected variables can be used to measure how well each method accurately estimates the regression coefficients. Table 2 presents a summary of the selection results from three different Bayesian procedures for three simulated data sets. It appears that the Bayesian Lasso and Elastic-net selected fewer variables than 44 when 99% or 95% credible intervals were used, so the sensitivity are relatively low for these two procedures. The specificities of these methods were 1 or close to 1 due to a large number of variables. The Bayesian GRACE selected the exact 44 variables with perfect selection results. The MSE of the Bayesian GRACE estimates were much smaller, and specifically the MSE were 100–200 times smaller for the simulated data set with correlation of 0.9. Overall, we observed that the Bayesian GRACE overwhelmed the other methods in terms of both estimation and the variable selection. In particular, it showed better performance when the predictors are highly correlated. The perfect selection results from the Bayesian GRACE also attributed to the large sample sizes in our simulations.

Table 2.

Summary of model selection by 99% and 95% Bayesian credible intervals for Bayesian Lasso, Bayesian Elastic-net, and Bayesian GRACE for three data sets based on Model 1 with different correlation structures. Mean squared errors (MSEs) are also presented.

Correlation	Method	Selection	# of Variables	Sensitivity	Specificity	MSE
0.1	B.Lasso	99%	29	0.66	1.00	0.014
		95%	36	0.82	1.00	0.010
	B.Elastic-net	99%	28	0.64	1.00	0.015
		95%	35	0.80	1.00	0.010
	B.GRACE	99%	44	1.00	1.00	0.0024
		95%	44	1.00	1.00	0.0024
0.5	B.Lasso	99%	31	0.70	1.00	0.0144
		95%	35	0.80	1.00	0.0120
	B.Elastic-net	99%	31	0.71	1.00	0.014
		95%	36	0.82	1.00	0.011
	B.GRACE	99%	44	1.00	1.00	0.0008
		95%	44	1.00	1.00	0.0008
0.9	B.Lasso	99%	23	0.52	1.00	0.044
		95%	32	0.73	1.00	0.027
	B.Elastic-net	99%	22	0.50	1.00	0.045
		95%	32	0.73	1.00	0.023
	B.GRACE	99%	44	1.00	1.00	0.0002
		95%	44	1.00	1.00	0.0002

For Model 2, we assume the following true regression coefficients,

$$\text{Model}2:{\beta }_j=\{\begin{array}{cc}(2,\frac{2}{\sqrt{10}}{u}_1,\dots ,\frac{2}{\sqrt{10}}{u}_{10}),& 1\le j\le 11\\ (-2,\frac{-2}{\sqrt{10}}{u}_1,\dots ,\frac{-2}{\sqrt{10}}{u}_{10}),& 12\le j\le 22\\ (4,\frac{4}{\sqrt{10}}{u}_1,\dots ,\frac{4}{\sqrt{10}}{u}_{10}),& 23\le j\le 33\\ (-4,\frac{-4}{\sqrt{10}}{u}_1,\dots ,\frac{-4}{\sqrt{10}}{u}_{10}),& 34\le j\le 44\\ 0,& j\ge 45,\end{array}$$

where ${\sum }_{k=1}^{10}{u}_k^2=1$, so the error variance $\frac{1}{4}{\sum }_j{\beta }_j^2=20$ remains unchanged. We simply generated u_k by setting u_k = k and dividing each by ${(10{\sum }_{k=1}^{10}{u}_k^2)}^{-1/2}$. Y and X_j are generated in the same way as for Model 1, but we fix the network correlation at 0.5.

Similar to analysis for Model 1, the DIC was used for choosing the tuning parameters λ₁ and λ₂. All three Bayesian procedures selected the sparsity tuning parameter λ₁ = 50, and the smoothness tuning parameter λ₂ = 1 and 10² were selected by the Bayesian Elastic-net and Grace, respectively. Under these selected tuning parameters, Figure 3 compares the squared errors of median estimates of the three different Bayesian procedures. It is clear that the Bayesian Grace still resulted in the smallest estimation error, especially for those relevant variables. Additionally, we computed the 99%, 95%, and 90% Bayesian credible regions from the MCMC runs for all three Bayesian procedures to assess the selection performance. Table 3 shows the summary of the selected variables based on these different credible intervals. The mean squared errors of the median estimates in the table are based only on the selected variables, i.e., we set the median estimates of unselected variables to zero. We observed that the Bayesian Grace procedure selected more relevant variables with higher sensitivities and smaller MSEs, even though they all employee the same sparsity tuning parameter.

Figure 3.

The squared errors of posterior median estimates for the regression coefficients of the Bayesian Lasso, the Bayesian Elastic-net, and the Bayesian GRACE for simulated data set from Model 2 with a correlation of 0.5 between the TFs and the genes that they regulate. The x-axis represents the variable index, where the first 44 variables are relevant.

Table 3.

Summary of model selection by 99%, 95%, and 90% Bayesian credible intervals for Bayesian Lasso, Bayesian Elastic-net, and Bayesian GRACE for simulated data set based on Model 2 with a correlation of 0.5 between the TFs and the genes that they regulate. Mean squared errors (MSEs) are also presented.

Method	Selection	# of Variables	Sensitivity	Specificity	MSE
B.Lasso	99%	28	0.64	1.00	0.0099
	95%	31	0.70	1.00	0.0079
	90%	34	0.77	1.00	0.0069
B.Elastic-net	99%	28	0.64	1.00	0.0098
	95%	31	0.70	1.00	0.0078
	90%	34	0.77	1.00	0.0068
B.GRACE	99%	30	0.68	1.00	0.0057
	95%	36	0.82	1.00	0.0039
	90%	38	0.86	1.00	0.0032

5 Real Data Analysis

Neuroblastoma (NB) is a common and lethal pediatric malignancy, but despite significant effort the genetic events that initiate tumorigenesis were until recently unknown [20]. We had hypothesized that NB is a complex disease that results from the interaction of mutant alleles with relatively low to moderate effect on tumor initiation. To identify these genetic variants, Maris et al. [21] reported a GWAS of NB where 1032 neuroblastoma cases and 2043 controls of European descent were genotyped using the Illumina 550K SNP chips and they observed a significant association between NB and the common minor alleles of three consecutive SNPs at chromosome band 6p22 and containing the predicted genes FLJ22536 and FLJ44180 (p-value = 1.71×10⁻⁹ to 7.01×10⁻¹⁰; allelic odds ratio, 1.39 to 1.40) using single SNP trend tests. Homozygosity for the at-risk G allele of the most significantly associated SNP, rs6939340, resulted in an increased likelihood of NB development (odds ratio, 1.97; 95% confidence interval, 1.58 to 2.45).

To demonstrate the Bayesian GRACE procedure, we reanalyzed the 1000 SNPs on chromosome 6 around the 6p22 region where the three SNPs were identified based on the single SNP analysis [21]. To account for the LD among the SNPs, we created a weighted LD graph linking these 1000 SNPs, where two SNPs are linked if their LD measurement r² is greater 0.4. We assumed a linear regression model with response being 1 for the cases and −1 for the controls. In addition, we coded the SNP genotypes as 0,1, and 2 for 0, 1 and 2 minor alleles and normalized the genotypes such that ΣX_j = 0 and $\sum X_j^2=3075$ for the jth SNP where the sum is over all the individuals. The linear model was repeatedly fitted with λ₁ = 1, 10, 50, 100, 200, and λ₂ = 1, 10, 10², 10³, 10⁴, 10⁵ and the DIC was computed each of the combinations of λ₁ and λ₂. The DIC selected the model with λ₁ = 100 and λ₂ = 10⁵.

The 95% Bayesian credible intervals for the median estimates of these 1000 SNPs are presented in Figure 4, showing that most of intervals include zeros and most of the SNPs are not associated with the NB risk. The bottom three plots in Figure 4 show the 99%, 95%, and 90% Bayesian credible intervals for the median estimates of the top 30 SNPs ranked by the absolute values of the estimates along with SNP names, respectively. In the 99% intervals only one SNP, rs4712653 was selected, and the 95% and 90% credible intervals selected 4 and 5 more SNPs, respectively. They are rs6939340, rs9295536, rs9466269, rs11759745, and rs12210008 (selected only when the 90% credible interval was used). The first 4 SNPs and rs4712653 are all linked together and have high LD scores between each other. Three of these SNPs, rs4712653, rs6939340 and rs9295536 were also identified by single SNP analysis [21]. The SNP rs12210008 has the highest median estimate but with relatively large standard error and was therefore only selected when the 90% credible interval was used. This SNP, which is in the neurensin 1 (NRSN1) gene, is not linked with any other SNPs identified. The gene NRSN1 is related to human nervous system development and neuron projection, which may be related to the risk of NB. This deserves further validation.

Figure 4.

Analysis of 1000 SNPs from a case-control genetic association study of neuroblastoma. The upper plot shows the posterior median estimates ( ) and the 95% Bayesian credible intervals ( ) for the 1000 SNPs. The lower plots show the posterior median estimates ( ) of the top 30 SNPs ranked by the absolute values of median estimates and their 99%, 95%, and 90% Bayesian credible intervals ( ), respectively.

6 Discussion and Conclusions

We have introduced a Bayesian approach for graph-constrained estimation and regularization for high-dimensional linear regressions where the predictors are the nodes of a graph. Such a graph-constrained regularization procedure utilizes the prior biological information encoded on graphs in identifying the genes or sub-networks or the SNPs that are related to the outcomes. An efficient Gibbs sampling procedure is developed to obtain the posterior distributions of the regression coefficients, which can be used to assess the variability of the estimates. The Gibbs sampling algorithm is very fast and is feasible for analyzing high dimensional data. When the regression coefficients are smooth w.r.t. the graphical structures, the proposed procedure performed better than the Bayesian Lasso or Bayesian Elastic-net procedure both in identifying the relevant variables and also in prediction. This agrees with what was observed by Li and Li [11] when comparing the graph-constrained regularization with Lasso or Elastic net procedures.

Compared to the GRACE, the Bayesian GRACE provides Bayesian credible intervals of the parameter estimates that can be used for statistical inferences. Due to availability of the closed-form conditional distributions in the Gibbs sampling steps, the computational burden is not too excessive. However, different from the standard regularization procedures that can directly result in zero estimates for some covariates and therefore directly lead to variable selection, the Bayesian formulation of the regularization procedures such as that of [7] and our proposed procedure do not directly result in zero estimates of the coefficients. When p is small, it is reasonable to consider the use of Bayesian credible intervals to select the relevant variables. However, when p is large and greater than the sample size n, selecting the variables based on the credible intervals can also be problematic since it depends the choice of the intervals. In this paper, we used the DIC to select the tuning parameters λ₁ and λ₂. This seems to provide sensible results for simulated and real data sets. We have also tried BIC and found that it tends to select much fewer number of variables. Alternatively, we can sample λ₁ and λ₂ in the Gibbs sampler with appropriate prior distributions for λ₁ and λ₂. For example, Park and Casella [7] assigned a Gamma prior on ${\lambda }_1^2$ in Bayesian Lasso model, so the resulting conjugacy allows easy extension of the Gibbs sampler. However, estimation of the hyper-parameters in the prior distributions can also be a problem.

The Bayesian regularization procedure proposed in this paper and that of Park and Casella [7] are different from the class of Bayesian variable selection procedures [22–24]. Instead of obtaining the posterior probability of whether a variable should be in a model, the regularization methods obtain the posterior distributions of the regression coefficients by assuming appropriate prior on these coefficients to reflect the desired penalty terms as in penalized regressions. Methods to incorporate the prior graph information into Bayesian variable selection have also been recently developed and studied [25–27]. The key of such approaches is to assume a Markov random field prior for Bernoulli random variables to indicate whether the covariates should be included in the model. In contrast, our approach assumes a combination of Laplacian and Gaussian Markov random field prior on the regression coefficients. It would be interesting to compare the performances of these different Bayesian approaches for variable selection in high dimensional settings.