Sparse Variational Analysis of Linear Mixed Models for Large Data Sets

Abstract

It is increasingly common to be faced with longitudinal or multi-level data sets that have large numbers of predictors and/or a large sample size. Current methods of fitting and inference for mixed effects models tend to perform poorly in such settings. When there are many variables, it is appealing to allow uncertainty in subset selection and to obtain a sparse characterization of the data. Bayesian methods are available to address these goals using Markov chain Monte Carlo (MCMC), but MCMC is very computationally expensive and can be infeasible in large p and/or large n problems. As a fast approximate Bayes solution, we recommend a novel approximation to the posterior relying on variational methods. Variational methods are used to approximate the posterior of the parameters in a decomposition of the variance components, with priors chosen to obtain a sparse solution that allows selection of random effects. The method is evaluated through a simulation study, and applied to an epidemiological application.

1. Introduction

It is often of interest to fit a hierarchical model in settings involving large numbers of predictors (p) and/or large sample size (n). For example, in a large prospective epidemiology study, one may obtain longitudinal data for tens of thousands of subjects, while also collecting ~ 100 predictors. Even in more modest studies, involving thousands of subjects, the number of predictors collected is often large. Unfortunately, current methods for inference in mixed effects models are not designed to accommodate large p and/or large n. This article proposes a method for obtaining sparse approximate Bayes inferences in such problems using variational methods (Jordan et al., 1999; Jaakkola and Jordan, 2000).

For concreteness we focus on the linear mixed effects (LME) model (Laird and Ware, 1982), though the proposed methods can be applied directly in many other hierarchical models. When considering LMEs in settings involving moderate to large p, it is appealing to consider methods that encourage sparse estimation of the random effects covariance matrix. There are a variety of methods available in the literature, including approaches based on Bayesian methods implemented with MCMC (Chen and Dunson, 2003; Kinney and Dunson, 2007; Frühwirth-Schnatter and Tüchler, 2008) and methods based on fast shrinkage estimation (Foster et al., 2007).

Frequentist procedures encounter convergence problems (Pennell and Dunson, 2007) and MCMC based methods tend to be computationally intensive and not to scale well as p and/or n increases. The methods relying on stochastic search variable selection (SSVS) algorithms (George and McCulloch, 1997) face difficulties when p increases beyond ~ 30 in linear regression applications, with the computational burden substantially greater in hierarchical models involving random effects selection. Approaches have been proposed to make MCMC implementations of hierarchical models feasible in large data sets (Huang and Gelman, 2008; Pennell and Dunson, 2007). However, these approaches do not solve the large p problem or allow sparse estimation or selection of random effects covariances. In addition, the algorithms are still time consuming to implement.

Model selection through shrinkage estimation has gained much popularity since the Lasso of Tibshirani (1996). Similar shrinkage effects were later obtained through hierarchical modeling of the regression coefficients in the Bayesian paradigm. A few examples of these are Tipping (2001); Bishop and Tipping (2000); Figueiredo (2003); Park and Casella (2008). Most approaches have relied on maximum a posteriori (MAP) estimation. MAP estimation produces a sparse point estimate with no measure of uncertainty, motivating MCMC and variational methods.

It would be appealing to have a fast approach that could be implemented much more rapidly in cases involving moderate to large data sets and numbers of variables, while producing sparse estimates and allowing approximate Bayesian inferences on predictor effects. In particular, it would be very appealing to have an approximation to the marginal posteriors instead of simply obtaining a point estimate. Basing inferences on point estimates does not account for uncertainty in the estimation process, and hence is not useful in applications, such as epidemiology.

One possibility is to rely on a variational approximation to the posterior distribution (Jordan et al., 1999; Jaakkola and Jordan, 2000; Bishop and Tipping, 2000). Within this framework, we develop a method for sparse covariance estimation relying on a decomposition and the use of heavy-tailed priors in a related manner to Tipping (2001), though they did not consider estimation of covariance matrices.

2. Variational inference

Except in very simple conjugate models, the marginal likelihood of the data is not available analytically. As an alternative to MCMC and Laplace approximations (Tierney and Kadane, 1986), a lower-bound on marginal likelihoods may be obtained via variational methods (Jordan et al., 1999) yielding approximate posterior distributions on the model parameters. Let θ be the vector of all unobserved quantities in the model and y be the observed data. Given a density q(θ), the marginal log-likelihood can be decomposed as

$$\text{log}p(\mathbf{y})=\underset{\mathrm{L}}{\underbrace{{\displaystyle \int q(\mathit{\theta })\text{log}\frac{p(\mathbf{y},\mathit{\theta })}{q(\mathit{\theta })}d\mathit{\theta }}}}+\text{KL}(q\Vert p),$$ (1)

where p(y, θ) is an unnormalized posterior density of θ and KL(q‖p) denotes the Kullback-Leibler divergence between the true posterior p(θ|y) and q(θ). Since this quantity is strictly non-negative and is equal to 0 only when p(θ|y) = q(θ), the first term in (1) constitutes a lower-bound on log p(y). Maximizing the first term in the right hand side of (1) is equivalent to minimizing the second term in the right hand side, suggesting that q(θ) is an approximation to the posterior density p(θ|y).

Following Bishop and Tipping (2000) we consider a factorized form

$$q(\mathit{\theta })={\displaystyle \prod _i{q}_i({\mathit{\theta }}_i),}$$ (2)

where θ_i is a sub-vector of θ and there are no restrictions on the form of q_i(θ_i). Then the lower-bound can be maximized with respect to q_i(θ_i) yielding an iterative procedure

$$q_i({\theta }_i)=\frac{\text{exp}{\langle \text{log}p(\mathbf{y},\mathit{\theta })\rangle }_{{\theta }_{j\ne i}}}{{\displaystyle \int \text{exp}{\langle \text{log}p(\mathbf{y},\mathit{\theta })\rangle }_{{\theta }_{j\ne i}}d{\theta }_i}},$$ (3)

where 〈․〉_θj≠i denotes the expectation with respect to the distributions q_j(θ _j) for j ≠ i. Due to conjugacy, these expectations will be easily evaluated yielding standard distributions for q_i(θ_i) with parameters expressed in terms of the moments of θ_j. Thus the procedure will consist of initializing the expectations required and re-iterating through them updating the expectations with respect to the densities provided by (3).

Due to the factorized form in (2), information pertaining to the dependence structure among θ_is is lost with these approximations. This comes as a trade-off to the computational advantages. It is this factorized form and the use of conjugate priors that help us identify q(θ_i) as well-known densities, making the required moments readily available. However, considering that this factorization will be realized in a block form, we will be able to preserve the dependence structure within the blocks, i.e. within each θ_i. Potentially, the approximation may become less accurate with increasing dimension if there are important dependencies among the θ_is.

In the following text G, IG, IW, N, U, W will respectively denote gamma, inverse gamma, inverse Wishart, normal, uniform and Wishart distributions.

3. The Model

3.1. The Standard Model

Suppose there are n subjects under study, with n_i observations for the ith subject. For subject i at observation j, let y_ij denote the response, let x_ij and z_ij denote p × 1 and q × 1 vectors for predictors. Then, a linear mixed effects model can be written as

$${\mathbf{y}}_i={\mathbf{X}}_i\mathit{\alpha }+{\mathbf{Z}}_i{\mathit{\beta }}_i+{\mathit{\epsilon }}_i,$$ (4)

where y_i = (y_i1, …, y_ini)′, X_i = (x_i1, …, x_ini)′, Z_i = (z_i1, …, z_ini)′, α is a p×1 vector of unknown fixed effects, β_i is a q×1 vector unknown subject-specific random effects with β_i ~ N(0,D), and the elements of the residual vector, ε_i, are N (0, σ²I).

Given the formulation in (4), the joint density of the observations given the model parameters can be written as

$$p(\mathbf{y}|\mathit{\alpha },\mathit{\beta },{\sigma }^2)={(2\pi {\sigma }^2)}^{-{\displaystyle {\sum }_{i=1}^n{n}_i/2}}\text{exp}\left\{-\frac{1}{2{\sigma }^2}{\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^{n_i}}{(y_{\mathit{\text{ij}}}-{\mathbf{x}}_{\mathit{\text{ij}}}^{\prime }\mathit{\alpha }-{\mathbf{z}}_{\mathit{\text{ij}}}^{\prime }{\mathit{\beta }}_i)}^2\right\}.$$ (5)

For conditionally-conjugate priors, inference is straightforward both in exact (via MCMC) and approximate (via variational methods) cases. To save space, the variational approximations to the posteriors of the model parameters will be given only for the shrinkage model explained in the following section.

3.2. The Shrinkage Model

Let D = ΛBΛ where B is a symmetric, positive-definite matrix and Λ = diag(λ₁, …, λ_q) with λ_k ∈ ℝ with no further restrictions. This decomposition is not unique yet is sufficient to guarantee the positive-semidefiniteness of D.

Let us re-write (4) as

$${\mathbf{y}}_i={\mathbf{X}}_i\mathit{\alpha }+{\mathbf{Z}}_i\mathbf{\Lambda }{\mathbf{b}}_i+{\mathit{\epsilon }}_i,$$ (6)

where b_i ~ N(0, B) and λ_k acts as a scaling factor on the kth row and column of the random effects covariance D. Although the parameterization is redundant, it has been noted that the redundant parameterizations are often useful for computational reasons and for inducing new classes of priors with appealing properties (Gelman, 2006). The incorporation of λ_k allows greater control on adaptive predictor-dependent shrinkage, with values λ_k ≈ 0 (along with small corresponding diagonals in B) leading to the _kth predictor being effectively excluded from the random effects component of the model through setting the values in the kth row and column of D close to zero. This maintains the positive-semidefinite constraint. One issue with redundant parameterization is the lack of identifiability in a frequentist sense, i.e. it will lead to a likelihood which comprises multiple ridges along possible combinations of Λ and B. This does not create insurmountable difficulties for Bayesian procedures as a non-flat prior should take care of this problem. When MCMC is used, the sampling of λ_k and B would occur along these ridges where the prior assigns a positive density. Multiple modes will exist due to the fact that each λ_k may take either sign.

The variational procedure used will converge to one of multiple exchangeable modes which live on the aforementioned ridges in the posterior. The tracking of the lower-bound plays an important role to stop the iterative procedure. As the lower-bound stops its monotonic increase (according to some preset criterion), we stop the procedure and assume that any further change in λ_k and B will not change inferences as we are moving along one of these ridges.

The joint density of the observations now can be written as

$$p(\mathbf{y}|\mathit{\alpha },\mathbf{\Lambda },\mathbf{b},{\sigma }^2)={(2\pi {\sigma }^2)}^{-{\displaystyle {\sum }_{i=1}^n}\frac{n_i}{2}}\text{exp}\left\{-\frac{1}{2{\sigma }^2}{\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^{n_i}}{(y_{\mathit{\text{ij}}}-{\mathbf{x}}_{\mathit{\text{ij}}}^{\prime }\mathit{\alpha }-{\mathbf{z}}_{\mathit{\text{ij}}}^{\prime }\mathbf{\Lambda }{\mathbf{b}}_i)}^2\right\}$$ (7)

where $\mathbf{b}=({\mathbf{b}}_1^{\prime },\dots ,{\mathbf{b}}_n^{\prime })\prime$.

3.2.1. Priors and Posteriors

After this decomposition the joint (conditional) density of the observations remains almost identical, replacing β by Λb_i in (7). Notice that λ_k and b_ik are interchangeable which is going to allow us to model λ_k as redundant random-effects coefficients.

We will use independent t priors for α_k and λ_k due to its shrinkage reinforcing quality. This will be accomplished through the scale mixtures of normals (West, 1987) due to the conjugacy properties, i.e. α_k ~ N(0, a_k) and λ _k ~ N(0, ν_k) where ${\alpha }_k^{-1},{\nu }_k^{-1}~\mathrm{G}({\eta }_0,{\zeta }_0)$. Under this setup, we would hope that those α_k and λ_k corresponding to insignificant fixed and random effects would shrink towards the neighborhood of zero. This will allow us to obtain a much smaller set of fixed effects for prediction purposes as well as a much more compact covariance structure on the random-effects coefficients. The usefulness of this approach will be especially emphasized in high dimensional problems. We also set σ² ~ IG(c₀, d₀) and B ~ IW(n₀,Ψ₀).

The approximate marginal posterior distributions of the parameters, using (3), are obtained as $q(\mathit{\alpha })\stackrel{d}{=}\mathbf{N}(\hat{\mathit{\alpha }},\hat{\mathbf{A}}),q({\sigma }^2)\stackrel{d}{=}\text{IG}(\hat{c},\hat{d}),q({\mathbf{b}}_i)\stackrel{d}{=}\mathbf{N}({\hat{\mathbf{b}}}_i,{\hat{\mathbf{B}}}_i),q(\mathit{\lambda })\stackrel{d}{=}\mathbf{N}(\hat{\mathit{\lambda }},\hat{\mathbf{V}}),q(\mathbf{B})\stackrel{d}{=}\text{IW}(\hat{n},\hat{\mathbf{\Psi }}),q({\alpha }_k^{-1})\stackrel{d}{=}G(\hat{\eta },{\hat{\zeta }}_k)\text{and}q({\nu }_k^{-1})\stackrel{d}{=}G({\eta }^{*},{\zeta }_k^{*})$ where

$$\begin{array}{cc}\hat{\mathit{\alpha }}\hfill & =\langle {\sigma }^{-2}\rangle \hat{\mathbf{A}}{\displaystyle \sum _{i=1}^n}{\mathbf{X}}_i^{\prime }({\mathbf{y}}_i-{\mathbf{Z}}_i\langle \mathbf{\Lambda }\rangle \langle {\mathbf{b}}_i\rangle ),\hfill \\ \hat{\mathbf{A}}\hfill & ={\langle {\sigma }^{-2}\rangle }^{-1}{({\displaystyle \sum _{i=1}^n}{\mathbf{X}}_i^{\prime }{\mathbf{X}}_i+{\langle {\sigma }^{-2}\rangle }^{-1}\langle {\mathbf{A}}^{-1}\rangle )}^{-1},\hfill \\ \hat{c}\hfill & ={\displaystyle \sum _{i=1}^n}{n}_i/2+c_0,\hfill \\ \hat{d}\hfill & ={\displaystyle \sum _{i=1}^n}\left({\mathbf{y}}_i^{\prime }{\mathbf{y}}_i-2\langle \mathit{\alpha }\rangle \prime {\mathbf{X}}_i^{\prime }{\mathbf{y}}_i-2\langle {\mathbf{b}}_i\rangle \prime \langle \mathbf{\Lambda }\rangle {\mathbf{Z}}_i^{\prime }{\mathbf{y}}_i+{\displaystyle \sum _{k=1}^p}{\mathbf{x}}_{\mathit{\text{ik}}}^{\prime }\langle \mathit{\alpha }\mathit{\alpha }\prime \rangle {\mathbf{x}}_{\mathit{\text{ik}}}+{\displaystyle \sum _{k=1}^q}{\mathbf{Z}}_{\mathit{\text{ik}}}^{\prime }\langle \mathit{\lambda }\mathit{\lambda }\prime \rangle •\langle {\mathbf{b}}_i{\mathbf{b}}_i^{\prime }\rangle {\mathbf{Z}}_{\mathit{\text{ik}}}+2\langle \mathit{\alpha }\rangle \prime {\mathbf{X}}_i^{\prime }{\mathbf{Z}}_i\langle \mathbf{\Lambda }\rangle \langle {\mathbf{b}}_i\rangle \right)/2+d_0,\hfill \\ {\hat{\mathbf{b}}}_i\hfill & =\langle {\sigma }^{-2}\rangle {\hat{\mathbf{B}}}_i\langle \mathbf{\Lambda }\rangle {\mathbf{Z}}_i^{\prime }({\mathbf{y}}_i-{\mathbf{X}}_i\langle \mathit{\alpha }\rangle ),\hfill \\ {\hat{\mathbf{B}}}_i\hfill & ={\langle {\sigma }^{-2}\rangle }^{-1}{(\langle \mathit{\lambda }\mathit{\lambda }\prime \rangle •{\mathbf{Z}}_i^{\prime }{\mathbf{Z}}_i+{\langle {\sigma }^{-2}\rangle }^{-1}\langle {\mathbf{B}}^{-1}\rangle )}^{-1},\hfill \\ {\hat{\mathit{\lambda }}}_i\hfill & =\langle {\sigma }^{-2}\rangle \hat{\mathbf{V}}{\displaystyle \sum _{i=1}^n}\text{diag}(\langle {\mathbf{b}}_i\rangle ){\mathbf{Z}}_i^{\prime }({\mathbf{y}}_i-{\mathbf{X}}_i\langle \mathit{\alpha }\rangle ),\hfill \\ \hat{\mathbf{V}}\hfill & ={\langle {\sigma }^{-2}\rangle }^{-1}{({\displaystyle \sum _{i=1}^n}\langle {\mathbf{b}}_i,{\mathbf{b}}_i^{\prime }\rangle •{\mathbf{Z}}_i^{\prime }{\mathbf{Z}}_i+{\langle {\sigma }^{-2}\rangle }^{-1}\langle {\mathbf{V}}^{-1}\rangle )}^{-1},\hfill \\ \hat{n}\hfill & =n+n_0,\hat{\mathbf{\Psi }}={\displaystyle \sum _{i=1}^n}\langle {\mathbf{b}}_i{\mathbf{b}}_i^{\prime }\rangle +{\mathbf{\Psi }}_0,\hfill \\ \hat{\eta }\hfill & =1/2+{\eta }_0,{\hat{\zeta }}_k=\langle {\alpha }_k^2\rangle /2+{\zeta }_0,\hfill \\ {\eta }^{*}\hfill & =1/2+{\eta }_0,{\hat{\zeta }}_k^{*}=\langle {\zeta }_k^2\rangle /2+{\zeta }_0,\hfill \end{array}$$

λ = diag(Λ), A = diag(a_k : k = 1, …, p), V = diag(ν_k : k = 1, …, q), (•) denotes the Hadamard product and diag(․), depending on its argument, either builds a vector from the diagonal elements of a matrix or builds a diagonal matrix using the components of a vector as the diagonal elements of that matrix.

The required moments are $\langle \mathit{\alpha }\rangle =\hat{\mathit{\alpha }},\langle \mathit{\alpha }\mathit{\alpha }\prime \rangle =\hat{\mathbf{A}}+\hat{\mathit{\alpha }}\hat{\mathit{\alpha }}\prime ,\langle {\mathbf{b}}_i\rangle ={\hat{\mathbf{b}}}_i,\langle {\mathbf{b}}_i{\mathbf{b}}_i^{\prime }\rangle ={\hat{\mathbf{B}}}_i+{\hat{\mathbf{b}}}_i{\hat{\mathbf{b}}}_i^{\prime },\langle {\sigma }^{-2}\rangle =\hat{c}/\hat{d},\langle \mathbf{\Lambda }\rangle =\text{diag}(\hat{\mathit{\lambda }}),\langle \mathit{\lambda }\mathit{\lambda }\prime \rangle =\hat{\mathbf{V}}+\hat{\mathit{\lambda }}\hat{\mathit{\lambda }}\prime ,\langle a_k^{-1}\rangle =\hat{\eta }/{\hat{\zeta }}_k,\langle {\nu }_k^{-1}\rangle ={\eta }^{*}/{\zeta }_k^{*},\langle {\mathbf{B}}^{-1}\rangle =\hat{n}{\hat{\mathbf{\Phi }}}^{-1}$. The iterative procedure that (3) implies is then to initiate these moments and cycle through them until convergence is reached.

After simplifications, the expression for the lower-bound, L, is given by

$$\mathrm{L}=\frac{1}{2}\left\{-{\displaystyle \sum _{i=1}^n}{n}_i\text{log}(2\pi )+q(n+1)+p+\text{log}|\mathrm{𝕍}(\mathit{\alpha })|+\text{log}|\mathrm{𝕍}(\mathit{\lambda })|+{\displaystyle \sum _{i=1}^n}\text{log}|\mathrm{𝕍}({\mathbf{b}}_i)|+q(\hat{n}-n_0)\text{log}2+\text{log}\frac{{|{\mathbf{\Psi }}_0|}^{n_0}}{{|\hat{\mathbf{\Psi }}|}^{\hat{n}}}\right\}+\text{log}\frac{{\mathrm{\Gamma }}_q(\hat{n}/2)}{{\mathrm{\Gamma }}_q(n_0/2)}+\text{log}\frac{d_0^{c_0}}{{\hat{d}}^{\hat{c}}}+\text{log}\frac{\mathrm{\Gamma }(\hat{c})}{\mathrm{\Gamma }(c_0)}+{\displaystyle \sum _{j=1}^p}\text{log}\frac{{\zeta }_0^{{\eta }_0}}{{\hat{\zeta }}_j^{\hat{n}}}+{\displaystyle \sum _{j=1}^q}\text{log}\frac{{\zeta }_0^{{\eta }_0}}{{\hat{\zeta }}_j^{*{\eta }^{*}}}+p\text{log}\frac{\mathrm{\Gamma }(\hat{\eta })}{\mathrm{\Gamma }({\eta }_0)}+q\text{log}\frac{\mathrm{\Gamma }({\eta }^{*})}{\mathrm{\Gamma }({\eta }_0)}.$$ (8)

4. Simulations

Focusing on the model of Section 3.2, we assess the performance in estimating the fixed effects and random effects covariance. We specify two alternatives for the number of subjects, n = {400, 2000}, three alternatives for the number of potential covariates, p = {4, 20, 60}, q = p, and three alternatives for the underlying sparsity corresponding respectively to the number of potential covariates, p′ = {.75p, .50p, .25p}, q′ = p′, where p′ and q′ denote the number of active covariates in the underlying model. We generate n_i = 8, i = 1, …, n observations per subject and set α_1:p′ = 1 and α_(p′+1):p = 0. In addition, we let x_ij1 = 1, x_ij(2:p) ~ N_p−1 (0,C) and C ~ W(p − 1, I_p−1); z_ij = x_ij; σ² ~ U(1, 3); D_{1:q′×1:q′} ~ W(q′, I_q′) and the rest of the entries along the dimensions (q′ + 1) : q are 0.

We run the variational procedure both for the standard and shrinkage models on 100 independently generated data sets. For the standard model, the priors are specified as α ~ N (0, A₀), σ⁻² ~ G(c₀, d₀) and D ~ IW(n₀,Ψ₀) where α₀ = 0, A₀ = 1000I (Chen and Dunson, 2003), c₀ = 0.1, d₀ = 0.001 (Smith and Kohn, 2002), n₀ = q and Φ₀ = I to reflect our vague prior information on α, σ⁻² and D. All these priors are proper yet specify very vague information about the parameters relative to the likelihoods that are observed in this simulation. For the shrinkage model, we choose c₀ = η₀ = 0.1, d₀ = ζ₀ = 0.001, n₀ = q and Ψ₀ = I to express our vague information on σ⁻², $a_k^{-1},{\nu }_k^{-1}$ and B. It is important that we refrain from using improper priors on the higher level parameters, i.e. $a_k^{-1},{\nu }_k^{-1}$, for meaningful marginal likelihoods as the limiting cases of these conjugate priors will lead to the impropriety of the posterior distribution and consequently the decomposition in (1) will lose its meaning.

The boxplots in Figure 1 (a) and (b) give the quadratic losses in the estimation of α and D respectively arising from the standard and the shrinkage models. As the dimension of the problem increases and the underlying model becomes sparser, the advantage of the shrinkage model is highly pronounced.

Figure 1.

Quadratic loss for (a) α and (b) D. The vertical axis is given in log-scale.

5. Real Data Application

We apply the proposed method to US Collaborative Perinatal Project (CPP) data on maternal pregnancy smoking in relation to child growth (Chen et al., 2005). Chen et al. (2005) examine the relationship between maternal smoking habits during pregnancy and childhood obesity within n = 34866 children in the CPP using generalized estimating equations (GEE) (Liang and Zeger, 1986). The size of the data, in particular the number of subjects, hinders a random effects analysis (Pennell and Dunson, 2007).

Having removed the missing observations we were left with 28211 subjects and 115811 observations. We set aside 211 observations across the subjects as a hold-out sample to test the performance of our procedure which leaves us with 115600 observations to train our model with. The main predictors considered in the model are, child’s age, recruitment center, race, whether ever breast-fed, number of prior live births by mother, socioeconomic index, mothers BMI, number of cigarettes smoked per day, mother’s age, mother’s education, marital status of mother at recruitment and trimester at recruitment. Dummy variables were created for the categorical variables. We also include a quadratic term for age and interaction terms between all the predictors (except for center) and age. Our design matrix, X = Z, (with a column of 1s for the intercept term) has 72 columns. Each column of X = Z is scaled to have unit length. The response was continuous, the weight of the children in kilograms. A detailed description of the data is available in Chen et al. (2005).

We apply our shrinkage model to the data. Figure 2 give the 99% credible sets for the fixed effect coefficients and for the diagonals of the random effects covariance matrix respectively. We can see for both fixed effects coefficients and the diagonals of the random effects covariance, many credible sets are concentrated around 0. Figure 3 (a) also gives the point estimates and 99% credible sets for the hold-out sample. Here the R² on the test set was found to be 94.8%.

Figure 2.

Vertical lines represent 99% credible sets for (a) the fixed effect coefficients for 72 predictors used in the model and (b) the diagonal elements of the random effects covariance matrix. (c) and (d) provide a closer look at those credible sets that are closer to zero.

Figure 3.

(a) gives the predicted vs. observed response (wight in kg) plot where black circles represent the point estimates for the shrinkage model, dashed line is the 45° line and the solid line is the linear fit between the predicted and observed values. Shaded area gives the 99% point-wise credible region for the mean response. (b) tracks the lower-bound (upper) and log ψ for convergence (lower) over time. Vertical dashed line marks the iteration/time the pre-specified convergence criterion is reached (ψ = 10⁻⁶).

The computational advantage of the procedure is undeniable. For the shrinkage model, the algorithm was implemented in MATLAB on a computer with a 2.8 GHz processor and 12 GB RAM. Figure 3(d) tracks the lower-bound and the relative error between two subsequent lower-bound values, ψ = |L^(t) − L^(t−1)|/|L^(t)|, for convergence where L^(t) denotes the lower-bound evaluated at iteration t. The preset value of ψ = 10⁻⁶ is reached after 2485 iterations which takes 2.4 days. It should be noted that the computational intensity for one iteration is almost identical to a Gibbs sampling scenario, which suggests, if Gibbs sampling procedure were to be used, only 2485 samples would have been drawn. Considering the burn-in period required for convergence and the thinning of the chain to obtain less correlated draws, this number is far from sufficient. Thus, with data sets this large or larger, MCMC is not a computationally feasible option.

6. Conclusion

Here we provided a fast approximate solution to fully Bayes inference to be used in the analysis of large longitudinal data sets. The proposed parameterization also allows for identifying the predictors that contribute as fixed and/or random effects. Although this parameterization leads to an unidentifiable likelihood, and would also cause the so-called label-switching problem with the application of Gibbs sampling, the variational approach allows us to converge to one of many solutions which lead to identical inferences. The utility of the new parameterization is justified through a simulation study. The application to a large epidemiological data set also demonstrates computational advantages obtained through the proposed method over conventional sampling techniques.

The computational advantages are contingent upon the availability of the required moments in analytical form which are easily obtained with the conjugate priors we considered. When the likelihood is no longer normal, these moments may not be available in analytical form and we may have to resort in some numerical solution. In such cases computational advantages may not be as pronounced. As a special case, the same computational advantages should be observed in linear mixed models with a binary outcome through latent variable modeling of Albert and Chib (1993). A similar approach for shrinkage estimation in probit models was considered in Armagan (2009).