Fixed and Random Effects Selection in Mixed Effects Models

SUMMARY

We consider selecting both fixed and random effects in a general class of mixed effects models using maximum penalized likelihood (MPL) estimation along with the smoothly clipped absolute deviation (SCAD) and adaptive LASSO (ALASSO) penalty functions. The maximum penalized likelihood estimates are shown to posses consistency and sparsity properties and asymptotic normality. A model selection criterion, called the IC_Q statistic, is proposed for selecting the penalty parameters (Ibrahim, Zhu and Tang, 2008). The variable selection procedure based on IC_Q is shown to consistently select important fixed and random effects. The methodology is very general and can be applied to numerous situations involving random effects, including generalized linear mixed models. Simulation studies and a real data set from an Yale infant growth study are used to illustrate the proposed methodology.

1. Introduction

In the analysis of mixed effects models, a primary objective is to assess significant fixed effects and/or random effects of the outcome variable. For instance, when simultaneously selecting both random and fixed effects, that is, when selecting mixed effects, it is common to use a selection procedure (e.g., forward or backward elimination), coupled with a selection criterion, such as AIC and/or BIC based on the observed data log-likelihood, to compare a set of candidate models (Keselman et al., 1998; Gurka, 2006; Liang, Wu, and Zou, 2008; Ibrahim, Zhu, and Tang, 2008; Claeskens and Consentino, 2008). Zhu and Zhang (2006) proposed a testing procedure based on a class of test statistics for a general mixed effects model to test the homogeneity hypothesis that all of the variance components are zero. Such methods, however, suffer from a serious deficiency in that it is infeasible to simultaneously select significant random and fixed (mixed) effects from a large number of possible models (Fan and Li, 2001; Fan and Li, 2002). To overcome such a deficiency, variable selection procedures based on penalized likelihood methods, such as the Smoothly Clipped Absolute Deviation (SCAD) (Fan and Li, 2001) and the Adaptive Lasso (ALASSO) (Zou, 2006), may be developed to select mixed effects.

Compared to the large body of literature on variable selection procedures, we make several novel contributions in this paper. This is one of the few papers on developing selection methods for selecting mixed effects in a large class of mixed effects models. Most variable selection procedures are developed for various parametric models and semiparametric models with/without random effects and/or unobserved data (Fan and Li, 2002, 2004; Cai et al., 2005; Qu and Li, 2006; Zhang and Lu, 2007; Ni, Zhang, and Zhang, 2009; Johnson, Lin, and Zeng, 2008; Garcia, Ibrahim, and Zhu 2010a, 2010b), but all these procedures have only been used for the selection of significant fixed effects. The only exception is the recent work by Krishna (2009) and Bondell, Krishna, and Ghosh (2010), in which only the linear mixed model is considered. We use a novel reparametrization to reformulate the selection of mixed effects into the problem of grouped variable selection in models with heavy ‘missing’ data, where the missing data is represented by the random effects. This reparametrization makes it possible to use penalized likelihood methods to select both fixed and random effects. Compared to most variable selection methods for linear models, we must address additional challenges due to the presence of missing observations for each subject. A computational challenge here is to directly maximize the observed data log-likelihood function along with the SCAD or ALASSO penalties to select both fixed and random effects and to calculate their estimates. The observed data log-likelihood for complicated mixed effects models is often not available in closed form, and is computationally intractable because it may involve high dimensional integrals which are difficult to approximate. When selecting random effects, this maximization is further complicated because one must eliminate the corresponding row and column of an insignificant random effect and constrain the remaining matrix to be positive definite. Another challenge is to select appropriate penalty parameters in order to produce estimates having proper asymptotic properties (Fan and Li, 2001), whereas existing selection criteria (Kowalchuck et al., 2004; Gurka, 2006; Liang, Wu, and Zou, 2008; Claeskens and Consentino, 2008) are computationally difficult for general mixed effect models.

The goal of this paper is to develop a simultaneous fixed and random effects selection procedure based on the SCAD and ALASSO penalties for application to longitudinal models, correlated models, and/or mixed effects models. We reformulate the problem of selecting mixed effects and develop a method based on the IC_Q criterion to select the penalty parameters. We also specify the penalty parameters in the SCAD and ALASSO penalty functions as a hyperparameter, and then we use the Expectation Maximization (EM) algorithm to simultaneously optimize the penalized likelihood function and estimate the penalty parameters. Under some regularity conditions, we establish the asymptotic properties of the maximum penalized likelihood estimator and the consistency of the IC_Q-based penalty selection procedure.

To motivate the proposed methodolgy, we consider a dataset from a Yale infant growth study (Wasserman and Leventhal, 1993; Stier et al., 1993). The objective of this study is to investigate the relationship between maternal cocaine dependency and child maltreatment (physical abuse, sexual abuse, or neglect). This study had a total of 298 children from the cocaine exposed and unexposed groups. The outcome variable is infant weight (in pounds), which is obtained over several time points. Seven covariates were considered: day of visit, age of mother, gestational age of infant, race, previous pregnancies, gender of infant, and cocaine exposure. Each child had different number and pattern of visits during the study. We consider the mixed effects model by using the seven covariates as fixed effects and the first three covariates as random effects. Our objective in this analysis is to select the most important predictors of infant weight as well as select significant random effects. The selection of random effects is crucial in this application, as it is not at all clear whether a random intercept model will suffice or whether the longitudinal model should also contain random slope effects. Moreover, there is large number of covariates to select from in the fixed effects component of the model. The selection can be done by our penalized likelihood method, which includes a penalty function (SCAD or ALASSO) with a random effect and an IC_Q penalty estimate. More details regarding the analyses of these data set is given in Section 5.

The rest of the paper is organized as follows. Section 2 introduces the general development for maximizing the penalized likelihood function and selecting the penalty parameters. Section 3 examines the asymptotic properties of the maximum penalized likelihood (MPL) estimator and the IC_Q penalty selection procedure. Section 4 presents a simulation study to examine the finite sample performance of the maximum penalized likelihood estimate. An real data analysis of the Yale infant growth study is given in Section 5. Section 6 concludes the paper with some discussion.

2. Mixed effects selection for mixed effects models

2.1 Model Formulation

Suppose we observe n independent observations (y₁, X₁),…, (y_n, X_n), where y_i is an n_i × 1 vector of responses or repeated measures and X_i is an n_i × p matrix of fixed covariates for i = 1,…, n. We assume independence among the different (y_i, X_i)’s and

$$E[{\mathbf{y}}_i|{\mathbf{b}}_i,{\mathbf{X}}_i;\mathit{\theta }]=g({\mathbf{X}}_i\mathit{\beta }+{\mathbf{Z}}_i\mathbf{\Gamma }{\mathbf{b}}_i),$$ (1)

where b_i is a q × 1 vector of unobserved random effects, θ denotes all the unknown parameters, Γ is a q × q lower triangular matrix, g(·) is an known link function, β = (β₁,…, β_p)^T is a p × 1 vector of regression coefficients, and Z_i is an n_i × q matrix composed of the columns of X_i. In practice, it is common to assume that the conditional distribution of y_i given (b_i, X_i), denoted by f(y_i|b_i, X_i; θ), belongs to the exponential family, such as the binomial, normal, and Poisson (Little and Schluchter (1985), and Ibrahim and Lipsitz (1996)). For notational simplicity, the random effects b_i ~ N_q(0, I_q) are assumed to follow a multivariate normal distribution with zero mean and a q × q covariance matrix I_q. Equivalently, Γb_i ~ N_q(0, D = ΓΓ^T) and Γ is the Cholesky composition of the q × q matrix D. We allow the possibility of D being positive semi-definite so that certain components of Γb_i may not be random but 0 with probability 1.

2.2 EM Algorithm for Maximizing the Penalized Likelihood

Selecting mixed effects involves identifying the nonzero components of β, determining the nonrandom elements of Γb_i, and simultaneously estimating all nonzero parameters. We propose to maximize the penalized likelihood function given by

$$\text{PL}(\mathit{\theta })=\ell (\mathit{\theta })-n{\displaystyle \sum _{j=1}^p{\varphi }_{{\lambda }_j}(|{\beta }_j|)-n{\displaystyle \sum _{k=1}^q{\varphi }_{{\lambda }_{p+k}}(\Vert {\gamma }_k\Vert ),}}$$ (2)

where $\ell (\mathit{\theta })={\displaystyle {\sum }_{i=1}^n{\ell }_i(\mathit{\theta })}$, in which ℓ_i(θ) = log ∫ f(y_i, b_i|X_i; θ) db_i is the observed-data log-likelihood for the ith individual, λ_j is the penalty parameter of β_j, and the penalty function ϕ_λj(·) is a nonnegative, nondecreasing, and differentiable function on (0, ∞) (Fan and Li, 2001; Zou, 2006). In addition, the k × 1 vector γ_k consists of all nonzero elements of the k-th row of the lower triangular q × q matrix Γ, $\Vert {\gamma }_k\Vert ={({\gamma }_k^T{\gamma }_k)}^{1/2}$, and λ_p+k is the group penalty parameter corresponding to the whole k-th row of Γ. The structure in (2) ensures that certain estimates of β are zero (Fan and Li, 2001), which are insignificant predictors of the outcome variable, and the other covariates are significant predictors. The penalization of γ_k is performed in a group manner in order to preserve the positive definite constraint on D such that the estimates of the parametric vector γ_k either are all not zero or all equal to zero (Yuan and Li, 2006). If all the elements of γ_k are zero, then the k-th row of Γ is zero and the k-th element of Γb_i is not random.

Similar to Chen and Dunson (2003), we reparametrize the linear predictor as

$${\mathbf{X}}_i\mathit{\beta }+{\mathbf{Z}}_i\mathbf{\Gamma }{\mathbf{b}}_i=({\mathbf{X}}_i({\mathbf{b}}_i^T\otimes {\mathbf{Z}}_i){\mathbf{J}}_q)\left(\begin{array}{c}\hfill \mathit{\beta }\hfill \\ \hfill \gamma \hfill \end{array}\right)={\mathbf{U}}_i\mathit{\delta },$$ (3)

where J_q is the q² × q(q + 1)/2 matrix which transforms γ to vec(Γ), i.e. vec(Γ) = J_qγ. By reparametrizing the linear predictor this way, the selection of mixed effects is equivalent to the problem of grouped variable selection in regression models with missing covariates, while the random effects in the design matrix U_i can be interpreted as the “missing covariates”. Using this reparametrization, we can apply the variable selection methods proposed in Garcia, Ibrahim, and Zhu (2010a; 2010b) to select important mixed effects.

Because the observed-data log-likelihood function usually involves intractable integration, we develop a Monte Carlo EM algorithm to compute the maximum penalized likelihood estimator of θ, denoted by θ̂_λ, for each λ = (λ₁,…,λ_p+q). Denote the complete and observed data for subject i by d_c,i = (y_i, X_i, b_i) and d_o,i = (y_i, X_i), respectively, and the entire complete and observed data by d_c and d_o, respectively. At the s-th iteration, given θ^(s), the E step is to evaluate the penalized Q-function, given by

$$Q_{\lambda }(\mathit{\theta }|{\mathit{\theta }}^{(s)})={\displaystyle \sum _{i=1}^{n}E\{\text{log}f({\mathbf{d}}_{i,c};\mathit{\theta })|{\mathbf{d}}_o;{\mathit{\theta }}^{(s)}\}-n{\displaystyle \sum _{j=1}^p{\varphi }_{{\lambda }_j}(|{\beta }_j|)-n{\displaystyle \sum _{k=1}^q{\varphi }_{{\lambda }_{p+k}}(\Vert {\gamma }_k\Vert })}}$$ (4)

$$=Q_1(\mathit{\theta }|{\mathit{\theta }}^{(s)})-n{\displaystyle \sum _{j=1}^p{\varphi }_{{\lambda }_j}(|{\beta }_j|)-n{\displaystyle \sum _{k=1}^q{\varphi }_{{\lambda }_{p+k}}(\Vert {\gamma }_k\Vert })+Q_2({\mathit{\theta }}^{(s)}),}$$ (5)

where θ = (δ^T, ξ^T)^T, in which ξ represents all other parameters other than δ, d_i,c = (y_i, b_i, X_i), and

$$Q_1(\mathit{\theta }|{\mathit{\theta }}^{(s)})={\displaystyle \sum _{i=1}^n{\displaystyle \int \{\text{log}f({\mathbf{y}}_i|{\mathbf{b}}_i,}}{\mathbf{X}}_i;\mathit{\delta },\mathit{\xi })\}f({\mathbf{b}}_i|{\mathbf{d}}_{i,o};{\mathit{\theta }}^{(s)})d{\mathbf{b}}_i,$$ (6)

$$Q_2({\mathit{\theta }}^{(s)})={\displaystyle \sum _{i=1}^n{\displaystyle \int \{\text{log}f({\mathbf{b}}_i)\}f({\mathbf{b}}_i|{\mathbf{d}}_{i,o};{\mathit{\theta }}^{(s)})d{\mathbf{b}}_i.}}$$ (7)

Since the integrals in (6) and (7) are often intractable, we approximate these integrals by taking a Markov chain Monte Carlo (MCMC) sample of size L from the density f(b_i|d_i,o; θ^(s)) (See Ibrahim, Chen, and Lipsitz, 1999). Let ${\mathbf{b}}_i^{(s,l)}$ be the l-th simulated value at the s-th iteration of the algorithm. The integrals in (6) can be approximated as,

$$Q_1(\mathit{\theta }|{\mathit{\theta }}^{(s)})=\frac{1}{L}{\displaystyle \sum _{l=1}^L{\displaystyle \sum _{i=1}^n\text{log}f({\mathbf{y}}_i|{\mathbf{b}}_i^{(s,l)},{\mathbf{X}}_i;\mathit{\theta }).}}$$ (8)

The M step involves maximizing

$$Q_{1,\lambda }(\mathit{\theta }|{\mathit{\theta }}^{(s)})=Q_1(\mathit{\theta }|{\mathit{\theta }}^{(s)})-n{\displaystyle \sum _{j=1}^p{\varphi }_{{\lambda }_j}(|{\beta }_j|)-n{\displaystyle \sum _{k=1}^q{\varphi }_{{\lambda }_{p+k}}(\Vert {\gamma }_k\Vert })}$$ (9)

with respect to (δ, ξ). Maximizing Q_1,λ(δ, ξ|θ^(s)) with respect to ξ is straightforward and can be done using a standard optimization algorithm, such as the Newton-Raphson algorithm (Little and Schluchter, 1985; Ibrahim, 1990; Ibrahim and Lipsitz, 1996). Maximizing Q_1,λ with respect to δ is difficult because Q_1,λ is a nondifferentiable and nonconcave function of δ respectively (Zou and Li, 2008).

In order to maximize Q_1,λ, following Fan and Li (2001), a second order Taylor’s series approximation of Q_1,λ centered at the value δ^(s) is used. Using this approximation, Q_1,λ resembles a penalized weighted least squares regression, so algorithms for minimizing penalized least squares can be used (Fan and Li, 2001; Hunter and Li, 2005). We use a modification of the local linear approximation algorithm (LLA) (Zou and Li, 2008) to incorporate grouped penalization. For γ_k, we use an approximation centered at ${\gamma }_k^{(s)}$ as follows:

$${\varphi }_{{\lambda }_{p+k}}(\Vert {\gamma }_k\Vert )\approx {\displaystyle \sum _{t=1}^k\left\{\frac{{\varphi }_{{\lambda }_{p+k}}(\Vert {\gamma }_k^{(s)}\Vert )|{\gamma }_{\mathit{\text{kt}}}^{(s)}|}{\Vert {\gamma }_k^{(s)}\Vert }\right\}}|{\gamma }_{\mathit{\text{kt}}}|,$$ (10)

where γ_kt is the t-th element of the vector γ_k and we assume $\Vert {\gamma }_k^{(s)}\Vert >0$. If $\Vert {\gamma }_k^{(s)}\Vert =0$, then we let ${\gamma }_k^{(s+1)}=0$. Using this approximation, Q_1,λ resembles a penalized regression with an L₂ penalty, so the methods for performing the lasso can be used to maximize Q_1,λ (Tibshirani, 1996; Fu, 1998).

Let ${\mathbf{\xi }}^{(s+1)}=\underset{\mathit{\xi }}{\text{argmax}}{Q}_{1,\lambda }({\mathit{\delta }}^{(s)},\mathit{\xi }|{\mathit{\theta }}^{(s)})\text{ and }{\mathit{\delta }}^{(s+1)}=\underset{\mathit{\delta }}{\text{argmax}}{Q}_{1,\lambda }(\mathit{\delta },{\mathit{\xi }}^{(s+1)}|{\mathit{\theta }}^{(s)})$. Due to the Taylor’s series approximation of Q₁ and the LLA of ϕ_λj, θ^(s+1) = (δ^(s+1), ξ^(s+1)) may not necessarily be the maximizer of Q_λ (θ|θ^(s)). By implementing the Expectation Conditional-Maximization (ECM) algorithm (Meng and Rubin, 1993), however, we can find a θ^(s+1) such that Q_λ(θ^(s+1)|θ^(s)) ≥ Q_λ(θ^(s)|θ^(s)) instead of directly maximizing Q_λ(θ|θ^(s)). This process is iterated until convergence and the value at convergence, denoted by θ̂_λ, maximizes the penalized observed data log likelihood function.

2.3 Penalty Parameter Selection Procedure

To ensure that θ̂_λ has good properties, the penalty parameter λ has to be appropriately selected. Two commonly used criteria for selection of the penalty parameter include the Generalized Cross Validation (GCV) and BIC criteria (Wang et al., 2007). These criteria cannot be easily computed in the presence of random effects, because they are functions of observed data quantities whose expressions may require intractable integrals. Moreover, it has been shown in Wang et al. (2007) that even in the simple linear model, the GCV criterion can lead to significant overfit.

We propose two methods to select the penalty parameter: an IC_Q criterion and a random effects penalty selection method. The IC_Q criterion (Ibrahim, Zhu, and Tang, 2008) selects the optimal λ by minimizing

$${\text{IC}}_Q(\mathit{\lambda })=-2Q({\widehat{\mathit{\theta }}}_{\lambda }|{\widehat{\mathit{\theta }}}_0)+c_n({\widehat{\mathit{\theta }}}_{\lambda }),$$

where ${\widehat{\mathit{\theta }}}_0=\underset{\mathit{\theta }}{\text{argmax}}\ell (\mathit{\theta })$ is the unpenalized maximum likelihood estimate and c_n(θ) is a function of the data and the fitted model. For instance, if c_n equals twice the total number of parameters, then we obtain an AIC-type criterion; alternatively, we obtain a BIC-type criterion when c_n(θ) = dim(θ) × log n. Moreover, in the absence of random effects, IC_Q(λ) reduces to the usual AIC or BIC criteria. As in the EM algorithm, we can draw a set of samples from f(b_i|d_i,o; θ̂₀) for i = 1,…, n in order to estimate Q(θ̂_λ|θ̂₀) for any λ.

The random effects penalty estimator is calculated under the assumption that δ is distributed as a random effect vector in a hierarchical model. The quantity λ can be regarded as a hyperparameter vector in the distribution of δ, denoted by f(δ|λ, n). Then, λ can be estimated by maximizing the marginal likelihood with respect to (ξ, λ), which is given by

$${\displaystyle \int {\displaystyle \prod _{i=1}^n{\displaystyle \int f(y_i|{\mathbf{X}}_i,{\mathbf{b}}_i,\mathit{\delta };\mathit{\xi })f({\mathbf{b}}_i)f(\mathit{\delta }|\mathit{\lambda },n)d{\mathbf{b}}_{i}d\mathit{\delta }={\displaystyle \prod _{i=1}^n{\displaystyle \int f({\mathbf{d}}_{i,o}|\mathit{\xi }})f(\mathit{\delta }|\mathit{\lambda },n)d\mathit{\delta },}}}}$$ (11)

where f(δ|λ, n) is defined by

$$f(\mathit{\delta }|\mathit{\lambda },n)={\displaystyle \prod _{j=1}^p\text{exp}\{-n{\varphi }_{{\lambda }_j}(|{\beta }_j|)\}{\displaystyle \prod _{k=1}^q\text{exp}\{-n{\varphi }_{{\lambda }_{p+k}}(\Vert {\gamma }_k\Vert })\}/\{C(\mathit{\lambda },n)\},}$$

and C(λ, n) is the normalizing constant of f(δ|λ, n). The resulting estimate of λ, denoted by λ̂_RE, from the maximization of (11), is the random effects penalty estimator. Treating δ as missing data, the Monte Carlo EM algorithm can be used to maximize (11) with respect to (ξ, λ).

We consider the SCAD and ALASSO penalty functions for determining λ. The ALASSO penalty is defined by

$${\varphi }_{{\lambda }_j}(|{\beta }_j|)={\lambda }_j|{\beta }_j|\text{ for }j=1,\cdots ,p,{\varphi }_{{\lambda }_{p+k}}(\Vert {\gamma }_k\Vert )={\lambda }_{p+k}\Vert {\gamma }_k\Vert \text{ for }k=1,\cdots ,q.$$

Typical values of λ_j are λ_j = λ₀₁|β̂_j|⁻¹ and ${\lambda }_{p+k}={\lambda }_{02}\sqrt{k}{\Vert {\widehat{\gamma }}_k\Vert }^{-1}$, where β̂_j and γ̂_k are the unpenalized maximum likelihood (ML) estimates. The multiplier $\sqrt{k}$ normalizes the penalty parameter γ_k in order to accommodate the varying sizes of γ_k. When λ_j = λ₀₁ and ${\lambda }_{p+k}={\lambda }_{02}\sqrt{k}$, the ALASSO reduces to the LASSO penalty.

The SCAD penalty (Fan and Li, 2001) is a nonconcave function defined by ϕ_λ (0) = 0 and for $|\beta |>0,{\varphi }_{\lambda }^{\prime }(|\beta |)=\lambda 1(|\beta |\le \lambda )+\frac{(a\lambda -|\beta |)+}{a-1}1(|\beta |>\lambda )$, where t₊ denotes the positive part of t and a = 3.7. Because the integral of the negative exponential of the ALASSO and SCAD penalties is not finite, i.e. ${\displaystyle {\int }_{-\infty }^{\infty }\text{exp}\{-n{\varphi }_{\lambda }(\Vert {\lambda }_k\Vert )\}}d{\gamma }_k=\infty$, the expression exp{−nϕ_λ(‖λ_k‖)} is defined in a bounded space to ensure that f(δ|λ, n) is a proper density. Since a closed form expression of λ̂_RE is unavailable for both the ALASSO and SCAD penalties, we use the Newton Raphson algorithm along with the ECM algorithm to estimate λ̂_RE.

3. Theoretical Results

In this section, we establish the asymptotic theory of the MPL estimator and the consistency of the penalty selection procedure based on IC_Q. Suppose $\mathit{\beta }={\left({\mathit{\beta }}_{(1)}^T,{\mathit{\beta }}_{(2)}^T\right)}^T$, where β₍₁₎ and β₍₂₎ are, respectively, p₁ × 1 and (p − p₁) × 1 subvectors. Let ${\mathit{\beta }}^{*}={\left({\mathit{\beta }}_{(1)}^{*T},{\mathit{\beta }}_{(2)}^{*T}\right)}^T$ denote the true value of β. Without loss of generality, we assume that ${\mathit{\beta }}_{(2)}^{*}=0$ and all of the components of ${\mathit{\beta }}_{(1)}^{*}$ are not equal to zero. Similarly let $\gamma ={\left({\gamma }_1^T,\dots ,{\gamma }_k^T\right)}^T={\left({\gamma }_{(1)}^T,{\gamma }_{(2)}^T\right)}^T$ where ${\gamma }_{(1)}^T={\left({\gamma }_1^T,\dots ,{\gamma }_{q_1}^T\right)}^T,{\gamma }_{(2)}^T={\left({\gamma }_{q_1+(1)}^T,\dots ,{\gamma }_q^T\right)}^T$ and γ₍₁₎ and γ₍₂₎ are q₁(q₁ + 1)/2 × 1 and {q − q₁(q₁ + 1)/2} × 1 subvectors respectively. Let ${\gamma }^{*}={\left({\gamma }_{(1)}^{*T},{\gamma }_{(2)}^{*T}\right)}^T$ denote the true value of γ. Without loss of generality, we assume that ${\gamma }_{(2)}^{*}=0$ and some of the components of each ${\gamma }_k^{*}$ are not equal to zero for k = 1,…, q₁.

Let 𝒮 = {j₁₁,…, j_1d1; j₂₁, …, j_2d2} be a candidate model containing the j₁₁-th, …, j_1d-th columns of X and the j₂₁-th, …, j_2d2-th columns of Z. Thus, 𝒮_F = {1,…, p; 1,…,q} and 𝒮_T = {1,…, p₁; 1,…, q₁} denote the full and true covariate models, respectively. If 𝒮 misses at least one important covariate, that is 𝒮 ⊅ 𝒮_T, then 𝒮 is referred to as an underfitted model; however, if 𝒮 ⊃ 𝒮_T and 𝒮 ≠ 𝒮_T, then 𝒮 is an overfitted model. The unpenalized and penalized ML estimators of θ = (β^T, γ^T, ξ)^T, denoted by θ̂_S and θ̂_λ, respectively, are defined as

$${\widehat{\mathit{\theta }}}_S=\underset{\mathit{\theta }:{\beta }_j\ne 0,\forall j\in 𝒮}{\text{argmax}}\ell (\mathit{\theta })\text{ and }{\widehat{\mathit{\theta }}}_{\lambda }=\underset{\mathit{\theta }}{\text{argmax}}\left\{\ell (\mathit{\theta })-n{\displaystyle \sum _{j=1}^p{\varphi }_{{\lambda }_j}(|{\beta }_j|)-n{\displaystyle \sum _{k=1}^q{\varphi }_{{\lambda }_{p+k}}(\Vert {\gamma }_k\Vert })}\right\},$$

, and particularly θ̂_𝕊F = θ̂₀. We obtain the following theorems whose assumptions and proofs can be found in the web-based supplementary document.

THEOREM 1

Under assumptions (C1)–(C7) in the supplementary document, we have

1. θ̂_λ − θ* = O_p(n^−1/2) as n → ∞, where θ* is the true value of θ;

2. Sparsity: P(β̂_(2)λ = 0, γ̂_(2)λ = 0) → 1;

3. Asymptotic normality: $\sqrt{n}\{{({\widehat{\mathit{\beta }}}_{(1)\lambda }^T,{\widehat{\lambda }}_{(1)\lambda }^T,{\widehat{\mathit{\xi }}}_{\lambda }^T)}^T-{({\mathit{\beta }}_{(1)}^{*T},{\gamma }_{(1)}^{*T},{\mathit{\xi }}_{(1)}^{*T})}^T\}$ is asymptotically normal with mean and covariance matrix defined in the supplement.

Theorem 1 states that by appropriately choosing the penalty λ, there exists a root-n estimator of θ, θ̂_λ, and that this estimator must possess the sparsity property, i.e. β̂_(2)λ = 0, γ̂_(2)λ = 0 in probability. Moreover, ${({\widehat{\mathit{\beta }}}_{(1)\lambda }^T,{\widehat{\lambda }}_{(1)\lambda }^T,{\widehat{\mathit{\xi }}}_{\lambda }^T)}^T$ is asymptotically normal.

We investigate whether the IC_Q(λ) criterion can consistently select the correct model. For each λ ∈ R^p+, (β̂_λ, γ̂_λ) naturally defines a candidate model 𝕊_λ = {j : β̂_λ,j ≠ 0; k : ‖γ̂_λ,k‖ ≠ 0}. Generally, 𝒮_λ can be either underfitted, overfitted, or true. Therefore, R^p+ can be partitioned into three mutually exclusive regions $R_u^{p+}=\{\mathit{\lambda }\in R^{p+}:{𝒮}_{\mathit{\lambda }}\not\supset {𝒮}_T\},R_t^{p+}=\{\mathit{\lambda }\in R^{p+}:{𝒮}_{\mathit{\lambda }}={𝒮}_T\},\text{ and }{R}_o^{p+}=\{\mathit{\lambda }\in R^{p+}:{𝒮}_{\mathit{\lambda }}\not\supset {𝒮}_T,{𝒮}_{\mathit{\lambda }}\ne {𝒮}_T\},$. Furthermore, if we can choose a reference penalty parameter sequence ${\{{\lambda }_n\in R^{p+}\}}_{n=1}^{\infty }$, which satisfies the conditions of Theorem 1, then 𝒮_λn = 𝒮_T in probability.

To select λ we first calculate

$${\text{dIC}}_Q({\mathit{\lambda }}_{2,}{\mathit{\lambda }}_1)={\text{IC}}_Q({\mathit{\lambda }}_2)-{\text{IC}}_Q({\mathit{\lambda }}_1)=-2Q({\widehat{\mathit{\theta }}}_{{\lambda }_2}|{\widehat{\mathit{\theta }}}_0)+c_n({\widehat{\mathit{\theta }}}_{{\lambda }_2})+2Q({\widehat{\mathit{\theta }}}_{{\lambda }_1}|{\widehat{\mathit{\theta }}}_0)-c_n({\widehat{\mathit{\theta }}}_{{\lambda }_1})$$

for any two λ₁ and λ₂. We assume 𝒮_λ2 ⊃ 𝒮_λ1 and choose the model 𝒮_λ1 resulting from using the penalty value λ₁ if dIC_Q(λ₂, λ₁) ≥ 0, otherwise we choose the model 𝒮λ₂.

Define ${\delta }_Q({\mathit{\lambda }}_1,{\mathit{\lambda }}_2)=E\{Q({\mathit{\theta }}_{{𝒮}_{{\lambda }_1}}^{*}|{\mathit{\theta }}^{*})\}-E\{Q({\mathit{\theta }}_{{𝒮}_{{\lambda }_2}}^{*}|{\mathit{\theta }}^{*})\}$, and δ_c(λ₂, λ₁) = c_n(θ̂_λ2) − c_n(θ̂_λ1), where ${\mathit{\theta }}_{𝒮}^{*}$ is defined in the supplementary document.

THEOREM 2

Under assumptions (C1)–(C7) in the supplementary document, we have the following results.

1. If for all 𝒮_λ ⊅ 𝒮_T, $\underset{n}{\text{lim}\text{inf}}{\delta }_Q(\mathit{\lambda },0)/n>0$ and δ_c(λ, 0) = o_p(n), then dIC_Q(λ, 0) > 0 in probability.

2. If $E\{Q({\mathit{\theta }}_{{𝒮}_{{\lambda }_1}}^{*}|\widehat{{\mathit{\theta }}_0})\}-E\{Q({\mathit{\theta }}_{{𝒮}_{{\lambda }_2}}^{*}|\widehat{{\mathit{\theta }}_0})\}=O_p(n^{1/2})\mathit{\text{ and }}Q({\widehat{\mathit{\theta }}}_{{\lambda }_t}|{\widehat{\mathit{\theta }}}_0)-E\{Q({\mathit{\theta }}_{{𝒮}_{{\lambda }_t}}^{*}|\widehat{{\mathit{\theta }}_0})\}=O_p(n^{1/2})$ for t = 1, 2, then dIC_Q(λ₂, λ₁) > 0 in probability as n^−1/2δ_c(λ₂, λ₁) converges to ∞ in probability.

3. If Q(θ̂_λ1 |θ̂₀) − Q(θ̂_λ2 |θ̂₀) = O_p(1), then dIC_Q(λ₂, λ₁) > 0 in probability as δ_c(λ₂, λ₁) converges to ∞ in probability.

Theorem 2 has some important implications. Theorem 2(a) shows that IC_Q(λ) chooses all significant covariates with probability 1. Because ${𝒮}_0\subset R_t^p\cup R_o^p$, the optimal model selected by minimizing IC_Q(λ) will not select a λ with 𝒮_λ ⊅ 𝒮_T because dIC_Q(λ, 0) > 0 in probability. Therefore, the IC_Q(λ) criterion selects all significant covariates with probability tending to 1. Generally, the most commonly used c_n(θ), such as 2dim(θ), dim(θ) log(n), and K log log(n) (K > 0), satisfy the condition δ_c(λ, 0) = o_p(n). The condition $\underset{n}{\text{lim}\text{inf}}{n}^{-1}{\delta }_Q(\mathit{\lambda },0)>0$ ensures that IC_Q(λ) chooses a model with large $E\{Q({\mathit{\theta }}_{𝒮}^{*}|{\mathit{\theta }}^{*})\}$. This condition is analogous to condition 2 in (Wang et al., 2007), which elucidates the effect of underfitted models. The term $n^{-1}E\{Q({\mathit{\theta }}^{*}|{\mathit{\theta }}^{*})\}-n^{-1}E\{Q({\mathit{\theta }}_{𝒮}^{*}|{\mathit{\theta }}^{*})\}$ can be written as

$$n^{-1}\ell ({\mathit{\theta }}^{*})-n^{-1}\ell ({\mathit{\theta }}_S^{*})+n^{-1}E\{H({\mathit{\theta }}^{*}|{\mathit{\theta }}^{*})\}-n^{-1}E\{H({\mathit{\theta }}_S^{*}|{\mathit{\theta }}^{*})\},$$ (12)

where

$$H({\mathit{\theta }}_1|{\mathit{\theta }}_2)={\displaystyle \sum _{i=1}^n{\displaystyle \int \text{log}\{f({\mathbf{b}}_i|{\mathbf{d}}_{o,i};{\mathit{\theta }}_1)\}f({\mathbf{b}}_{\mathit{\text{im}}}|{\mathbf{d}}_{o,i};{\mathit{\theta }}_2)d{\mathbf{b}}_{\mathit{\text{im}}}.}}$$ (13)

By Jensen’s inequality, the third and fourth terms of (12) are greater than zero and the first and second terms must be greater than zero for large n. Thus, lim inf_n n⁻¹δ_Q(λ, 0) ≥ 0 in probability.

If λ₁ and λ₂ have the same average $n^{-1}E\{Q({\mathit{\theta }}_{{𝒮}_{\lambda }}^{*}|{\mathit{\theta }}^{*})\}$, that is, lim inf_n n⁻¹δ_Q(λ₂, λ₁) = 0, then Theorem 2 (b) and (c) indicate that IC_Q(λ) picks out the smaller model 𝒮_λ1 when δ_c(λ₂, λ₁) increases to ∞ at a certain rate (e.g., log(n)). For example, for the BIC-type criterion, δ_c(λ₂, λ₁) = {dim(θ̂_𝒮λ2) − dim(θ̂_𝒮λ1)} log(n) ≥ log(n) since we assume 𝒮_λ2 ⊃ 𝒮_λ1. The AIC-type criterion, for which c_n(θ) = 2 × dim(θ), however, does not satisfy this condition. Thus, similar to the AIC criterion with no random effects, IC_Q(λ) with c_n(θ) = 2 × dim(θ) tends to overfit.

4. Simulation Study

We use simulations to examine the finite sample performance of the maximum penalized likelihood estimates using our proposed penalty estimators and compare them to the unpenalized ML estimate. Our objectives for these simulations are to 1) compare the random effects and IC_Q penalty estimators and 2) to compare the SCAD, LASSO, and ALASSO penalty functions.

To do this, we simulated a data set consisting of n independent observations according to the model y_i = X_iβ+Z_iΓb_i+σε_i, i = 1,…, n, where b_i and ε_i are independent and standard multivariate normal random vectors, and β = (3, 2, 1.5, 0, 0, 0, 0, 0)^T. Moreover, ΓΓ^T = D is a 3 × 3 matrix, such that the (r, s) element of D is ρ^|r−s|. The matrix X_i is a 12 × 8 matrix of independent rows, where each row of X_i has mean zero and covariance matrix Σ_xx whose (r, s) element is ρ^|r−s|. The matrix Z_i was set equal to X_i.

We considered six different settings: (n = 50, σ = 3), (n = 50, σ = 1), (n = 100, σ = 3), (n = 100, σ = 1), (n = 200, σ = 3), and (n = 200, σ = 1) with a value of ρ = .5 for all settings. For each setting, one design matrix was simulated and 100 data sets (y_i, X_i) for i = 1,…, n were generated.

For each simulated data set, the maximum penalized likelihood (MPL) estimate using the SCAD, LASSO and ALASSO penalties was computed using the random effects and IC_Q penalty estimates. These estimates are denoted as SCAD-RE, SCAD-IC_Q, LASSO-RE, LASSO-IC_Q, ALASSO-RE, and ALASSO-IC_Q, respectively. For the IC_Q estimate, the BIC-type criterion, c_n(θ) = dim(θ) log n, was used. For the Monte Carlo EM algorithm, 2000 Monte Carlo iterations were used within each iteration of EM. For the SCAD and LASSO penalties, we set λ_j = λ₀₁, for j = 1,… 8, and ${\lambda }_{8+k}={\lambda }_{02}\sqrt{k}$, for k = 1,…,3 while for the ALASSO penalty, λ_j = λ₀₁|β̂_j|⁻¹, for j = 1,… 8, and ${\lambda }_{8+k}={\lambda }_{02}\sqrt{k}{\Vert {\widehat{\gamma }}_k\Vert }^{-1}$ for k = 1,…,3 where β̂_j, and γ̂_k are the unpenalized ML estimates of β_j and γ_k respectively, and the penalty (λ₀₁, λ₀₂) was estimated using the IC_Q and random effects penalty selection methods.

For each estimate, the penalized estimate of β and D were computed, denoted as β̂_λ and D̂_λ respectively, and the mean square error ME(β̂_λ) = (β̂_λ − β)^TΣ_xx(β̂_λ − β)^T and the quadratic loss error ME(D̂_λ) = trace[(D̂_λ − D)²]^1/2 were computed. The ratio of the model error of the MPL estimate to that of the unpenalized ML estimate, ME(β̂_λ)/ME(β̂₀) and ME(D̂_λ)/ME(D̂₀), were computed for each data set and the median of the ratios over the 100 simulated data sets, denoted as MRME, was calculated. The MRME of the true model is also reported. In addition, we report two types of errors regarding the fixed and random effects. ZERO₁ is the mean number of type I errors (an effect is truly not significant or random but the corresponding MPL estimate indicates it is significant or random) and ZERO₂ is the mean number of type II errors (an effect is truly significant or random but the corresponding MPL estimate indicates it is not significant or random).

For the MPL estimates, MRME values greater than one indicate that the estimate performs worse than the ML estimate, values near one indicate it performs as good as the ML estimate, while values near the ‘true’ MRME value indicate optimal performance. The values ZERO₁ and ZERO₂ can be interpreted as estimates of the probability of overfit and underfit, respectively, and the value 1 − ZERO₁ − ZERO₂ is an estimate of the probability of selecting the true model. Ideally, one would like to have MPL estimates with small ZERO₁ and ZERO₂ values and small MRME values. Overall, the MRME values of all of the MPL estimates were less than or equal to one, which indicates that regardless of the sample size or noise level, the MPL estimates perform better than the ML estimate. Across all samples sizes and noise levels, the MRME values of the MPL estimates using the random effects penalty estimates was higher than the MPL estimates using the IC_Q penalty estimates. For the IC_Q MPL estimates, as the noise level decreases from σ = 3 to σ = 1, the MRME values increase. For a fixed noise level, the MRME values at sample sizes of n = 50 and n = 200 are comparable but there is a slight decrease in the MRME values at sample sizes of n = 100. This indicates that the MPL estimates perform better, relative to the MLE, at low noise levels and near sample sizes of n = 100. The MPL estimates using the random effects penalty estimate tended to overfit significantly. On average, the MPL estimate using the ALASSO penalty function had smaller estimation error and overfit than the LASSO estimate. For estimating fixed effects, the SCAD-IC_Q estimate has, on average, smaller estimation error and overfit than the other estimates. For estimating the random effects, the ALASSO-IC_Q has smaller error and overfit.

5. Yale Infant Growth Study

We applied the proposed methodology to the Yale infant growth study of Wasserman and Leventhal (1993) and Stier et al. (1993). The Yale infant growth data were collected to study whether cocaine exposure during pregnancy leads to the maltreatment of infants after birth, such as physical and sexual abuse. A total of 298 children were recruited from two subject groups (cocaine exposure group and unexposed group). Throughout the study different children had different numbers and patterns of visits during the study period. The multivariate response was weight of the infant at each visit. Let y_ij denote the weight (in pounds) at the j-th visit of infant i, for i = 1,…, 298, j = 1,…, n_i and let y_i = (y_i1,…, y_ini). The covariates used were: x_ij1 = day of visit, x_ij2 = age (in years) of mother, x_ij3 = gestational age (in weeks) of infant, x_ij4 = race (2 levels: African American and other, coded as 1 and 0), x_ij5 = previous pregnancies (2 levels: no and yes, coded as 1 and 0), x_ij6 = gender of infant (2 levels: male and female, coded as 1 and 0), x_ij7 = cocaine exposure (2 levels: yes and no, coded as 1 and 0). The design matrix X_i is a n_i × 8 matrix with the j-th row equal to (1, x_ij1, x_ij2, x_ij3, x_ij4, x_ij5, x_ij6, x_ij7), Z_i is a n_i × 3 matrix composed of the first 3 continuous covariates of X_i, i.e., the j-th row of Z_i is (x_ij1, x_ij2, x_ij3), and therefore q = 3 here. All covariates were centered in the analysis for numerical stability. Further, we assume that [y_i|X_i; β, D] is normally distributed with mean E(y_i) = X_iβ + Z_iΓb_i, where ΓΓ^T = D and [y_ij |X_i; β, D] and [y_ij′|X_i; β, D] are independent for j ≠ j′.

The objective of this analysis was to determine the significant predictors of infant weight and the significant random effects. Because the ALASSO penalty outperformed the LASSO penalty in the simulations, only the SCAD and ALASSO penalty functions were used along with the IC_Q and random effects penalty estimates. Note that the intercept term was not penalized. For the SCAD, λ_j = λ₀₁ for j = 2,…, 8 and ${\lambda }_{8+k}={\lambda }_{02}\sqrt{k}$, for k = 1,…, 3, while for the ALASSO penalty, λ_j = λ₀₁|β̂_j|⁻¹ for j = 2, …, 8 and ${\lambda }_{8+k}={\lambda }_{02}\sqrt{k}{\Vert {\widehat{\gamma }}_k\Vert }^{-1}$, for k = 1,…, 3, where β̂_j and γ̂_k are the unpenalized ML estimates of β_j and γ_k, respectively, and (λ₀₁, λ₀₂) was estimated using the IC_Q and random effects penalty selection methods.

The results of the analysis are presented in Table 2. The MPL estimates using the SCAD penalty identify visit, gestational age of infant, gender of infant and cocaine exposure as significant predictors of infant weight, and visit as significant random effect. These estimates coincide with the results of the maximum likelihood analysis which identify the same fixed and random effects as significant (significant effects by MLE analysis are indicated by a * in Table 2). The results of using the SCAD with two different sets of penalty estimates are similar. Although the estimates using SCAD with the IC_Q penalty estimates do not shrink the random-effect variances for age and gestational age to 0, these variance estimates are relatively smaller than that of the visit random effect, which still identifies the correct random-effect. The MPL estimate using the ALASSO penalty shrunk two more coefficients of the fixed effects to zero: gender and cocaine. Although these two effects are identified as significant in the MLE, we see that their corresponding MLE estimates are smaller relative to the other significant fixed effects. The estimates using the ALASSO penalty with the IC_Q penalty estimates are close to that of the RE penalty estimates. The MPL estimates using the ALASSO penalty identify visit and gestational age of infant as significant fixed effects, and visit as a significant random effect.

Table 2.

Maximum penalized likelihood estimates of Yale infant grown data comparing SCAD and ALASSO penalty functions with random effects and IC_Q penalty estimates

Fixed Estimatea (Variance Estimate of Random Effectb)
		SCAD		ALASSO
Variable	MLEc	RE	ICQ	RE	ICQ
Intercept	7.002* (-)	6.924 (-)	6.988 (-)	6.913 (-)	6.913 (-)
Visit	2.641* (0.230*)	2.576 (0.087)	2.617 (0.109)	2.543 (0.040)	2.548 (0.067)
Age	−0.035 (0.017)	0.000 (0.000)	0.000 (0.007)	0.000 (0.000)	0.000 (0.000)
Gestation	0.528* (0.017)	0.424 (0.000)	0.455 (0.011)	0.322 (0.000)	0.424 (0.000)
Race	−0.060 (-)	0.000 (-)	0.000 (-)	0.000 (-)	0.000 (-)
Pregnant	−0.004 (-)	0.000 (-)	0.000 (-)	0.000 (-)	0.000 (-)
Gender	0.139* (-)	0.022 (-)	0.033 (-)	0.000 (-)	0.000 (-)
Cocaine	0.103* (-)	0.016 (-)	0.022 (-)	0.000 (-)	0.000 (-)
σ2 d	0.512 (-)	0.552 (-)	0.527 (-)	0.612 (-)	0.594 (-)
ICQe	9223.7	11507.32	9660.013	11999.01	11773.25

^ais estimate of β

^bis estimate of diag(D)

^c* indicates significant effects by MLE analysis

^dis the variance estimate of error term of the linear mixed model

^eis a measure of goodness of fit

6. Discussion

We have proposed a general method which performs simultaneous fixed and random effects selection as well as estimation. Under certain regularity conditions and appropriate assumptions on the penalty parameters, the maximum penalized likelihood estimate possesses oracle properties. We have used two methods for estimating the penalty parameters, the random effects and IC_Q penalty selection methods, and showed that under an appropriate choice of c_n(θ), the IC_Q penalty estimate chooses all the significant fixed and random effects with probability 1. Since penalized likelihood methods have been shown to perform poorly in finite samples, simulations were performed to examine the finite sample properties of the maximum penalized likelihood estimators and the performance of the Monte Carlo EM algorithm. In the simulations, the SCAD and ALASSO penalty functions using the IC_Q penalty estimate performed best and had significantly less estimation error than the maximum likelihood estimate. Unlike previous implementations of the random effects penalty estimate (Garcia, Ibrahim, and Zhu, 2010a, 2010b), the simulations and real data analysis results show that for mixed effects regression models, the random effects penalty estimate has significant overfit. For estimating fixed effects, the SCAD-IC_Q estimate had, on average, smaller estimation error and overfit, while for estimating random effects, the ALASSO-IC_Q had smaller error and overfit.

Many aspects of this work warrant further research and investigation. Recent developments have shown that there may be more than one plausible scheme for formulating the grouped penalty in the penalized likelihood (Zhao et al., 2009; Breheny and Huang, 2009). To select significant random effects using a cholesky parametrization of the covariance matrix of the random effects requires that each row of the cholesky matrix to be penalized as a group. Other parameters, however, can be grouped and penalized in various ways. For instance, it is possible to group parameters corresponding to the fixed effects if one is interested in determining whether a particular group of fixed effects is significant or not. It is also possible to use different penalty functions for each group of parameters.

The objective of this paper was to perform simultaneous selection of fixed and random effects. To the best of our knowledge, this is the first paper to propose this type of methodology. In the existing literature, (Gurka, 2006; Chen and Dunson, 2003; Daniels and Kass, 1999, 2001), the predominant approach to mixed effects selection has been to fix either the mean model or the covariance structure of the random effects and then either test variance components or perform variable selection on the mean model (Keselman et al., 1998). This approach, since it fixes certain parts of the model, makes assumptions regarding the model structure which may not inappropriate. A possible reason that simultaneous mixed effects selection may not have been pursued before is perhaps due to the numerical complexity inherent in the model fitting algorithms. With penalized likelihood methods, however, simultaneous mixed effects selection is straightforward to implement and no assumptions are necessary regarding any part of the model.

As it stands, calculating the IC_Q penalty estimator is slightly demanding. An alternative to IC_Q penalty parameter estimation is to select the penalty parameter which optimizes other criteria developed in mixed effects models such as those in Claeskens and Consentino (2008) and Liang, Wu, and Zou (2008). We will formally study these issues in future work.

Table 1.

Simulation results of linear mixed effects models comparing SCAD, LASSO, and ALASSO penalty functions with random effect and IC_Q penalty estimates

β Estimate (D Estimate)
Model	Method	MRME	ZERO1	ZERO2
n = 50, σ = 3	SCAD-RE	0.576 (0.980)	0.11 (0.94)	0.00 (0.00)
	SCAD-ICQ	0.552 (0.259)	0.01 (0.09)	0.00 (0.01)
	LASSO-RE	0.983 (0.988)	0.99 (1.00)	0.00 (0.00)
	LASSO-ICQ	0.605 (0.241)	0.04 (0.10)	0.00 (0.01)
	ALASSO-RE	0.949 (0.983)	0.80 (1.00)	0.00 (0.00)
	ALASSO-ICQ	0.597 (0.263)	0.01 (0.13)	0.00 (0.01)
	True	0.559 (0.228)	0.00 (0.00)	0.00 (0.00)
n = 50, σ = 1	SCAD-RE	0.906 (0.803)	0.58 (1.00)	0.00 (0.00)
	SCAD-ICQ	0.869 (0.461)	0.03 (0.13)	0.00 (0.00)
	LASSO-RE	0.997 (0.996)	0.99 (1.00)	0.00 (0.00)
	LASSO-ICQ	0.884 (0.438)	0.04 (0.08)	0.00 (0.00)
	ALASSO-RE	0.983 (0.989)	0.81 (1.00)	0.00 (0.00)
	ALASSO-ICQ	0.858 (0.441)	0.03 (0.10)	0.00 (0.00)
	True	0.846 (0.439)	0.00 (0.00)	0.00 (0.00)
n = 100, σ = 3	SCAD-RE	0.571 (0.970)	0.13 (0.93)	0.00 (0.00)
	SCAD-ICQ	0.565 (0.219)	0.01 (0.04)	0.00 (0.00)
	LASSO-RE	0.993 (0.994)	0.99 (1.00)	0.00 (0.00)
	LASSO-ICQ	0.584 (0.232)	0.01 (0.04)	0.00 (0.00)
	ALASSO-RE	0.949 (0.987)	0.81 (1.00)	0.00 (0.00)
	ALASSO-ICQ	0.574 (0.205)	0.01 (0.04)	0.00 (0.00)
	True	0.513 (0.196)	0.00 (0.00)	0.00 (0.00)
n = 100, σ = 1	SCAD-RE	0.895 (0.803)	0.57 (1.00)	0.00 (0.00)
	SCAD-ICQ	0.820 (0.452)	0.01 (0.07)	0.00 (0.00)
	LASSO-RE	0.999 (0.997)	0.99 (1.00)	0.00 (0.00)
	LASSO-ICQ	0.835 (0.478)	0.03 (0.08)	0.00 (0.00)
	ALASSO-RE	0.982 (0.989)	0.82 (1.00)	0.00 (0.00)
	ALASSO-ICQ	0.839 (0.415)	0.02 (0.06)	0.00 (0.00)
	True	0.832 (0.392)	0.00 (0.00)	0.00 (0.00)
n = 200, σ = 3	SCAD-RE	0.553 (0.987)	0.13 (0.94)	0.00 (0.00)
	SCAD-ICQ	0.554 (0.245)	0.01 (0.07)	0.00 (0.00)
	LASSO-RE	0.995 (0.996)	0.99 (1.00)	0.00 (0.00)
	LASSO-ICQ	0.617 (0.244)	0.05 (0.09)	0.00 (0.00)
	ALASSO-RE	0.934 (0.992)	0.78 (1.00)	0.00 (0.00)
	ALASSO-ICQ	0.603 (0.237)	0.02 (0.11)	0.00 (0.00)
	True	0.546 (0.218)	0.00 (0.00)	0.00 (0.00)
n = 200, σ = 1	SCAD-RE	0.902 (0.833)	0.55 (1.00)	0.00 (0.00)
	SCAD-ICQ	0.853 (0.487)	0.01 (0.12)	0.00 (0.00)
	LASSO-RE	0.998 (0.998)	0.99 (1.00)	0.00 (0.00)
	LASSO-ICQ	0.873 (0.554)	0.07 (0.20)	0.00 (0.00)
	ALASSO-RE	0.982 (0.991)	0.79 (1.00)	0.00 (0.00)
	ALASSO-ICQ	0.871 (0.468)	0.02 (0.11)	0.00 (0.00)
	True	0.839 (0.408)	0.00 (0.00)	0.00 (0.00)