A Note on Conditional AIC for Linear Mixed-Effects Models

Summary

The conventional model selection criterion AIC has been applied to choose candidate models in mixed-effects models by the consideration of marginal likelihood. Vaida and Blanchard (2005) demonstrated that such a marginal AIC and its small sample correction are inappropriate when the research focus is on clusters. Correspondingly, these authors suggested to use conditional AIC. The conditional AIC is derived under the assumptions of the variance-covariance matrix or scaled variance-covariance matrix of random effects being known. We develop a general conditional AIC but without these strong assumptions. This allows Vaida and Blanchard’s conditional AIC to be applied in a wide range. Simulation studies show that the proposed method is promising.

1. INTRODUCTION

Linear mixed-effects (LME) models (Laird and Ware, 1982), as a powerful tool for the analysis of longitudinal data, have been paid more and more attentions because they can incorporate within-cluster and between-cluster variations into consideration. Statistical estimation and inference for LME models have widely been studied and applied in literature (Vonesh and Chinchilli, 1996; Pinheiro and Bates, 2000; Verbeke and Molenberghs, 2000). A fundamental question in LME models, model selection, seems to be disregarded, however. Traditional selection criteria such as AIC (Akaike, 1973) and BIC (Schwarz, 1978) for cross-sectional data have been parallelly applied for the selection of LME models without justification (Pinheiro and Bates, 2000; Ngo and Brand, 2002). This deficiency was recently noticed by Vaida and Blanchard (2005). These authors explicitly elucidated that, when the researchers’ focus is on clusters instead of population, the traditional AIC and its small sample correction AIC_C are not appropriate, and suggested the conditional Akaike information and the corresponding model selection criterion: conditional AIC. However, in deriving the conditional AIC, they required that the variance-covariance matrix of random effects should be known when the variance of the measurement error term is known, or the scaled variance-covariance matrix of random effects should be known when the variance of the measurement error term is unknown. These requirements may limit the use of the conditional AIC. The objective of this note is to remove Vaida and Blanchard’s assumptions and to propose a more general conditional AIC. This will allow Vaida and Blanchard’s conditional AIC to be applied in a wide range. This note considers the case of known error variance. For the case of unknown error variance, a discussion can be found in Liang et al. (2006) which is available from the authors upon request.

2. GENERAL CONDITIONAL AIC FOR LME MODELS

Assume the data y_i from m clusters to be modelled by the following LME model:

$${\mathbf{y}}_i={\mathbf{X}}_i\mathit{\beta }+{\mathbf{Z}}_i{\mathbf{b}}_i+{\mathit{\epsilon }}_i,i=1,\dots ,m,$$ (1)

where y_i is an n_i × 1 vector of observations for cluster i, β is a p × 1 vector of fixed effects, b_i is a q × 1 vector of random effects for cluster i, X_i and Z_i are the n_i × p and n_i × q design matrices for the fixed and random effects of full column rank, respectively, and ε_i is the disturbance. We assume that b_i and ε_i are independently and normally distributed with mean of zero and variance-covariance matrices of G and σ²I_ni, respectively, where I_ni is an n_i × n_i identity matrix. Let $N={\sum }_{i=1}^m{n}_i$be the total number of observations, and let θ be the vector of parameters in the model, including β, σ² and the parameters in G. Model (1) can be written as

$$\mathbf{y}=\mathbf{X}\mathit{\beta }+\mathbf{Zb}+\mathit{\epsilon },\mathbf{b}\sim N(0,\mathbf{G}),$$ (2)

where $\mathbf{y}={({\mathbf{y}}_1^{\text{T}},\dots ,{\mathbf{y}}_m^{\text{T}})}^{\text{T}}$ is an N × 1 vector of observations, $\mathbf{X}={({\mathbf{X}}_1^{\text{T}},\dots ,{\mathbf{X}}_m^{\text{T}})}^{\text{T}}$ is an N × p matrix of rank p, Z = diag(Z₁, …, Z_m) is an N × r block-diagonal matrix of rank r = mq, $\mathbf{b}={({\mathbf{b}}_1^{\text{T}},\dots ,{\mathbf{b}}_m^{\text{T}})}^{\text{T}},\mathit{\epsilon }={({\mathit{\epsilon }}_1^{\text{T}},\dots ,{\mathit{\epsilon }}_m^{\text{T}})}^{\text{T}}$, and G = diag(G, …, G) is a r × r block-diagonal matrix. Denote the joint density function of y and b under model (2) by g(y, b | θ). Thus, given b, the conditional likelihood is g(y | θ, b) and the marginal likelihood is g(y | θ) = ∫ g(y, b | θ)db. Let the true conditional distribution of y is f(y | u), where u is the true random effects vector with distribution p(u), and f(y, u) be the joint density of y and u. Then Vaida and Blanchard (2005) defined the conditional Akaike information as follows.

Definition 1

The conditional Akaike information is defined to be

$$\begin{array}{l}\text{cAI}=-2E_{f(\mathbf{y},\mathbf{u})}{E}_{f({\mathbf{y}}^{\ast }\mid \mathbf{u})}logg\{{\mathbf{y}}^{\ast }\mid \widehat{\mathit{\theta }}(\mathbf{y}),\widehat{\mathbf{b}}(\mathbf{y})\}\\ =\int -2logg\{{\mathbf{y}}^{\ast }\mid \widehat{\mathit{\theta }}(\mathbf{y}),\widehat{\mathbf{b}}(\mathbf{y})\}f({\mathbf{y}}^{\ast }\mid \mathbf{u})f(\mathbf{y},\mathbf{u})d{\mathbf{y}}^{\ast }d\mathbf{y}d\mathbf{u},\end{array}$$ (3)

where y* is the prediction dataset which is independent of y conditional on u and from the same distribution f(· | u) as y, θ̂(y) and b^(y) are the estimators of θ and b, respectively.

The following theorem derives an unbiased estimator of cAI when the variance σ² is known. The proof is given in the Appendix. Let θ̂(y) and b^(y) be the maximum likelihood and the empirical Bayes estimators of θ and b, respectively.

Theorem 1

Assume that the data y have true density f(y | u) = g(y | θ₀, u) for some θ₀ and some random effect u with distribution p(u). Let the data be modelled by (2) with densities denoted by g(y | θ, b) and p(b). If σ² is known, then an unbiased estimator of the cAI in (3) is given by

$$\text{cAIC}=-2logg\{\mathbf{y}\mid \widehat{\mathit{\theta }}(\mathbf{y}),\widehat{\mathbf{b}}(\mathbf{y})\}+2{\mathrm{\Phi }}_0(\mathbf{y}),$$ (4)

where ${\mathrm{\Phi }}_0(\mathbf{y})={\sum }_{i=1}^N\partial {\widehat{y}}_i/\partial y_i=\text{tr}(\partial {\widehat{\mathbf{y}}}^{\text{T}}/\partial \mathbf{y})$, and y_i and ŷ_i are the i-th components of y and the fitted vector ŷ = Xβ̂+ Zb̂, respectively.

From (4), it is seen that unlike for linear fixed-effects models, the penalty term generally depends on the observed data y for LME models. The calculation on the penalty function Φ₀(y) involves the first partial derivatives ∂ŷ/∂y_i (i = 1, …, N) which can be directly calculated or numerically approximated by {ŷ_i (y + he_i) − ŷ_i(y)}/h, where h is a small number and e_i is the N × 1 vector with the i-th component of one and other components of zero.

Remark 1

Vaida and Blanchard (2005) developed a neat result (Theorem 1, p355) for the case of the known G when σ² is known. However, they claimed that no unbiased estimator for cAI such as (9) of their paper (see (5) below) exists for the unknown G. Our Theorem 1 provides an unbiased estimator of cAI for the unknown G when σ² is known.

Corollary 1

(Vaida and Blanchard 2005) Under the assumptions of Theorem 1, further assume that G is known. Then an unbiased estimator of the cAI is

$$\text{cAIC}=-2logg\{\mathbf{y}\mid \widehat{\mathit{\theta }}(\mathbf{y}),\widehat{\mathbf{b}}(\mathbf{y})\}+2\rho ,$$ (5)

where ρ = tr(H₁), H₁ is the “hat” matrix mapping the observed data vector y into the fitted vector ŷ, that is, ŷ = H₁y.

Proof

See the Appendix.

An intuitive explanation on ρ, the penalty term when both σ² and G are known, can be provided as follows: From the definition of H₁ (see the proof of Corollary 1), it can be shown that

$$\rho =p+\sum _{i=1}^{r_0}\frac{{\lambda }_i}{1+{\lambda }_i},$$

where λ₁, …, λ_r0 are the non-zero eigenvalues of the matrix ${\mathbf{D}}_0^{1/2}{\mathbf{Z}}^{\text{T}}(\mathbf{I}-{\mathbf{P}}_{\mathbf{X}})\mathbf{Z}{\mathbf{D}}_0^{1/2}$ with D₀ = σ⁻²G and P_X = X(X^TX)⁻¹X^T. Note that in the scenario of Corollary 1, only β is unknown. So the first term on the right-hand side of the above formula is the total number of parameters in LME model. Thus, unlike for the usual linear fixed-effects model, the penalty term is not only the number of unknown parameters for LME model. The second term in the expression of ρ is the extra penalty due to random effects. Also, observe that this term is smaller than the number of random effects, r, showing that the extra penalty is not the number of random effects terms, although these random effects may be independent (note that the covariate matrix Z in model (2) can be non-block diagonal). Further, when G is unknown, Vaida and Blanchard (2005) suggested to use the observed ρ̂ = tr{H₁(Ĝ)}, where Ĝ is the maximum likelihood estimator of G. Observe that when G is unknown, we have ŷ = H₁(Ĝ)y. So from Theorem 2.1, the exact penalty term when G is unknown will be

$${\mathrm{\Phi }}_0(\mathbf{y})=\widehat{\rho }+{\mathbf{1}}^{\text{T}}\mathbf{H}(\widehat{\mathbf{G}})\mathbf{y},$$

where 1 is the N × 1 vector of ones, and

$$\mathbf{H}(\widehat{\mathbf{G}})=\left(\begin{array}{lll}\frac{\partial h_{11}(\widehat{\mathbf{G}})}{\partial y_1}\hfill & \cdots \hfill & \frac{\partial h_{1N}(\widehat{\mathbf{G}})}{\partial y_1}\hfill \\ \cdots \hfill & \cdots \hfill & \cdots \hfill \\ \frac{\partial h_{N1}(\widehat{\mathbf{G}})}{\partial y_N}\hfill & \cdots \hfill & \frac{\partial h_{NN}(\widehat{\mathbf{G}})}{\partial y_N}\hfill \end{array}\right)$$

with h_ij(Ĝ) being the (i, j)-th element of the matrix H₁(Ĝ) (here we write H as a function of Ĝ but it may depend on y not only through Ĝ). The second term 1^TH(Ĝ)y is the additional penalty due to the variability of estimating unknown G.

Remark 2

In Theorem 1 and Corollary 1, the assumption of f(y | u) = g(y | θ₀, u) means that the true model is included in the candidate model family. This is a traditional assumption in deriving model selection criterion (see, for example, Akaike, 1973; Hurvich and Tsai, 1989; Burnham and Anderson, 1998; and Hurvich et al., 1998). The further assumption of G being known in Corollary 1 implies that the covariate matrices for random effects under the true and candidate models are in fact exactly the same. The removal of this further assumption shows that the covariate matrices for random effects under the true and candidate models can be different. Further, in the proof of Theorem 1, the expression of μ (= Xβ₀ + Zu, where β₀ is the true parameter for fixed effects) is not useful. This means that the traditional assumption that the candidate models include the true one can even be removed. As an example, if the data y come from a LME model mentioned in Vaida and Blanchard (2005): y = Pα + Qv + e with v ~ N(0, S), $\mathbf{e}\sim N(0,{\sigma }_0^2{\mathbf{I}}_N)$, and P and Q containing covariates different from X and Z, then Theorem 1 and Corollary 1 still hold.

3. SIMULATION STUDY

In this section, we describe simulation results to study the behavior of the proposed method under small and moderate sample sizes. To make a comparison, we generate data from the framework that Vaida and Blanchard (2005) used, that is, the data are generated from the model

$$y_{ij}=({\beta }_0+{\beta }_1{t}_j)+(b_{0i}+b_{1i}{t}_j)+{\epsilon }_{ij},i=1,\dots ,m=10,j=1,\dots ,n_i,$$

where β₀ = −2.78, β₁ = −0.186, t_j = 5j, (b_0i, b_1i)^T follows a normal distribution with mean of zero and variance-covariance matrix of $\left(\begin{array}{cc}0.0367& -0.00126\\ -0.00126& 0.00279\end{array}\right)$, ε_ij are iid with N(0, σ²). In our simulation experiments, similarly to Vaida and Blanchard (2005), we consider σ = 0.0705, 0.141, and 0.282 and the following three scenarios (i) j = 0, 1, …, 5, giving n_i = 6; (ii) j = 0, 1, …, 25, giving n_i = 26; and (iii) j = 0, 1, …, 50, giving n_i = 51. For each of the nine configurations, 500 independent sets of data are generated. We mainly compare the estimates of the bias correction (BC, which is defined as cAI = E_{f(y, u)}{−2 log g(y | θ̂, b̂)} + 2BC) based on our proposed method, Φ₀(y), and Vaida and Blanchard’s (2005) method, ρ̂ with the true BC values.

Table 1 summarizes the results of this small simulation study. The results obtained are in accord with the theory. The estimated values based on the proposed method and Vaida and Blanchard’s (2005) method are both close to the BC values, and generally, the larger the sample size, the closer. However, it is worthy of emphasizing that the estimated values based on the former are consistently closer to the true BC values than those based on the latter, showing that our method is promising.

Table 1.

Simulation study. Comparison of BC and its two estimates, ρ̂ and Φ₀(y) based on 500 runs.

ni	σ	BC	ρ̂	Φ0(y)
6	0.0705	19.549	19.994	19.38
26	0.0705	19.875	19.999	19.837
51	0.0705	19.926	19.999	19.891
6	0.141	17.638	19.731	18.253
26	0.141	19.339	19.976	19.355
51	0.141	19.547	19.986	19.597
6	0.282	15.818	16.944	15.436
26	0.282	17.832	19.265	17.927
51	0.282	18.723	19.763	18.648

4. CONCLUDING REMARKS

This note removed the assumption on the variance-covariance matrix of random effects being known in the conditional AIC of Vaida and Blanchard (2005) and developed a more general conditional AIC. This would substantially enlarge the use of the conditional AIC in LME model selection.

It is worthy of noting that the derivation of (A2) in the Appendix does not require the assumption that the candidate models include the true one. This means that when the error variance ${\sigma }_0^2$ under the true model is known, to derive a reasonable model selection criterion, this traditional assumption is not necessary. Further analysis shows that this conclusion is still true if ${\sigma }_0^2$ is the same as the error variance under the candidate model. Also, the assumption of the true model being included in the candidate model family is needed only in the derivation of the estimator of ${\sigma }_0^2$ when it is unknown (c.f., Liang et al., 2006). Noting that ${\sigma }_0^2$ is a nuisance parameter, this explains in part why the commonly used AIC and AIC_C in fixed-effects models often perform well even the candidate model family does not include the true model, although these selection criteria were derived under the above traditional assumption.

Different from the derivation in the model selection literature, we made use of the integration by part technique, which was used to obtain risk-unbiased estimators before (Stein, 1981; Lu and Berger, 1989), to derive the selection criterion for LME models. It can be seen that our method can also be applied to obtain marginal AIC based on the marginal likelihood and overall AIC based on the joint likelihood for LME models, and AIC_C for nonparametric regression models (Hurvich et al., 1998) and single-index models (Naik and Tsai, 2001) etc. Further, the principle of this note may be extended to generalized mixed-effects models, and applied to select smoothing parameters in the semiparametric regression. These topics warrant our future researches.

APPENDIX

Proof of Theorem 1

Denote μ = Xβ₀ + Zu, where β₀ is the true parameter for fixed effects. Then it is readily seen that

$$\begin{array}{l}\text{cAI}=-2E_{f(\mathbf{y},\mathbf{u})}{E}_{f({\mathbf{y}}^{\ast }\mid \mathbf{u})}\text{log}g\{{\mathbf{y}}^{\ast }\mid \widehat{\mathit{\theta }}(\mathbf{y}),\widehat{\mathbf{b}}(\mathbf{y})\}\\ =E_{f(\mathbf{y},\mathbf{u})}\left\{Nlog(2\pi {\sigma }^2)+N+\frac{1}{{\sigma }^2}{(\widehat{\mathbf{y}}-\mathit{\mu })}^{\text{T}}(\widehat{\mathbf{y}}-\mathit{\mu })\right\}.\end{array}$$

Also, we have

$$E_{f(\mathbf{y},\mathbf{u})}\{-2logg(\mathbf{y}\mid \widehat{\mathit{\theta }},\widehat{\mathbf{b}})\}=E_{f(\mathbf{y},\mathbf{u})}\left\{Nlog(2\pi {\sigma }^2)+\frac{1}{{\sigma }^2}{(\mathbf{y}-\widehat{\mathbf{y}})}^{\text{T}}(\mathbf{y}-\widehat{\mathbf{y}})\right\}.$$

Thus, after some calculations, we obtain

$$\begin{array}{ll}\text{cAI}-E_{f(\mathbf{y},\mathbf{u})}\{-2logg(\mathbf{y}\mid \widehat{\mathit{\theta }},\widehat{\mathbf{b}})\}& =\frac{2}{{\sigma }^2}{E}_{f(\mathbf{y},\mathbf{u})}\{{(\mathbf{y}-\mathit{\mu })}^{\text{T}}\widehat{\mathbf{y}}\}\\ & =\frac{2}{{\sigma }^2}{E}_{p(\mathbf{u})}{E}_{f(\mathbf{y}\mid \mathbf{u})}\left\{\sum _{i=1}^N(y_i-{\mu }_i){\widehat{y}}_i\right\},\end{array}$$ (A1)

where μ_i is the i-th component of μ.

Note that under the true model, for given u, y follows a normal distribution with the mean μ and variance-covariance matrix σ²I_N. Assuming that ŷ_i is a continuous function with piecewise continuous partial derivatives with respect to y, it can be shown from the integration by part that

$$E_{f(\mathbf{y}\mid \mathbf{u})}\left\{\sum _{i=1}^N(y_i-{\mu }_i){\widehat{y}}_i\right\}={\sigma }^2{E}_{f(\mathbf{y}\mid \mathbf{u})}\left(\sum _{i=1}^N\frac{\partial {\widehat{y}}_i}{\partial y_i}\right),$$

providing each expectation on the right-hand side exists (see also Stein, 1981; and Lu and Berger, 1989). Therefore, (A1) becomes

$$\begin{array}{ll}\text{cAI}-E_{f(\mathbf{y},\mathbf{u})}\{-2logg(\mathbf{y}\mid \widehat{\mathit{\theta }},\widehat{\mathbf{b}})\}& =2E_{p(\mathbf{u})}{E}_{f(\mathbf{y}\mid \mathbf{u})}\left(\sum _{i=1}^N\frac{\partial {\widehat{y}}_i}{\partial y_i}\right)\\ & =2E_{f(\mathbf{y},\mathbf{u})}\{{\mathrm{\Phi }}_0(\mathbf{y})\}.\end{array}$$ (A2)

Thus, an unbiased estimator of the cAI is given by cAIC in (4) and this completes the proof of Theorem 1.

Proof of Corollary 1

From Hodges and Sargent (2001) or Vaida and Blanchard (2005), when σ² and G are known, the fitted vector is

$$\widehat{\mathbf{y}}=\mathbf{X}\widehat{\mathit{\beta }}+\mathbf{Z}\widehat{\mathbf{b}}={\mathbf{H}}_{\mathbf{1}}\mathbf{y},$$

where H₁ = (X Z)(M^T M)⁻¹(X Z)^T with

$$\mathbf{M}=\left(\begin{array}{cc}\mathbf{X}& \mathbf{Z}\\ \mathbf{O}& -\mathbf{\Delta }\end{array}\right)$$

and Δ being some r × r matrix such that σ⁻²G = (Δ^TΔ)⁻¹. Thus,

$${\mathrm{\Phi }}_0(\mathbf{y})=\text{tr}\left(\frac{\partial {\widehat{\mathbf{y}}}^{\text{T}}}{\partial \mathbf{y}}\right)=\text{tr}({\mathbf{H}}_{\mathbf{1}})=\rho .$$