Variable Selection for Semiparametric Mixed Models in Longitudinal Studies

Summary

We propose a double-penalized likelihood approach for simultaneous model selection and estimation in semiparametric mixed models for longitudinal data. Two types of penalties are jointly imposed on the ordinary log-likelihood: the roughness penalty on the nonparametric baseline function and a nonconcave shrinkage penalty on linear coefficients to achieve model sparsity. Compared to existing estimation equation based approaches, our procedure provides valid inference for data with missing at random, and will be more efficient if the specified model is correct. Another advantage of the new procedure is its easy computation for both regression components and variance parameters. We show that the double penalized problem can be conveniently reformulated into a linear mixed model framework, so that existing software can be directly used to implement our method. For the purpose of model inference, we derive both frequentist and Bayesian variance estimation for estimated parametric and nonparametric components. Simulation is used to evaluate and compare the performance of our method to the existing ones. We then apply the new method to a real data set from a lactation study.

1. Introduction

Semiparametric mixed models (SPMMs; Diggle et al., 2002; Zhang et al., 1998) are useful extension to linear mixed models (Lard and Ware, 1982; Verbeke and Molenberghs, 2000; Diggle et al., 2002) and provide a flexible framework for analysis of longitudinal data. Many authors have studied semiparametric models for longitudinal data in various setup (e.g., Wang, 1998; He et al., 2002; Diggle et al., 2002; Ruppert et al., 2003; Fan and Li, 2004; Chen and Jin, 2006). An SPMM uses parametric fixed effects to represent covariate effects and a smooth function to model the time effect, modeling the within-subject correlation using random effects and stochastic processes. Zhang et al. (1998) adopted a penalized likelihood approach based on smoothing splines, which is computationally efficient and can be conveniently implemented in standard software.

In longitudinal studies, often times there are a large number of covariates, but usually not all of them are predictive to the response. For example, in a longitudinal lactation study (Sowers et al., 1993), one hundred fifteen pregnant women were initially enrolled to the study and were scheduled to measure the bone mineral density (BMD) at lumbar spine at four postpartum time points within 18 months. Meanwhile, measurements of many covariates describing participants’ physical characteristics, lactation practice and hormonal environment were also taken. One of the objectives of the study is to investigate the pattern of BMD at lumbar spine for postpartum women and to identify from those potential variables the covariates that are associated with the BMD at lumbar spine. Preliminary analysis indicated that on average the BMD at lumbar spine initially declined and then gradually rebounded, and a parametric function may not be adequate to describe this pattern. This motivates the use of an SPMM to model the association of the postpartum BMD at lumbar spine and other covariates and the research to conduct variable selection in SPMMs.

Many variable selection methods have been developed for linear regression models for independent data, such as best subset selection, stepwise selection and shrinkage methods, etc. However, little work had been done for semiparametric models in the longitudinal data settings until the appearance of a recent work of Fan and Li (2004). In the pioneering paper of Fan and Li (2004), they first proposed two estimation procedures to initially estimate regression coefficients: different-based estimator (DBE) and profile least squares. Then they used the local polynomial regression technique to estimate the nonparametric component, and variable selection was achieved by imposing the SCAD (Fan and Li, 2001) penalty on parametric linear covariate effects. As shown in Fan and Li (2004), their methods ignore the correlation in longitudinal data and are therefore very effective in the class of working independent estimators. It is well-known that the estimating equation approach is robust to the mis-specification of the correlation structure in the data when there is no missing data or the missing data mechanism is missing completely at random (MCAR). However, there are some missing data from the motivating lactation study (see Section 6 for more details). To guard against the possible problem associated with missing data, we propose in this paper a double penalized likelihood approach by explicitly taking into account data correlation while conducting model selection and estimation. In particular, our procedure uses random effects to describe the subject-specific effects and uses Gaussian stochastic processes to model the extra temporal correlation among observations within subjects. Since our approach is likelihood based, it will be robust to missing at random (MAR) mechanism and the resulting estimators will be more efficient when the models on the mean and variance structures are correctly specified. Moreover, our method is based on the smoothing spline framework for nonparametric component estimation, which allows us to reformulate the entire problem as a linear mixed effects model (LMM) and hence greatly facilitate the computation. We also show that the variance parameters can be jointly estimated with the smoothing parameters in a unified fashion.

To avoid terminology confusion, we would like to point out that the notion of “double penalty” has been used by various researchers under different contexts. For example, Lin and Zhang (1999) used double penalized quasi-likelihood approach to make inference for generalized additive mixed models, where roughness penalties were imposed on additive nonparametric functions and a quadratic penalty was applied to the random effects. Zou and Hastie (2005) proposed the elastic net for linear model estimation by using a combined penalty of the form ${\lambda }_1{\sum }_j\mid {\beta }_j\mid +{\lambda }_2{\sum }_j{\beta }_j^2$, where β_j are regression coefficients. Lu and Zhang (2006) and Lu (2006) proposed the functional smooth lasso for functional linear model y_i = ∫ x_i(t)f(t)dt + ε_i with i.i.d. ε_i’s, and their procedure has a double penalty on the nonparametric component f as λ₁ ∫ |f(t)|dt+λ₂ ∫{f″(t)}²dt. Different from all the methods above, our procedure involves a roughness penalty for the nonparametric component and a shrinkage SCAD penalty for variable selection of parametric components.

The rest of the article is organized as follows. Section 2 first describes the SPMM framework and useful notations. We then introduce the main method, the double-penalized likelihood method for SPMMs. Section 3 describes the computational algorithm for finding the double penalized likelihood estimators. Section 4 derives the variance estimates for parametric and nonparametric components, from both frequentist and Bayesian perspectives. In Section 5 we demonstrate the effectiveness of our method through simulation studies. We illustrate our method through the application to the data from the lactation study in Section 6. We conclude the article with a discussion in Section 7.

2. Double Penalized Likelihood

2.1 Framework and Notation

Suppose that in a longitudinal study there are m subjects, with the ith subject having n_i observations over time. Denote by y_ij (i = 1,…, m, j = 1,…, n_i) the response at time point t_ij. Consider the following semiparametric mixed model

$$y_{ij}=f(t_{ij})+{\mathbf{x}}_{ij}^T\mathit{\beta }+{\mathbf{z}}_{ij}^T{\mathbf{b}}_i+U_i(t_{ij})+{\epsilon }_{ij},$$ (1)

where f(t) is an arbitrary twice-differentiable smooth function, β is the d × 1 fixed effects vector of potentially a large number of covariates x_ij from which important covariates are to be selected, b_i is an c × 1 vector of subject-specific random effects of the corresponding covariates z_ij, U_i(t) is an independent mean zero Gaussian process modeling extra underlying serial correlation, and ε_ij ~ N (0, σ²) are independent measurement errors. Further assume b_i ~ N {0, G(φ)}, where G(φ) is a positive definite matrix parametrized by vector φ. The Gaussian process U_i(t) has zero mean and variance-covariance function cov{U_i(t), U_i(s)} = γ (ξ, α; t, s) for a specific parametric function γ(·) that depends on a parameter vector ξ and α used to characterize the variance and correlation of the process U_i(t). Zhang et al. (1998) considered several forms of Gaussian processes for modeling various within-subject serial correlation. For example, a stationary Ornsterin-Uhlenbeck (OU) process can be used to model homogeneous within-subject covariance structure with constant variance and exponentially decaying correlation: corr{U_i(t), U_i(s)} = exp(−α|t − s|) (Diggle et al., 2002). If we assume that the variance function changes over time, for instance, in the form of exp(ξ₀ + ξ₁t), then the OU process generalizes to a non-homogeneous Ornsterin-Uhlenbeck (NOU) process. We further assume that b_i, U_i(t) and ε_ij are mutually independent. Note that an SPMM does not have to contain every single term as given in (1), and it can be easily extended to a more complicated model such as the one where ε_ij’s are correlated with a parametric variance-covariance matrix.

We define some matrix notations for convenience. Define Y_i = (y_i1,…, y_ini)^T, and X_i, Z_i, U_i, ε_i similarly (i = 1,…, m). The total number of observations is $n={\sum }_{i=1}^m{n}_i$. Let ${\mathit{t}}^{\mathbf{0}}={(t_1^0,\dots ,t_r^0)}^T$ be an r × 1 vector of ordered distinct values of {t_ij}’s, and N be the incidence matrix mapping t⁰ to {t_ij}. Further denote $\mathbf{Y}={(Y_1^T,\dots ,Y_m^T)}^T$ and X, N, ε similarly, and Z = diag{Z₁,…, Z_m}. Then model (1) can be written in matrix format as Y = Xβ + Nf + Zb + U + ε, where $\mathbf{f}={\{f(t_1^0),\dots ,f(t_r^0)\}}^T,\mathbf{b}={({\mathbf{b}}_1^T,\dots ,{\mathbf{b}}_m^T)}^T\sim \text{N}\{\mathbf{0},\mathcal{G}(\phi )\}$ with variance-covariance matrix (φ) = diag{G,…, G}; $\mathbf{U}={({\mathbf{U}}_1^T,\dots ,{\mathbf{U}}_m^T)}^T\sim \text{N}\{0,\mathbf{\Gamma }(\mathit{\xi },\alpha )\}$ with variance-covariance matrix Γ(ξ, α) = diag{Γ₁,…, Γ_m} and the (j, j′)th element (j, j′= 1,…, n_i) of Γ_i being γ(ξ, α; t_ij, t_ij′); and ε ~ N(0, σ²I_n).

2.2 Double Penalized Likelihood

For fixed variance components θ = (φ^T,ξ^T, α, σ²)^T, the log-likelihood function of (β, f) is (up to a constant)

$$\ell (\beta ,\mathbf{f};\mathbf{Y})=-\frac{1}{2}log\mid \mathbf{V}\mid -\frac{1}{2}{(\mathbf{Y}-\mathbf{X}\mathit{\beta }-\mathbf{Nf})}^T{\mathbf{V}}^{-1}(\mathbf{Y}-\mathbf{X}\mathit{\beta }-\mathbf{Nf}),$$ (2)

where V = diag{V₁,…, V_m} and ${\mathbf{V}}_i={\mathbf{Z}}_i\mathbf{D}{\mathbf{Z}}_i^T+{\mathbf{\Gamma }}_i+{\sigma }^2{\mathbf{I}}_{n_i\times n_i}$. To achieve both model selection and estimation in SPMM (1), we propose to maximize the double penalized (log)likelihood (DPL) function:

$${\ell }_{dp}(\mathit{\beta },f(·);\mathbf{Y})=\ell (\beta ,\mathbf{f};\mathbf{Y})-\frac{{\lambda }_1}{2}{\int }_{T_1}^{T_2}{\{f^{″}(t)\}}^{2}dt-n\sum _{j=1}^d{p}_{{\lambda }_2}(\mid {\beta }_j\mid ),$$ (3)

where λ₁ > 0 is a smoothing parameter controlling the balance between the goodness of fit and the roughness of the estimated f(t), and T₁ and T₂ specify the range of t; p_λ2 (·) is a shrinkage penalty function with λ₂ > 0 controlling the amount of shrinkage. In order for the use of same λ₂ for all β’s to be reasonable, all covariates x are assumed to be standardized. We call the DPL maximizers (β̂, f̂) maximum double-penalized likelihood estimators (MDPLEs). It is easy to show that f̂ is a natural cubic spline estimate for f(t). There are many choices for the shrinkage penalty p_λ2 (·) in (3), and we adopt the smoothly clipped absolute deviation penalty (SCAD) due to its desirable theoretical properties (Fan and Li, 2001, 2004). The SCAD satisfies $p_{{\lambda }_2}^{\prime }(\omega )={\lambda }_2[I(\omega \le {\lambda }_2)+{(a{\lambda }_2-\omega )}_{+}/\{(a-1){\lambda }_2\}I(\omega >{\lambda }_2)]$, where ω > 0 and a > 1 is another tuning parameter, usually taken a priori (Fan and Li, 2001). Selection of two tuning parameters λ₁ and λ₂ usually needs a two-dimensional grid search, which can be time consuming in practice. In order to reduce the computational cost on parameter tuning, we reformulate the problem using a linear mixed model (LMM) representation; in the new representation, λ₁ is treated as the inverse of a variance component and estimated jointly with other variance components using the REML approach. As shown in Section 3, our procedure only requires one-dimensional parameter tuning on λ₂. We discuss the details in the next section.

3. Computational Algorithm

In this section, we formulate a linear mixed model (LMM) representation for the SPMM, and describe the computational procedures for obtaining parameter estimates based on this LMM representation.

3.1 Linear Mixed Model Representation

By the fact that f̂(t) is a cubic smoothing spline and theorem (2.1) of Green and Silverman (1994), the DPL function (3) can be re-written as

$${\ell }_{dp}(\mathit{\beta },f(·);\mathbf{Y})=\ell (\beta ,\mathbf{f};\mathbf{Y})-\frac{{\lambda }_1}{2}{\mathbf{f}}^T\mathbf{Kf}-n\sum _{j=1}^d{p}_{{\lambda }_2}(\mid {\beta }_j\mid ),$$ (4)

where K is the non-negative definite smoothing matrix. Following Green (1987), we have f = Tδ+Ba, where T = [1, t⁰] and 1 is an r×1 vector of 1’s, δ and a are 2×1 and (r−2)×1 vectors respectively, and B = L(L^T L)⁻¹ with L being an r×(r−2) full rank matrix satisfying K = LL^T and L^T T = 0. Note that f^T Kf = a^T a and it yields an equivalent double-penalized log-likelihood ${\ell }_{dp}(\mathit{\beta },\mathit{\delta },\mathbf{a};\mathbf{Y})=-(1/2)log\mid \mathbf{V}\mid -(1/2){(\mathbf{Y}-{\mathbf{X}}_{\ast }{\mathit{\beta }}_{\ast }-{\mathbf{B}}_{\ast }\mathbf{a})}^T{\mathbf{V}}^{-1}(\mathbf{Y}-{\mathbf{X}}_{\ast }{\mathit{\beta }}_{\ast }-{\mathbf{B}}_{\ast }\mathbf{a})-({\lambda }_1/2){\mathbf{a}}^T\mathbf{a}-n{\sum }_{j=1}^d{p}_{{\lambda }_2}(\mid {\beta }_j\mid )$, where X_* = [NT, X], B_* = NB, β_* = (δ^T, β^T)^T. For fixed β_* and given λ₁, λ₂ and θ, maximizing the new DPL with respect to a gives $\widehat{\mathbf{a}}={({\mathbf{B}}_{\ast }^T{\mathbf{V}}^{-1}{\mathbf{B}}_{\ast }+{\lambda }_{1}I)}^{-1}{\mathbf{B}}_{\ast }^T{\mathbf{V}}^{-1}(\mathbf{Y}-{\mathbf{X}}_{\ast }\mathit{\beta })$. Substituting â back to the new DPL, some algebra (Harville, 1977) then leads to an equivalent objective function (up to a constant)

$${\ell }_{dp}({\mathit{\beta }}_{\ast };\mathbf{Y})=-\frac{1}{2}{(\mathbf{Y}-{\mathbf{X}}_{\ast }{\mathit{\beta }}_{\ast })}^T{\mathbf{V}}_{\ast }^{-1}(\mathbf{Y}-{\mathbf{X}}_{\ast }{\mathit{\beta }}_{\ast })-n\sum _{j=1}^d{p}_{{\lambda }_2}(\mid {\beta }_j\mid ),$$ (5)

where ${\mathbf{V}}_{\ast }=\mathbf{V}+\tau {\mathbf{B}}_{\ast }{\mathbf{B}}_{\ast }^T$. The objective function (5) can be regarded as the penalized log-likelihood function of β_* from the following linear mixed model with the same SCAD penalty for β:

$$\mathbf{Y}={\mathbf{X}}_{\ast }{\mathit{\beta }}_{\ast }+{\mathbf{B}}_{\ast }\mathbf{a}+{\epsilon }_{\ast },$$ (6)

where β_* = (δ^T, β^T)^T are new fixed effects, a is treated as random effects distributed as a ~ N(0, τI) with τ = 1/λ₁, ε_* = Zb + U + ε distributed as N(0, V), and θ_* = (τ, θ^T)^T are the variance components. It is then equivalent to conducting variable selection for x in the modified LMM (6). Based on the selected variables and estimated β̂_*, we can use δ̂ and â to construct the smoothing spline fit f̂(t). The LMM representation suggests that the inverse of the smoothing parameter τ can be treated as a variance component and jointly estimated with θ using the maximum likelihood or restricted maximum likelihood (REML) approach, which has been discussed by many authors (e.g. Wahba, 1985; Kohn et al., 1991; Speed, 1991; Zhang et al., 1998; Lin and Zhang, 1999).

3.2 Iterative Ridge Regression

For fixed parameters λ₁, λ₂ and θ, direct maximization of (5) is still difficult due to the singularity of the SCAD function. Following Fan and Li (2001, 2004), we adopt the iterative ridge regression approach via the local quadratic approximation (LQA). For a small constant ξ and an initial value $\mid {\widehat{\beta }}_{\ast j}^0\mid \ge \xi$, we have $\{p_{{\lambda }_2}(\mid {\widehat{\beta }}_{\ast j}\mid ){\}}^{\prime }=p_{{\lambda }_2}^{\prime }(\mid {\widehat{\beta }}_{\ast j}\mid )sgn({\widehat{\beta }}_{\ast j})\approx \left\{p_{{\lambda }_2}^{\prime }(\mid {\widehat{\beta }}_{\ast j}^0\mid )/\mid {\widehat{\beta }}_{\ast j}^0\mid \right\}{\widehat{\beta }}_{\ast j}$. Taylor expansion leads to the following optimization problem

$$\underset{\mathit{\beta }}{max}{\ell }_{dp}({\widehat{\mathit{\beta }}}_{\ast }\mid {\widehat{\mathit{\beta }}}_{\ast }^0)\approx -\frac{1}{2}{(\mathbf{Y}-{\mathbf{X}}_{\ast }{\widehat{\mathit{\beta }}}_{\ast })}^T{\mathbf{V}}_{\ast }^{-1}(\mathbf{Y}-{\mathbf{X}}_{\ast }{\widehat{\mathit{\beta }}}_{\ast })-\frac{1}{2}{\widehat{\mathit{\beta }}}_{\ast }^T{\mathbf{\sum }}_{{\lambda }_2}({\widehat{\mathit{\beta }}}_{\ast }^0){\widehat{\mathit{\beta }}}_{\ast },$$

where ${\mathbf{\sum }}_{{\lambda }_2}(\mathit{\beta })=\text{diag}\{0,0,p_{{\lambda }_2}^{\prime }(\mid {\beta }_1\mid )/\mid {\beta }_1\mid ,\dots ,p_{{\lambda }_2}^{\prime }(\mid {\beta }_d\mid )/\mid {\beta }_d\mid \}$. For fixed θ_* = (τ, θ^T)^T apply the Newton-Raphson method to maximize ${\ell }_{dp}({\widehat{\mathit{\beta }}}_{\ast }\mid {\widehat{\mathit{\beta }}}_{\ast }^0)$ and get the updating formula

$${\widehat{\mathit{\beta }}}_{\ast }={\left\{{\mathbf{X}}_{\ast }^T{\mathbf{V}}_{\ast }^{-1}{\mathbf{X}}_{\ast }+n{\mathbf{\sum }}_{{\lambda }_2}({\widehat{\mathit{\beta }}}_{\ast }^0)\right\}}^{-1}{\mathbf{X}}_{\ast }^T{\mathbf{V}}_{\ast }^{-1}\mathbf{Y},$$ (7)

where ${\mathbf{V}}_{\ast }^{-1}={\mathbf{V}}^{-1}-{\mathbf{V}}^{-1}{\mathbf{B}}_{\ast }{({\tau }^{-1}{\mathbf{I}}_r+{\mathbf{B}}_{\ast }^T{\mathbf{V}}^{-1}{\mathbf{B}}_{\ast })}^{-1}{\mathbf{B}}_{\ast }^T{\mathbf{V}}^{-1}$, which is computationally more efficient when r (number of distinct t_i’s) is much smaller than n (total sample size). Following each updating step, we set β̂_j = 0 for |β̂_j| < η, where η is a small threshold, say η = 10⁻⁵. The non-zero coefficients correspond to important covariates. The iterative ridge regression converges when ${max}_j\{\mid {\widehat{\beta }}_{\ast j}^{(k)}-{\widehat{\beta }}_{\ast j}^{(k-1)}\mid /\mid {\widehat{\beta }}_{\ast j}^{(k-1)}\mid \}<\mathit{tol}$, where tol is a small tolerance.

Remark 1

Very recently, (Zou and Li, 2008) proposed the local linear approximation (LLA) algorithm to solve a SCAD problem. Applying the LLA algorithm to (5) leads to the following optimization problem

$$\underset{\mathit{\beta }}{max}{\ell }_{dp}({\widehat{\mathit{\beta }}}_{\ast }\mid {\widehat{\mathit{\beta }}}_{\ast }^0)\approx -\frac{1}{2}{(\mathbf{Y}-{\mathbf{X}}_{\ast }{\widehat{\mathit{\beta }}}_{\ast })}^T{\mathbf{V}}_{\ast }^{-1}(\mathbf{Y}-{\mathbf{X}}_{\ast }{\widehat{\mathit{\beta }}}_{\ast })-\sum _{j=1}^d{p}_{{\lambda }_2}^{\prime }(\mid {\widehat{\beta }}_j^0)\mid {\beta }_j\mid ,$$

which provides an alternative way to solve the problem.

3.3 Iterative Variable Selection Algorithm

We now outline an iterative algorithm that alternatively eliminates unimportant variables and updates parameter estimates. First we fit the full linear mixed model (6) in SAS by including all covariates in the model, and compute the initial values (β̂_*[1], â_[1]) and (τ̂_[1], θ̂ _[1]). For a given λ₂, we propose the following iterative variable selection algorithm:

Step 1

Initialize with s = 1 and (β̂_*[1], â_[1], τ̂_[1], θ̂_[1]).

Step 2

Compute β̂_*[s+1] using the iterative ridge regression (7) based on current values of β̂_*[s] and (τ̂_[s], θ̂_[s]).

Step 3

Obtain τ̂_[s+1] and θ̂ _[s+1] using REML based on important variables.

Step 4

Let s = s + 1. Go to Step 2 until the selected covariates converge to a stable set.

Now we describe the REML estimation procedure in Step 3. Denote by X_[s] the subset of important variables selected from X at the sth iteration. The REML log-likelihood of (τ, θ) at this iteration is

$${\ell }_R^{[s]}(\tau ,\mathit{\theta };\mathbf{Y})=-\frac{1}{2}log\mid {\mathbf{V}}_{\ast }\mid -\frac{1}{2}log\mid {\mathbf{X}}_{\ast [s]}^T{\mathbf{V}}_{\ast }^{-1}{\mathbf{X}}_{\ast [s]}\mid -\frac{1}{2}{(\mathbf{Y}-{\mathbf{X}}_{\ast [s]}{\widehat{\mathit{\beta }}}_{\ast [s]})}^T{\mathbf{V}}_{\ast }^{-1}(\mathbf{Y}-{\mathbf{X}}_{\ast [s]}{\widehat{\mathit{\beta }}}_{\ast [s]}),$$

where X_*[s] = [NT, X_[s]], and ${\widehat{\mathit{\beta }}}_{\ast [s]}={({\mathbf{X}}_{\ast [s]}^T{\mathbf{V}}_{\ast }^{-1}{\mathbf{X}}_{\ast [s]})}^{-1}{\mathbf{X}}_{\ast [s]}^T{\mathbf{V}}_{\ast }^{-1}\mathbf{Y}$ is the MLE of β_* based on the selected important variables X_[s]. Differentiating ${\ell }_R^{[s]}(\tau ,\mathbf{V};\mathbf{Y})$ with respect to τ and θ_k (the kth element of θ) and using the identity ${\mathbf{V}}_{\ast }^{-1}(\mathbf{Y}-{\mathbf{X}}_{\ast [s]}{\widehat{\mathit{\beta }}}_{[s]})={\mathbf{V}}^{-1}(\mathbf{Y}-{\mathbf{X}}_{[s]}{\widehat{\mathit{\beta }}}_{[s]}-\mathbf{N}\widehat{\mathbf{f}})$, we obtain the REML estimating equations for τ and θ (Harville, 1977):

$$-\frac{1}{2}\text{tr}(\mathbf{P}{\mathbf{B}}_{\ast }{\mathbf{B}}_{\ast }^T)+\frac{1}{2}{(\mathbf{Y}-{\mathbf{X}}_{[s]}{\widehat{\mathit{\beta }}}_{[s]}-\mathbf{N}\widehat{\mathbf{F}})}^T{\mathbf{V}}^{-1}{\mathbf{B}}_{\ast }{\mathbf{B}}_{\ast }^T{\mathbf{V}}^{-1}(\mathbf{Y}-{\mathbf{X}}_{[s]}{\widehat{\mathit{\beta }}}_{[s]}-\mathbf{N}\widehat{\mathbf{F}})=0$$ (8)

$$-\frac{1}{2}\text{tr}(\mathbf{P}\frac{\partial \mathbf{V}}{\partial {\theta }_k})+\frac{1}{2}{(\mathbf{Y}-{\mathbf{X}}_{[s]}{\widehat{\mathit{\beta }}}_{[s]}-\mathbf{N}\widehat{\mathbf{f}})}^T{\mathbf{V}}^{-1}\frac{\partial \mathbf{V}}{\partial {\theta }_i}{\mathbf{V}}^{-1}(\mathbf{Y}-{\mathbf{X}}_{[s]}{\widehat{\mathit{\beta }}}_{[s]}-\mathbf{N}\widehat{\mathbf{f}})=0,$$ (9)

where $\mathbf{P}={\mathbf{V}}_{\ast }^{-1}-{\mathbf{V}}_{\ast }^{-1}{\mathbf{X}}_{\ast [s]}{({\mathbf{X}}_{\ast [s]}^T{\mathbf{V}}_{\ast }^{-1}{\mathbf{X}}_{\ast }^T)}^{-1}{\mathbf{X}}_{\ast [s]}^T{\mathbf{V}}_{\ast }^{-1}$. Since V_* is no longer block-diagonal, direct inverse of V_* may be intractable. We can tackle this problem by using the identity: P = V⁻¹ − V⁻¹ℋD⁻¹ℋ^T V⁻¹, where ℋ is the coefficient matrix of the following system of equations

$$\left[\begin{array}{cc}{\mathbf{X}}_{\ast }^T\mathbf{W}{\mathbf{X}}_{\ast }& {\mathbf{X}}_{\ast }^T\mathbf{W}{\mathbf{B}}_{\ast }\\ {\mathbf{B}}_{\ast }^T\mathbf{W}{\mathbf{X}}_{\ast }& {\mathbf{B}}_{\ast }^T\mathbf{W}{\mathbf{B}}_{\ast }+{\lambda }_1\mathbf{I}\end{array}\right]\left[\begin{array}{c}{\mathit{\beta }}_{\ast }\\ \mathbf{a}\end{array}\right]=\left[\begin{array}{c}{\mathbf{X}}_{\ast }^T\mathbf{W}\mathbf{Y}\\ {\mathbf{B}}_{\ast }^T\mathbf{W}\mathbf{Y}\end{array}\right].$$

The Fisher scoring algorithm can then be used to solve (8) and (9) for τ and θ.

The aforementioned algorithm is based on fixed SCAD tuning parameters. Fan and Li (2001) recommended to use a = 3.7 as it minimizes the Bayesian risk, and thus we set a = 3.7 in our implementation. We propose to tune λ₂ using the Bayesian Information Criterion (BIC) (Schwarz, 1978). For a fixed λ₂, let X₁ and β₁ be the covariate matrix and coefficients respectively corresponding to the q variables selected by the iterative variable selection algorithm. Fit LMM (6) using the important variables to get Ŷ= SY, where S is a smoother matrix with q₁ = trace(S). Then BIC(λ₂) = −2ℓ₁ + q₁ log n, where ℓ₁ = −(n/2) log(2π) − 1/2 log |V| − (1/2)(Y − X₁β̂₁ − Nf̂)^TV⁻¹(Y − X₁β̂₁ − Nf̂). The generalized cross validation (GCV; Craven and Wahba, 1979) can also be used for tuning λ₂. Wang et al. (2007) compared BIC and GCV for selecting the SCAD tuning parameter and suggested that BIC leads to consistent model selection, whereas GCV tends to have an overfitting effect. Model selection results in our simulation studies also favor BIC, and therefore we chose BIC for tuning λ₂.

4. Frequentist and Bayesian Standard Errors

We derive the frequentist and Bayesian variance formulas for β̂ and f̂. The proposed variance estimates are evaluated via simulations. From frequentists’ point of view, var(Y|t, x) = V. At convergence, we can write β̂_* = (δ̂^T, β̂^T)^T as an approximately linear function of $\mathbf{Y}:{\widehat{\mathit{\beta }}}_{\ast }={\left\{{\mathbf{X}}_{\ast }^T{\mathbf{V}}_{\ast }^{-1}{\mathbf{X}}_{\ast }+n{\mathbf{\sum }}_{{\lambda }_2}(\widehat{\mathit{\beta }})\right\}}^{-1}{\mathbf{X}}_{\ast }^T{\mathbf{V}}_{\ast }^{-1}\mathbf{Y}\equiv \mathbf{Q}\mathbf{Y}$. Let $\mathbf{Q}={({\mathbf{Q}}_1^T,{\mathbf{Q}}_2^T)}^T$, where Q₁ and Q₂ are partitions of Q with dimensions corresponding to (δ^T, β^T)^T, so that δ̂ = Q₁Y, and β̂ = Q₂Y. The estimated variance-covariance matrix for β̂ is given by

$${\widehat{var}}_F(\widehat{\mathit{\beta }})={\mathbf{Q}}_2\widehat{var}(\mathbf{Y}){\mathbf{Q}}_2^T={\mathbf{Q}}_2\widehat{\mathbf{V}}{\mathbf{Q}}_2^T.$$ (10)

Denote â (β_*) = S_aY, where S_a = A˜ (I − X_*Q) and $\stackrel{\sim }{\mathbf{A}}={({\lambda }_1\mathbf{I}+{\mathbf{B}}_{\ast }^T{\mathbf{V}}^{-1}{\mathbf{B}}_{\ast })}^{-1}{\mathbf{B}}_{\ast }^T{\mathbf{V}}^{-1}$. Therefore f̂ = Tδ̂ + Bâ = (TQ₁ + BS_a)Y and its variance-covariance matrix estimate is

$${\widehat{var}}_F(\widehat{\mathbf{f}}\mid t,\mathbf{x})={\widehat{\sigma }}^2(\mathbf{T}{\mathbf{Q}}_1+\mathbf{B}{\mathbf{S}}_a){(\mathbf{T}{\mathbf{Q}}_1+\mathbf{B}{\mathbf{S}}_a)}^T.$$ (11)

From a Bayesian perspective, the double penalized likelihood structure indicates that f(t) has a prior in the form of f = Tδ + Ba, with a ~ N(0, τI) and a flat prior for δ. It follows from the LQA in Section 3.2 that the important coefficients β has a prior with log-density kernel equal to −β^TΣ_λ2β/2. The definition of the SCAD penalty implies that some diagonal elements of Σ_λ2 can be zero, corresponding to those |β_j| > aλ₂. Assume after reordering, Σ_λ2 = diag{0, Σ₂₂}, where Σ₂₂ has positive diagonal elements. Then β can be partitioned into ${({\mathit{\beta }}_1^T,{\mathit{\beta }}_2^T)}^T$, where ${\mathit{\beta }}_1^{d_1\times 1}$ can be regarded as “fixed” effects and ${\mathit{\beta }}_2^{d_2\times 1}$ as “random” effects with ${\mathit{\beta }}_2\sim \text{N}(\mathbf{0},{\mathbf{\sum }}_{22}^{-1})$. The matrix X is partitioned into [X₁, X₂] accordingly. Now we reformulate the mixed model (6) as: Y = NTδ + X₁β₁ + X₂β₂ + B_*a + ε_*, or Y = _*γ + Z_*b_* + ε_*, where _* = [NT, X₁], $\mathit{\gamma }={({\mathit{\delta }}^T,{\mathit{\beta }}_1^T)}^T$, Z_* = [X₂, B_*], and ${\mathbf{b}}_{\ast }={({\mathit{\beta }}_2^T,{\mathbf{a}}^T)}^T$ are the new random effects distributed as N(0, Σ_b) with ${\mathbf{\sum }}_b=\text{diag}\{{\mathbf{\sum }}_{22}^{-1},\tau {\mathbf{I}}_{r-2}\}$, and ε_* = Zb + U + ε distributed as N(0, V). Following Henderson (1975), the variance-covariance matrix of γ̂ and b̂_* under this new mixed model is estimated by

$$var\left(\begin{array}{c}\widehat{\mathit{\gamma }}\\ {\widehat{\mathbf{b}}}_{\ast }-{\mathbf{b}}_{\ast }\end{array}\right)={\mathbf{C}}_{\ast }^{-1}={\left(\begin{array}{cc}{\mathcal{X}}_{\ast }^T{\mathbf{V}}^{-1}{\mathcal{X}}_{\ast }& {\mathcal{X}}_{\ast }^T{\mathbf{V}}^{-1}{\mathbf{Z}}_{\ast }\\ {\mathbf{Z}}_{\ast }^T{\mathbf{V}}^{-1}{\mathcal{X}}_{\ast }& {\mathbf{Z}}_{\ast }^T{\mathbf{V}}^{-1}{\mathbf{Z}}_{\ast }+{\mathbf{\sum }}_b^{-1}\end{array}\right)}^{-1},$$

where C_* is the corresponding coefficient matrix of Henderson’s mixed model equations. From this it is straightforward to construct the Baysian variance-covariance matrices of β̂ and f̂. Denote by A_β1,β2 the matrix formed by the block matrices of ${\mathbf{C}}_{\ast }^{-1}$ corresponding to β₁ and β₂. Similarly denote by A_δ,a the matrix corresponding to δ and a. Then the Bayesian variance-covariance matrices for β̂ and f̂ are

$${var}_B(\widehat{\mathit{\beta }})={\mathbf{A}}_{{\mathit{\beta }}_1,{\mathit{\beta }}_2},$$ (12)

$${var}_B(\widehat{\mathbf{f}})=[\mathbf{T},\mathbf{B}]{\mathbf{A}}_{\mathit{\delta },\mathbf{a}}{[\mathbf{T},\mathbf{B}]}^T.$$ (13)

These Bayesian variance estimates can be viewed to account for the biases in β̂ and f̂ due to the imposed penalties (Wahba, 1983). We evaluate and compare the empirical performance of frequentist and Bayesian variance estimates in the next section.

5. Simulation Studies

We conduct simulation to evaluate the performance of our DPL procedure and compare this new procedure with the SCAD and LASSO methods in Fan and Li (2004) in both full data and missing data cases. The SCAD and LASSO estimates are computed as Fan and Li’s (2004) method implemented in their original Matlab programs. Following Fan and Li (2004), we set the bandwidth h for local polynomial regression equal to h₀× interquartile range of observed t_ij’s. We varied the value of h₀ and found h₀ = 0.1 is a reasonable choice in terms of model errors (defined later) in most cases. Therefore we used h₀ = 0.1 for SCAD and LASSO estimates.

We consider three scenarios in the simulation: full data, missing at random (MAR) data and independent data. For the full data scenario, we simulate longitudinal data from the following semiparametric mixed model:

$$y_{ij}=f(t_{ij})+{\mathit{x}}_i^T\mathit{\beta }+b_{0i}+U_i(t_{ij})+{\epsilon }_{ij},$$ (14)

where i = 1,…, m (m = 40 or 60), and j = 1,…, 10. We consider a staggered entry design: the subjects are equally divided into 5 groups with each group entering at time point 1 through 5; each subject has 10 equally spaced (5 time points in between) measurements. We choose f(t) = 4 sin(2πt/4) as the baseline function. By design, there are 50 knots for the smoothing spline fit, and the time t ranges from 0 to 4. The true regression coefficients are β = (.8, .8, 0, 0, .8, 0, 0, 0)^T, with eight mutually independent covariates (x₁,…, x₈) generated from a standard normal distribution. The random intercept b_0i ~ N(0, v₁) represents among-subject variation, and we set v₁ = 0.36. We choose a stationary Ornstein-Uhlenbeck (OU) process U(t) with a constant variance ${\sigma }_u^2$ and an exponentially decaying serial correlation: corr {U_i(t), U_i(s)} = exp(−α|t − s|) =ρ^|t−s|, where ρ = 0.4 and ${\sigma }_u^2=0.5$. We also simulate the measurement error ε_ij from $\text{N}(0,{\sigma }_{\epsilon }^2)$, with ${\sigma }_{\epsilon }^2=0.25$. In order to investigate the impact on the performance of the DPL under a mis-specified correlation structure, we also apply the proposed DPL approach by assuming a random slope (of t) instead of the random intercept in model (14). This approach is labeled as DPL-MIS in Table 1.

Table 1.

Model selection and estimation results. model (14). The number in parentheses are standard errors. MRME is the median relative model error in 100 simulation runs. “Corr” and “Inc” respectively denote the average number of correct and incorrect zeros. The number in the parentheses of “Corr. (5)” and “Inc. (0)” are the corresponding measures for the true model.

			Zero coef.		Proportion of
Scenario	Method	MRME	Corr.(5)	Inc.(0)	Under-fit	Correct-fit	Over-fit
Full data m = 40	DPL	0.31	4.87	0.04	0.03	0.87	0.10
	DPL-MIS	0.35	4.77	0.03	0.03	0.78	0.19
	SCAD	0.44	4.79	0	0	0.82	0.18
	LASSO	0.43	4.77	0	0	0.81	0.19
Full data m = 60	DPL	0.33	4.90	0	0	0.94	0.06
	DPL-MIS	0.32	4.86	0	0	0.91	0.09
	SCAD	0.46	4.76	0	0	0.80	0.20
	LASSO	0.45	4.74	0	0	0.78	0.22
MAR data m = 40	DPL	0.24	4.83	0.06	0.04	0.83	0.14
	SCAD	0.58	4.53	0.02	0.02	0.66	0.34
	LASSO	0.59	4.48	0.01	0.01	0.60	0.40
Ind. data m = 40	DPL	0.19	4.93	0	0	0.93	0.07
	SCAD	0.14	4.96	0	0	0.96	0.04
	LASSO	0.15	4.96	0	0	0.96	0.04

For missing data scenario, we maintain m = 40 and simulate MAR data using the main model (14) and the drop-out model in Section 13.3 of Diggle et al. (2002). The ith (i = 1, 2, ···, 40) subject has no missing value in the first six observations y_i1,…, y_i6 and for j = 7,…, 10, the drop-out probability p_ij of subject i at the jth time point depends on the last previous observation through a logit model: logit(p_ij) = −1 − 2y_i,j−1. This yielded approximately 30% missing data out of the total 400 observations in the full data. For independent data scenario, we remove the random intercept and the stochastic process in model (14) and simulate independent data for m = 40, and then apply the DPL approach assuming an over-parametrized SPMM with a random intercept.

Following Wang et al. (2007), we use the median relative model error (MRME) to evaluate the overall performance of a procedure for model selection and estimation in Table 1. By the simulation design described above, we define the overall model error (ME) of a procedure as

$$\text{ME}={(\widehat{\mathit{\beta }}-\mathit{\beta })}^T(\widehat{\mathit{\beta }}-\mathit{\beta })+{\int }_0^T{\{\widehat{f}(t)-f(t)\}}^{2}dt/T,$$

where [0, T = 4] specifies the range of the variable t, and the second term is approximated by averaging {f̂(t)−f(t)}² over the design knots. The MRME is then the median of ratios of the ME of a selected model to the commone ME of the estimates from the full model fitted by the REML approach of Zhang et al. (1998). To present a more comprehensive picture, we also use other criteria in Table 1 for performance evaluation. The columns labeled by “Corr.” and “Inc.” denote the average numbers of correct and incorrect zeros in the coefficient estimates over 100 simulation runs. The column labeled by “Under-fit” corresponds to the proportion of excluding any nonzero coefficients. Similarly, we report the proportion of selecting the exact subset model in the column “Correct-fit” and the proportion of including all three important variables plus some noise variables in the column “Over-fit”.

As can be seen from Table 1, DPL performs very well in terms of all evaluation criteria. The DPL out-performs SCAD and LASSO in terms the overall MRME criterion, even for the cases of mis-specified correlation structures. Although DPL occasionally is slightly more likely than SCAD and LASSO to under-fit the true model by shrinking some important coefficients to zero, it gives a very competitive performance. In the case of data with missing at random mechanism, the performance of SCAD and LASSO deteriorates as expected, particularly with respect to MRME due the poor performance of f̂ (t) (Figure 2), while the performance of DPL is not affected by the data missing at random. Although the DPL with a mis-specified correlation structure does not perform as well as the correctly specified DPL, it still gives satisfactory performance and outperforms SCAD and LASSO for a moderate sample size (m = 60). In the independent data scenario, although SCAD and LASSO have slightly better performance (as expected), the over-parameterized DPL still delivers a very competitive performance in terms of all evaluation criteria. The estimated variance for random intercept for the over-parameterized DPL is 0.002, indicating that the DPL approach is flexible to produce rather accurate variance component estimates with an over-parameterized correlation structure.

Figure 2.

Plots for estimated f(t) in the missing at random data (m=40) scenario based on 100 samples. Details are as in Figure 1.

Table 2 summaries the estimated (β₁,β₂,β₅), their relative biases, empirical and model-based standard errors and 95% coverage probabilities. Since SCAD and LASSO give similar numerical results, we only report DPL and SCAD results here. Compared with SCAD (and LASSO), our estimated β_j’s have smaller biases in all cases except the independent data scenario. The frequentist and Bayesian standard error formulas derived in Section 4 perform well in most cases: they are close to the empirical estimates and the 95% coverage probability rates are around the nominal level. One observation is that the derived model-based standard errors tend to be under-estimated. We observe that when sample sizes increases (m = 60), the differences get smaller. This suggests that the derived standard error formulas may work well when sample size is reasonably large. Although not reported here due to space limit, the estimates from DPL with mis-specified correlation structures are still very good compared to those under correct model specification. Table 3 presents the estimated variance components using REML based on the modified LMM. When the correlation model is correctly specified, the biases become smaller when sample size increases. The variance component estimates are biased under mis-specified correlation structures, as we expected.

Table 2.

Estimated regression coefficients for important covariates in model (14), standard errors and coverage probabilities (CPs) of 95% confidence intervals

					Model-based SE		95% CP
Method	Model Parameter	Point Estimate	Relative Bias	Empirical SE	Freq.	Bayes.	Freq.	Bayes.
Scenario 1: Full data, m = 40
DPL	β1	0.789	−0.014	0.167	0.129	0.129	0.93	0.93
	β2	0.807	0.008	0.156	0.135	0.135	0.92	0.92
	β5	0.787	−0.016	0.150	0.130	0.130	0.91	0.91
SCAD	β1	0.775	−0.031	0.141	0.116	-	0.89	-
	β2	0.789	−0.014	0.141	0.117	-	0.91	-
	β5	0.764	−0.045	0.145	0.115	-	0.82	-
Scenario 2: Full data, m = 60
DPL	β1	0.795	−0.006	0.098	0.103	0.103	0.94	0.94
	β2	0.807	0.009	0.101	0.105	0.105	0.93	0.93
	β5	0.795	−0.007	0.105	0.103	0.103	0.93	0.93
SCAD	β1	0.783	−0.021	0.105	0.096	-	0.90	-
	β2	0.795	−0.006	0.108	0.097	-	0.90	-
	β5	0.780	−0.025	0.115	0.097	-	0.89	-
Scenario 3: Missing data, m = 40
DPL	β1	0.780	−0.026	0.177	0.144	0.144	0.93	0.93
	β2	0.795	−0.006	0.151	0.151	0.151	0.96	0.96
	β5	0.782	−0.022	0.144	0.144	0.144	0.93	0.93
SCAD	β1	0.765	−0.043	0.163	0.120	-	0.81	-
	β2	0.776	−0.029	0.169	0.120	-	0.86	-
	β5	0.755	−0.056	0.169	0.115	-	0.85	-
Scenario 4: Independent data, m = 40
DPL	β1	0.773	−0.034	0.045	0.026	0.026	0.71	0.71
	β2	0.777	−0.029	0.048	0.028	0.028	0.77	0.77
	β5	0.773	−0.034	0.049	0.027	0.027	0.81	0.81
SCAD	β1	0.798	−0.002	0.028	0.025	-	0.91	-
	β2	0.804	0.005	0.033	0.027	-	0.87	-
	β5	0.800	0.000	0.028	0.025	-	0.91	-

Table 3.

Estimated variance components in the simulation study. Re. bias denotes relative bias.

	Full data (m = 40)		Full data (m = 60)
Parameter	Estimate (SD)	Re. bias	Estimate (SD)	Re. bias
DPL
v1(intercept)	0.36 (0.22)	0.00	0.33 (0.17)	−0.08
ρ	0.38 (0.23)	−0.05	0.40 (0.20)	0.00
${\sigma }_u^2$	0.58 (0.19)	0.16	0.56 (0.14)	0.12
${\sigma }_{\epsilon }^2$	0.21 (0.11)	−0.16	0.22 (0.09)	−0.12
DPL-MIS
v1(slope)	0.08 (0.18)	-	0.06 (0.11)	-
ρ	0.72 (0.09)	0.80	0.72 (0.08)	0.80
${\sigma }_u^2$	0.85 (0.17)	0.70	0.84 (0.15)	0.68
${\sigma }_{\epsilon }^2$	0.38 (0.09)	0.52	0.38 (0.08)	0.52

The estimated baseline function f̂(t) is also evaluated through visualization. We plot and compare the estimated f(t) and point-wise biases by DPL, LASSO and SCAD, for the full data (m = 40) and missing at random data scenarios. For our DPL estimate f̂, we also plot the point-wise empirical and model-based frequentist and Bayesian standard errors, and coverage probability of 95% confidence intervals. Figure 1(b) shows that in full data (m = 40) case, our approach yields smaller overall biases than the LASSO and SCAD. It can be seen from Figure 1(c) and Figure 1(d) that our standard error formulas for f̂ work well. The Bayesian standard errors are always larger in that they account for the biases due to the imposed penalties.

Figure 1.

Plots for estimated f(t) in the full data (m=40) scenario based on 100 samples. Plots (a) and (b) show the averaged fit and point-wise bias by the DPL, SCAD and LASSO methods; plot (c) shows the DPL Bayesian and frequentist standard errors against empirical estimates; and plot (d) plots the averaged coverage probability rates for 95% confidence intervals.

Figure 2 depicts the results for missing at random data. As shown in Figure 2(a) and Figure 2(b), SCAD and LASSO produced large biases due to dropped out subjects after the sixth time point. In contrast, our DPL method maintains consistent performance in estimation of f(t) despite 30% missing data. In Figure 2(c), we notice the disparity between the empirical and the estimated standard errors after t ≥ 7 where the dropout starts occurring, which is due to the smaller sample size caused by the missing data.

6. Application to Longitudinal Lactation Study

In this section, we apply the proposed model selection and estimation procedure for SPMMs to the longitudinal data from the lactation study introduced in Section 1. The detailed description of the study can be found in Sowers et al. (1993). Briefly, the study originally enrolled 115 pregnant women with 0 or 1 parity, who had no intent to breast-feed their incoming babies or intended to breast-feed for at least 6 months. The study participants were then scheduled to have BMD at lumbar spine measured at 4 time points after the birth of their babies. The scheduled time points are 2 weeks, 6 months, 12 months and 18 months postpartum. However, due to various logistic reasons, the actual observation times deviate somewhat from the schedule and several participants missed some scheduled visits. For each woman who made the scheduled visit, the information about her physical activity, dietary calcium intake, lactation practice was obtained. At the same time, blood sample was drawn and assay was conducted to measure various serum hormones including prolactin, parathyroid hormone (PTH) and parathyroid hormone-related peptide (PTHrP), traditionally thought to be related to calcium mobilization for lactating mothers. Since only about 56% of the PTHrP measurements were above the detection limit, PTHrP is dichotomized according to whether or not it was above the limit. One of the study objectives is to identify covariates from this pool of variables that are associated with the postpartum BMD at lumbar spine.

Preliminary analysis indicates that random intercept is sufficient to account for the correlation in the postpartum BMD at lumbar spine and that on average it declined in the first 4 months, then gradually rebounded to a level close to that at week 2 postpartum. In other words, a parametric function may not be appropriate to describe the pattern of postpartum BMD at lumbar spine. This motivates us to consider variable selection in the following SPMM

$$y_{ij}=f(t_{ij})+{\mathbf{x}}_{ij}^T\beta +b_i+{\epsilon }_{ij},$$ (15)

where y_ij is the postpartum BMD (g/cm²) at lumbar spine for woman i measured at time t_ij, f(t) is a non-parametric function, x_ij is a 11 × 1 vector of 11 variables measured at t_ij: lactation practice (breast-feed or partially breast-feed), physical activity index (mets), body mass index (weight in kilogram/square of height in meters), dietary calcium intake (mg), menstruation status, prolactin (ng/ml), PTH (ng/ml), dichotomized PTHrP as well as the baseline age (year) and infant’s birth weight (pound), $b_i\sim \text{N}(0,{\sigma }_b^2)$ is the woman-specific random effect and ${\epsilon }_{ij}\sim \text{N}(0,{\sigma }_{\epsilon }^2)$ is the independent residual error. For numerical stability and ease of interpretation, the continuous covariates are centered by their means. Furthermore, centered dietary calcium intake was divided by 1000, centered BMI, physical activity index and prolactin were divided by 100. After excluding missing data, ninety six women remained in the analysis with 313 total observations. The number of distinct time points is 71.

We apply the DPL procedure to model (15), and the following variables are selected: dietary calcium intake, menstruation status, prolactin, PTH, baseline age and infant’s birth weight. This finding is consistent to those in the epidemiology literature. The estimates of the corresponding regression coefficients as well as their estimated standard errors are presented in Table 4. It is seen from this table that after adjusting for time effect all selected variables are positively associated with the postpartum BMD at lumbar spine except the baseline age. The estimated nonparametric function f(t) and its 95% pointwise confidence bands given by the frequentist and Bayesian variance estimation are presented in Figure 3. We can see that the BMD in lumber spline decreases in the first six months, starts increasing from the seventh month and reaches a level comparable to the baseline at the end of the study. Frequentist and Bayesian methods give almost identical standard errors, as shown in Figure 3.

Table 4.

Estimated coefficients and frequentist and Bayesian standard errors under model (15) for data from the longitudinal lactation study

Variable	Full Model β̂(SEfreq, SEBayes)	Selected Model β̂ (SEfreq, SEBayes)
Breastfeed	−0.0203 (0.0091, 0.0092)	0(0, 0)
Partial breastfeed	−0.0020 (0.0090, 0.0091)	0(0, 0)
PTHrP above detection	0.0067 (0.0064, 0.0064)	0(0, 0)
Activity index	−0.0034 (0.0071, 0.0072)	0(0, 0)
BMI	0.1928 (0.1820, 0.1822)	0(0, 0)
Calcium	0.0261 (0.0059, 0.0059)	0.0050 (0.0021, 0.0057)
Menstruation status	0.0244 (0.0076, 0.0076)	0.0101 (0.0040, 0.0072)
Prolactin	0.0173 (0.0057, 0.0057)	0.0055 (0.0021, 0.0053)
PTH	0.0419 (0.0176, 0.0180)	0.0208 (0.0143, 0.0180)
Age	−0.0053 (0.0013, 0.0013)	−0.0013 (0.0001, 0.0013)
Birth weight	0.0404 (0.0046, 0.0046)	0.0305 (0.0028, 0.0039)

Figure 3.

Plot of estimated baseline functions f(t) in the selected model of (15) and the 95% pointwise confidence intervals. The dotted lines correspond to the frequentist confidence interval, and the dashed lines are given by the Bayesian confidence interval.

7. Discussion

In this paper we have proposed a new double penalized likelihood (DPL) approach for selecting important parametric fixed effects in semiparametric mixed models (SPMM) for longitudinal data. The DPL is equipped with two penalty terms: the roughness penalty for f(t) and a shrinkage penalty for the fixed covariate effects β. Maximizing the DPL leads to a parsimonious model and a smoothing spline fit for f(t). We cast the SPMM into a modified linear mixed model framework and proposed an iterative variable selection algorithm for the computation. Within the LMM framework, the inverse of the smoothing parameter is treated as an extra variance component and can be conveniently estimated jointly with other variance components using REML approach. Simulations demonstrated that our method gives very competitive performance in terms of variable selection and parameter estimation, compared with the SCAD and LASSO method in Fan and Li (2004). Under the correct model specification of an SPMM, including the correct specification on the conditional mean structure of the longitudinal data given covariates, random effects and stochastic processes, and the variance structures of the random effects and the stochastic process, our method is more efficient in terms of model selection and parameter estimation than other existing methods which ignore the correlation structure of the data and use working independence correlation matrix. Model diagnostics through exploratory analysis and visualizations are often useful to ensure a properly specified variance structure for the data. Furthermore, when the data set contains missing data subject to missing at random, our DPL estimates still perform consistently. The usage of the proposed method is demonstrated through its application to the data from the longitudinal lactation study.

The proposed method is for selecting important parametric fixed effects in an SPMM. In the future we hope to conduct variance component selection as well. We also plan to extend our DPL methodology to generalized semiparametric mixed models to incorporate other types of endpoints and more likelihood structures. We hope to derive theoretical properties for the estimators. As pointed out by a referee, Zou and Li (2008) recently proposed the local linear approximation (LLA) algorithm and a one-step LLA approach for solving penalized log-likelihood, which potentially can be used in place of LQA to get computationally and statistically more efficient DPL estimators.