Statistical Learning Methods for Longitudinal High-dimensional Data

Abstract

Recent studies have collected high-dimensional data longitudinally. Examples include brain images collected during different scanning sessions and time-course gene expression data. Because of the additional information learned from the temporal changes of the selected features, such longitudinal high-dimensional data, when incorporated with appropriate statistical learning techniques, are able to more accurately predict disease status or responses to a therapeutic treatment. In this article, we review recently proposed statistical learning methods dealing with longitudinal high-dimensional data.

1 Introduction

Current biomedical technology enables the collection of high-dimensional data longitudinally to gain understanding of genomic, proteomic, and in vivo neural processing properties over time. The temporal changes in high-profile biological properties may provide insight into disease diagnosis, progression, or recovery. Depending on the types of outcomes and clinical needs, the goals of longitudinal high-dimensional data include clustering and classification, survival analysis, multilevel regression and time series modeling.

By “longitudinal data,” we indicate two types of data collections: (1) high-dimensional profiles are collected at multiple times points during the study but the response variable is only collected at the end of the study as a final outcome; and (2) both the high-dimensional predictor variables and response variable are collected at multiple times points during the study. The desired methodology for high-dimensional longitudinal data would take advantage of the additional data to determine temporal trends of features and incorporate the temporal effects into learning methods and models that allow for repeated measurements. Recent research has developed several strategies to analyze high-dimensional longitudinal data using different statistical learning techniques, including support vector machines, non-parametric Bayesian methods, and shrinkage methods for different purposes. To address different objectives in the context of different data structures, we review several recent methods for high-dimensional longitudinal data. Across these models, the key challenges are determining how to extract features in high-dimensional space and incorporate the temporal effects for more accurate prediction.

In this paper, we review a set of methods for high-dimensional longitudinal data, with focus on longitudinal support vector machine and penalized linear mixed effects models. We begin with basic concepts of each method, and then introduce how recent high-dimensional longitudinal data analysis methods extended from original model. We also review the computational strategies and algorithm implementations for these methods.

2 Methods

In this section, we will review several current statistical methods for use with both types of longitudinal high-dimensional data.

2.1 Longitudinal Support Vector Classifier - LSVC

We first review the statistical methods to deal with the first type of longitudinal high-dimensional data. The support vector (SVC) classifier is a robust and effective machine learning method that has been widely used for high-dimensional data analysis (Mitchell et al., 2004, Vapnik, 1996). Also, SVC has been applied to handle spatial-temporal high-dimensional data, Mourao-Miranda et al, 2007 first use singular value decomposition to obtain the linear combination of spatial and temporal effects and then apply the component as input of SVC. Recently, Chen and Bowman, 2011 developed a SVC based method for high-dimensional data measured at multiple time points. Suppose that longitudinal high-dimensional data is collected from N subjects at T measurement time points, and p represents the dimensionality of data. The expanded feature matrix for longitudinal high-dimensional data becomes TN by p. For features x_i,t collected for subject i at time t, the goal is to classify each individual x̃_i = {x_i,1, x_i,2, …, x_i,T}′ to certain groups y_i ∈ {−1, 1}, outcomes only collected at the end of the study.

Linear trends of change are characterized: x_s = x_i,1 + β₁x_i,2 + β₂x_i,3… + β_T−1x_i,T, where β = (1, β₁, β₂, …, β_T−1)′ is an unknown parameter vector. Such trend information is highly desired to improve the classification accuracy, usually not available. Thus, a key challenge of building a classifier of longitudinal high dimensional data is jointly estimating the separating hyperplane parameters α and the temporal trend parameters β. Chen and Bowman, 2011 first proposed a novel longitudinal support vector classifier (LSVC) to solve the problem using quadratic programming.

The LSVC is extended from the conventional support vector classifier (SVC), and it augments the cross-sectional high-dimensional feature space to a longitudinal high-dimensional feature space. The method seeks to construct an objective function by incorporating both temporal trend parameters and separating the hyperplane parameters. In the paper, the authors first note the augmented Gram matrix as

$$\begin{array}{l}\mathbf{G}={({\stackrel{\sim }{\mathbf{X}}}_{t=1}+{\mathit{\beta }}_1{\stackrel{\sim }{\mathbf{X}}}_{t=2}+\dots +{\mathit{\beta }}_{T-1}{\stackrel{\sim }{\mathbf{X}}}_{t=T})}^T({\stackrel{\sim }{\mathbf{X}}}_{t=1}+{\mathit{\beta }}_1{\stackrel{\sim }{\mathbf{X}}}_{t=2}+\dots +{\mathit{\beta }}_{T-1}{\stackrel{\sim }{\mathbf{X}}}_{t=T})\\ ={({\stackrel{\sim }{\mathbf{X}}}_m{\mathit{\beta }}_m)}^T({\stackrel{\sim }{\mathbf{X}}}_m{\mathit{\beta }}_m)\\ ={\mathit{\beta }}_m^T{\mathbf{G}}_m{\mathit{\beta }}_m,\end{array}$$

where

$${\mathbf{G}}_m=\left[\begin{array}{ccc}{\stackrel{\sim }{\mathbf{X}}}_{t=1}^T{\stackrel{\sim }{\mathbf{X}}}_{t=1}& \cdots & {\stackrel{\sim }{\mathbf{X}}}_{t=1}^T{\stackrel{\sim }{\mathbf{X}}}_{t=T}\\ ⋮& \ddots & ⋮\\ {\stackrel{\sim }{\mathbf{X}}}_{t=T}^T{\stackrel{\sim }{\mathbf{X}}}_{t=1}& \cdots & {\stackrel{\sim }{\mathbf{X}}}_{t=T}^T{\stackrel{\sim }{\mathbf{X}}}_{t=T}\end{array}\right],$$

X̃_m = [X̃_t=1, X̃_t=2, …, X̃_t=T]^T represents the p × TN longitudinal high dimensional features, with components X̃_t=k = (y₁x_1,t=k, y₂x_2,t=k, …, y_Nx_N,t=k) to be data from N subjects each with p features at time point k. The corresponding β_m is a TN × N matrix, and ${\mathit{\beta }}_m^T=[{\mathbf{I}}_{N\times N},{\beta }_1{\mathbf{I}}_{N\times N},{\beta }_2{\mathbf{I}}_{N\times N},\dots ,{\beta }_{T-1}{\mathbf{I}}_{N\times N}]$. Similar to the conventional SVC, the objective function of LSVC is also subject to maximize the margins in the following equation:

$$\underset{{\mathit{w}}_{nv}}{min}\frac{1}{2}{\Vert {\mathit{w}}_{nv}\Vert }^2+\mathcal{C}\sum _{i=1}^N{\xi }_i,\text{for}i=1,\dots ,N$$ (2.1)

where w_nv is the estimate of separating hyperplane parameters with, by assuming that the temporal trend parameters are known, longitudinal high-dimensional features x_i = x_i,1 + β₁x_i,2 + β₂x_i,3… + β_T−1x_i,T.

After the incorporation of the temporal trend parameters ${\mathit{w}}_{nv}={\displaystyle \sum _{s=1}^N}{y}_s{\alpha }_s{({\stackrel{\sim }{\mathit{x}}}_s{\mathit{\beta }}_m)}^T$, the Langrange (Wolfe) dual function becomes:

$$\text{subject}\text{to}\begin{array}{l}\underset{\mathit{\alpha }}{min}\frac{1}{2}{\mathit{\alpha }}_m^T{\mathbf{G}}_m{\mathit{\alpha }}_m-{\mathbf{1}}^T\mathit{\alpha }\hfill \\ C\ge {\mathit{\alpha }}_m(i)\ge 0,\hfill \\ \sum _t^T\sum _i^N{\mathit{\alpha }}_m[i+(t-1)N]y_i=0,\hfill \end{array}$$ (2.2)

for i = 1, …, N and t = 1, …, T − 1.

Provided with α_m, the separating hyperplane parameter becomes

$${\mathit{w}}_{nv}=\sum _{i=1}^n{y}_i{\mathit{\alpha }}_m(i)({\mathit{x}}_{i,1}+{\beta }_1{\mathit{x}}_{i,2}+{\beta }_2{\mathit{x}}_{i,3}\dots +{\beta }_{T-1}{\mathit{x}}_{i,T}),$$

where α_m,i = (α_m(i), α_m(i + N), …, α_m(i + (T − 1)N)) and ${\mathit{w}}_{nv}={\sum }_{i=1}^n{y}_i{\mathit{\alpha }}_{m,i}{\stackrel{\sim }{\mathit{x}}}_i$.

Given w_nv, the intercept term then becomes $b=\frac{1}{N}{\displaystyle \sum _{i=1}^N}({\mathit{w}}_{nv}·{({\stackrel{\sim }{\mathit{x}}}_i{\mathit{\beta }}_m)}^T-y_i)$, in which β_m can be estimated based on α_m. At last, the separating hyperplane can be used to classify each subject by

$$h(\stackrel{\sim }{\mathit{x}})={\mathit{w}}_{nv}·{(\stackrel{\sim }{\mathit{x}}{\mathit{\beta }}_m)}^T+b.$$ (2.3)

Model Estimation

To estimate α and β vectors, the authors suggest to reparameterize the first part of the objective function in 2.6 as:

$$f={\left(\begin{array}{c}\mathit{\alpha }\\ \beta \mathit{\alpha }\end{array}\right)}^T\left[\begin{array}{cc}{\mathbf{G}}_m^{0,0}& {\mathbf{G}}_m^{0,1}\\ {\mathbf{G}}_m^{1,0}& {\mathbf{G}}_m^{1,1}\end{array}\right]\left(\begin{array}{c}\mathit{\alpha }\\ \beta \mathit{\alpha }\end{array}\right)$$

where

$${\mathbf{G}}_m=\left[\begin{array}{cc}{\mathbf{G}}_m^{0,0}& {\mathbf{G}}_m^{0,T}\\ {\mathbf{G}}_m^{T,0}& {\mathbf{G}}_m^{T,T}\end{array}\right],$$

and ${\mathbf{G}}_m^{0,0}$ is the N × N submatrix in the left top corner of the matrix G_m for the baseline data ( ${\stackrel{\sim }{\mathbf{X}}}_{t=1}^T{\stackrel{\sim }{\mathbf{X}}}_{t-1}$), the Gram matrix of SVC.

The objective function has been proven to be convex, and an iterative quadratic programming (QP) procedure is developed for optimization: (1) start with initial values of β and use QP to optimize 2.3 to obtain α; (2) use the updated α obtained in step 1 and apply QP again to estimate β; and (3) repeat the above two steps until convergence. The convergence of the iterative algorithm can be achieved because a unique solution exists.

Nonlinear Kernel Functions

The authors also provide solutions for nonlinear kernels. The Gram matrix of a nonlinear kernel is

$$\stackrel{\sim }{\mathbf{K}}({\stackrel{\sim }{\mathit{x}}}_i,{\stackrel{\sim }{\mathit{x}}}_{i^{\prime }})=\left[\begin{array}{ccc}\mathbf{K}({\stackrel{\sim }{\mathit{x}}}_{i,1},{\stackrel{\sim }{\mathit{x}}}_{i^{\prime },1})& \cdots & \mathbf{K}({\stackrel{\sim }{\mathit{x}}}_{i,1},{\stackrel{\sim }{\mathit{x}}}_{i^{\prime },T})\\ ⋮& \ddots & ⋮\\ \mathbf{K}({\stackrel{\sim }{\mathit{x}}}_{i,T},{\stackrel{\sim }{\mathit{x}}}_{i^{\prime },1})& \cdots & \mathbf{K}({\stackrel{\sim }{\mathit{x}}}_{i,T},{\stackrel{\sim }{\mathit{x}}}_{i^{\prime },T})\end{array}\right],$$

where 〈βK(·, x̃_i,t), K(·, x̃_i′,t)〉 = βK(x̃_i,t, x̃_i′,t), and K(·, x̃_i,t) indicates the reproducing kernel map of x̃_i,t (Wahba, 1990). The separating hyperplane with a nonlinear kernel becomes

$$h(\stackrel{\sim }{\mathit{x}})=\sum _{s=1}^N{y}_s{\mathit{\alpha }}_m\stackrel{\sim }{\mathbf{K}}(\stackrel{\sim }{\mathit{x}},{\stackrel{\sim }{\mathit{x}}}_s){\mathit{\beta }}_m^T+b,$$

where b is obtained by $b=\frac{1}{N}{\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{i^{\prime }=1}^N}(y_i{y}_i^{\prime }{\mathit{\alpha }}_m\stackrel{\sim }{\mathbf{K}}({\stackrel{\sim }{\mathit{x}}}_i,{\stackrel{\sim }{\mathit{x}}}_{i^{\prime }}){\mathit{\beta }}_m^T-y_i)$. Therefore, the nonlinear kernel does not increase the complexity of estimating β. In addition, the authors discussed the variable selection based on the predictors’ effect on the objective function which is a “wrapper” method (Guyon et al., 2003; Hastie and Tibshirani, 2004).

To demonstrate the use and potential advantages of LSVC, the authors apply the method to a simulation study and a data example from the Alzheimer’s disease Neuroimaging Initiative. The results show that by leveraging the additional longitudinal information LSVC achieves higher accuracy than methods using only cross-sectional data and methods that combine longitudinal data by naively expanding the feature space.

2.2 Penalized Linear Mixed Effects Models

Linear mixed effects models can be used in the analysis of clustered or longitudinal data. Those models estimate the relationship between the dependent variable and the fixed effects and random effects of independent variables by considering both means and covariances. With the improvement of data collection and storage technology, a large number of independent variables are available and can be included in the model. Inference and prediction of such a model becomes too complex and infeasible when the number of predictors increases. One challenge is how to choose significant predictors while excluding variables that have no true effects on the outcome. An example is the Trial of Activity for Adolescent Girls (TAAG) study, which determined the effectiveness of a school and community based intervention on the physical activities of girls from 6 middle schools in Maryland (Young et al., 2013). A large group of girls were followed up for four years from year 2006 to 2009 and asked to take a survey with hundreds of questions at multiple time points to measure the change of physical activities. A linear mixed effects model can be fitted to take into account the clustering effects (6 middle schools) and temporal effects (four years) as well as the fixed effects such as race, socio-economic status, and other questions in the survey.

To construct a linear mixed effects model, consider the ith subject in a longitudinal study with n subjects, each having observations at m_i time points for a total of $N={\sum }_{i=1}^n{m}_i$ observations. The linear mixed effects model can be written by

$$y_{ij}={\mathit{x}}_{ij}^T\mathit{\beta }+{\mathit{z}}_{ij}^T{\mathit{b}}_i+{\epsilon }_{ij},j=1,\dots ,m_i$$

where y_ij is the response variable at the jth time point, x_ij is the vector of p fixed effects, z_ij is the vectors of q random effects at the jth time point, β is the p parameter vector for the fixed effects, b_i is the q parameter vector for random effects, and ε_ij is the i.i.d. random error from N (0, σ²). The random effects parameter b_i are i.i.d. multivariate normal variables following MV N (0, σ²Π), where Π is the covariance matrix, and are independent of ε_ij. Using the matrix notation, the model can be simplified as

$${\mathit{Y}}_i={\mathit{X}}_i\mathit{\beta }+{\mathit{Z}}_i{\mathit{b}}_i+{\mathit{\epsilon }}_i,$$ (2.4)

where Y_i = (y_i1, …, y_imi)^T, X_i = (x_i1, …, x_imi)^T is the m × p design matrix of fixed effects, Z_i = (z_i1, …, z_imi)^T is the m × q design matrix of random effects, and ε_i = (ε_i1, …, ε_imi)^T is an i.i.d. random error following N(0, σ²I_mi).

Previous methods for penalized estimation of fixed effects include Efron et al., 2004, Zou and Hastie, 2005, and Bondell and Reich, 2008. Previous methods for selection of random effects include Stram and Lee, 1994, Lin, 1997, Hall and Praestgaard, 2001, and Chen, 2003. In this review we focus on three penalized linear mixed effects models, which, unlike many previous approaches, select both fixed effects and random effects simultaneously. The first model is developed by Bondell et al., 2010, maximizing a penalized joint likelihood problem. The second model is introduced by Fan and Li, 2012, uses a proxy matrix in maximizing a penalized profile likelihood for fixed and random effects separately. Third, the paper of Li et al., 2012 optimizes maximum likelihood estimator with separate penalization methods for fixed and random effects. A side-by-side comparison of these models is summarized in Table 1.

Table 1.

Comparison of three penalized linear mixed effects models

	1. Joint Penalization	2. Independent Selection	3. Double Penalization
Authors	Bondell et al. (2010)	Fan and Li (2012)	Li et al. (2012)
Regularizations	1	1	2
Objective Functions	ℓ(β, Π, σ2) − λP(β, b)	Fixed: ℓ(β) − λ1P1 (β) Random: ℓ(b, σ2) λ2P2(b)	ℓ(β, Π, σ2) − λ1P1(β) − λ2P2(b)
Penalties	1 total (ALASSO)	1 fixed (SCAD) 1 random (SCAD)	1 fixed (ALASSO) 1 random (L-2 norm)
Covariance Structure	Modified Cholesky decomposition for covariance matrix	Use of proxy matrix to substitute for unknown covariance matrix	Cholesky decomposition for covariance matrix
Algorithms	EM algorithm	LARS/Elastic net	New efficient algorithm with two quadratic components
High-Dimensional Data	EM algorithm is not efficient for large number of predictors	p and q can diverge to ∞, but must reduce dimensions of fixed effects ignoring random effects first	p and q can diverge to 1

Model 1 (Bondell et al., 2010)

Method 1 estimates fixed effects, random effects, and the covariance structure of the selected random effects simultaneously in a model with one penalty function. Equation (2.4) can be reparameterized using a modified Cholesky decomposition to factorize the covariance matrix of the random effects, Π. Through this factorization, Π = DΓΓ^TD, where Γ is a q × q lower triangular matrix with 1’s on the diagonal and whose (l, r)th element is given by γ_lr and D = diag(d₁, d₂, …, d_q) is a diagonal matrix. After this reparameterization, the linear mixed effects model (2.4) becomes:

$${\mathit{Y}}_i={\mathit{X}}_i\mathit{\beta }+{\mathit{Z}}_i\mathit{D}\mathbf{\Gamma }{\mathit{b}}_i+{\mathit{\epsilon }}_i.$$

The covariance matrix of b_i is now expressed in terms of vector d = (d₁, d₂, …, d_q)^T and the free elements of Γ, denoted by vector γ = (γ_lr : l = 1, …, q : r = l + 1, …, q)^T. Setting any d_l = 0 will set the corresponding lth row and column of the covariance matrix Π to 0 and therefore remove the lth random effect from the model.

After reparameterizing the model and treating b as given, maximizing the log-likelihood function is equivalent to minimizing the conditional expectation of ||y − ZD̃Γ̃b − Xβ||², where D̃ = I_m ⊗ D and Γ̃ = I_m ⊗ Γ. Rearranging the terms and adding the Adaptive Least Absolute Selection and Shrinkage Operator (ALASSO, Zou, 2006) penalty, the goal is to minimize the quadratic problem:

$$Q({\mathit{\beta }}^T,{\mathit{d}}^T,{\mathit{\gamma }}^T\mid \mathit{y},\mathit{b})={\Vert \mathit{y}-\mathit{X}\mathit{\beta }-\mathit{Z}\text{Diag}(\stackrel{\sim }{\mathbf{\Gamma }}\mathit{b})({\mathbf{1}}_{\mathit{q}}\otimes I_m)\Vert }^2+\lambda \left(\sum _{j=1}^p\frac{\mid {\beta }_i\mid }{\mid {\widehat{\beta }}_j\mid }+\sum _{k=1}^k\frac{\mid d_k\mid }{\mid {\widehat{d}}_k\mid }\right),$$ (2.5)

where β̂_j and d̂_k are ordinary least squares estimates and 1_q is a column vector of ones of length q. The problem (2.5) can be solved using the EM algorithm developed by Laird and Ware (1982) and Laird, Lange, and Stram (1987).

For high-dimensional data, it would be necessary to reduce the dimension of the data before using the method. This could be accomplished by using previous methods of penalized variable selection on the fixed effects, while ignoring the random effects, and vice versa. The resulting reduced model could then be applied to this joint penalty problem for further simultaneous selection of fixed and random effects. While this method is effective at selecting fixed and random effects, the EM algorithm that it uses is not efficient and may not be plausible when the number of predictors is too large due to its slow convergence rate and computational burden.

Model 2 (Fan and Li, 2012)

This method selects important fixed and random effects independently in two separate models. Proxy matrices are used to account for the unknown variance-covariance structure of the random effects during the selections. Stacking X_i, b_i, y_i, and ε_i and setting Z = diag(Z₁, …, Z_n) with corresponding Π̃ = diag(Π, …, Π), the linear mixed effects model in (2.4) can be rewritten as

$$\mathit{y}=\mathit{X}\mathit{\beta }+\mathit{Zb}+\mathit{\epsilon }.$$

For the fixed effects parameter β, it is necessary to minimize the penalized likelihood equation

$$Q(\mathit{\beta })=\frac{1}{2}{(\mathit{y}-\mathit{X}\mathit{\beta })}^T{\mathit{P}}_{\mathit{z}}(\mathit{y}-\mathit{X}\mathit{\beta })+n\sum _{j=1}^p{P}_{\lambda }(\mid {\beta }_j\mid ),$$ (2.6)

where P_z = (I + σ⁻²ZΠ̃Z^T)⁻¹ and the penalty function P_λ(|β_j|) is the Smoothed Clipped Absolute Deviation (SCAD, Fan and Li, 2001) penalty with tuning parameter λ. It is important to note that the problem Q(β) depends on the unknown parameters Π̃ and σ². To overcome this obstacle, a proxy matrix P̃_z = (I + Z Z^T)⁻¹, where = (log n)I, is substituted into (2.6) for P_z. Since this regularization function is quadratic, it can be solved through previous methods for penalized least-squares, such as the LARS algorithm (Efron et al., 2004).

The selection of the random effects is accomplished through Bayesian methods of deriving the restricted posterior distribution of the random effects, and penalizing this solution of the restricted posterior mode. The resulting regularization problem to be minimized is

$$Q(\mathit{b})=\frac{1}{2}{(\mathit{y}-\mathit{Zb})}^T{\mathit{P}}_{\mathit{x}}(\mathit{y}-\mathit{Zb})+\frac{1}{2}{\sigma }^2{\mathit{b}}^{\mathit{T}}\stackrel{\sim }{\mathbf{\Pi }}+\mathit{b}+n\sum _{k=1}^q{P}_{\lambda }(\mid b_k\mid ),$$ (2.7)

where P_x = I − X(X^TX)⁻¹X^T, the penalty function P_λ(|b_k|) is again the SCAD penalty with tuning parameter λ, and Π̃⁺ is the Moore-Penrose generalized inverse of Π̃. Again, Π̃ and σ² are unknown so a the proxy matrix = diag(M, …, M), with M = (log n)I, is substituted for σ⁻²Π̃, so the regularization problem in (2.7) becomes:

$$Q(\mathit{b})=\frac{1}{2}{(\mathit{y}-\mathit{Zb})}^T{\mathit{P}}_{\mathit{x}}(\mathit{y}-\mathit{Zb})+\frac{1}{2}{\mathit{b}}^{\mathit{T}}\mathcal{M}\mathit{b}+n\sum _{k=1}^q{P}_{\lambda }(\mid b_k\mid ).$$

This problem is similar to the penalized quadratic function of adaptive elastic net (Zou and Zhang, 2009), so it can be solved through modification of this algorithm.

For high-dimensional data where N ≤ p, the dimension of fixed effects must be lowered to below the sample size before using the above methods. This can be done by first using penalized least squares methods on the fixed effects while ignoring random effects. Using these selected fixed effects, the random effects can be estimated using the regularization problem (2.7). Next, using these selected random effects from the second step, the fixed effects regularization problem (2.6) can be used to select from the remaining fixed effects. The second and third steps can be repeated, as needed, to further reduce the dimensionality of the data.

Model 3 (Li et al., 2012)

The final method selects and estimates fixed effects, random effects, and the covariance structure of the selected random effects simultaneously in a linear mixed effects model using two penalty functions. Using the model (2.4), When N > p, a modified log-likelihood incorporating the Restricted Maximum Likelihood (REML) is

$${\ell }_{nM}(\mathit{\beta },\mathbf{\Pi },{\sigma }^2)=-\frac{1}{2}\sum _{i=1}^{n}log\mid {\sigma }^2{\mathit{V}}_i\mid -\frac{1}{2}log{\sigma }^{-2}\sum _{i=1}^n{\mathit{X}}_i^T{\mathit{V}}_i^{-1}{\mathit{X}}_i-\frac{1}{2{\sigma }^2}\sum _{i=1}^n{({\mathit{Y}}_i-{\mathit{X}}_i\mathit{\beta })}^T{\mathit{V}}_i^{-1}({\mathit{Y}}_i-{\mathit{X}}_i\mathit{\beta }),$$ (2.8)

where ${\mathit{V}}_i={\mathit{I}}_{m_i}+{\mathit{Z}}_i\mathbf{\Pi }{\mathit{Z}}_i^T$. In high dimensional settings when N ≤ p, the restricted term in (2.8) will become singular so the following full log-likelihood must be used

$${\ell }_{nF}(\mathit{\beta },\mathbf{\Pi },{\sigma }^2)=-\frac{1}{2}\sum _{i=1}^{n}log\mid {\sigma }^2{\mathit{V}}_i\mid -\frac{1}{2{\sigma }^2}\sum _{i=1}^n{({\mathit{Y}}_i-{\mathit{X}}_i\mathit{\beta })}^T{\mathit{V}}_i^{-1}({\mathit{Y}}_i-{\mathit{X}}_i\mathit{\beta }).$$ (2.9)

The maximum likelihood can be found for this equation to obtain the parameters. Adding penalty functions, the regularization problem to maximize is

$$Q_n(\mathit{\beta },\mathbf{\Pi },{\sigma }^2)={\ell }_n(\mathit{\beta },\mathbf{\Pi },{\sigma }^2)-{\lambda }_1{P}_1(\beta )-{\lambda }_2{P}_2(\mathit{D}),$$ (2.10)

where ℓ_n(θ) is (2.8) or (2.9) depending on if N > p or N ≤ p, respectively. The first penalty function, λ₁P₁(β), is for the fixed effects and is an ALASSO penalty. For the random effects, a Cholesky decomposition is performed such that Π = LL^T where L is a lower triangular matrix with positive diagonal elements. If any diagonal element of L_kk = 0, then the corresponding random effect b_k is also 0 and is removed from the model. The second penalty function in (2.10) is set to be an L − 2 penalty function (Yuan and Lin, 2006) with an adaptive weight added so that the regularization problem is

$$Q_n(\mathit{\beta },\mathbf{\Pi },{\sigma }^2)={\ell }_n(\mathit{\beta },\mathbf{\Pi },{\sigma }^2)-{\lambda }_1\sum _{j=1}^p\frac{\mid {\beta }_j\mid }{\mid {\widehat{\beta }}_j\mid }-\sum _{k=2}^q\frac{\sqrt{L_{k1}^2+\dots L_{kq}^2}}{\Vert {\widehat{L}}_{(k)}\Vert },$$ (2.11)

where β̂_j and ||L̂_(k)|| are ordinary least squares estimates. An estimation of the variance σ² (Lind-strom and Bates, 1988) can be substituted into the model, allowing (2.11) to be maximized in terms of only L and β.

The problem (2.11) can be solved by a new algorithm that iteratively updates two quadratic optimization functions for the random and fixed effects. This has proven to be more efficient than the EM algorithm, which cannot handle large numbers of predictors. When the maximum likelihood is used, the new algorithm is proven to be a consistent estimator for high-dimensional data, where p and q can diverge at an exponential rate with the sample size n.

3 Summary

In this article, we described two types of longitudinal high-dimensional data that researchers often encounter in current biomedical research and reviewed several recently developed statistical methods to deal with these two types of data. First, we introduced a kernel method for classification for the first type of longitudinal high-dimensional data and the corresponding computational strategy for parameter estimation. Second, we reviewed three mixed effect shrinkage models for the other type of longitudinal high-dimensional data. In the review, we compared the model setups, computational strategies, and advantages and shortcomings of the methods.