HIGH-DIMENSIONAL FACTOR REGRESSION FOR HETEROGENEOUS SUBPOPULATIONS

Abstract

In modern scientific research, data heterogeneity is commonly observed owing to the abundance of complex data. We propose a factor regression model for data with heterogeneous subpopulations. The proposed model can be represented as a decomposition of heterogeneous and homogeneous terms. The heterogeneous term is driven by latent factors in different subpopulations. The homogeneous term captures common variation in the covariates and shares common regression coefficients across subpopulations. Our proposed model attains a good balance between a global model and a group-specific model. The global model ignores the data heterogeneity, while the group-specific model fits each subgroup separately. We prove the estimation and prediction consistency for our proposed estimators, and show that it has better convergence rates than those of the group-specific and global models. We show that the extra cost of estimating latent factors is asymptotically negligible and the minimax rate is still attainable. We further demonstrate the robustness of our proposed method by studying its prediction error under a mis-specified group-specific model. Finally, we conduct simulation studies and analyze a data set from the Alzheimer’s Disease Neuroimaging Initiative and an aggregated microarray data set to further demonstrate the competitiveness and interpretability of our proposed factor regression model.

1. Introduction

Data heterogeneity is an important issue in modern complex data analysis. In practice, data heterogeneity may come from variables or samples. More specifically, multi-modality/source data have heterogeneity among the variables, because they may correspond to different types of measurements. For example, in biomedical imaging, people may acquire both MRI and PET images (Zhang et al. (2011)). In genomics studies, measurements are collected from different sources, such as mRNA and miRNA (Muniategui et al. (2013)). In addition to variable heterogeneity, data heterogeneity can also arise from samples. For example, there can be subpopulations, batch and clustering effects, or outliers in the data (Bühlmann (2016)), potentially violating the standard independent and identically distributed (i.i.d.) assumption. Ignoring such heterogeneity can lead to poor estimation and prediction. Hence, it is important to take data heterogeneity into account during the modeling process.

In this study, we are interested in data heterogeneity that comes from subgroup populations. For example, in the Alzheimer’s Disease (AD) study, subjects can have five subtypes: Normal Control (NC), Significant Memory Concern (SMC), Early Mild Cognitive Impairment (eMCI), Late Mild Cognitive Impairment (lMCI), and AD, where these subtypes are ordered by disease severity. Owing to data heterogeneity, it can be difficult to build accurate and interpretable predictive models on such data using traditional statistical techniques. A global model that fits a single regression model to all the data may be restrictive because it ignores the group label information, whereas fitting distinct regression models in each group may not be optimal because this does not capture shared information across groups. Hence, a statistical regression model that can recover interpretable globally shared and group-specific signals in the data is required to handle such heterogeneous data. In the literature, varying coefficient models (Hastie and Tibshirani (1993)) and mixed-effects models (Pinheiro and Bates (2000)) are useful in addressing data heterogeneity. However, these models can be computationally expensive to use in practice, especially when the dimension is too high. More recently, Vicari and Vichi (2013) proposed a general regression model to account for both between-cluster and within-cluster variation. Meinshausen and Bühlmann (2015) proposed a maxmin-effects approach under the mixture model. Zhao, Cheng and Liu (2016) proposed a partially linear regression framework to model massive heterogeneous data. Tang and Song (2016) and Ma and Huang (2017) proposed fused penalties to estimate regression coefficients in order to identify subpopulations. Wang, Liu and Shen (2018) proposed a locally weighted penalized model by incorporating a progression score in the local kernels. However, these models are not designed to characterize the globally shared and group-specific structures. Thus, it is desirable to build a model that can identify such structures, quantify prediction errors, and draw interpretable and generalizable scientific conclusions.

There is a large body of literature on data heterogeneity for unsupervised learning. Principal component analysis (PCA) (Wold, Esbensen and Geladi (1987)) techniques are popular, owing to their computational simplicity and theoretical soundness. The joint and individual variations explained (JIVE) method (Lock et al. (2013)) decomposes joint and individual low-rank signals across multiple sources of data. More recent extensions of JIVE can be found in Feng et al. (2018); Gaynanova and Li (2019); Park and Lock (2020). These methods can be extended easily to decompose data from multiple subgroups. Zhou et al. (2015) proposed a matrix factorization framework for common and individual feature extraction for multi-block data.

Closely related to PCA, another popular technique for handling data heterogeneity is factor models. Factor models are useful unsupervised learning tools that model the dependence between multiple variables. The relationship between PCA and factor models is well studied in the literature (Joliffe and Morgan (1992); Stock and Watson (2002); Bai and Ng (2002)). Factor models assume that the variations among the variables are driven by latent factors residing in a low-dimensional space. More recently, Fan et al. (2018) proposed a factor model framework to model the heterogeneity from different subgroups. They used the factor model in the context of Gaussian graphical models to estimate common and individual graphs from different groups. Their structural assumption on the data matrices can be generalizable to predictive modeling.

Here, we focus on supervised learning, and propose a novel factor regression model for heterogeneous data with jointly shared and group-specific structures. We assume that the leading factors in each group drive the majority of variation, which contributes to the heterogeneity effects. After the majority of the variation has been removed, the residual signals are assumed to be homogeneous across subgroups; that is, they have the same covariance matrix. Under this framework, the predictors in the proposed model can be decomposed into heterogeneous factors and homogeneous signals. Correspondingly, in our proposed model, the regression coefficients associated with the factors are group specific, whereas the regression coefficients associated with the homogeneous signals are shared across groups. We use PCA to estimate the factors and homogeneous signals. Because the estimated factors and homogeneous signals are orthogonal, their coefficients can be estimated separately. The low-dimensional heterogeneous regression coefficients can be estimated directly using the ordinary least squares (OLS) method. After projecting the responses on the estimated factors in each group, their residuals can be aggregated together to perform a global regression. When the dimension is high, the homogeneous signals’ coefficients are difficult to estimate. Following given penalization methods (Hoerl and Kennard (2000); Tibshirani (1996); Zou and Hastie (2005)), we propose a flexible penalized least squares method to solve for the high-dimensional coefficients. In the least squares problem, we use the adaptive thresholding estimator (Cai and Liu (2011)) to estimate the covariance of the homogeneous signals. For prediction, we propose a data-driven trace maximization step to estimate the factors and homogeneous signals in the test set before applying our model for prediction.

We establish the estimation consistency for our proposed estimators using either an ${\ell }_2$ or ${\ell }_1$ penalty. In terms of the prediction accuracy, we study the prediction error of our method in both theoretical and simulation studies, and demonstrate that the proposed model attains a good balance between a global model and a group-specific model. Furthermore, we show that our method is robust when the underlying model is group specific, and has comparable prediction performance with respect to the group-specific model. We apply our method to an Alzheimer’s Disease Neuroimaging Initiative (ADNI) data set and an aggregated microarray data set to show the competitiveness of our model in terms of model prediction and interpretability.

The rest of paper is organized as follows. In Section 2, we introduce the factor decomposition of heterogeneous and homogeneous signals and a corresponding regression model. In Section 3, we introduce the model estimation and a data-driven approach to estimate the factors in the testing data for prediction. In Section 4, we study the estimation and prediction consistency of our proposed method, and compare it with those of group-specific and global models under different scenarios. In Section 5, we conduct simulated experiments to evaluate the performance of our model under different settings, and compare them with that of the global and group-specific models. In Section 6, we apply our model to the ADNI data to predict the clinical score. We conclude the paper with a discussion in Section 7.

2. Motivation and Model Framework

Factor models are useful for modeling the dependence between multiple variables, if these variables are driven by some latent factors. For heterogeneous data, the subgroup heterogeneity can be captured by the group-specific latent factors. After removing such latent factors, different subgroups can be viewed as homogeneous samples for a joint analysis. In this section, we first motivate our proposed model by introducing two simple models in Section 2.1. Then, we briefly review the factor decomposition for heterogeneous data and propose our new factor regression model in Section 2.2.

2.1. Motivation

We first introduce some notation. Assume that the data come from $G$ groups. There are $n_g$ samples in the gth group, each having the same set of $p$ explanatory variables. Let ${\left\{{\mathbf{X}}_g,{\mathbf{Y}}_g\right\}}_{g=1}^G$ be the observations from $G$ groups, where ${\mathbf{X}}_g\in {\mathbb{R}}^{n_g\times p}$ is the data matrix and ${\mathit{Y}}_g\in {\mathbb{R}}^{n_g}$ is the response vector.

There are two commonly used approaches in the regression setup for heterogeneous subpopulations. On the one hand, ignoring the group information, one can use a global model:

$$\mathit{Y}={\mathbf{\mu }}^{\mathrm{*}}+\mathbf{X}{\mathit{\beta }}^{\mathrm{*}}+\mathbf{ϵ},$$ (2.1)

where $\mathit{Y}={\left({\mathit{Y}}_1^{\prime },\dots ,{\mathit{Y}}_G^{\prime }\right)}^{\prime }$ and $\mathbf{X}={\left({\mathbf{X}}_1^{\prime },\dots ,{\mathbf{X}}_G^{\prime }\right)}^{\prime }$. In this model, all the subgroups share the same intercept and regression coefficients. The global model ignores the heterogeneity from subgroups and may be too restrictive. On the other hand, by modeling each group separately, one may consider a group-specific model:

$${\mathit{Y}}_g={\mathit{\mu }}_g^{\mathrm{*}}+{\mathbf{X}}_g{\mathit{\beta }}_g^{\mathrm{*}}+{\mathit{ϵ}}_{g.}$$ (2.2)

However, this model may not be efficient because it ignores the shared information across subgroups. These global and group-specific models motivate us to consider a model in between, under which the group-specific heterogeneity and homogeneity across subgroups can both be accounted for. This can be achieved by using a factor model that decomposes covariates into the heterogeneous and homogeneous components.

2.2. Factor model framework

To model the heterogeneous effect introduced by groups, assume that the data matrix ${\mathbf{X}}_g$ can be decomposed as

$${\mathbf{X}}_g={\mathbf{F}}_g{\mathbf{\Lambda }}_g+{\mathbf{U}}_g,$$ (2.3)

where ${\mathbf{F}}_g\in {\mathbb{R}}^{n_g\times K_g}$ is the factor matrix, ${\mathbf{\Lambda }}_g\in {\mathbb{R}}^{K_g\times p}$ is the loading matrix, and ${\mathbf{U}}_g\in {\mathbb{R}}^{n_g\times p}$ denotes the homogeneous signals, also known as idiosyncratic errors in the factor model literature (Bai and Ng (2008)). The number of random factors $K_g$ can vary among groups.

Denote the ith row of ${\mathbf{X}}_g,{\mathbf{F}}_g$, and ${\mathbf{U}}_g$ by ${\mathbf{x}}_{g,i},{\mathbf{f}}_{g,i}$, and ${\mathbf{u}}_{g,i}$ respectively. By (2.3), we have ${\mathbf{x}}_{g,i}={\mathbf{\Lambda }}_g^{\prime }{\mathbf{f}}_{g,i}+{\mathbf{u}}_{g,i}$. We assume ${\mathbf{f}}_{g,i}$ and ${\mathbf{u}}_{g,i}$ are uncorrelated and satisfy $\mathbb{E}\left({\mathbf{f}}_{g,i}\right)=0,\mathrm{c}\mathrm{o}\mathrm{v}\left({\mathbf{f}}_{g,i}\right)={\mathbf{I}}_{K_g\times K_g},\mathbb{E}\left({\mathbf{u}}_{g,i}\right)=0$, and $\mathrm{c}\mathrm{o}\mathrm{v}\left({\mathbf{u}}_{g,i}\right)={\mathbf{\Sigma }}_u$. Hence, for each sample in group $g$, we have $\mathrm{c}\mathrm{o}\mathrm{v}\left({\mathbf{x}}_{g,i}\right)={\mathbf{\Lambda }}_g^{\prime }{\mathbf{\Lambda }}_g+{\mathbf{\Sigma }}_u$, which is the sum of the group-specific low-rank matrix ${\mathbf{\Lambda }}_g^{\prime }{\mathbf{\Lambda }}_g$ capturing the group-specific heterogeneity, and the matrix ${\mathbf{\Sigma }}_u$ that is homogeneous across different groups.

We adopt the approximate factor model (Stock and Watson (2002)) by assuming that ${\mathbf{\Sigma }}_u$ is sparse. Its sparsity can be characterized by $m_p$, defined as

$$m_p=\underset{i\le p}{\mathrm{m}\mathrm{a}\mathrm{x}} \sum _{j=1}^p I\left({\sigma }_{u,ij}\ne 0\right),$$

which is the maximum number of nonzero entries in the row of ${\mathbf{\Sigma }}_u$.

Under the decomposition (2.3), we have the following regression model for the gth group:

$${\mathit{Y}}_g={\mathbf{\mu }}_g^{\mathrm{*}}+{\mathbf{F}}_g{\mathbf{\gamma }}_g^{\mathrm{*}}+{\mathbf{U}}_g{\mathit{\beta }}^{\mathrm{*}}+{\mathbf{ϵ}}_g.$$ (2.4)

Here, ${\mathbf{\mu }}_g^{\mathrm{*}}$ is the true group mean vector, ${\mathbf{\gamma }}_g^{\mathrm{*}}\in {\mathbb{R}}^{K_g}$ denotes the true group-specific coefficients for ${\mathbf{F}}_g,{\mathit{\beta }}^{\mathrm{*}}\in {\mathbb{R}}^p$ denotes the common coefficients shared across $G$ groups for ${\mathbf{U}}_g$, and ${\mathit{ϵ}}_g$ is the noise term and has variance ${\sigma }^2$. In (2.4), ${\mathbf{\gamma }}_g^{\mathrm{*}}$ vary across $G$ groups, and they characterize the heterogeneity induced by the factors in the regression model. Moreover, the group mean term ${\mathbf{\mu }}_g^{\mathrm{*}}$ contributes to the heterogeneity in the regression model (2.4). When the heterogeneous effect is removed from (2.4), we have the same coefficients ${\mathit{\beta }}^{\mathrm{*}}$ for ${\mathbf{U}}_g$ across $G$ groups.

From (2.4), we can see that the heterogeneity is modeled by ${\mathbf{\mu }}_g^{\mathrm{*}}+{\mathbf{F}}_g{\mathbf{\gamma }}_g^{\mathrm{*}}$. After adjusting this heterogeneous term, the remainder term ${\mathbf{U}}_g{\mathit{\beta }}^{\mathrm{*}}$ is homogeneous. Model (2.4) implies that, for the response $y_{g,i}$ of the ith subject in group $g$, we have $\mathrm{v}\mathrm{a}\mathrm{r}\left(y_{g,i}\right)={\mathbf{\gamma }}_g^{\mathrm{*}\prime }{\mathbf{\gamma }}_g^{\mathrm{*}}+{\mathit{\beta }}^{\mathrm{*}\prime }{\mathbf{\Sigma }}_u{\mathit{\beta }}^{\mathrm{*}}+{\sigma }^2$. This decomposition shows that the variance can be decomposed as the sum of a group-specific part ${\mathbf{\gamma }}_g^{\mathrm{*}\prime }{\mathbf{\gamma }}_g^{\mathrm{*}}$, a homogeneous part ${\mathit{\beta }}^{\mathrm{*}\prime }{\mathbf{\Sigma }}_u{\mathit{\beta }}^{\mathrm{*}}$, and the background noise ${\sigma }^2$. This decomposition allows us to account for the heterogeneity among subgroups, while also borrowing information across subgroups to model homogeneous effects.

One special case of our proposed model (2.4) is when there is no group-specific factor; that is, ${\mathbf{F}}_g=0$. Then, it reduces to a mean-specific model:

$${\mathit{Y}}_g={\mathbf{\mu }}_g^{\mathrm{*}}+{\mathbf{X}}_g{\mathit{\beta }}^{\mathrm{*}}+{\mathit{ϵ}}_g.$$ (2.5)

This model lies between the global model (2.1) and the group-specific model (2.2). It is different from (2.1) because it adjusts the group mean. It is different from (2.2) because different groups share the common regression coefficients. We refer to (2.5) as the “Factor-0” model.

3. Model Estimation and Prediction

In this section, we introduce the model estimation procedure and a data-driven way to estimate the factors in the testing data for prediction. The overall training procedure consists of two steps. First, we estimate the factors and homogeneous signals from the training data. Second, we estimate the regression coefficients using the estimated factors and homogeneous signals. In Section 3.1, we introduce how the factors can be estimated using a PCA. In Section 3.2, we introduce our procedure for estimating the model parameters. After the model is trained, in Section 3.3, we propose a data-driven procedure to estimate the factors in the testing data in order to make predictions.

3.1. Factor model estimation

For group $g$, the estimation of ${\mathbf{F}}_g$ and ${\mathbf{\Lambda }}_g$ can be formulated into the following optimization problem:

$$\begin{gathered} \underset{{\mathbf{F}}_g,{\mathbf{\Lambda }}_g}{\mathrm{m}\mathrm{i}\mathrm{n}} {∥{\mathbf{X}}_g-{\mathbf{F}}_g{\mathbf{\Lambda }}_g∥}_F, \\ \text{s.t.}{\mathbf{F}}_g^{\prime }{\mathbf{F}}_g=n_g\mathbf{I},{\mathbf{\Lambda }}_g{\mathbf{\Lambda }}_g^{\prime }\text{is}\text{diagonal}, \end{gathered}$$ (3.1)

where $\parallel \cdot {\parallel }_F$ denotes the matrix Frobenius norm. The solution to (3.1) can be obtained by performing the eigendecomposition of the matrix ${\mathbf{X}}_g{\mathbf{X}}_g^{\prime }$. Following the standard PCA procedure, we estimate ${\mathbf{F}}_g$ by ${\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g$, where the kth column of ${\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g$ is $\sqrt{n_g}$ times the eigenvector corresponding to the kth largest eigenvalue of ${\mathbf{X}}_g{\mathbf{X}}_g^{\prime }$. Then, the loading matrix ${\mathbf{\Lambda }}_g$ can be estimated by regressing ${\mathbf{X}}_g$ on ${\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g$ to obtain ${\stackrel{\mathbf{ˆ}}{\mathbf{\Lambda }}}_g={\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g^T{\mathbf{X}}_g/n_g$. The homogeneous signal matrix ${\mathbf{U}}_g$ can hence be estimated by the residual matrix ${\stackrel{\mathbf{ˆ}}{\mathbf{U}}}_g={\mathbf{X}}_g-{\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g{\stackrel{\mathbf{ˆ}}{\mathbf{\Lambda }}}_g$.

We now consider estimating the number of factors $K_g$. In the literature, several estimators have been proposed to solve this problem (Bai and Ng (2002); Lam and Yao (2012); Ahn and Horenstein (2013)). We consider the following estimator:

$${\hat{K}}_g=\underset{k\le K_{\mathrm{m}\mathrm{a}\mathrm{x}}}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}}\frac{{\lambda }_k\left({\mathbf{X}}_g{\mathbf{X}}_g^{\prime }\right)}{{\lambda }_{k+1}\left({\mathbf{X}}_g{\mathbf{X}}_g^{\prime }\right)},$$ (3.2)

where ${\lambda }_k(\cdot )$ denotes the kth largest eigenvalue (Lam and Yao (2012)). Here, $K_{\text{max}}$ is a pre-determined upper bound for the number of factors. This estimator has been shown to be a consistent estimator (Ahn and Horenstein (2013)) for the true $K_g$, and is simple to implement in practice.

3.2. Estimation of regression coefficients

Given ${\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g$ and ${\stackrel{\mathbf{ˆ}}{\mathbf{U}}}_g$, as discussed in Section 3.1, we can estimate the model parameters ${\mu }_g^{\mathrm{*}},{\mathbf{\gamma }}_g^{\mathrm{*}}$, and ${\mathit{\beta }}^{\mathrm{*}}$. The factor decomposition (2.3) projects the original signals onto the low-dimensional space spanned by ${\mathbf{F}}_g$ and the space spanned by ${\mathbf{U}}_g$, which is orthogonal to ${\mathbf{F}}_g$. Owing to the properties of an eigendecomposition, we have ${\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g$ and ${\stackrel{\mathbf{ˆ}}{\mathbf{U}}}_g$ orthogonal to each other. Hence, we can estimate the regression coefficients ${\mathbf{\gamma }}_g^{\mathrm{*}}$ and ${\mathit{\beta }}^{\mathrm{*}}$ in (2.4) separately. Given ${\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g,{\mu }_g^{\mathrm{*}}$ and ${\mathbf{\gamma }}_g^{\mathrm{*}}$ can be estimated by the following OLS estimators:

$${\hat{\mu }}_g={\bar{Y}}_g,{\hat{\gamma }}_g=\frac{{\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g^T{\mathbf{Y}}_g}{n_g},$$ (3.3)

where ${\bar{Y}}_g$ denotes the sample mean of the response in group $g$.

Note that the factor matrix ${\mathbf{F}}_g$ and the coefficients ${\mathbf{\gamma }}_g^{\mathrm{*}}$ are not separately identifiable, because for any orthogonal matrix ${\mathbf{H}}_g$, we have ${\mathbf{F}}_g{\mathbf{\gamma }}_g^{\mathrm{*}}={\mathbf{F}}_g{\mathbf{H}}_g^{\prime }{\mathbf{H}}_g{\mathbf{\gamma }}_g^{\mathrm{*}}$. Hence, $\left({\mathbf{F}}_g,{\mathit{\gamma }}_g^{\mathrm{*}}\right)$ cannot be identified from $\left({\mathbf{F}}_g{\mathbf{H}}_g^{\prime },{\mathbf{H}}_g{\mathbf{\gamma }}_g^{\mathrm{*}}\right)$. In practice, it does not matter which one is used, because the linear space spanned by the columns of ${\mathbf{F}}_g{\mathbf{H}}_g^{\prime }$ is the same as that spanned by those of ${\mathbf{F}}_g$.

For homogeneous regression coefficients ${\mathit{\beta }}^{\mathrm{*}}$, because they are shared across groups, we can aggregate the residuals from the response and the factor projection to perform a global regression to estimate ${\mathit{\beta }}^{\mathrm{*}}$. Denote the aggregated residual vectors from the response as $\tilde{\mathit{Y}}={({\tilde{\mathit{Y}}}_1^{\prime },\dots ,{\tilde{\mathit{Y}}}_G^{\prime })}^{\prime }$, where ${\tilde{\mathit{Y}}}_g={\mathit{Y}}_g-{\stackrel{\mathbf{ˆ}}{\mathbf{\mu }}}_g-{\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g{\stackrel{\mathbf{ˆ}}{\mathbf{\gamma }}}_g$. Let $\mathbf{U}={\left({\mathbf{U}}_1^{\prime },\dots ,{\mathbf{U}}_G^{\prime }\right)}^{\prime }$ and $\stackrel{\mathbf{ˆ}}{\mathbf{U}}={({\stackrel{\mathbf{ˆ}}{\mathbf{U}}}_1^{\prime },\dots ,{\stackrel{\mathbf{ˆ}}{\mathbf{U}}}_G^{\prime })}^{\prime }$. We solve the following penalized quadratic minimization problem to estimate ${\mathit{\beta }}^{\mathrm{*}}$:

$$\underset{\mathit{\beta }}{\mathrm{m}\mathrm{i}\mathrm{n}} \frac{1}{2}\left({\mathit{\beta }}^{\prime }{\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_u\mathit{\beta }-\frac{2}{n}{\tilde{\mathit{Y}}}^{\prime }\stackrel{\mathbf{ˆ}}{\mathbf{U}}\mathit{\beta }\right)+\lambda P(\mathit{\beta }),$$ (3.4)

where $P(\mathit{\beta })$ is a penalty function and $\lambda$ is a tuning parameter, the optimal value of which is chosen using cross-validation. In particular, we consider an ${\ell }_1$ penalty that $P(\mathit{\beta })={\sum }_{j=1}^p \left|{\beta }_j\right|$ and an ${\ell }_2$ penalty that $P(\mathit{\beta })={\sum }_{j=1}^p {\beta }_j^2$, and denote the corresponding solutions of (3.4) as ${\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda }^{lasso}$ and ${\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda }^{ridge}$, respectively. In (3.4), ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_u$ is an estimator of ${\mathbf{\Sigma }}_u$. To obtain such an estimator, we use the adaptive thresholding method (Cai and Liu (2011)). More specifically, let ${\hat{\sigma }}_{ij}=(1/n){\sum }_{g=1}^G {\sum }_{t=1}^{n_g} {\hat{u}}_{g,ti}{\hat{u}}_{g,tj}$ and ${\hat{\theta }}_{ij}=(1/n){\sum }_{g=1}^G {\sum }_{t=1}^{n_g} {\left({\hat{u}}_{g,ti}{\hat{u}}_{g,tj}-{\hat{\sigma }}_{ij}\right)}^2$, where ${\hat{u}}_{g,ti}$ is the $(t,i)$ th element of ${\stackrel{\mathbf{ˆ}}{\mathbf{U}}}_g$. We have

$${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_u={({\hat{\sigma }}_{ij}^{𝒯})}_{p\times p},{\hat{\sigma }}_{ij}^{𝒯}=\left\{\begin{array}{ll}{\hat{\sigma }}_{ii},& i=j,\\ s_{ij}\left({\hat{\sigma }}_{ij}\right),& i\ne j,\end{array}\right.$$ (3.5)

where $s_{ij}(\cdot )$ is any thresholding function that satisfies that for all $z\in \mathbb{R}$,

$$s_{ij}\left(z\right)=0\text{when}\left|z\right|<{\tau }_{ij},\text{and}\left|s_{ij}\left(z\right)-z\right|\le {\tau }_{ij}\text{when}\left|z\right|\ge {\tau }_{ij}.$$ (3.6)

Here, ${\tau }_{ij}=D{\omega }_n\sqrt{{\hat{\theta }}_{ij}}$ is an adaptive threshold, where ${\omega }_n=1/\sqrt{p}+\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}p/n}$. The purpose of using such a thresholding estimator is to ensure ${\mathbf{\Sigma }}_u$ can be consistently estimated when $p>n$. In Section S3.1 of the Supplementary Material, we perform a sensitivity study on the choice of $D$, and find that the prediction performance of our method is not sensitive to $D$. Thus, we recommend choosing $D$ to be a fixed number, rather than tuning it. When $p<n,{\mathbf{\Sigma }}_u$ does not have to be sparse. In this case, we find it is safe to choose $D=0$; see Section S3.1 of the Supplementary Material.

We summarize the overall training procedure as follows:

1. For $g=1,\dots ,G$:

   1. Estimate $K_g$ from (3.2).

   2. Perform a PCA on ${\mathbf{X}}_g{\mathbf{X}}_g^{\prime }$ to obtain ${\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g$. Estimate ${\mu }_g^{\mathrm{*}}$ and ${\mathbf{\gamma }}_g^{\mathrm{*}}$ from (3.3).

   3. Compute the projection matrix ${\mathbf{P}}_g={\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g{\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g^{\prime }/n_g$.

2. Let $\mathbf{H}=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left\{\mathbf{I}-{\mathbf{P}}_1,\dots ,\mathbf{I}-{\mathbf{P}}_G\right\}$ be the block diagonal matrix. Compute the aggregated signals $\stackrel{\mathbf{ˆ}}{\mathbf{U}}=\mathbf{H}\mathbf{X},\tilde{\mathit{Y}}=\mathbf{H}(\mathit{Y}-\stackrel{\mathbf{ˆ}}{\mathbf{\mu }})$, where $\stackrel{\mathbf{ˆ}}{\mathbf{\mu }}={\left({\stackrel{\mathbf{ˆ}}{\mathbf{\mu }}}_1^{\prime },\dots ,{\stackrel{\mathbf{ˆ}}{\mathbf{\mu }}}_G^{\prime }\right)}^{\prime }$. Estimate ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_u$ from $\stackrel{\mathbf{ˆ}}{\mathbf{U}}$ using (3.5). Solve the optimization problem (3.4) to estimate ${\mathit{\beta }}^{\mathrm{*}}$.

In practice, it can be desirable to have an automatic way to choose between the proposed factor regression model (2.4) and the group-specific model (2.2). We provide an effective rule of thumb in Section S2 in the Supplementary Material.

3.3. Prediction

After training the model, in order to make predictions on the testing data, we need to estimate the factors and homogeneous signals in the testing data. In practice, they are not observable. We provide a data-driven procedure to estimate them based on the estimated loading matrix. Let ${\mathbf{X}}_{g,\mathrm{*}}\in {\mathbb{R}}^{n_{g,\mathrm{*}}\times p}$ denote the testing data matrix from group $g$. We aim to estimate the factor matrix ${\mathbf{F}}_{g,\mathrm{*}}\in {\mathbb{R}}^{n_{g,\mathrm{*}}\times K_g}$ and the homogeneous signal matrix ${\mathbf{U}}_{g,\mathrm{*}}\in {\mathbb{R}}^{n_{g,\mathrm{*}}\times p}$. Note that the number of columns in ${\mathbf{F}}_{g,\mathrm{*}}$ is the same as that in ${\mathbf{F}}_g$.

Motivated by (3.1), we assume that the training and testing data from the same group have the same factor decomposition with the same loading matrix ${\mathbf{\Lambda }}_g$. Hence, given ${\stackrel{\mathbf{ˆ}}{\mathbf{\Lambda }}}_g$ from the training data, we propose estimating ${\mathbf{F}}_{g,\mathrm{*}}$ by solving

$$\begin{gathered} \underset{{\mathbf{F}}_{g,\mathrm{*}}}{\mathrm{m}\mathrm{i}\mathrm{n}} {∥{\mathbf{X}}_{g,\mathrm{*}}-{\mathbf{F}}_{g,\mathrm{*}}{\stackrel{\mathbf{ˆ}}{\mathbf{\Lambda }}}_g∥}_F, \\ \text{s.t.}{\mathbf{F}}_{g,\mathrm{*}}^{\prime }{\mathbf{F}}_{g,\mathrm{*}}=n_{g,\mathrm{*}}\mathbf{I}. \end{gathered}$$ (3.7)

Note that (3.7) can be formulated as a trace maximization problem, the solution of which is given by ${\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_{g,\mathrm{*}}=\sqrt{n_{g,\mathrm{*}}}{\tilde{\mathbf{V}}}_g{\tilde{\mathbf{U}}}_g^{\prime }$, where ${\tilde{\mathbf{V}}}_g$ and ${\tilde{\mathbf{U}}}_g$ come from a singular value decomposition with ${\stackrel{\mathbf{ˆ}}{\mathbf{\Lambda }}}_g{\mathbf{X}}_{g,\mathrm{*}}^{\prime }={\tilde{\mathbf{U}}}_g{\mathbf{S}}_g{\tilde{\mathbf{V}}}_g^{\prime }$.

4. Theoretical Properties

We study the statistical properties of the proposed estimator. Without loss of generality, we assume that ${\mu }_g^{\mathrm{*}}=0$ for any $g\in \{1,\dots ,G\}$, so that (2.4) reduces to

$${\mathit{Y}}_g={\mathbf{F}}_g{\gamma }_g^{\mathrm{*}}+{\mathbf{U}}_g{\mathit{\beta }}^{\mathrm{*}}+{\mathit{ϵ}}_g.$$ (4.1)

We establish the following theoretical results. First, we prove in Theorem 1 that the proposed estimators are consistent up to a rotation of the true parameters. As a corollary, we give an upper bound of the prediction error for the proposed method. Second, we show in Theorems 2 and 3 that if (4.1) is true, the group-specific model and the global model yield worse predictions than those of our proposed method. On the other hand, we show in Theorem 4 that even if one assumes each group has a distinct model, our method can have the same prediction error as the group-specific model when $p$ is sufficiently large. Thus, our method is robust to model mis-specification.

First, we introduce some notation. For a matrix $\mathbf{A}\in {\mathbb{R}}^{p\times p}$, let ${\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}\left(\mathbf{A}\right)$ and ${\lambda }_{\mathrm{m}\mathrm{a}\mathrm{x}}\left(\mathbf{A}\right)$ denote its minimum and maximum eigenvalues, respectively. Let $\parallel \mathbf{A}{\parallel }_F=\sqrt{\mathrm{t}\mathrm{r}\left({\mathbf{A}}^{\prime }\mathbf{A}\right)},\parallel \mathbf{A}\parallel ={\lambda }_{\mathrm{m}\mathrm{a}\mathrm{x}}\left({\mathbf{A}}^{\prime }\mathbf{A}\right),\parallel \mathbf{A}{\parallel }_1={\mathrm{m}\mathrm{a}\mathrm{x}}_{j\le p} {\sum }_{i=1}^p \left|a_{ij}\right|$, and $\parallel \mathbf{A}{\parallel }_{\mathrm{m}\mathrm{a}\mathrm{x}}={\mathrm{m}\mathrm{a}\mathrm{x}}_{i,j\le p} \left|a_{ij}\right|$ denote its Frobenius, ${\ell }_2,{\ell }_1$, and elementwise maximum norms, respectively. For a vector $\mathit{b}\in {\mathbb{R}}^p$, let $\parallel \mathit{b}\parallel =\sqrt{{\sum }_{j=1}^p b_j^2},\parallel \mathit{b}{\parallel }_1={\sum }_{j=1}^p \left|b_j\right|$, and $\parallel \mathit{b}{\parallel }_{\mathrm{\infty }}={\mathrm{m}\mathrm{a}\mathrm{x}}_{j\le p} \left|b_j\right|$ denote its ${\ell }_2,{\ell }_1$, and maximum norms, respectively, and define its support as $\left\{j:b_j\ne 0\right\}$. Furthermore, we let $n_{\mathrm{m}\mathrm{a}\mathrm{x}}={\mathrm{m}\mathrm{a}\mathrm{x}}_{g\le G} n_g,n={\sum }_{g=1}^G n_g$, and $\left[m\right]=\{1,\dots ,m\}$ for a general positive integer $m$. In addition, we introduce the following definitions.

Definition 1.

A vector $\mathit{\beta }\in {\mathbb{R}}^p$ is called $s$-sparse if and only if its support’s cardinality is at most $s$.

Definition 2 (RE Condition).

A matrix $\mathbf{\Sigma }$ is said to satisfy the restricted eigenvalue (RE) condition if and only if there exists a positive constant $\kappa$, such that ${\mathit{\beta }}^{\prime }\mathbf{\Sigma }\mathit{\beta }\ge \kappa \parallel \mathit{\beta }{\parallel }^2$ for any $\mathit{\beta }\in \mathbb{C}\left(S\right)=\left\{\mathit{\beta }\in {\mathbb{R}}^p:{∥{\mathit{\beta }}_{S^c}∥}_1\le 3{∥{\mathit{\beta }}_S∥}_1\right\}$, where $S\subset \left[p\right]$ and $S^c$ denotes its complement.

4.1. Consistency of the factor regression method

To establish the consistency of our proposed method, we need to impose the following conditions.

Assumption 1 (Pervasiveness).

There exist positive constants $C_{\mathrm{m}\mathrm{i}\mathrm{n}}$ and $C_{\mathrm{m}\mathrm{a}\mathrm{x}}>0$ such that, for any $g\in \left[G\right]$,

$$C_{\mathrm{m}\mathrm{i}\mathrm{n}}<{\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}\left(p^{-1}{\mathbf{\Lambda }}_g{\mathbf{\Lambda }}_g^{\prime }\right)<{\lambda }_{\mathrm{m}\mathrm{a}\mathrm{x}}\left(p^{-1}{\mathbf{\Lambda }}_g{\mathbf{\Lambda }}_g^{\prime }\right)<C_{\mathrm{m}\mathrm{a}\mathrm{x}}.$$

Assumption 2.

For any $g\in \left[G\right]$, assume that ${\left\{{\mathbf{f}}_{g,i}\right\}}_{i\le n_g}$ and ${\left\{{\mathbf{u}}_{g,i}\right\}}_{i\le n_g}$ are i.i.d. sub-Gaussian random variables with zero means and covariances ${\mathbf{I}}_{K_g\times K_g}$ and ${\mathbf{\Sigma }}_u$, respectively. More explicitly, assume for any $\mathbf{\alpha }\in {\mathbb{R}}^{K_g}$, $\mathbf{\gamma }\in {\mathbb{R}}^p$, and $s>0$, there exists $C>0$ such that $\mathbb{P}\left(\left|{\mathbf{\alpha }}^{\prime }{\mathbf{f}}_{g,i}\right|>s\right)\le \mathrm{e}\mathrm{x}\mathrm{p}(-C{s}^2/\parallel \mathbf{\alpha }{\parallel }^2)$ and $\mathbb{P}\left(\left|{\gamma }^{\prime }{\mathbf{u}}_{g,i}\right|>s\right)\le \mathrm{e}\mathrm{x}\mathrm{p}(-C{s}^2/\parallel \gamma {\parallel }^2)$. Morever, assume ${\left\{{\mathbf{f}}_{g,i}\right\}}_{i\le n_g}$ are uncorrelated with ${\left\{{\mathbf{u}}_{g,i}\right\}}_{i\le n_g}$.

Assumption 3.

There exist positive constants $c_1$ and $c_2$ such that ${\lambda }_{min}\left({\mathbf{\Sigma }}_u\right)>c_1$ and ${∥{\mathbf{\Sigma }}_u∥}_1<c_2$.

Assumption 4.

For any $g\in \left[G\right]$, $j\in \left[p\right]$, and $i_1,i_2,i\in \left[n_g\right]$, there exists a positive constant $M$ such that

1. ${∥{\mathbf{\lambda }}_{g,j}∥}_{\mathrm{\infty }}<M$, where ${\mathbf{\lambda }}_{g,j}$ denotes the jth column of ${\mathbf{\Lambda }}_g$

2. $\mathbb{E}{[p^{-1/2}\{{\mathbf{u}}_{g,i_1}^{\prime }{\mathbf{u}}_{g,i_2}-\mathbb{E}({\mathbf{u}}_{g,i_1}^{\prime }{\mathbf{u}}_{g,i_2})\}]}^4<M$;

3. $\mathbb{E}{∥p^{-1/2}{\sum }_{j=1}^p {\mathbf{\lambda }}_{g,j}{u}_{g,ij}∥}^4<M$.

Assumption 1 is a typical pervasiveness assumption to ensure that the latent factors can be well estimated by the PCA method (Bai and Ng (2013); Fan, Liao and Mincheva (2013)). Such an assumption assumes that the latent factors affect a large proportion of variables, and is commonly used in the factor analysis literature. Assumption 2 is a typical sub-Gaussian assumption on the latent factors and the idiosyncratic components. Assumption 3 is a regularity condition on ${\mathbf{\Sigma }}_u$. Assumption 4 is a collection of technical conditions needed to establish the factor estimation consistency. Such conditions are commonly used in the factor analysis literature (Bai (2003); Bai and Ng (2008); Fan, Liao and Mincheva (2013)). Given these conditions, we show that under model (4.1), the proposed estimators are consistent.

Theorem 1.

Suppose Assumptions 1–3 hold, $\mathrm{l}\mathrm{o}\mathrm{g}p=o\left(n^{2/39}\right),n=o\left(p^2\right)$, and $m_p{\omega }_n=o\left(1\right)$. Then, it follows that

1. $∥{\hat{\gamma }}_g-{\mathbf{H}}_g{\gamma }_g^{\mathrm{*}}∥=O_P\left(1/\sqrt{n_g}+1/\sqrt{p}\right)$, where ${\stackrel{\mathbf{ˆ}}{\mathbf{\gamma }}}_g$ is as defined in (3.3), ${\mathbf{H}}_g={\stackrel{\mathbf{ˆ}}{\mathbf{D}}}_g^{-1}{\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g^{\prime }{\mathbf{F}}_g{\mathbf{\Lambda }}_g{\mathbf{\Lambda }}_g^{\prime }$, and ${\stackrel{\mathbf{ˆ}}{\mathbf{D}}}_{\mathrm{g}}$ is a ${\hat{K}}_g\times {\hat{K}}_g$ diagonal matrix consisting of the ${\hat{K}}_g$ largest eigenvalues of ${\mathbf{X}}_g{\mathbf{X}}_g^{\prime }$.

2. In (3.4), if we choose an ${\ell }_2$ penalty and $\lambda =C\mathrm{m}\mathrm{a}\mathrm{x}\{n_{\mathrm{m}\mathrm{a}\mathrm{x}}^{3/4}/n,\sqrt{n_{\mathrm{m}\mathrm{a}\mathrm{x}}p}/n\}$, for some large enough constant $C$, we have

   $$\parallel {\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda }^{ridge}-{\mathit{\beta }}^{\mathrm{*}}\parallel =O_P\left(\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}^{3/4}}{n}+\frac{\sqrt{n_{\mathrm{m}\mathrm{a}\mathrm{x}}p}}{n}+m_p{\omega }_n\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}}{n}\right).$$ (4.2)
3. Assuming that ${\mathit{\beta }}^{\mathrm{*}}$ is s-sparse, ${\mathbf{\Sigma }}_u$ satisfies the $RE$ condition, and ${s\omega }_n=o\left(1\right)$, if we choose an ${\ell }_1$ penalty in (3.4) and $\lambda =C{\omega }_n(m_p+\sqrt{n_{\mathrm{m}\mathrm{a}\mathrm{x}}/n})$, for some large enough constant $C$, then we have

   $$\parallel {\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda }^{lasso}-{\mathit{\beta }}^{\mathrm{*}}\parallel =O_P\left(\sqrt{s}\left(m_p{\omega }_n+\sqrt{\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}}{n}}{\omega }_n\right)\right).$$ (4.3)

Statement $\left(a\right)$ shows that ${\stackrel{\mathbf{ˆ}}{\mathbf{\gamma }}}_g$ is consistent to ${\mathbf{\gamma }}_g^{\mathrm{*}}$ up to a rotation given by ${\mathbf{H}}_g$. When the latent factors are known, the oracle convergence rate of ${\stackrel{\mathbf{ˆ}}{\mathbf{\gamma }}}_g$ is $O_P\left(1/\sqrt{n_g}\right)$. Compared with this oracle rate, the extra term of $O_P(1/\sqrt{p})$ is essentially due to the estimation error of the latent factors; see Lemma 1(a). When $p\gg n$, such a term is ignorable and the oracle rate can be attained. This is because, in that situation, many variables can be used to estimate the latent factors. The error in estimating the latent factors is so small that it does not affect the convergence rate of ${\stackrel{\mathbf{ˆ}}{\mathbf{\gamma }}}_g$. This is essentially due to the blessing of dimensionality property of a factor analysis, which has been studied in Li et al. (2018). Statements (b) and (c) show that the proposed penalized estimator in (3.4) is consistent to ${\mathit{\beta }}^{\mathrm{*}}$, regardless of whether an ${\ell }_1$ or ${\ell }_2$ penalty is imposed. To simplify the discussion, assume that $n_1=\cdots =n_G,m_p$, and $G$ are bounded. Then, the convergence rates in (4.2) and (4.3) reduce to

$$\parallel {\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda }^{ridge}-{\mathit{\beta }}^{\mathrm{*}}\parallel =O_P\left(\frac{1}{n^{1/4}}+\sqrt{\frac{p}{n}}\right),\parallel {\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda }^{lasso}-{\mathit{\beta }}^{\mathrm{*}}\parallel =O_P\left(\sqrt{\frac{s}{p}}+\sqrt{\frac{s\mathrm{l}\mathrm{o}\mathrm{g}p}{n}}\right).$$ (4.4)

Hsu, Kakade and Zhang (2014) show that the minimax rate of a Ridge estimator in a linear regression model is $O_P(\sqrt{p/n})$ if no sparsity assumption is assumed. Compared with this minimax rate, our method has an extra term of $O_P(1/n^{1/4})$, which is again due to the error when estimating the latent factors; see Lemma 4. However, when $p\gg n$, such a term is ignorable and the minimax rate can be obtained. A similar conclusion can be drawn for the Lasso estimator. In (4.4), the term of $O_P(\sqrt{s\mathrm{l}\mathrm{o}\mathrm{g}p/n})$ agrees with the minimax rate of the standard Lasso problem (Raskutti, Wainwright and Yu (2011)). The extra term of $O_P(\sqrt{s/p})$ comes from the estimation error ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_u$; see Fan, Liao and Mincheva (2013). This term is ignorable when $p\gg n$, in which case the minimax rate is attained.

Let ${\stackrel{\mathit{ˆ}}{\mathit{Y}}}_{g,\lambda }^{ridge}={\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g{\hat{\gamma }}_g+{\stackrel{\mathbf{ˆ}}{\mathbf{U}}}_g{\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_{\lambda }^{ridge}$ and ${\stackrel{\mathit{ˆ}}{\mathit{Y}}}_{g,\lambda }^{lasso}={\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g{\hat{\gamma }}_g+{\stackrel{\mathbf{ˆ}}{\mathbf{U}}}_g{\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_{\lambda }^{lasso}$ denote the predicted values of ${\mathit{Y}}_g$, where ${\stackrel{\mathit{ˆ}}{\mathit{\gamma }}}_g$ is given in (3.3), ${\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_{\lambda }^{ridge}$ and ${\stackrel{\mathit{ˆ}}{\mathit{\beta }}}_{\lambda }^{lasso}$ are the Ridge and Lasso estimators, respectively, solved from (3.4), and ${\stackrel{\mathbf{ˆ}}{\mathbf{F}}}_g$ and ${\stackrel{\mathbf{ˆ}}{\mathbf{U}}}_g$ are as described in Section 3.1. The following corollary gives the upper bounds of the corresponding in-sample prediction errors.

Corollary 1.

Under the assumptions of Theorem 1, we have

$$\begin{gathered} ∥\frac{1}{n_g}\{{\stackrel{\mathit{ˆ}}{\mathit{Y}}}_{g,\lambda }^{ridge}-\mathbb{E}\left({\mathit{Y}}_g\mid {\mathbf{F}}_g,{\mathbf{U}}_g\right)\}∥= \\ {O}_P\left(\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}^{3/4}}{n\sqrt{n_g}}+\frac{1}{n}\sqrt{\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}p}{n_g}}+m_p{\omega }_n\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}}{n\sqrt{n_g}}\right)+O_P\left(\frac{\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}{n}_g\mathrm{l}\mathrm{o}\mathrm{g}p}}{n_g}+\frac{1}{n_g^{1/4}\sqrt{p}}\right), \end{gathered}$$ (4.5)

$$\begin{gathered} ∥\frac{1}{n_g}\{{\stackrel{\mathit{ˆ}}{\mathit{Y}}}_{g,\lambda }^{lasso}-\mathbb{E}\left({\mathit{Y}}_g\mid {\mathbf{F}}_g,{\mathbf{U}}_g\right)\}∥= \\ {O}_P\left(\sqrt{\frac{s}{n_g}}\left(m_p{\omega }_n+\sqrt{\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}}{n}}{\omega }_n\right)\right)+O_P\left(\frac{\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}{n}_g\mathrm{l}\mathrm{o}\mathrm{g}p}}{n_g}+\frac{1}{n_g^{1/4}\sqrt{p}}\right). \end{gathered}$$ (4.6)

Again, if we assume $n_1=\cdots =n_G,m_p$, and $G$ are bounded, these results reduce to

$$∥\frac{1}{n_g}\{{\stackrel{\mathit{ˆ}}{\mathit{Y}}}_{g,\lambda }^{ridge}-\mathbb{E}\left({\mathit{Y}}_g\mid {\mathbf{F}}_g,{\mathbf{U}}_g\right)\}∥=O_P\left(\frac{1}{n^{1/4}\sqrt{p}}+\frac{\sqrt{p}}{n}\right),$$ (4.7)

$$\begin{gathered} ∥\frac{1}{n_g}\{{\stackrel{\mathit{ˆ}}{\mathit{Y}}}_{g,\lambda }^{lasso}-\mathbb{E}\left({\mathit{Y}}_g\mid {\mathbf{F}}_g,{\mathbf{U}}_g\right)\}∥ \\ =O_P\left(\frac{1}{n^{1/4}\sqrt{p}}+\sqrt{\frac{s}{np}}+\frac{\sqrt{\log n\mathrm{l}\mathrm{o}\mathrm{g}p}}{n}+\frac{\sqrt{s\mathrm{l}\mathrm{o}\mathrm{g}p}}{n}\right). \end{gathered}$$ (4.8)

In (4.7), the term of $O_P(\sqrt{p}/n)$ agrees with the minimax rate of the prediction error given by the Ridge estimator in a standard linear regression problem (Dicker (2016); Dobriban and Wager (2018)). In (4.8), the term of $O_P(\sqrt{s\mathrm{l}\mathrm{o}\mathrm{g}p}/n)$ agrees with the prediction error given by the Lasso estimator in the standard setting (Bickel, Ritov and Tsybakov (2009)). All other terms are ignorable when $p\gg n$.

In conclusion, these results show that our proposed estimators can have the same convergence rates as the Ridge and Lasso estimators have under the standard homogeneous linear regression model, which is simpler than the heterogeneous model we have considered.

4.2. Consistency of group-specific and global models

We study the statistical properties of the group-specific and global models when the underlying model follows (4.1). We show that, in this case, our proposed method has an advantage over these two models in terms of the prediction error. We rewrite (4.1) as

$${\mathit{Y}}_g={\tilde{\mathbf{X}}}_g{\mathit{\beta }}^{\mathrm{*}}+{\mathbf{F}}_g{\mathbf{\delta }}_g+d_p{\mathbf{U}}_g{\mathit{\beta }}^{\mathrm{*}}+{\mathit{ϵ}}_g,$$ (4.9)

where ${\tilde{\mathbf{X}}}_g=p^{-1/2}{\mathbf{X}}_g,{\mathbf{\delta }}_g={\gamma }_g^{\mathrm{*}}-p^{-1/2}{\mathbf{\Lambda }}_g{\mathit{\beta }}^{\mathrm{*}}$, and $d_p=1-p^{-1/2}$. Here, we standardize ${\mathbf{X}}_g$ by dividing it by $p^{1/2}$. This is because the pervasiveness assumption means that $∥{\mathbf{X}}_g∥$ is unbounded, which is different from the typical linear regression model. Therefore, we rescale it to be ${\tilde{\mathbf{X}}}_g$. Then, the group-specific model seeks to solve

$${\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{g,\lambda }=\underset{\mathit{\beta }}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}\frac{1}{2n_g}{∥{\mathit{Y}}_g-{\tilde{\mathbf{X}}}_g\mathit{\beta }∥}^2+\lambda P(\mathit{\beta }),$$ (4.10)

whereas the global model seeks to solve

$${\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda ,global}=\underset{\mathit{\beta }}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}\frac{1}{2n}\parallel \mathit{Y}-\tilde{\mathbf{X}}\mathit{\beta }{\parallel }^2+\lambda P\left(\mathit{\beta }\right),$$ (4.11)

where $\tilde{\mathbf{X}}={({\tilde{\mathbf{X}}}_1^{\prime },\dots ,{\tilde{\mathbf{X}}}_G^{\prime })}^{\prime },\lambda$ is a tuning parameter and $P(\mathit{\beta })$ is a general penalty function. Similar to (3.4), we choose either an ${\ell }_1$ or an ${\ell }_2$ penalty, and denote the corresponding solutions as ${\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{g,\lambda }^{lasso},{\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda ,global}^{lasso}$ and ${\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{g,\lambda }^{ridge},{\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda ,global}^{ridge}$ respectively. Next, we give the convergence rates of the estimators in the group-specific and global models in Theorems 2 and 3, respectively.

Theorem 2.

Suppose Assumptions 1–3 hold and $\mathrm{l}\mathrm{o}\mathrm{g}p=o\left(n\right)$. Then, it follows that

1. If we use an ${\ell }_2$ penalty in (4.10) and choose $\lambda =C/\sqrt{p}$, for some large enough constant $C$, we have

   $$\parallel {\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{g,\lambda }^{ridge}-{\mathit{\beta }}^{\mathrm{*}}\parallel =O_P\left(\sqrt{p}∥{\mathbf{\delta }}_g∥+d_p\left(1+\sqrt{\frac{p}{n_g}}\right)+\sqrt{\frac{p}{n_g}}\right).$$ (4.12)
2. Assuming that ${\mathit{\beta }}^{\mathrm{*}}$ is s-sparse, ${\mathbf{\Lambda }}_g^{\prime }{\mathbf{\Lambda }}_g/\sqrt{p}$ satisfies the $RE$ condition, and $s\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}p/\left(n_{g}p\right)}=o\left(1\right)$, if we use an ${\ell }_1$ penalty in (4.10) and choose $\lambda =C\left\{\left(1+\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}p/n_g}\right)\left(d_p+∥{\mathbf{\delta }}_g∥\right)+\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}p/n_g}\right\}/\sqrt{p}$, for some large enough constant $C$, we have

   $$\parallel {\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{g,\lambda }^{lasso}-{\mathit{\beta }}^{\mathrm{*}}\parallel =O_P\left(\sqrt{s}\left\{\left(1+\sqrt{\frac{\mathrm{l}\mathrm{o}\mathrm{g}p}{n_g}}\right)\left(d_p+∥{\mathit{\delta }}_g∥\right)+\sqrt{\frac{\mathrm{l}\mathrm{o}\mathrm{g}p}{n_g}}\right\}\right).$$ (4.13)

Let ${\stackrel{\mathbf{ˆ}}{\mathit{Y}}}_{g,\lambda }^{ridge}={\tilde{\mathbf{X}}}_g{\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{g,\lambda }^{ridge}$ and ${\stackrel{\mathbf{ˆ}}{\mathit{Y}}}_{g,\lambda }^{lasso}={\tilde{\mathbf{X}}}_g{\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{g,\lambda }^{lasso}$ be the predicted values of ${\mathit{Y}}_g$, where ${\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{g,\lambda }^{ridge}$ and ${\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{g,\lambda }^{lasso}$ are the Ridge and Lasso solutions, respectively, to (4.10). We have the following upper bounds of their in-sample prediction errors.

Corollary 2.

Under the assumptions of Theorem 2, we have

$$\begin{gathered} \parallel \frac{1}{n_g}\{{\stackrel{\mathbf{ˆ}}{\mathit{Y}}}_{g,\lambda }^{ridge}-\mathbb{E}\left({\mathit{Y}}_g\mid {\mathbf{F}}_g,{\mathbf{U}}_g\right)\}\parallel \\ =O_P\left(\sqrt{\frac{p}{n_g}}∥{\mathbf{\delta }}_g∥+d_p\left(\frac{1}{\sqrt{n_g}}+\frac{\sqrt{p}}{n_g}\right)+\frac{\sqrt{p}}{n_g}\right), \end{gathered}$$ (4.14)

$$\begin{gathered} \parallel \frac{1}{n_g}\{{\stackrel{\mathbf{ˆ}}{\mathit{Y}}}_{g,\lambda }^{lasso}-\mathbb{E}\left({\mathit{Y}}_g\mid {\mathbf{F}}_g,{\mathbf{U}}_g\right)\}\parallel \\ =O_P\left(\sqrt{\frac{s}{n_g}}\left\{\left(1+\sqrt{\frac{\mathrm{l}\mathrm{o}\mathrm{g}p}{n_g}}\right)\left(d_p+∥{\mathbf{\delta }}_g∥\right)+\sqrt{\frac{\mathrm{l}\mathrm{o}\mathrm{g}p}{n_g}}\right\}\right). \end{gathered}$$ (4.15)

Theorem 3.

Suppose Assumptions 1–3 hold and $\mathrm{l}\mathrm{o}\mathrm{g}p=o\left(n\right)$. Then, it follows that

1. If we use an ${\ell }_2$ penalty in (4.11) and choose $\lambda =C/\sqrt{p}$, for some large enough constant $C$, we have

   $$\parallel {\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda ,\text{global}}^{ridge}-{\mathit{\beta }}^{\mathrm{*}}\parallel =O_P\left(\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}\sqrt{p}}{n}\sum _{g=1}^G ∥{\mathbf{\delta }}_g∥+d_p\left(\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}}{n}+\frac{\sqrt{n_{\mathrm{m}\mathrm{a}\mathrm{x}}p}}{n}\right)+\frac{\sqrt{n_{\mathrm{m}\mathrm{a}\mathrm{x}}p}}{n}\right).$$ (4.16)
2. Assuming that ${\mathit{\beta }}^{\mathrm{*}}$ is s-sparse, ${\mathbf{\Lambda }}_g^{\prime }{\mathbf{\Lambda }}_g/\sqrt{p}$ satisfies the $RE$ condition, and $s\sqrt{\mathrm{l}\mathrm{o}\mathrm{g}p/\left(n_{g}p\right)}=o\left(1\right)$ for any $g\in \left[G\right]$, if we use an ${\ell }_1$ penalty in (4.11) and choose $\lambda =C[\{n_{\mathrm{m}\mathrm{a}\mathrm{x}}/(n\sqrt{p})+(1/n)\sqrt{n_{\mathrm{m}\mathrm{a}\mathrm{x}}\mathrm{l}\mathrm{o}\mathrm{g}p/p}\}(d_p+{\sum }_{g=1}^G \parallel {\mathit{\delta }}_g\parallel )+(1/n)\sqrt{n_{\mathrm{m}\mathrm{a}\mathrm{x}}\mathrm{l}\mathrm{o}\mathrm{g}p/p}]$, for some large enough constant $C$, we have

   $$\begin{gathered} \parallel {\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda ,global}^{lasso}-{\mathit{\beta }}^{\mathrm{*}}\parallel \\ =O_P\left(\sqrt{s}\left\{\left(\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}}{n}+\frac{\sqrt{n_{\mathrm{m}\mathrm{a}\mathrm{x}}\mathrm{l}\mathrm{o}\mathrm{g}p}}{n}\right)\left(d_p+\sum _{g=1}^G ∥{\mathit{\delta }}_g∥\right)+\frac{\sqrt{n_{\mathrm{m}\mathrm{a}\mathrm{x}}\mathrm{l}\mathrm{o}\mathrm{g}p}}{n}\right\}\right). \end{gathered}$$ (4.17)

Let ${\stackrel{\mathbf{ˆ}}{\mathit{Y}}}_{g,\lambda }^{ridge}={\tilde{\mathbf{X}}}_g{\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda ,global}^{ridge}$ and ${\stackrel{\mathbf{ˆ}}{\mathit{Y}}}_{g,\lambda }^{lasso}={\tilde{\mathbf{X}}}_g{\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda ,global}^{lasso}$ be the predicted values of ${\mathit{Y}}_g$, where ${\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda ,global}^{ridge}$ and ${\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda ,global}^{lasso}$ are the Ridge and Lasso solutions, respectively, (4.11). We have the following upper bounds for their in-sample prediction errors.

Corollary 3.

Under the assumptions of Theorem 3, we have

$$\begin{gathered} ∥\frac{1}{n_g}\{{\stackrel{\mathit{ˆ}}{\mathit{Y}}}_{g,\lambda }^{ridge}-\mathbb{E}\left({\mathit{Y}}_g\mid {\mathbf{F}}_g,{\mathbf{U}}_g\right)\}∥ \\ =O_P\left(\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}}{n}\sqrt{\frac{p}{n_g}}{\sum }_{g^{\prime }=1}^G ∥{\mathit{\delta }}_{g^{\prime }}∥+\frac{1}{n}\sqrt{\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}p}{n_g}}+d_p\left(\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}}{n_{\sqrt{n_g}}}+\frac{1}{n}\sqrt{\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}p}{n_g}}\right)\right), \end{gathered}$$ (4.18)

$$\begin{gathered} ∥\frac{1}{n_g}\left\{{\stackrel{\mathit{ˆ}}{\mathit{Y}}}_{g,\lambda }^{lasso}-\mathbb{E}\left({\mathit{Y}}_g\mid {\mathbf{F}}_g,{\mathbf{U}}_g\right)\right\}∥ \\ =O_P\left(\sqrt{\frac{s}{n_g}}\left\{\frac{\sqrt{n_{\mathrm{m}\mathrm{a}\mathrm{x}}\mathrm{l}\mathrm{o}\mathrm{g}p}}{n}+\left(\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}}{n}+\frac{\sqrt{n_{\mathrm{m}\mathrm{a}\mathrm{x}}\mathrm{l}\mathrm{o}\mathrm{g}p}}{n}\right)\left(d_p+\sum _{g^{\prime }=1}^G ∥{\mathit{\delta }}_{g^{\prime }}∥\right)\right\}\right). \end{gathered}$$ (4.19)

Under (4.1), $∥{\mathbf{\delta }}_g∥\le ∥{\mathbf{\gamma }}_g^{\mathrm{*}}∥+p^{-1/2}∥{\mathbf{\Lambda }}_g∥\parallel {\mathit{\beta }}^{\mathrm{*}}\parallel =O(1)$, for all $g\in \left[G\right]$ and $d_p=O\left(1\right)$. Thus, if we assume that $n_1=\cdots =n_G$ and $G$ is bounded, then (4.14) and (4.18) further reduce to $\parallel \left(1/n_g\right)\{{\stackrel{\mathbf{ˆ}}{\mathit{Y}}}_{g,\lambda }^{ridge}-\mathbb{E}\left({\mathit{Y}}_g\mid {\mathbf{F}}_g,{\mathbf{U}}_g\right)\}\parallel =O_P(\sqrt{p/n})$ for the Ridge estimator. Compared with the predictor error of our Ridge estimator, which is $O_P(\sqrt{p}/n)$, these two methods are worse by a factor of $\sqrt{n}$, owing to the mis-specified model (4.1). Similarly for the Lasso estimator, when $n_1=\cdots =n_G$ and $G$ is bounded, (4.15) and (4.19) reduce to $\parallel \left(1/n_g\right)\{{\stackrel{\mathit{ˆ}}{\mathit{Y}}}_{g,\lambda }^{lasso}-\mathbb{E}\left({\mathit{Y}}_g\mid {\mathbf{F}}_g,{\mathbf{U}}_g\right)\}\parallel =O_P(\sqrt{s/n}+\sqrt{s\mathrm{l}\mathrm{o}\mathrm{g}p}/n)$. Compared with our Lasso estimator, they have an extra term of $\sqrt{s/n}$, which also comes from the model mis-specification and is nonignorable.

4.3. Robustness

In this section, we assume each group follows a distinct model

$${\mathit{Y}}_g={\tilde{\mathbf{X}}}_g{\mathit{\beta }}_g^{\mathrm{*}}+{\mathit{ϵ}}_g,$$ (4.20)

and examine how well our method performs under this model assumption. In other words, we study how robust our method is under model mis-specification. Here, we still use the rescaled ${\tilde{\mathbf{X}}}_g$ as the design matrix. We rewrite (4.20) as ${\mathit{Y}}_g=p^{-1/2}{\mathbf{F}}_g{\mathbf{\Lambda }}_g{\mathit{\beta }}_g^{\mathrm{*}}+p^{-1/2}{\mathbf{U}}_g{\mathit{\beta }}_g^{\mathrm{*}}+{\mathit{ϵ}}_g$. Compared with (4.1), we see that $p^{-1/2}{\mathbf{\Lambda }}_g{\mathit{\beta }}_g^{\mathrm{*}}$ and $p^{-1/2}{\mathit{\beta }}_g^{\mathrm{*}}$ can be viewed as ${\gamma }_g^{\mathrm{*}}$ and ${\mathit{\beta }}^{\mathrm{*}}$, respectively, in our model. Under the model assumption in (4.20), we have the following results.

Theorem 4.

Suppose Assumptions 1–3 hold, $\mathrm{l}\mathrm{o}\mathrm{g}p=o\left(n^{2/39}\right),n=o\left(p^2\right)$, and $m_p{\omega }_n=o\left(1\right)$. Then, for any $g\in \left[G\right]$, it follows that

1. $∥{\hat{\gamma }}_g-p^{-1/2}{\mathbf{H}}_g{\mathbf{\Lambda }}_g{\mathit{\beta }}_g^{\mathrm{*}}∥=O_P\left(1/\sqrt{n_g}+1/\sqrt{p}\right)$, where ${\mathbf{H}}_g$ is as defined in Theorem 1.

2. If an ${\ell }_2$ penalty in (3.4) is used and $\lambda =O(\mathrm{m}\mathrm{a}\mathrm{x}\{n_{\mathrm{m}\mathrm{a}\mathrm{x}}^{3/4}\sqrt{p}/n,\sqrt{n_{\mathrm{m}\mathrm{a}\mathrm{x}}}p/n\})$, then

   $$\Vert {\widehat{\mathit{\beta }}}_{\lambda }^{ridge}-\frac{1}{\sqrt{p}}{\mathit{\beta }}_g^{*}\Vert =O_P\left(\frac{\sqrt{n_{\max }p}}{n}+\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}^{3/4}}{n}\right)+\sum _{g^{\prime }=1}^G{O}_P\left(\frac{n_{g^{\prime }}}{n\sqrt{p}}\Vert {\mathit{\beta }}_{g^{\prime }}^{*}-{\mathit{\beta }}_g^{*}\Vert \right).$$
3. Assuming that ${\mathit{\beta }}_g^{\mathrm{*}}$ is s-sparse and ${\mathbf{\Sigma }}_u$ satisfies the $RE$ condition, if we use an ${\ell }_1$ penalty in (3.4) and choose $\lambda =C\{{\omega }_n\sqrt{n_{\mathrm{m}\mathrm{a}\mathrm{x}}/n}+n_{\mathrm{m}\mathrm{a}\mathrm{x}}/(n\sqrt{p}){\sum }_{g^{\prime }=1}^G \parallel {\mathit{\beta }}_g^{\mathrm{*}}-{\mathit{\beta }}_{g^{\prime }}^{\mathrm{*}}\parallel \}$, for some large enough constant $C$, we have

   $$∥{\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda }^{lasso}-\frac{1}{\sqrt{p}}{\mathit{\beta }}_g^{\mathrm{*}}∥=O_P\left(\sqrt{s}\left(\sqrt{\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}}{n}}{\omega }_n+\frac{n_{\mathrm{m}\mathrm{a}\mathrm{x}}}{n\sqrt{p}}\sum _{g^{\prime }=1}^G ∥{\mathit{\beta }}_g^{\mathrm{*}}-{\mathit{\beta }}_{g^{\prime }}^{\mathrm{*}}∥\right)\right).$$

Let ${\stackrel{\mathbf{ˆ}}{\mathit{Y}}}_{g,\lambda }^{ridge}$ and ${\stackrel{\mathbf{ˆ}}{\mathit{Y}}}_{g,\lambda }^{lasso}$ be the same as in Corollary 1. Using Theorem 4, we give the upper bounds of the in-sample prediction errors given by our proposed method, when the underlying model follows (4.20).

Corollary 4.

Under the assumptions of Theorem 4, for each $g\in \left[G\right]$, we have

$$\begin{gathered} ∥\frac{1}{n_g}\{{\stackrel{\mathbf{ˆ}}{\mathit{Y}}}_{g,\lambda }^{ridge}-\mathbb{E}({\mathit{Y}}_g\mid {\tilde{\mathbf{X}}}_g)\}∥ \\ =O_P\left(\frac{1}{n_g}\right)+O_P\left(\frac{1}{\sqrt{n_{g}p}}\right)+O_P\left(\frac{1}{\sqrt{n_g}}\parallel {\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda }^{ridge}-\frac{1}{\sqrt{p}}{\mathit{\beta }}_g^{\mathrm{*}}\parallel \right), \end{gathered}$$ (4.21)

$$\begin{gathered} ∥\frac{1}{n_g}\{{\stackrel{\mathbf{ˆ}}{\mathit{Y}}}_{g,\lambda }^{lasso}-\mathbb{E}({\mathit{Y}}_g\mid {\tilde{\mathbf{X}}}_g)\}∥ \\ =O_P\left(\frac{1}{n_g}\right)+O_P\left(\frac{1}{\sqrt{n_{g}p}}\right)+O_P\left(\frac{1}{\sqrt{n_g}}\parallel {\stackrel{\mathbf{ˆ}}{\mathit{\beta }}}_{\lambda }^{lasso}-\frac{1}{\sqrt{p}}{\mathit{\beta }}_g^{\mathrm{*}}\parallel \right). \end{gathered}$$ (4.22)

When $n_1=\cdots =n_G$ and $G$ is bounded, (4.21) and (4.22) further reduces to

$$\begin{gathered} ∥\frac{1}{n_g}\{{\stackrel{\mathit{ˆ}}{\mathit{Y}}}_{g,\lambda }^{ridge}-\mathbb{E}\left({\mathit{Y}}_g\mid {\tilde{\mathbf{X}}}_g\right)\}∥ \\ =O_P\left({\sum }_{g^{\prime }=1}^G \frac{1}{\sqrt{np}}∥{\mathit{\beta }}_g^{\mathrm{*}}-{\mathit{\beta }}_{g^{\prime }}^{\mathrm{*}}∥\right)+O_P\left(\frac{\sqrt{p}}{n}\right)=O_P\left(\frac{\sqrt{p}}{n}\right), \end{gathered}$$ (4.23)

$$\begin{gathered} \parallel \frac{1}{n_g}\{{\widehat{Y}}_{g,\lambda }^{lasso}-\mathbb{E}(Y_g\mid {\tilde{\mathit{X}}}_g)\}\parallel \\ =O_P(\sum _{g^{\prime }=1}^G\sqrt{\frac{s}{np}}\parallel {\beta }_g^{\ast }-{\beta }_{g^{\prime }}^{\ast }\parallel )+O_P\left(\sqrt{\frac{s}{np}}+\frac{\sqrt{s\log p}}{n}\right). \end{gathered}$$ (4.24)

We compare these convergence rates with those given by the group-specific model. Because the true model (4.20) is a special case of (4.9), by treating $d_p=0$ and ${\mathbf{\delta }}_g=0$, it follows from Theorem 2 that the prediction errors of the group-specific model are $O_P\left(\sqrt{p}/n_g\right)$ and $O_P\left(\sqrt{s\mathrm{l}\mathrm{o}\mathrm{g}p}/n_g\right)$, when using a Ridge or a Lasso estimator, respectively. Comparing then with (4.23) and (4.24), we find that the Ridge estimator of our model has the same rate as the group-specific Ridge estimators; see (4.23). For the Lasso estimator, when $p$ is small, our model converges at a rate of $\sqrt{s/\left(np\right)}$, which is slower than that of the group-specific model by a factor of $\sqrt{n/\left(p\mathrm{l}\mathrm{o}\mathrm{g}p\right)}$. The reason is that our model estimates $G^{-1}{\sum }_{g^{\prime }=1}^G {\mathit{\beta }}_{g^{\prime }}^{\mathrm{*}}$, instead of ${\mathit{\beta }}_g^{\mathrm{*}}$, and needs to estimate ${\mathbf{\Sigma }}_u$, which introduces an extra error of $O_P(\sqrt{s/(np)})$. However, when $p\gg n$, all these terms are negligible, and our model has the same convergence as the group-specific model. In conclusion, we have shown that even if the true model is group-specific, our method still provides comparable prediction to that of the group-specific model, especially when the dimension $p$ is high.

5. Simulation Studies

In this section, we perform two simulation studies to compare our proposed model with the global, group-specific, and Factor-0 models. In both studies, we choose $G=3,p=200,K_g=3$, and $n_g=100$, for any $g\in \left[G\right]$, generate 600 training samples to train all four models, and evaluate their mean squared error (MSE) in an independent test set of 600 samples. Additional simulation studies on other choices of $K_g$ can be found in Section S3.4 in the Supplementary Material. We repeat the simulations 50 times. In setting 1, we generate data from our proposed model. In setting 2, we generate different models for different groups.

5.1. Setting 1: under proposed model

We first generate data from the proposed model in (2.4). For any $g\in \left[G\right]$, we generate ${\left\{{\mathbf{f}}_{g,i}\right\}}_{i\le n_g}$ as i.i.d. samples from $𝒩\left(0,{\mathbf{I}}_{K_g}\times K_g\right)$. We set

$${\mathbf{\Lambda }}_g=\left[\begin{array}{c}{\mathbf{\Lambda }}_g^{1\prime }{\mathbf{\Lambda }}_g^1{\mathbf{\Lambda }}_g^{1\prime }{\mathbf{\Lambda }}_g^2\\ {\mathbf{\Lambda }}_g^{2\prime }{\mathbf{\Lambda }}_g^1{\mathbf{\Lambda }}_g^{2\prime }{\mathbf{\Lambda }}_g^2\end{array}\right].$$

To ensure ${\mathbf{\Lambda }}_g$ satisfies the pervasiveness assumption (Assumption 1), we first choose a positive-definite matrix $\mathbf{R}\mathrm{*}{\mathbf{s}}_g{\mathbf{s}}_g^{\prime }$, where $\mathbf{R}=\left(r_{ij}\right)$ with $r_{ij}={0.1}^{|i-j|},{\mathbf{s}}_g={\left(\sqrt{{\lambda }_{g,1}},\dots ,\sqrt{{\lambda }_{g,K_g}}\right)}^{\prime },\left({\lambda }_{\mathrm{1,1}},{\lambda }_{\mathrm{1,2}},{\lambda }_{\mathrm{1,3}}\right)=\left(\mathrm{7.0,3.5,1.2}\right)$, $\left({\lambda }_{\mathrm{2,1}},{\lambda }_{\mathrm{2,2}},{\lambda }_{\mathrm{2,3}}\right)=\left(\mathrm{10,3.9,1.2}\right)$, $\left({\lambda }_{\mathrm{3,1}},{\lambda }_{\mathrm{3,2}},{\lambda }_{\mathrm{3,3}}\right)=\left(\mathrm{13,3.9,1.1}\right)$, and $\mathrm{*}$ denotes elementwise matrix multiplication. Additional simulation studies on other choices of ${\lambda }_{g,1},\dots ,{\lambda }_{g,K_g}$ can be found in Section S3.2 in the Supplementary Material. Then, we perform an eigendecomposition on it to obtain $\mathbf{R}\mathrm{*}{\mathbf{s}}_g{\mathbf{s}}_g^{\prime }={\mathbf{V}}_g{\mathbf{D}}_g{\mathbf{V}}_g^{\prime }$, where ${\mathbf{D}}_g$ is the diagonal matrix consisting of its eigenvalues. Next, we set ${\mathbf{\Lambda }}_g^1={\mathbf{Q}}_g{\mathbf{D}}_g^{1/2}{\mathbf{V}}_g^{\prime }$, where ${\mathbf{Q}}_g$ is a random orthonormal matrix, and ${\mathbf{\Lambda }}_g^2={\mathbf{Q}}_g{\mathit{T}}_g$, where ${\mathit{T}}_g$ is a $K_g\times \left(p-K_g\right)$ matrix with elements randomly generated from $\mathrm{U}\mathrm{n}\mathrm{i}\mathrm{f}(-1/\mathrm{20,1}/20)$. This construction of ${\mathbf{\Lambda }}_g$ ensures that it has spiked eigenvalues, as required by the pervasiveness assumption, and its rank is $K_g$. We further generate ${\left\{{\mathbf{u}}_{g,i}\right\}}_{i\le n_g}$ as i.i.d. samples from $𝒩\left(0,{\mathbf{\Sigma }}_u\right)$, where ${\mathbf{\Sigma }}_u$ is a diagonal matrix with diagonal elements all equal to 0.03. For the coefficients in (2.4), we choose ${\mu }_g^{\mathrm{*}}=g$ for $g=1,2,3$. We set ${\gamma }_1^{\mathrm{*}}=(h,h,2h{)}^{\prime },{\gamma }_2^{\mathrm{*}}=(h,2h,h{)}^{\prime }$, and ${\gamma }_3^{\mathrm{*}}=(2h,h,h{)}^{\prime }$, where we let $h$ change so that, as it increases, the between-group heterogeneity increases accordingly. We consider two settings of ${\mathit{\beta }}^{\mathrm{*}}$. For a sparse ${\mathit{\beta }}^{\mathrm{*}}$, we set ${\mathit{\beta }}^{\mathrm{*}}={\left({\mathbf{2}}_{10},{\mathbf{0}}_{90},-{\mathbf{2}}_{10},{\mathbf{0}}_{90}\right)}^{\prime }$, where ${\mathit{m}}_L$ denotes an $L$-dimensional vector with elements all equal to $m$; for a dense ${\mathit{\beta }}^{\mathrm{*}}$, we set ${\mathit{\beta }}^{\mathrm{*}}={\left({\mathbf{1}}_{80},{\mathbf{0}}_{20},-{\mathbf{1}}_{80},{\mathbf{0}}_{20}\right)}^{\prime }$. Finally, we generate the error term $ϵ$ as i.i.d samples from $𝒩\left(\mathrm{0,4}\right)$.

Under this model generation scheme, Figure 1 shows how the MSEs of these four methods change as $h$ varies. When ${\mathit{\beta }}^{\mathrm{*}}$ is sparse, all methods use an ${\ell }_1$ penalty; when ${\mathit{\beta }}^{\mathrm{*}}$ is dense, all methods use an ${\ell }_2$ penalty. The shaded areas represent the standard errors of the MSEs in the 50 simulations. The optimal tuning parameters in these methods are chosen using 10-fold cross-validation. It is clearly seen that for most $h$, our model performs best. Owing to the model mis-specification, the group-specific model loses some efficiency in estimating the homogeneous part of (4.9) separately, and the global model entirely ignores the heterogeneity. The Factor-0 model adjusts for group means; therefore, it is better than the global model. However, it is still worse than the proposed full model, indicating that some additional heterogeneity has not been fully taken into account in the Factor-0 model. When $h$ increases, the true model (2.4) becomes more group-specific, and less homogeneity can be used to estimate the common ${\mathit{\beta }}^{\mathrm{*}}$. In this case, the group-specific model gradually outperforms our method. They both become much better than the global and the Factor-0 models. The estimation errors on ${\gamma }_g^{\mathrm{*}}$ and ${\mathit{\beta }}^{\mathrm{*}}$ are reported in Tables S2 and S3 in the Supplementary Material.

Figure 1.

The MSE curves given by the four models. The left panel represents the results for a sparse ${\mathit{\beta }}^{\mathrm{*}}$, and the right panel represents the results for a dense ${\mathit{\beta }}^{\mathrm{*}}$.

5.2. Setting 2: under group-specific model

We generate different models for different groups and inspect how robust our model is under such a scenario. We generate ${\mathbf{f}}_{g,i}$ as we did in the first study and ${\mathbf{u}}_{g,i}$ as i.i.d samples from $𝒩\left(0,{\mathbf{\Sigma }}_u\right)$, where ${\mathbf{\Sigma }}_u=\left({\sigma }_{u,ij}\right)$, with ${\sigma }_{u,ij}={0.1}^{|i-j|}0.03$ if $|i-j|\le 2$, and ${\sigma }_{u,ij}=0$ otherwise. Additional simulation studies on ${\left\{{\mathbf{f}}_{g,i}\right\}}_{i\le n_g}$ and ${\left\{{\mathbf{u}}_{g,i}\right\}}_{i\le n_g}$ generated from more general sub-Gaussian distributions for both settings can be found in Section S3.3 in the Supplementary Material. For ${\mathbf{\Lambda }}_g$, we set ${\mathbf{\Lambda }}_g={\tilde{\mathbf{Q}}}_g\mathrm{*}{\mathbf{s}}_g$, where ${\mathbf{s}}_g$ is as in the first study, and ${\tilde{\mathbf{Q}}}_g$ is a random $K_g\times p$ orthonormal matrix. Then, we use these elements to generate ${\mathit{X}}_g$ according to (2.3) and normalize it to obtain the design matrix ${\tilde{\mathit{X}}}_g$. Given ${\tilde{\mathit{X}}}_g$, for any $g\in \left[G\right]$, we generate ${\mathit{Y}}_g$ from (4.20) by setting ${\mu }_g=g$ for $g\in \left[G\right]$, generating $\mathit{ϵ}$ as i.i.d. samples from $N\left(\mathrm{0,4}\right)$, and choosing two kinds of ${\mathit{\beta }}_g^{\mathrm{*}}$. For sparse ${\mathit{\beta }}_g^{\mathrm{*}}$, we set ${\mathit{\beta }}_1^{\mathrm{*}}=\left(10h,10h,-10h,{\mathbf{10}}_5,{\mathbf{0}}_{187},{\mathbf{10}}_5\right),{\mathit{\beta }}_2^{\mathrm{*}}=\left(10h,-10h,10h,{\mathbf{10}}_5,{\mathbf{0}}_{187},{\mathbf{10}}_5\right)$, and ${\mathit{\beta }}_3^{\mathrm{*}}=\left(-10h,10h,10h,{\mathbf{10}}_5,{\mathbf{0}}_{187},{\mathbf{10}}_5\right)$. For dense ${\mathit{\beta }}_g^{\mathrm{*}}$, we set ${\mathit{\beta }}_1^{\mathrm{*}}=\left(10h,10h,-10h,{\mathbf{1}}_{80},{\mathbf{0}}_{37},{\mathbf{1}}_{80}\right),{\mathit{\beta }}_2^{\mathrm{*}}=\left(10h,-10h,10h,{\mathbf{1}}_{80}\right.,\left.{\mathbf{0}}_{37},{\mathbf{1}}_{80}\right)$, and ${\mathit{\beta }}_3^{\mathrm{*}}=\left(-10h,10h,10h,{\mathbf{1}}_{80},{\mathbf{0}}_{37},{\mathbf{1}}_{80}\right)$.

Under this model generation scheme, Figure 2 shows the MSE curves of the four methods, which are computed the same way as in the first study. For sparse ${\mathit{\beta }}_g^{\mathrm{*}}$, when $h$ is small, the differences between the group-specific, the Factor-0, and our method are marginal, which agrees with what we proved in Corollary 4. When $h$ gets larger, the group difference dominates. In this case, the group-specific model gives the best prediction, although our model is not far off. Compared with these two models, the global and the Factor-0 models are much worse because they fail to recognize the group difference. For a dense ${\mathit{\beta }}^{\mathrm{*}}$, when $h$ is small, all other models have similar performance, except for the global model. As $h$ gets larger, our model becomes slightly worse than the group-specific model for the same reason discussed in the sparse case. However, the performance of the Factor-0 model deteriorates much faster. In conclusion, this study shows that our method’s performance is still acceptable, even when the underlying models in the various groups are different. The estimation errors on ${\mathit{\beta }}_g^{\mathrm{*}}$ are reported in Table S4 in the Supplementary Material.

Figure 2.

The MSE curves given by the four models. The left panel represents the results for sparse ${\mathit{\beta }}_g^{\mathrm{*}}$, and the right panel represents the results for dense ${\mathit{\beta }}_g^{\mathrm{*}}$.

6. Application to ADNI Data Analysis

AD is an irreversible neurodegenerative disease that results in a loss of mental functions caused by a deterioration of the brain. It is the most common cause of dementia among people over the age of 65, affecting an estimated 5.5 million Americans, yet no prevention methods or cures have been discovered. The ADNI was started in 2004 with the goal of tracking the progression of the disease using biomarkers, and using clinical measures to assess the brain’s function over the course of the disease states. In this section, we apply our method to the ADNI data. We are interested in predicting the ADAS-Cog scores using structural magnetic resonance imaging (MRI) scans. All subjects in our analysis are from the ADNI2 phase of the study. In total, there are 697 subjects in our analysis and five groups: NC, SMC, eMCI, lMCI, and AD, ordered by disease severity. The MRI images were preprocessed using anterior commissure-posterior commissure correction, intensity inhomogeneity correction, skull stripping, cerebellum removal based on registration with atlas, spatial segmentation, and registration. After registration, we obtain MRI data with 93 regions of interest (ROIs). For each of the 93 ROIs, we compute the volume of gray matter as a feature. As a result, for each subject, we finally obtain 93 MRI features. Our goal is to predict the ADAS-Cog scores using the 93 MRI features, together with the group information.

We randomly partition the whole data set into two parts: 75% for training the model, and the rest for testing the performance. We repeat the random split 100 times. The testing MSEs and the corresponding standard errors are reported in Table 1 (overall performance) and Table 2 (groupwise performance). We compare four models: the global model (2.1), the group-specific model (2.2), the Factor-0 model (2.5), and our proposed model, as shown in (2.4). For each model, we use three penalty functions, the ${\ell }_2$ penalty (Ridge), the ${\ell }_1$ penalty (Lasso), and the Elastic Net (EN) penalty with the bridging parameter 0.5.

Table 1.

Overall MSEs for the four models.

Penalty	Global	Group-specific	Factor-0	Proposed
Ridge	27.52 (0.33)	15.70 (0.19)	15.17 (0.18)	15.04 (0.18)
EN	28.23 (o.33)	16.26 (0.2l)	15.47 (0.18)	15.40 (0.18)
Lasso	28.27 (0.34)	16.39 (0.23)	15.49 (0.19)	15.45 (0.18)

Table 2.

Groupwise MSEs for the four models.

Group	Global	Group-specific	Factor-0	Proposed
Penalty = Ridge
NC	16.66 (0.38)	6.24 (0.09)	6.50 (0.10)	6.19 (0.10)
SMC	14.52 (0.31)	6.68 (0.15)	6.43 (0.15)	6.54 (0.15)
eMCI	18.37 (0.41)	10.26 (0.19)	9.84 (0.19)	9.82 (0.19)
lMCI	19.17 (0.38)	16.75 (0.32)	15.61 (0.30)	15.92 (0.32)
AD	73.55 (0.38)	41.25 (0.32)	40.00 (0.30)	39.28 (0.32)
Penalty = Elastic Net
NC	16.79 (0.38)	6.45 (0.09)	6.40 (0.11)	6.37 (0.09)
SMC	15.46 (0.38)	7.12 (0.09)	6.78 (0.11)	6.96 (0.09)
eMCI	18.65 (0.38)	10.59 (0.09)	10.13 (0.11)	10.22 (0.09)
lMCI	20.26 (0.38)	18.32 (0.09)	16.14 (0.11)	16.43 (0.09)
AD	75.00 (0.38)	41.49 (0.09)	40.54 (0.11)	39.64 ( 0.09)
Penalty = Lasso
NC	16.69 (0.38)	6.49 (0.09)	6.41 (0.11)	6.37 (0.09)
SMC	15.57 (0.38)	7.16 (0.09)	6.84 (0.11)	7.05 (0.09)
eMCI	18.44 (0.38)	10.73 (0.09)	10.17 (0.11)	10.26 (0.09)
lMCI	20.36 (0.38)	18.53 (0.09)	16.21 (0.11)	16.50 (0.09)
AD	75.40 (0.38)	41.73 (0.09)	40.47 (0.11)	39.68 (0.09)

As shown in Tables 1 and 2, our proposed models achieve promising performance in most cases. The global model performs worst, because it does not use the label information at all. The group-specific model does not perform as well as our proposed models, because it does not borrow information across different groups. Note that the Factor-0 model achieves great improvement over the global model, which demonstrates that the difference on group means is the main source of the heterogeneous effect on the clinical scores across the five groups. It is seen in Table 2 that our model achieves the greatest improvement on the AD patients over the other models, which indicates that the effects of the heterogeneous factors identified in the AD group are much stronger than those in other groups. This appears to be reasonable, because the brain structure of AD patients is significantly more impaired.

Our model has good interpretations. In this real data set, we can interpret variations due to identified factors as disease-specific variations, and the variation due to the homogeneous signals as the disease-shared variation among all groups. Figure 3 gives heatmaps of ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{x,g}=\left(1/n_g\right){\mathbf{X}}_g^{\prime }{\mathbf{X}}_g$ (the top row), where ${\mathbf{\Sigma }}_{x,g}=\mathrm{c}\mathrm{o}\mathrm{v}\left({\mathbf{x}}_{g,i}\right)$, and ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{u,g}$ (the bottom row), which is obtained by applying an adaptive soft threshold to ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{x,g}-{\stackrel{\mathbf{ˆ}}{\mathbf{\Lambda }}}_g^{\prime }{\stackrel{\mathbf{ˆ}}{\mathbf{\Lambda }}}_g$. The left, middle, and right columns of Figure 3 are for the NC, eMCI, and AD groups, respectively. From Figure 3, we can see that the bottom row looks more homogeneous than the top row. We further represent brain connections using precision matrices estimated from Gaussian graphical models (Cai, Liu and Luo (2011)). See Section S4 in the Supplementary Material.

Figure 3.

Heatmaps of ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{x,g}$ and ${\stackrel{\mathbf{ˆ}}{\mathbf{\Sigma }}}_{u,g}$ in NC, eMCI and AD groups.

7. Conclusion

We have proposed a factor regression model for heterogeneous data with sub-populations. Our proposed model decomposes the predictors into heterogeneous components driven by latent factors and homogeneous components. We assume the group-specific latent factors explain the main heterogeneous variations and, consequently, their associated coefficients can differ by groups. The homogeneous components share the same covariance matrix and, as a result, they share the same regression coefficients. Because the factors are unobserved, we first estimate them using a standard PCA procedure. We use an OLS to directly estimate the group-specific coefficients. For the homogeneous regression coefficients, we propose a flexible penalized least squares solution. For model prediction, we also propose a data-driven procedure to estimate the factors for the testing data. Theoretical studies on the estimation and prediction consistency under ${\ell }_2$ and ${\ell }_1$ penalties are established. We show that our proposed model is robust under the group-specific model. Extensive simulation studies further demonstrate the competitive performance of our proposed model over the global model and the group-specific model, and our proposed model achieves a good balance between the two. Finally, we apply the proposed method to an ADNI data set for clinical score prediction, and demonstrate that our model has good prediction power and meaningful interpretation. One interesting future direction is to extend the method to include other outcomes, such as categorical or count data.