Embracing the Blessing of Dimensionality in Factor Models

Abstract

Factor modeling is an essential tool for exploring intrinsic dependence structures among high-dimensional random variables. Much progress has been made for estimating the covariance matrix from a high-dimensional factor model. However, the blessing of dimensionality has not yet been fully embraced in the literature: much of the available data are often ignored in constructing covariance matrix estimates. If our goal is to accurately estimate a covariance matrix of a set of targeted variables, shall we employ additional data, which are beyond the variables of interest, in the estimation? In this article, we provide sufficient conditions for an affirmative answer, and further quantify its gain in terms of Fisher information and convergence rate. In fact, even an oracle-like result (as if all the factors were known) can be achieved when a sufficiently large number of variables is used. The idea of using data as much as possible brings computational challenges. A divide-and-conquer algorithm is thus proposed to alleviate the computational burden, and also shown not to sacrifice any statistical accuracy in comparison with a pooled analysis. Simulation studies further confirm our advocacy for the use of full data, and demonstrate the effectiveness of the above algorithm. Our proposal is applied to a microarray data example that shows empirical benefits of using more data. Supplementary materials for this article are available online.

1. Introduction

With the advance of modern information technology, it is now possible to track millions of variables or subjects simultaneously. To discover the relationship among them, the estimation of a high-dimensional covariance matrix Σ has recently received a great deal of attention in the literature. Researchers proposed various regularization methods to obtain consistent estimators of Σ (Bickel and Levina 2008; Rothman et al. 2008; Lam and Fan 2009; Cai, Zhang, and Zhou 2010; Cai and Liu 2011). A key assumption for these regularization methods is that Σ is sparse, that is, many elements of Σ are small or exactly zero.

Different from such a sparsity condition, factor analysis assumes that the intrinsic dependence is mainly driven by some common latent factors (Johnson and Wichern 1992). For example, in modeling stock returns, Fama and French (1993) proposed the well-known Fama–French three-factor model. In the factor model, Σ has spiked eigenvalues and dense entries. In the high-dimensional setting, there are many recent studies on the estimation of the covariance matrix based on the factor model (Fan, Fan, and Lv 2008; Fan, Liao, and Mincheva 2011, 2013; Bai and Li 2012; Bai and Liao 2013), where the number of variables can be much larger than the number of observations.

The interest of this article is on the estimation of the covariance matrix for a certain set of variables using auxiliary data information. In the literature, we use only the data information on the variables of interest. In the data-rich environment today, substantially more amount of data information is indeed available, but is often ignored in statistical analysis. For example, we might be interested in understanding the covariance matrix of 50 stocks in a portfolio, yet the available data information is a time series of thousands of stocks. Similarly, an oncologist may wish to study the dependence or network structures among 100 genes that are significantly associated with a certain cancer, yet she has expression data for over 20,000 genes from the whole genome. Can we benefit from using much more rich auxiliary data?

The answer to the above question is affirmative when a factor model is imposed. Since the whole system is driven by a few common factors, these common factors can be inferred more accurately from a much larger set of data information (Fan, Liao, and Mincheva 2013), which is indeed a “blessing of dimensionality.” A major contribution of this article is to characterize how much the estimation of the covariance matrix of interest and also common factors can be improved by auxiliary data information (and under what conditions).

Consider the following factor model for all p observable data y_t = (y_1t, …, y_pt)′ ∈ ℝ^p at time t:

$$\begin{array}{cc}{\mathbf{y}}_t=\mathbf{B}{\mathbf{f}}_t+{\mathbf{u}}_t,& t=1,\dots ,T,\end{array}$$ (1)

where f_t ∈ ℝ^K is a K-dimensional vector of common factors, $\mathbf{B}={({\mathbf{b}}_1^{\prime },\dots ,{\mathbf{b}}_p^{\prime })}^{\prime }\in {\mathbb{R}}^{p\times K}$ is a factor loading matrix with b_i ∈ ℝ^K being the factor loading of the ith variable on the latent factor f_t, and u_t is an idiosyncratic error vector. In the above model, y_t is the only observable variable, while B is a matrix of unknown parameters, and (f_t, u_t) are latent random variables. Without loss of generality, we assume E(f_t) = E(u_t) = 0 and f_t and u_t are uncorrelated. Then, the model implied covariance structure is

$$\mathbf{\Sigma }=\mathbf{B}cov({\mathbf{f}}_t){\mathbf{B}}^{\prime }+{\mathbf{\Sigma }}_u,$$

where $\mathbf{\Sigma }=\mathrm{E}({\mathbf{y}}_t{\mathbf{y}}_t^{\prime })$ and ${\mathbf{\Sigma }}_u=\mathrm{E}({\mathbf{u}}_t{\mathbf{u}}_t^{\prime })$. Observe that B and f_t are not individually identifiable, since Bf_t = BHH′f_t for any orthogonal matrix H. To this end, an identifiability condition is imposed:

$$\begin{array}{ccc}cov({\mathbf{f}}_t)={\mathbf{I}}_K& \text{and}& {\mathbf{B}}^{\prime }{\mathbf{\Sigma }}_u^{-1}\mathbf{B}\text{is diagonal},\end{array}$$ (2)

which is a common assumption in the literature (Bai and Li 2012; Bai and Liao 2013).

Assume that we are only interested in a subset S among a total of p variables in model (1). We aim to obtain an efficient estimator of

$${\mathbf{\Sigma }}_S={\mathbf{B}}_S{\mathbf{B}}_S^{\prime }+{\mathbf{\Sigma }}_{u,S},$$

the covariance matrix of the s variables in S, where B_S is the sub-matrix of B with row indices in S and Σ_u,S is the submatrix of Σ_u with row and column indices in S. As mentioned above, the existing literature uses the following conventional method:

• Method 1: Use solely the s variables in the set S to estimate common factors f_t, the loading matrix B_S, the idiosyncratic matrix Σ_u,S, and the covariance matrix Σ_S.

This idea is apparently strongly influenced by the nonparametric estimation of the covariance matrix and ignores a large portion of the available data in the other p – s variables. An intuitively more efficient method is

• Method 2: Use all the p variables to obtain estimators of f_t, the loading matrix B, the idiosyncratic matrix Σ_u, and the entire covariance matrix Σ, and then restrict them to the variables of interest. This is the same as estimating f_t using all variables, and then estimating B_S and Σ_u,S based on the model (1) and the subset S with f_t being estimated (observed), and obtaining a plug-in estimator of Σ_S.

We will show that Method 2 is more efficient than Method 1 in the estimation of f_t and Σ_S as more auxiliary data information is incorporated. By treating common factor as an unknown parameter, we calculate its Fisher information that grows with more data being used in Method 2. In this case, a more efficient factor estimate can be obtained, for example, through weighted principal component (WPC) method (Bai and Liao 2013). The advantage of factor estimation is further carried over to the estimation of Σ_S by Method 2 in terms of its convergence rate. Moreover, if the number of total variables is sufficiently large, Method 2 is proven to perform as well as an “oracle method,” which observes all latent factors. This lends further support to our aforementioned claim of “blessing of dimensionality.” Such a best possible rate improvement is new to the existing literature, and counted as another contribution of this article. All these conclusions hold when the number of factors K is assumed to be fixed and known, while s, p, and T all tend to infinity.

The idea of using data as much as possible brings computational challenges. Fortunately, we observe that all the p variables are controlled by the same group of latent factors. Having said that, we can actually split p variables into smaller groups, and then use each group to estimate latent factors. The final factor estimate is obtained by averaging over these repeatedly estimated factors. Obviously, this divide-and-conquer algorithm can be implemented in a parallel computing environment, and thus produces factor estimators in a much more efficient way. On the other hand, our theory illustrates that this new method performs as well as the “pooled analysis,” where we run the method over the whole dataset. Simulation studies further demonstrate the boosted computational speed and satisfactory statistical performance.

The rest of the article is organized as follows. We compare the Fisher information of the factors by the two methods in Section 2. Section 3 describes the WPC method. As a main result, the convergence rates of Different estimators of Σ_S are further compared in Section 4 under various norms. Section 5 introduces the divide-and-conquer method for accelerating computation, while Section 6 presents all simulation results. Section 7 gives a microarray data example to illustrate our proposal. All technical proofs are delegated to the Appendix.

For any vector a, let a_S denote a sub-vector of a with indices in S. Denote ‖a‖ the Euclidean norm of a. For a symmetric matrix A ∈ ℝ^d×d, let A_I,j be the submatrix of A with row and column indices in I and J, respectively. We write A_S for A_S,S for simplicity. Let λ_j(A) be the jth largest eigenvalue of A. Denote ‖A‖ = max{|λ₁(A)|, |λ_d(A)|} the operator norm of A, ‖A‖_max = max_ij |a_ij| the max-norm of A, where a_ij is the (i, j)th entry of A, ${\Vert \mathbf{A}\Vert }_1={max}_i{\sum }_{j=1}^d|a_{ij}|$ the L₁ norm of A, ${\Vert \mathbf{A}\Vert }_F=\sqrt{\text{tr}({\mathbf{A}}^{\prime }\mathbf{A})}$ the Frobenius norm of A, and ‖A‖_M = d^−1/2‖M^−1/2AM^−1/2‖_F the relative norm of A to M, where the weight matrix M is assumed to be positive definite. For a non-square matrix C, let C_S be the submatrix of C with row indices in S.

2. Fisher Information of Common Factor

In this section, we treat the vector of common factors as a fixed unknown parameter, and compute its Fisher information matrices based on Method 1 and Method 2. In the computation, the loading matrix B is treated as deterministic in Proposition 2. In Proposition 3, the Fisher information is computed for each given B and then averaged over B by regarding it as a realization of a chance process, which bypasses the block diagonal assumption needed without taking average over B. In other sections, we adopt the convention regarding the factors as random and B as fixed. We start by calculating the Fisher information of θ_t := Bf_t, which serves as an intermediate step in obtaining that for f_t. For simplicity of notation, time t is suppressed in (y_t, f_t, u_t, θ_t) so that it becomes (y, f, u, θ) in this section.

Given a general density function of y, denoted as h(y; θ), the Fisher information of θ contained in full data is given by

$$I_p(\mathit{\theta })=\mathrm{E}\left[\right(\frac{\partial logh(\mathbf{y};\mathit{\theta })}{\partial \mathit{\theta }}\left){\left(\frac{\partial logh(\mathbf{y};\mathit{\theta })}{\partial \mathit{\theta }}\right)}^{\prime }\right].$$

When only data in S is used, the Fisher information of θ_S is given by

$$I_S({\mathit{\theta }}_S)=\mathrm{E}\left[\right(\frac{\partial log{h}_S({\mathbf{y}}_S;{\mathit{\theta }}_S)}{\partial {\mathit{\theta }}_S}\left){\left(\frac{\partial log{h}_S({\mathbf{y}}_S;{\mathit{\theta }}_S)}{\partial {\mathit{\theta }}_S}\right)}^{\prime }\right].$$

where h_S is the marginal density of y_S for the target set of variable S. Our first proposition shows that {I_p(θ)}_S, the submatrix of I_p(θ) restricted on S, dominates I_S(θ_S) under a mild condition.

Proposition 1. If h(y; θ) = h(y – θ) and the density function h(y – θ) satisfies the following regularity condition:

$${\nabla }_{{\mathbf{y}}_S}\int h({\mathbf{y}}_S-{\mathit{\theta }}_S,{\mathbf{y}}_{S^c}-{\mathit{\theta }}_{S^c})d{\mathbf{y}}_{S^c}=\int {\nabla }_{{\mathbf{y}}_S}h({\mathbf{y}}_S-{\mathit{\theta }}_S,{\mathbf{y}}_{S^c}-{\mathit{\theta }}_{S^c})d{\mathbf{y}}_{S^c},$$ (3)

then{I_p(θ)}_S ≥ I_S(θ_S) in the sense that {I_p(θ)}_S – I_S(θ_S) is positive semidefinite.

The regularity condition (3) is fairly mild, as illustrated in the following examples.

Example 1. In model 1, if u_S and u_S^c are independent, then (3) holds.

Example 2. If y follows an elliptical distribution that

$$h(\mathbf{y};\mathit{\theta })\propto g({(\mathbf{y}-\mathit{\theta })}^{\prime }{\mathbf{\Sigma }}^{-1}(\mathbf{y}-\mathit{\theta })),$$

where the mapping function g(t) : [0, ∞) → [0, ∞) satisfies that |g′(t)| ≤ cg(t) for some positive constant c, and E|y| < ∞, then (3) holds. Example 2 includes some commonly used multivariate distributions as its special cases, for example, the multivariate normal distribution and the multivariate t-distribution with degrees of freedom greater than 1. The proof is given in Appendix A.2.

We next compute the Fisher information of f based on the full dataset, denoted as I(f), and the partial dataset restricted on S, denoted as I_S(f). This can be done easily by noting that I(f) = B′I_p(θ)B. Indeed, the WPC estimators used in Methods 1 and 2 achieve such efficiency since their asymptotic variances are proven to be the inverse of I(f) and I_S(f), respectively; see Remark 1.

Proposition 2 shows that I(f) dominates I_S(f), if I_p(θ) is block-diagonal, that is, {I_p(θ)}_S,S^c = 0. Hence, common factors can be estimated more efficiently using additional data y_S^c. The above block-diagonal condition implies that the idiosyncratic error of additional variables cannot be confounded with that of the variables-of-interest. For example, if u is normal, then {I_p(θ)}_S,S^c = 0 indeed requires that u_S is independent of u_S^c.

Proposition 2. Under condition (3), if {I_p(θ)}_S,S^c = 0, I(f) ≥ I_S(f).

So far we treat B as being deterministic. Rather, Proposition 3 regards {b_i} as a realization of a chance process. Under this assumption, the expectation of I(f) over B is shown to always dominate that of I_S(f). In other words, we can claim that averaging over loading matrices, a larger dataset contains more information about the unknown factors.

Proposition 3. If ${\{{\mathbf{B}}_i\}}_{i=1}^p$ are iid random loadings with E(b_i) = 0 and (3) holds, then E[I(f)] ≥ E[I_S(f)], where the expectation is taken with respect to the distribution of B.

3. Efficient Estimation of Common Factor

In this section, we construct an efficient estimator of the common factors by showing that its asymptotic variance is exactly the inverse of its Fisher information. This together with the arguments in Section 2 enables us to draw a conclusion that using more data results in a more efficient factor estimator with a smaller asymptotic variance.

From a least-square perspective, when the loading matrix B is known, f_t can be estimated by the weighted least-squares: ${argmin}_{{\mathbf{f}}_t\in {\mathbb{R}}^K}{\sum }_{t=1}^T{({\mathbf{y}}_t-\mathbf{B}{\mathbf{f}}_t)}^{\prime }{\mathbf{\Sigma }}_u^{-1}({\mathbf{y}}_t-\mathbf{B}{\mathbf{f}}_t)$. In the high-dimensional setting (p ≫ T), we assume Σ_u is a sparse matrix and define its sparsity measurement as

$$\begin{array}{cc}{m}_p=\underset{i\le p}{max}\sum _{j\ne i}I({\sigma }_{u,ij}\ne 0),& \text{where}{\sigma }_{u,ij}\text{is the}(i,j)\text{th entry of}{\mathbf{\Sigma }}_u.\end{array}$$ (4)

In particular, we assume the following sparsity condition:

$$\begin{array}{ccc}{m}_p=o(min\{\frac{1}{p^{1/4}}\sqrt{\frac{T}{logp}},p^{1/4}\left\}\right)& \text{and}& \sum _{i=1}^p\sum _{j\ne i}I({\sigma }_{u,ij}\ne 0)=O(p).\end{array}$$ (5)

Now, we propose to solve the following constrained weighted least-square problem:

$$\begin{array}{c}(\widehat{\mathbf{B}},{\widehat{\mathbf{f}}}_1,\dots ,{\widehat{\mathbf{f}}}_T)=\underset{\mathbf{B},{\mathbf{f}}_t}{argmin}\sum _{t=1}^T{({\mathbf{y}}_t-\mathbf{B}{\mathbf{f}}_t)}^{\prime }{\stackrel{\sim }{\mathbf{\Sigma }}}_u^{-1}({\mathbf{y}}_t-\mathbf{B}{\mathbf{f}}_t),\\ \text{subject to}\frac{1}{T}\sum _{t=1}^T{\mathbf{f}}_t{\mathbf{f}}_t^{\prime }={\mathbf{I}}_K;{\mathbf{B}}^{\prime }{\stackrel{\sim }{\mathbf{\Sigma }}}_u^{-1}\mathbf{B}\text{is diagonal},\end{array}$$ (6)

where Σ̃_u is a regularized estimator of Σ_u to be discussed later. The above constraint is a sample analog of the identifiability condition (2). The involvement of the weight ${\stackrel{\sim }{\mathbf{\Sigma }}}_u^{-1}$ is to account for the heterogeneity among the data and leads to more efficient estimation of (B, f_t) (Choi 2012; Bai and Liao 2013).

Indeed, an initial estimator Σ̃_u of the idiosyncratic matrix Σ_u is needed for solving the constrained weighted least-square problem. We propose to obtain such an estimator by the following procedure, which is in the same spirit as the estimation of the idiosyncratic matrix in the POET method (Fan, Liao, and Mincheva 2013). Let ${\mathbf{S}}_y=T^{-1}{\sum }_{t=1}^T({\mathbf{y}}_t-\overline{\mathbf{y}}){({\mathbf{y}}_t-\overline{\mathbf{y}})}^{\prime }$ be the sample covariance of y and ${\{({\lambda }_i,{\zeta }_i)\}}_{i=1}^p$ be eigen-pairs of S_y with λ₁ ≥ λ₂ ≥ … ≥ λ_p. Denote $\mathbf{R}={\mathbf{S}}_y-{\sum }_{i=1}^K{\lambda }_i{\mathit{\zeta }}_i{\mathit{\zeta }}_i^{\prime }$. We estimate Σ_u by Σ̃_u, whose (i, j)th entry

$$\begin{array}{cc}{\widehat{\sigma }}_{u,ij}=\{\begin{array}{cc}{r}_{ij},& \text{for}i=j,\\ s_{ij}(r_{ij}),& \text{for}i\ne j,\end{array}& \text{where}\mathbf{R}=(r_{ij}),\end{array}$$

s_ij(r_ij) is a general entry-wise thresholding function (Antoniadis and Fan 2001) such that s_ij(z) = 0 if |z| ≤ τ_ij and |s_ij(z) – z| ≤ τ_ij for |z| > τ_ij. In our article, we choose hard-thresholding even though SCAD (Fan and Li 2001) and MCP (Zhang 2010) are also applicable. We specify the entry-wise thresholding level as

$$\begin{array}{ccc}{\tau }_{ij}(p)=C\sqrt{r_{ii}{r}_{jj}}\omega (p),& \text{where}& \omega (p)=\sqrt{\frac{logp}{T}}+\frac{1}{\sqrt{p}},\end{array}$$ (7)

and C is a constant chosen by cross-validation. The thresholding parameter C_ω(p) is applied to the correlation matrix. This is similar to the adaptive thresholding estimator for a general covariance matrix (Rothman, Levina, and Zhu 2009), where the entry-wise thresholding level depends on p.

With Σ̃_u being the thresholding estimator described above, the constrained weighted least-square problem (6) can be solved by the weighted principal component (WPC) method. The solution is given by

$$\begin{array}{ccc}\widehat{\mathbf{F}}={({\widehat{\mathbf{f}}}_1,\dots ,{\widehat{\mathbf{f}}}_T)}^{\prime }& \text{and}& {\widehat{\mathbf{B}}}^{\prime }=T^{-1}\mathbf{Y}\widehat{\mathbf{F}},\end{array}$$ (8)

where Y = (y₁, …, y_T) and the columns of F̂ are the eigenvectors corresponding to the largest K eigenvalues of the T × T matrix $\sqrt{T}{\mathbf{Y}}^{\prime }{\stackrel{\sim }{\mathbf{\Sigma }}}_u^{-1}\mathbf{y}$ (Bai and Liao 2013).

In the following, we give a result showing that the WPC estimator is asymptotically efficient. Indeed, Bai and Liao (2013) derive the asymptotic normality of f̂_t under the following conditions:

1. All eigenvalues of B′B/p are bounded away from zero and infinity as p → ∞;

2. There exists a K × K diagonal matrix Q such that ${\mathbf{B}}^{\prime }{\mathbf{\Sigma }}_u^{-1}\mathbf{B}/p\to \mathbf{Q}$. In addition, the diagonal elements of Q are distinct and bounded away from infinity.

3. For each fixed t ≤ T, ${({\mathbf{B}}^{\prime }{\mathbf{\Sigma }}_u^{-1}\mathbf{B})}^{-1/2}{\mathbf{B}}^{\prime }{\mathbf{\Sigma }}_u^{-1}{\mathbf{u}}_t\stackrel{d}{\to }N(\mathbf{0},{\mathbf{I}}_K)$, as p → ∞, together with the sparsity assumption (5), and some additional regularity conditions given in Section A.1. When √plog p = o(T), it is shown that

   $$\sqrt{p}({\widehat{\mathbf{f}}}_t-\mathbf{H}{\mathbf{f}}_t)\stackrel{D}{\to }N(\mathbf{0},{\mathbf{Q}}^{-1}),$$ (9)

   where H is a specific rotation matrix given by

   $$\mathbf{H}={\widehat{\mathbf{V}}}^{-1}{\widehat{\mathbf{F}}}^{\prime }F{B}^{\prime }{\stackrel{\sim }{\mathbf{\Sigma }}}_u^{-1}\mathbf{B}/T,$$ (10)

   and V̂ is a K × K diagonal matrix of the largest K eigenvalues of ${\mathbf{Y}}^{\prime }{\stackrel{\sim }{\mathbf{\Sigma }}}_u^{-1}\mathbf{y}/T$. The rotation matrix H is introduced here so that Hf_t is an identifiable quantity from the data. See more discussion about the identifiability in Remark 2.

Condition (i) is a “pervasive condition” requiring that the common factors affect a nonnegligible fraction of subjects. This is a common assumption for the principal components based methods (Fan, Liao, and Mincheva 2011; Bai and Liao 2013). In condition (ii), ${\mathbf{B}}^{\prime }{\mathbf{\Sigma }}_u^{-1}\mathbf{B}$ is indeed the Fisher information (under Gaussian errors) contained in p variables, while the limit Q can be viewed as an average information for each variable. Hence, the asymptotic normality in (9) shows that f̂_t is efficient as its asymptotic variance attains the inverse of the (averaged) Fisher information.

Remark 1. The results in Section 2 together with (9) imply that Method 2 is in general better than Method 1 in the estimation of common factors. To explain why, we consider two Different cases here. When p is an order of magnitude larger than s, where s is the number of variables of interest. Method 2 produces a better estimator of factors with a faster convergence rate. Even when p and s diverge at the same speed, the factor estimator based on Method 2 is shown to possess a smaller asymptotic variance, as long as Σ_u,S,S^c = 0. Recall that ${\mathbf{B}}^{\prime }{\mathbf{\Sigma }}_u^{-1}\mathbf{B}=I(\mathbf{f})$ and ${\mathbf{B}}_S^{\prime }{\mathbf{\Sigma }}_{u,S}^{-1}{\mathbf{B}}_S=I_S(\mathbf{f})$ under Gaussian errors, and they also correspond to the inverse of the asymptotic variance given by Methods 1 and 2, respectively. Then, Proposition 2 implies that Method 2 has a smaller asymptotic variance, if Σ_u,S,S^c = 0. Alternatively, if B is treated as being random, Proposition 3 immediately implies that $E({\mathbf{B}}_S^{\prime }{\mathbf{\Sigma }}_{u,S}^{-1}{\mathbf{B}}_S)\ge E({\mathbf{B}}^{\prime }{\mathbf{\Sigma }}^{-1}\mathbf{B})$. Therefore, even without the block diagonal assumption, Method 2 produces a more efficient factor estimate on average.

4. Covariance Matrix Estimation

One primary goal in this article is to obtain an accurate estimator of the covariance matrix ${\mathbf{\Sigma }}_S=\mathrm{E}({\mathbf{y}}_S{\mathbf{y}}_S^{\prime })$ for the variables-of-interest. In this section, we compare three Different estimation methods, namely, Methods 1, 2, and Oracle Method, in terms of their rates of convergence (under various norms). Obviously, these rates depend on how accurately the realized factors are estimated as demonstrated later.

Below we describe these three methods in full details.

• Method 1:

  1. Use solely the data in the subset S to obtain estimators of the realized factors F̂⁽¹⁾ and the loading matrix B̂₁ = T⁻¹Y_SF̂⁽¹⁾ based on (8);
  2. Let ${({\widehat{\mathbf{f}}}_t^{(1)})}^{\prime }$ be the tth row of F̂⁽¹⁾, ${({\widehat{\mathbf{b}}}_i^{(1)})}^{\prime }$ be the ith row of B̂₁, ${\widehat{u}}_{it}=y_{it}-{({\widehat{\mathbf{b}}}_i^{(1)})}^{\prime }{\widehat{\mathbf{f}}}_t^{(1)}$, and ${\widehat{\sigma }}_{ij}=\frac{1}{T}{\sum }_{t=1}^T{\widehat{u}}_{it}{\widehat{u}}_{jt}$. The (i, j)th entry of the idiosyncratic matrix estimator ${\widehat{\mathbf{\Sigma }}}_{u,S}^{(1)}$ of Σ_u,S is given by thresholding σ̂_ij at the level of $C{\widehat{\mathit{\theta }}}_{ij}^{1/2}\omega (s)$, where ω(s) is defined in (7) and ${\widehat{\mathit{\theta }}}_{ij}=\frac{1}{T}{\sum }_{t=1}^T{({\widehat{u}}_{it}{\widehat{u}}_{jt}-{\widehat{\sigma }}_{ij})}^2$;
  3. The final estimator is given by ${\widehat{\mathbf{\Sigma }}}_S^{(1)}={\widehat{\mathbf{B}}}_1{\widehat{\mathbf{B}}}_1^{\prime }+{\widehat{\mathbf{\Sigma }}}_{u,S}^{(1)}$.
• Method 2:

  1. Use all p variables to obtain the estimate F̂⁽²⁾ as given in (8) for the realized factors and then estimate the loading B_S by B̂₂ = T⁻¹Y_SF̂⁽²⁾;
  2. Follow the same procedure as in Method 1 to obtain the estimator ${\widehat{\mathbf{\Sigma }}}_{u,S}^{(2)}$ but based on F̂⁽²⁾ and B̂₂;
  3. The final estimator is given by ${\widehat{\mathbf{\Sigma }}}_S^{(2)}={\widehat{\mathbf{B}}}_2{\widehat{\mathbf{B}}}_2^{\prime }+{\widehat{\mathbf{\Sigma }}}_{u,S}^{(2)}$.
• Oracle Method:

  1. Estimate the loading by B̂_o = T⁻¹Y_SF, where F = (f₁, …, f_T)′ are the true factors.
  2. The idiosyncratic matrix estimator ${\widehat{\mathbf{\Sigma }}}_{u,S}^o$ is given by the same procedure as in Method 1, with ${\widehat{\mathbf{b}}}_i^{(1)}$ and ${\widehat{\mathbf{f}}}_t^{(1)}$ being replaced by ${\widehat{\mathbf{b}}}_i^o$ and f_t, respectively.
  3. The final estimator is given by ${\widehat{\mathbf{\Sigma }}}_S^o={\widehat{\mathbf{B}}}_o{\widehat{\mathbf{B}}}_o^{\prime }+{\widehat{\mathbf{\Sigma }}}_{u,S}^o$.

Theorem 1 depicts the estimation accuracy of Σ_S by the above three methods with respect to the following measurements:

$$\begin{array}{ccc}{\Vert {\widehat{\mathbf{\Sigma }}}_S-{\mathbf{\Sigma }}_S\Vert }_{{\mathbf{\Sigma }}_S},& {\Vert {\widehat{\mathbf{\Sigma }}}_S-{\mathbf{\Sigma }}_S\Vert }_{max},& \Vert {\widehat{\mathbf{\Sigma }}}_S^{-1}-{\mathbf{\Sigma }}_S^{-1}\Vert ,\end{array}$$

where ${\Vert {\widehat{\mathbf{\Sigma }}}_S-{\mathbf{\Sigma }}_S\Vert }_{{\mathbf{\Sigma }}_S}=p^{-1/2}{\Vert {\mathbf{\Sigma }}_S^{-1/2}{\widehat{\mathbf{\Sigma }}}_S{\mathbf{\Sigma }}_S^{-1/2}-{\mathbf{I}}_S\Vert }_F$ is a norm of the relative errors. Note that the results of Fan, Liao, and Mincheva (2013) cannot be directly used here since we employ the weighted principal component analysis to estimate the unobserved factors. This is expected to be more accurate than the ordinary principal component analysis, as shown in Bai and Liao (2013). Indeed, the technical proofs for our results are technically more involved than those by Fan, Liao, and Mincheva (2013).

We assume that s is much less than p, that is, s = o(p), but both tend to infinity. Under the pervasive condition (i), ‖Σ_S‖ ≥ cs and therefore diverges. For this reason, we consider the relative norm ‖Σ̂_S – Σ_S‖_Σs, instead of ‖Σ̂_S – Σ_S‖, and the operator norm $\Vert {\widehat{\mathbf{\Sigma }}}_S^{-1}-{\mathbf{\Sigma }}_S^{-1}\Vert$ for estimating the inverse. In addition, we consider another element-wise max norm ‖Σ̂_S – Σ_S‖_max. We show that if p is large with respect to s and T, Method 2 performs as well as the Oracle Method, both of which outperform Method 1. As a consequence, even if we are only interested in the covariance matrix of a small subset of variables, we should use all the data to estimate the common factors, which ultimately improves the estimation of Σ_S. In particular, we are able to specify an explicit regime of (s, p) under which the improvements are substantial. However, when s ≍ p, that is, they are in the same order, using more data does not show as dramatic improvements for estimating Σ_S. This is expected and will be clearly seen in the simulation section.

Before stating Theorem 1, we need a few preliminary results: Lemmas 1–3. Specifically, Lemma 1 presents the uniform convergence rates of the factor estimates by Methods 1 and 2. Based on that, Lemmas 2 and 3 further derive the estimation accuracy of factor loadings and idiosyncratic matrix by the three methods, respectively. These results together lead to the estimation error rates of Σ_S in Theorem 1 w.r.t. three measures defined above. Additional Lemmas supporting the proof are given in the Appendix. Again, these kinds of results cannot be obtained directly from Fan, Liao, and Mincheva (2013) due to our use of WPC.

Lemma 1. Suppose that conditions (i), (ii), the sparsity condition (5), and additional regularity conditions (iv)–(vii) in Section A.1 hold for both s and p. If √plog p = o(T) and T = o(s²), then we have

$$\begin{array}{cc}\underset{t\le T}{max}\Vert {\widehat{\mathbf{f}}}_t^{(1)}-{\mathbf{H}}_1{\mathbf{f}}_t\Vert =O_P(\frac{1}{\sqrt{T}}+\frac{T^{1/4}}{\sqrt{s}})& \text{and}\\ \underset{t\le T}{max}\Vert {\widehat{\mathbf{f}}}_t^{(2)}-{\mathbf{H}}_2{\mathbf{f}}_t\Vert =O_P(\frac{1}{\sqrt{T}}+\frac{T^{1/4}}{\sqrt{p}}),\end{array}$$

where ${\mathbf{H}}_1={\widehat{\mathbf{V}}}_1^{-1}{\widehat{\mathbf{F}}}^{{(1)}^{\prime }}\mathbf{F}{\mathbf{B}}_S^{\prime }{\stackrel{\sim }{\mathbf{\Sigma }}}_{u,S}^{-1}{\mathbf{B}}_S/T,{\mathbf{H}}_2={\widehat{\mathbf{V}}}_2^{-1}{\widehat{\mathbf{F}}}^{{(2)}^{\prime }}\mathbf{F}{\mathbf{B}}^{\prime }{\stackrel{\sim }{\mathbf{\Sigma }}}_{u,S}^{-1}\mathbf{B}/T$, V̂₁ is the diagonal matrix of the largest K eigenvalues of ${\mathbf{Y}}_S^{\prime }{\stackrel{\sim }{\mathbf{\Sigma }}}_{u,S}^{-1}{\mathbf{Y}}_S/T$ and V̂₂ is the diagonal matrix of the largest K eigenvalues of ${\mathbf{Y}}^{\prime }{\stackrel{\sim }{\mathbf{\Sigma }}}_u^{-1}\mathbf{y}/T$.

Remark 2. H₁ and H₂ correspond to the rotation matrix H defined in (10) using Methods 1 and 2, respectively. Recall that F = (f₁, …, f_T)′, then $\mathbf{H}{\mathbf{f}}_t=T^{-1}{\widehat{\mathbf{V}}}^{-1}\widehat{\mathbf{F}}{(\mathbf{B}{\mathbf{f}}_1,\dots ,\mathbf{B}{\mathbf{f}}_T)}^{\prime }{\stackrel{\sim }{\mathbf{\Sigma }}}_u^{-1}\mathbf{B}{\mathbf{f}}_t$. Note that Hf_t only depends on quantities V⁻¹F̂, ${\stackrel{\sim }{\mathbf{\Sigma }}}_u^{-1}$ and the identifiable component ${\{\mathbf{B}{\mathbf{f}}_t\}}_{t=1}^T$. Therefore, there is no identifiability issue regarding Hf_t. In other words, even though f_t itself may not be identifiable, an identifiable rotation of f_t can be consistently estimated by f̂_t.

Lemma 1 implies that Method 2 produces a better factor estimate if

$$0.5<{\gamma }_s<1.5\le {\gamma }_p<2,$$

by representing s and p as s ≍ T^γ_s and p ≍ T^γ_p.

It is not surprising that the estimation accuracy of loading matrix also varies among these three methods as shown in Lemma 2.

Lemma 2. Under conditions of Lemma 1,

$$\begin{gathered} \underset{i\le s}{max}\Vert {\widehat{\mathbf{b}}}_i^{(1)}-{\mathbf{H}}_1{\mathbf{b}}_i\Vert =O_P(w_1), \\ \text{where}{w}_1:=\frac{1}{\sqrt{s}}+\sqrt{\frac{logs}{T}},\underset{i\le s}{max}\Vert {\widehat{\mathbf{b}}}_i^{(2)}-{\mathbf{H}}_2{\mathbf{b}}_i\Vert =O_P(w_2), \\ \text{where}{w}_2:=\frac{1}{\sqrt{p}}+\sqrt{\frac{logs}{T}},\underset{i\le s}{max}\Vert {\widehat{\mathbf{b}}}_i^{(2)}-{\mathbf{b}}_i\Vert =O_P(w_o), \\ \text{where}{w}_o:=\sqrt{\frac{logs}{T}}. \end{gathered}$$

Similarly, Lemma 2 indicates that Method 2 performs as well as the Oracle Method, both of which are better than Method 1, that is, w₂ = w_o < w₁, if

$$0.5<{\gamma }_s<1\le {\gamma }_p<2,$$

by representing s and p in the order of T as above. We remark that the extra terms 1/√s and 1/√p in w₁ and w₂ (in comparison with the oracle rate w_o) are due to the factor estimation. Another preliminary result regarding the estimation of the identifiable component ${\mathbf{b}}_i^{\prime }{\mathbf{f}}_t$ is given in Lemma A.1.

Similar insights can be delivered from Lemma 3 on the estimation of Σ_u,S.

Lemma 3. Under conditions of Lemma 1, it holds that

$$\begin{array}{c}\Vert {\widehat{\mathbf{\Sigma }}}_{u,S}^{(1)}-{\mathbf{\Sigma }}_{u,S}\Vert =O_P(m_s{w}_1)=\Vert {\left({\widehat{\mathbf{\Sigma }}}_{u,S}^{(1)}\right)}^{-1}-{\mathbf{\Sigma }}_{u,S}^{-1}\Vert ,\\ \Vert {\widehat{\mathbf{\Sigma }}}_{u,S}^{(2)}-{\mathbf{\Sigma }}_{u,S}\Vert =O_P(m_s{w}_2)=\Vert {\left({\widehat{\mathbf{\Sigma }}}_{u,S}^{(2)}\right)}^{-1}-{\mathbf{\Sigma }}_{u,S}^{-1}\Vert ,\\ \Vert {\widehat{\mathbf{\Sigma }}}_{u,S}^o-{\mathbf{\Sigma }}_{u,S}\Vert =O_P(m_s{w}_o)=\Vert {\left({\widehat{\mathbf{\Sigma }}}_{u,S}^o\right)}^{-1}-{\mathbf{\Sigma }}_{u,S}^{-1}\Vert ,\end{array}$$

where m_s is defined as in (4) with p being replaced by s.

Now, we are ready to state our main result on the estimation of Σ_S based on the above preliminary results. From Theorem 1, it is easily seen that the comparison of the estimation accuracy of Σ_S among three methods is solely determined by the relative magnitude of w_o, w₁, and w₂. Therefore, we should use additional variables to estimate the factors if p is much larger than s in the sense that T/log s = O(p) and s log s = o(T) (implying w₂ = w_o < w₁).

Theorem 1. Under conditions of Lemma 1, it holds that

1. For the relative norm, ${\Vert {\widehat{\mathbf{\Sigma }}}_S^{(1)}-{\mathbf{\Sigma }}_S\Vert }_{{\mathbf{\Sigma }}_S}=O_P(\sqrt{s}{w}_1^2+m_s{w}_1)$, ${\Vert {\widehat{\mathbf{\Sigma }}}_S^{(2)}-{\mathbf{\Sigma }}_S\Vert }_{{\mathbf{\Sigma }}_S}=O_P(\sqrt{s}{w}_2^2+m_s{w}_2)$, and ${\Vert {\widehat{\mathbf{\Sigma }}}_S^o-{\mathbf{\Sigma }}_S\Vert }_{{\mathbf{\Sigma }}_S}=O_P(\sqrt{s}{w}_0^2+m_s{w}_o)$.

2. For the max-norm, ${\Vert {\widehat{\mathbf{\Sigma }}}_S^{(1)}-{\mathbf{\Sigma }}_S\Vert }_{max}=O_P(w_1)$, ${\Vert {\widehat{\mathbf{\Sigma }}}_S^{(2)}-{\mathbf{\Sigma }}_S\Vert }_{max}=O_P(w_2)$, and ${\Vert {\widehat{\mathbf{\Sigma }}}_S^o-{\mathbf{\Sigma }}_S\Vert }_{max}=O_P(w_o)$.

3. For the operator norm of the inverse matrix, $\Vert {({\widehat{\mathbf{\Sigma }}}_S^{(1)})}^{-1}-{\mathbf{\Sigma }}_S^{-1}\Vert =O_P(m_s{w}_1)$, $\Vert {({\widehat{\mathbf{\Sigma }}}_S^{(2)})}^{-1}-{\mathbf{\Sigma }}_S^{-1}\Vert =O_P(m_s{w}_2)$ and $\Vert {({\widehat{\mathbf{\Sigma }}}_S^o)}^{-1}-{\mathbf{\Sigma }}_S^{-1}\Vert =O_P(m_s{w}_o)$.

Remark 3. So far, we assumed that the number of factors K is fixed and known. A data-driven choice of K has been extensively studied in the econometrics literature, for example, by Bai and Ng (2002), Kapetanios (2010). To estimate K, we can adopt the method by Bai and Ng (2002) and propose a consistent estimator of K (by allowing p, T → ∞) as follows:

$$\widehat{K}=\underset{0\le k\le N}{argmin}log\left\{\frac{1}{pT}{\Vert \mathbf{Y}-T^{-1}\mathbf{Y}{\widehat{\mathbf{F}}}_k{\widehat{\mathbf{F}}}_k^{\prime }\Vert }_F^2\right\}+kg(p,T),$$

where N is a predefined upper bound, F̂_k is a T × k matrix whose columns are √T times the eigenvectors corresponding to the largest k eigenvalues of Y′Y, and g(p, T) is a penalty function. Two examples suggested by Bai and Ng (2002) are

$$\begin{array}{cc}g(T,p)=\frac{p+T}{pT}log\left(\frac{pT}{p+T}\right)& \text{or}\\ g(T,p)=\frac{p+T}{pT}log(min\{p,T\}).\end{array}$$

Under our assumptions (i)–(x), all conditions required by theorem 2 of Bai and Ng (2002) hold. Hence, their theorem implies that P(K̂ = K) → 1. Then, conditioning on the event that {K̂ = K}, our Theorem 1 still holds by replacing K with K̂. Other effective methods for selecting the number of factors include the eigen ratio method by Lam and Yao (2012) and Ahn and Horenstein (2013).

Remark 4. When K grows with p and T, Fan, Liao, and Mincheva (2013) gave the explicit dependence of the convergence rates on K for their proposed POET estimator. By adopting their technique, we can obtain the following results:

1. ${\Vert {\widehat{\mathbf{\Sigma }}}_S^{(1)}-{\mathbf{\Sigma }}_S\Vert }_{{\mathbf{\Sigma }}_S}=O_P(K\sqrt{s}{w}_1^2+K^3{m}_s{w}_1)$, ${\Vert {\widehat{\mathbf{\Sigma }}}_S^{(2)}-{\mathbf{\Sigma }}_S\Vert }_{{\mathbf{\Sigma }}_S}=O_P(K\sqrt{s}{w}_2^2+K^3{m}_s{w}_2)$, ${\Vert {\widehat{\mathbf{\Sigma }}}_S^o-{\mathbf{\Sigma }}_S\Vert }_{{\mathbf{\Sigma }}_S}=O_P(K\sqrt{s}{w}_o^2+K^3{m}_s{w}_o)$;

2. ${\Vert {\widehat{\mathbf{\Sigma }}}_S^{(1)}-{\mathbf{\Sigma }}_S\Vert }_{max}=O_P(K^3{w}_1)$, ${\Vert {\widehat{\mathbf{\Sigma }}}_S^{(2)}-{\mathbf{\Sigma }}_S\Vert }_{max}=O_P(K^3{w}_2)$, ${\Vert {\widehat{\mathbf{\Sigma }}}_S^o-{\mathbf{\Sigma }}_S\Vert }_{max}=O_P(K^3{w}_o)$;

3. $\Vert {({\widehat{\mathbf{\Sigma }}}_S^{(1)})}^{-1}-{\mathbf{\Sigma }}_S^{-1}\Vert =O_P(K^3{m}_s{w}_1)$, $\Vert {({\widehat{\mathbf{\Sigma }}}_S^{(2)})}^{-1}-{\mathbf{\Sigma }}_S^{-1}\Vert =O_P(K^3{m}_s{w}_2)$, $\Vert {({\widehat{\mathbf{\Sigma }}}_S^o)}^{-1}-{\mathbf{\Sigma }}_S^{-1}\Vert =O_P(K^3{m}_s{w}_o)$.

   Again, the rate difference among three types of estimators only depends on w_o, w₁, and w₂. Therefore, the same conclusion (when p is much larger than s, using additional variables improves the estimation of Σ_S) can still be made even if K diverges. As long as K diverges in the rate that $K=o(min\{1/(\sqrt{s}{w}_1^2),1/{(m_s{w}_1)}^{1/3}\})$, $K=o(1/w_1^{1/3})$ or K = o(1/(m_sw₁)^1/3), the same blessing of dimensionality phenomena persist in terms of estimation consistency in relative norm, max norm, or operator norm of the inverse, respectively.

5. Divide-and-Conquer Computing Method

As discussed previously, we prefer using auxiliary data information as much as possible even we are only interested in the covariance matrix of some particular set of variables. But this can bring up heavy computational burden. This concern motivates a simple divide-and-conquer scheme that splits all p variables in Y. Without loss of generality, assume that p rows of matrix Y can be evenly divided into M groups with p/M variables in each group. The s variables of interest can possibly be assigned to Different groups.

Divide-and-Conquer Computation Scheme

1. In the mth group, obtain the initial estimator Σ̃_u,m by using the adaptive thresholding method as described in Section 3 based on the data in the mth group only.

2. Denote Y_m as the data vector corresponding to the variables in the mth group and let F̂_m = (f̂_m,1, …, f̂_m,T)′, where its columns are the eigenvectors corresponding to the largest K eigenvalues of the T × T matrix $\sqrt{T}{\mathbf{Y}}_m^{\prime }{\stackrel{\mathbf{\sim }}{\mathbf{\Sigma }}}_{u,m}^{-1}{\mathbf{Y}}_m$. The computation in the above two steps can be done in a parallel manner.

3. Average ${\{{\widehat{\mathbf{f}}}_{m,t}\}}_{m=1}^M$ to obtain a single estimator of f_t as

   $${\overline{\mathbf{f}}}_t=\frac{1}{M}\sum _{m=1}^M{\widehat{\mathbf{f}}}_{m,t}.$$

   The loading matrix estimate is given by B̄_S = T⁻¹Y_SF̄, where F̄ = (f̄₁, …, f̄_T)′.

4. The idiosyncratic matrix is estimated as follows. Let ${\overline{\mathbf{f}}}_t^{\prime }$ be the tth row of F̄ and ${\overline{\mathbf{b}}}_i^{\prime }$ be the ith row of B̄_S. Let ${\widehat{u}}_{it}=y_{it}-{\overline{\mathbf{b}}}_i^{\prime }{\overline{\mathbf{f}}}_t$, ${\widehat{\sigma }}_{ij}=T^{-1}{\sum }_{t=1}^T{\widehat{u}}_{it}{\widehat{u}}_{jt}$, and ${\widehat{\mathit{\theta }}}_{ij}=T^{-1}{\sum }_{t=1}^T{({\widehat{u}}_{it}{\widehat{u}}_{jt}-{\widehat{\sigma }}_{ij})}^2$. The (i, j)th entry of Σ_u,S is given by thresholding σ̂_ij at the level of $C{\widehat{\mathit{\theta }}}_{ij}^{1/2}\omega (s)$, where ω(s) is defined as in (7) with p replaced by s.

5. The final estimator of the covariance matrix is given by

   $${\overline{\mathbf{\Sigma }}}_S={\overline{\mathbf{B}}}_S{\overline{\mathbf{B}}}_S^{\prime }+{\overline{\mathbf{\Sigma }}}_{u,S}.$$

We show that, if M is fixed,

$$\begin{array}{c}{\Vert {\overline{\mathbf{\Sigma }}}_S-{\mathbf{\Sigma }}_S\Vert }_{{\mathbf{\Sigma }}_S}=O_P(\sqrt{s}{w}_2^2+m_s{w}_2),\\ {\Vert {\overline{\mathbf{\Sigma }}}_S-{\mathbf{\Sigma }}_S\Vert }_{max}=O_P(w_2),\\ \Vert {({\overline{\mathbf{\Sigma }}}_S)}^{-1}-{\mathbf{\Sigma }}_S^{-1}\Vert =O_P(m_s{w}_2).\end{array}$$

These rates match the rates of ${\widehat{\mathbf{\Sigma }}}_S^{(2)}$ attained by Method 2, where all p variables are pooled together for the analysis. The proof is given in Appendix A.3. The simulation results in Section 6 further demonstrate that without sacrificing the estimation accuracy, the divide-and-conquer method runs much faster than Method 2. Therefore, the divide-and-conquer method is practically useful when dealing with massive dataset.

The main computational cost of our method comes from taking the inverse of Σ̃_u. For our Method 2, where all p variables are pooled together for the analysis, the computational complexity of the inversion is O(p³). On the other hand, for the divide-and-conquer method, the corresponding estimator Σ̃_u,m in the mth group only needs a computational cost of O((p/M)³) to be inverted. Then, the total computation complexity is O(p³/M²). Hence, the computational speed can be boosted by M²-fold. Such a computational acceleration can also be observed from simulation study results in Figure 1(d). Other operations like the eigen-decomposition on the T × T matrix $\sqrt{T}{\mathbf{Y}}^{\prime }{\stackrel{\sim }{\mathbf{\Sigma }}}_u^{-1}\mathbf{Y}$ do not have dominating computational cost, as we assume that p is much larger than T. When M grows too fast, the divide-and-conquer method may lose estimation efficiency compared with the pooled analysis (Method 2). However, considering its boost of computation, the divide-and-conquer method is practically useful when dealing with massive dataset.

Figure 1.

Estimation error by four methods and their computational time: the dotted lines represent the means over 100 simulations and the segments represent the corresponding standard deviations.

6. Simulations

We use simulated examples to compare the statistical performances of Methods 1, 2, and the Oracle Method. We fix the number of factors K = 3 and repeat 100 simulations for each combination of (s, p, T). The loading b_i, the factor f_t and the idiosyncratic error u_t are generated as follows:

• ${\mathbf{\{}{\mathbf{b}}_i\}}_{i=1}^p$ are iid from N_K(0, 5I_K).

• ${\{{\mathbf{f}}_t\}}_{t=1}^T$ are iid from N_K(0, I_K).

• ${\{{\mathbf{u}}_t\}}_{t=1}^T$ are iid from N_p(0, 50I_p).

The observations ${\{{\mathbf{y}}_t\}}_{t=1}^T$ are generated from (1) using b_i, f_t, and u_t from the above. Tables 1–4 report the estimation errors of the factors, the loading matrices, and the covariance-of-interest Σ_S in terms of Different measurements.

Table 1.

Comparison of three methods when s is much smaller than p(T = 200).

(s, p) Method	(50, 1000)			(50, 2000)
	M1	M2	ORA	M1	M2	ORA
‖Σ̂S – ΣS‖ΣS	0.271(0.014)	0.205(0.013)	0.204(0.013)	0.270(0.014)	0.201(0.013)	0.200(0.013)
$\Vert {\widehat{\mathbf{\Sigma }}}_S^{-1}-{\mathbf{\Sigma }}_S^{-1}\Vert$	0.016(0.003)	0.009(0.002)	0.009(0.002)	0.017(0.003)	0.009(0.002)	0.009(0.002)
‖Σ̂S – ΣS‖max	18.828(3.072)	17.460(3.237)	17.457(3.261)	18.076(2.697)	16.631(2.949)	16.623(2.950)
maxt≤T ‖f̂t – Hft‖	1.811(0.195)	0.445(0.046)	NA	1.870(0.236)	0.331(0.025)	NA
maxi≤s ‖b̂i – Hbi‖	8.064(0.694)	4.100(0.330)	3.858(0.274)	8.150(0.682)	3.932(0.292)	3.805(0.297)
${max}_{i\le s,t\le T}\Vert {\widehat{\mathbf{b}}}_i^{\prime }{\widehat{\mathbf{f}}}_t-{\mathbf{b}}_i^{\prime }{\mathbf{f}}_t\Vert$	11.375(1.262)	5.519(0.813)	5.268(0.843)	11.466(1.353)	5.253(0.776)	5.113(0.739)

NOTE: M1, M2, and ORA stand for Method 1, 2, and Oracle method, respectively.

Table 4.

Comparison of three methods when s is comparative to p(T = 400).

(s, p) Method	(800, 1000)			(800, 2000)
	M1	M2	ORA	M1	M2	ORA
‖Σ̂S – ΣS‖ΣS	0.193(0.004)	0.192(0.004)	0.189(0.004)	0.192(0.004)	0.190(0.004)	0.188(0.004)
$\Vert {\widehat{\mathbf{\Sigma }}}_S^{-1}-{\mathbf{\Sigma }}_S^{-1}\Vert$	0.008(0.001)	0.008(0.001)	0.008(0.001)	0.008(0.001)	0.008(0.001)	0.008(0.001)
‖Σ̂S – ΣS‖max	17.062(2.603)	17.051(2.612)	17.041(2.621)	16.919(2.182)	16.891(2206)	16.888(2209)
maxt≤T ‖f̂t – Hft‖	0.467(0.038)	0.423(0.036)	NA	0.466(0.038)	0.304(0.026)	NA
maxi≤s ‖b̂i – Hbi‖	11.009(0298)	10.850(0.302)	10225(0205)	10.934(0274)	10530(0213)	10.189(0.172)
${max}_{i\le s,t\le T}\Vert {\widehat{\mathbf{b}}}_i^{\prime }{\widehat{\mathbf{f}}}_t-{\mathbf{b}}_i^{\prime }{\mathbf{f}}_t\Vert$	5.367(0577)	5276(0560)	4.880(0528)	5293(0.411)	5.024(0.461)	4.894(0.420)

NOTE: M1, M2, and ORA stand for Method 1, 2, and Oracle method, respectively.

We see from Tables 1 and 2 that when s = 50 and p = 1000, 2000, Method 1 performs much worse than Method 2, for both T = 200 and T = 400. However, when s increases to 800 with p being the same, Tables 3 and 4 show that the improvement of Method 2 over Method 1 is less profound. This is expected as the set of interest already contains sufficiently rich information to produce an accurate estimator for realized factors. In general, we note that Method 2 is the most advantageous in the settings where s is much smaller than p. In addition, from Tables 3 and 4, we can tell that Method 2 comes closer to the Oracle method as p grows. In practice, we also observe that the WPC factor estimator performs better than the unweighted PC estimator when u_t is heteroscedastic. Due to the space limit, we choose not to present the simulation results in this model.

Table 2.

Comparison of three methods when s is much smaller than p(T = 400).

(s, p) Method	(50, 1000)			(50, 2000)
	M1	M2	ORA	M1	M2	ORA
‖Σ̂S – ΣS‖ΣS	0.186(0.009)	0.132(0.007)	0.131(0.007)	0.186(0.009)	0.131(0.008)	0.130(0.008)
$\Vert {\widehat{\mathbf{\Sigma }}}_S^{-1}-{\mathbf{\Sigma }}_S^{-1}\Vert$	0.011(0.002)	0.004(0.001)	0.004(0.001)	0.011(0.002)	0.004(0.001)	0.004(0.001)
‖Σ̂S – ΣS‖max	14.054(1.945)	11.922(2245)	11.891(2262)	14.180(2.154)	11.901(2.603)	11.900(2.604)
maxt≤T ‖f̂t – Hft‖	1.839(0.193)	0.417(0.036)	NA	1.843(0.198)	0.305(0.026)	NA
maxi≤s ‖b̂i – Hbi‖	6.960(0.584)	2.830(0200)	2.692(0.198)	7.024(0.605)	2.761(0.188)	2.692(0.194)
${max}_{i\le s,t\le T}\Vert {\widehat{\mathbf{b}}}_i^{\prime }{\widehat{\mathbf{f}}}_t-{\mathbf{b}}_i^{\prime }{\mathbf{f}}_t\Vert$	11.871(1.540)	4.138(0510)	3.824(0.501)	11.457(1.569)	4.088(0.516)	3.889(0.542)

NOTE: M1, M2, and ORA stand for Method 1, 2, and Oracle method, respectively.

Table 3.

Comparison of three methods when sis comparative to p(T = 200).

(s, p) Method	(800, 1000)			(800, 2000)
	M1	M2	ORA	M1	M2	ORA
‖Σ̂S – ΣS‖ΣS	0.440(0.006)	0.439(0.006)	0.435(0.006)	0.439(0.006)	0.436(0.006)	0.435(0.006)
$\Vert {\widehat{\mathbf{\Sigma }}}_S^{-1}-{\mathbf{\Sigma }}_S^{-1}\Vert$	0.062(0.009)	0.062(0.009)	0.062(0.009)	0.061(0.009)	0.061(0.009)	0.062(0.012)
‖Σ̂S – ΣS‖max	24565(2.626)	24562(2.609)	24567(2599)	24511(2.883)	24543(2.847)	24536(2.851)
maxt≤T ‖f̂t – Hft‖	0.488(0.047)	0.447(0.040)	NA	0.478(0.049)	0.337(0.038)	NA
maxi≤s ‖b̂i – Hbi‖	15550(0.488)	15.370(0.462)	14.418(0271)	15595(0551)	15.041(0.357)	14.398(0243)
${max}_{i\le s,t\le T}\Vert {\widehat{\mathbf{b}}}_i^{\prime }{\widehat{\mathbf{f}}}_t-{\mathbf{b}}_i^{\prime }{\mathbf{f}}_t\Vert$	6.745(0.611)	6.680(0.635)	6.405(0.630)	6.904(0.734)	6.697(0.763)	6588(0.737)

NOTE: M1, M2, and ORA stand for Method 1,2, and Oracle method, respectively.

For further comparison with the divide-and-conquer method, we vary T from 50 to 500 and set (s, p, M) as s = ⎿T^0.6⏌, p = ⎿T^1.4⏌, and M = ⎿T^0.2⏌. Figure 1 shows the estimation errors of the four methods together with the corresponding computational time. Again, when p is large, Method 2 performs as well as the Oracle Method, both of which greatly outperform Method 1. However, its computation becomes much slower in this case. In contrast, the divide-and-conquer method is much faster, while maintaining comparable performance as Method 2. In the extreme case that p is around 6000 (T = 500), the divide-and-conquer method can boost the speed by nine-fold for Method 2.

7. Real Data Example

We use a real data example to illustrate how Different utilization of available variables can affect the inference of the variables of interest. Krug et al. (2012) carried out a gene profiling study among 40 Portuguese and Spanish adults to identify key genetic risk factors for ischemic stroke. Among them, 20 subjects were patients having ischemic stroke and the others were controls. Their gene profiles were obtained using the GeneChip Human Genome U133 Plus 2.0 microarray. The data were available at Gene Expression Omnibus with access name “GSE22255.”

To judge how effectively the gene expression can distinguish ischemic stroke and controls, we applied the Linear Discriminant Analysis (LDA) to this dataset. We randomly chose 10 subjects as the test set and the rest as the training set. We repeated the random splitting for 100 runs. In each run, we selected the set of expressed Differentially (DE) genes with a threshold of over 1.2-fold change and a Q-value ≤ 0.05, which is a commonly used quantity to define DE genes (Storey 2002). An LDA rule was then learned from the training set using the selected genes and further applied to the test set for classifying cases and controls. The LDA rule classifies a subject as a case if

$$\widehat{\mathit{\delta }}\prime {\widehat{\mathbf{\Sigma }}}^{-1}(\mathbf{x}-\overline{\mathit{\mu }})\ge 0,$$ (11)

where δ̂ = μ̂₁ – μ̂₀ ∈ ℝ^s is the sample mean difference between the two groups (case–control), s is the number of selected genes, Σ̂ ∈ ℝ^s×s is an estimator of the true covariance matrix Σ of the selected genes, and μ̄ = (μ̂₁ + μ̂₀)/2. μ̄, δ̂, and Σ̂ are obtainesd from the training set and x is the gene expression of subjects in the test set.

As s can be larger than the sample size, the traditional LDA where Σ̂ is the sample covariance is no longer applicable. An alternative method to estimate Σ is adopting the factor model. Factor modeling is widely used in the genomics literature to model the dependencies among genes (Kustra, Shioda, and Zhu 2006; Carvalho et al. 2012). Several factors, like the natural pathway structure (Ogata et al. 2000) can be the latent factors affecting the correlation among genes. A few spiked eigenvalues of the sample covariance in Figure 2 also suggest the existence of potential latent factors in this dataset. Again, there are two ways using the factor model. One way is to use Method 1, where all procedures are done based on the selected genes only. The resulting rule is referred as “LDA-1” in Figure 3. Another way is to use auxiliary data as in Method 2. More specifically, it first uses data from all involved genes and subjects in the training set to estimate the latent factors. These estimated factors are then applied to the set of selected genes, where their loadings and idiosyncratic matrix estimators are obtained. Combing them together produces the covariance matrix estimator, which is still an s × s matrix. The resulting rule is referred as “LDA-2” in Figure 3. Recall that the only difference between the two rules is that they use Different covariance estimators.

Figure 2.

Eigen-values of the sample covariance matrix for GSE22255.

Figure 3.

Misclassification rates of LDA-1 and LDA-2 over 100 random splits: the dotted lines represent the means over 100 splits and the segments represent the corresponding standard deviations.

Figure 3 plots the average misclassification rates on the test set against the number of factors for the 100 random splits. It is clearly seen that LDA-2 gives better misclassification rates than LDA-1, which is solely due to a Different estimation of the covariance matrix. The results lend further support to our claim that using more data is beneficial.