Large Covariance Estimation by Thresholding Principal Orthogonal Complements

Abstract

This paper deals with the estimation of a high-dimensional covariance with a conditional sparsity structure and fast-diverging eigenvalues. By assuming sparse error covariance matrix in an approximate factor model, we allow for the presence of some cross-sectional correlation even after taking out common but unobservable factors. We introduce the Principal Orthogonal complEment Thresholding (POET) method to explore such an approximate factor structure with sparsity. The POET estimator includes the sample covariance matrix, the factor-based covariance matrix (Fan, Fan, and Lv, 2008), the thresholding estimator (Bickel and Levina, 2008) and the adaptive thresholding estimator (Cai and Liu, 2011) as specific examples. We provide mathematical insights when the factor analysis is approximately the same as the principal component analysis for high-dimensional data. The rates of convergence of the sparse residual covariance matrix and the conditional sparse covariance matrix are studied under various norms. It is shown that the impact of estimating the unknown factors vanishes as the dimensionality increases. The uniform rates of convergence for the unobserved factors and their factor loadings are derived. The asymptotic results are also verified by extensive simulation studies. Finally, a real data application on portfolio allocation is presented.

1 Introduction

Information and technology make large data sets widely available for scientific discovery. Much statistical analysis of such high-dimensional data involves the estimation of a covariance matrix or its inverse (the precision matrix). Examples include portfolio management and risk assessment (Fan, Fan and Lv, 2008), high-dimensional classification such as Fisher discriminant (Hastie, Tibshirani and Friedman, 2009), graphic models (Meinshausen and Bühlmann, 2006), statistical inference such as controlling false discoveries in multiple testing (Leek and Storey, 2008; Efron, 2010), finding quantitative trait loci based on longitudinal data (Yap, Fan, and Wu, 2009; Xiong et al. 2011), and testing the capital asset pricing model (Sentana, 2009), among others. See Section 5 for some of those applications. Yet, the dimensionality is often either comparable to the sample size or even larger. In such cases, the sample covariance is known to have poor performance (Johnstone, 2001), and some regularization is needed.

Realizing the importance of estimating large covariance matrices and the challenges brought by the high dimensionality, in recent years researchers have proposed various regularization techniques to consistently estimate Σ. One of the key assumptions is that the covariance matrix is sparse, namely, many entries are zero or nearly so (Bickel and Levina, 2008, Rothman et al, 2009, Lam and Fan 2009, Cai and Zhou, 2010, Cai and Liu, 2011). In many applications, however, the sparsity assumption directly on Σ is not appropriate. For example, financial returns depend on the equity market risks, housing prices depend on the economic health, gene expressions can be stimulated by cytokines, among others. Due to the presence of common factors, it is unrealistic to assume that many outcomes are uncorrelated. An alternative method is to assume a factor model structure, as in Fan, Fan and Lv (2008). However, they restrict themselves to the strict factor models with known factors.

A natural extension is the conditional sparsity. Given the common factors, the outcomes are weakly correlated. In order to do so, we consider an approximate factor model, which has been frequently used in economic and financial studies (Chamberlain and Rothschild, 1983; Fama and French 1993; Bai and Ng, 2002, etc):

$$y_{it}={\mathbf{b}}_i^{\prime }{\mathbf{f}}_t+u_{it}.$$ (1.1)

Here y_it is the observed response for the ith (i = 1, …, p) individual at time t = 1, …, T; b_i is a vector of factor loadings; f_t is a K × 1 vector of common factors, and u_it is the error term, usually called idiosyncratic component, uncorrelated with f_t. Both p and T diverge to infinity, while K is assumed fixed throughout the paper, and p is possibly much larger than T.

We emphasize that in model (1.1), only y_it is observable. It is intuitively clear that the unknown common factors can only be inferred reliably when there are sufficiently many cases, that is, p → ∞. In a data-rich environment, p can diverge at a rate faster than T. The factor model (1.1) can be put in a matrix form as

$${\mathbf{y}}_t={\mathbf{Bf}}_t+{\mathbf{u}}_t.$$ (1.2)

where y_t = (y_1t, …, y_pt)′, B = (b₁, …, b_p)′ and u_t = (u_1t, …, u_pt)′. We are interested in Σ, the p × p covariance matrix of y_t, and its inverse, which are assumed to be time-invariant. Under model (1.1), Σ is given by

$$\mathbf{\sum }=\mathbf{B}\text{cov}({\mathbf{f}}_t){\mathbf{B}}^{\prime }+{\mathbf{\sum }}_u,$$ (1.3)

where Σ_u = (σ_u,ij)_p×p is the covariance matrix of u_t. The literature on approximate factor models typically assumes that the first K eigenvalues of Bcov(f_t)B′ diverge at rate O(p), whereas all the eigenvalues of Σ_u are bounded as p → ∞. This assumption holds easily when the factors are pervasive in the sense that a non-negligible fraction of factor loadings should be non-vanishing. The decomposition (1.3) is then asymptotically identified as p → ∞. In addition to it, in this paper we assume that Σ_u is approximately sparse as in Bickel and Levina (2008) and Rothman et al. (2009): for some q ∈ [0, 1),

$$m_p=\underset{i\le p}{max}\sum _{j\le p}{\mid {\sigma }_{u,ij}\mid }^q$$

does not grow too fast as p → ∞. In particular, this includes the exact sparsity assumption (q = 0) under which m_p = max_i≤p Σ_j≤p I_(σu,ij≠0), the maximum number of nonzero elements in each row.

The conditional sparsity structure of (1.2) was explored by Fan, Liao and Mincheva (2011) in estimating the covariance matrix, when the factors {f_t} are observable. This allows them to use regression analysis to estimate ${\{{\mathbf{u}}_t\}}_{t=1}^T$. This paper deals with the situation in which the factors are unobservable and have to be inferred. Our approach is simple, optimization-free and it uses the data only through the sample covariance matrix. Run the singular value decomposition on the sample covariance matrix Σ̂_sam of y_t, keep the covariance matrix formed by the first K principal components, and apply the thresholding procedure to the remaining covariance matrix. This results in a Principal Orthogonal complEment Thresholding (POET) estimator. When the number of common factors K is unknown, it can be estimated from the data. See Section 2 for additional details. We will investigate various properties of POET under the assumption that the data are serially dependent, which includes independent observations as a specific example. The rate of convergence under various norms for both estimated Σ and Σ_u and their precision (inverse) matrices will be derived. We show that the effect of estimating the unknown factors on the rate of convergence vanishes when p log p ≫ T, and in particular, the rate of convergence for Σ_u achieves the optimal rate in Cai and Zhou (2012).

This paper focuses on the high-dimensional static factor model (1.2), which is innately related to the principal component analysis (PCA), as clarified in Section 2. This feature makes it different from the classical factor model with fixed dimensionality (e.g., Lawley and Maxwell 1971). In the last ten years, much theory on the estimation and inference of the static factor model has been developed, for example, Stock and Watson (1998, 2002), Bai and Ng (2002), Bai (2003), Doz, Giannone and Reichlin (2011), among others. Our contribution is on the estimation of covariance matrices and their inverse in large factor models.

The static model considered in this paper is to be distinguished from the dynamic factor model as in Forni, Hallin, Lippi and Reichlin (2000); the latter allows y_t to also depend on f_t with lags in time. Their approach is based on the eigenvalues and principal components of spectral density matrices, and on the frequency domain analysis. Moreover, as shown in Forni and Lippi (2001), the dynamic factor model does not really impose a restriction on the data generating process, and the assumption of idiosyncrasy (in their terminology, a p-dimensional process is idiosyncratic if all the eigenvalues of its spectral density matrix remain bounded as p → ∞) asymptotically identifies the decomposition of y_it into the common component and idiosyncratic error. The literature includes, for example, Forni et al. (2000, 2004), Forni and Lippi (2001), Hallin and Liška (2007, 2011), and many other references therein. Above all, both the static and dynamic factor models are receiving increasing attention in applications of many fields where information usually is scattered through a (very) large number of interrelated time series.

There has been extensive literature in recent years that deals with sparse principal components, which has been widely used to enhance the convergence of the principal components in high-dimensional space. d’Aspremont, Bach and El Ghaoui (2008), Shen and Huang (2008), Witten, Tibshirani, and Hastie (2009) and Ma (2011) proposed and studied various algorithms for computations. More literature on sparse PCA is found in Johnstone and Lu (2009), Amini and Wainwright (2009), Zhang and El Ghaoui (2011), Birnbaum et al. (2012), among others. In addition, there has also been a growing literature that theoretically studies the recovery from a low-rank plus sparse matrix estimation problem, see for example, Wright et al. (2009), Lin et al. (2009), Candès et al. (2011), Luo (2011), Agarwal, Nagahban, Wainwright (2012), Pati et al. (2012). It corresponds to the identifiability issue of our problem.

There is a big difference between our model and those considered in the aforementioned literature. In the current paper, the first K eigenvalues of Σ are spiked and grow at a rate O(p), whereas the eigenvalues of the matrices studied in the existing literature on covariance estimation are usually assumed to be either bounded or slowly growing. Due to this distinctive feature, the common components and the idiosyncratic components can be identified, and in addition, PCA on the sample covariance matrix can consistently estimate the space spanned by the eigenvectors of Σ. The existing methods of either thresholding directly or solving a constrained optimization method can fail in the presence of very spiked principal eigenvalues. However, there is a price to pay here: as the first K eigenvalues are “too spiked”, one can hardly obtain a satisfactory rate of convergence for estimating Σ in absolute term, but it can be estimated accurately in relative term (see Section 3.3 for details). In addition, Σ⁻¹ can be estimated accurately.

We would like to further note that the low-rank plus sparse representation of our model is on the population covariance matrix, whereas Candès et al. (2011), Wright et al. (2009), Lin et al. (2009)¹ considered such a representation on the data matrix. As there is no Σ to estimate, their goal is limited to producing a low-rank plus sparse matrix decomposition of the data matrix, which corresponds to the identifiability issue of our study, and does not involve estimation and inference. In contrast, our ultimate goal is to estimate the population covariance matrices as well as the precision matrices. For this purpose, we require the idiosyncratic components and common factors to be uncorrelated and the data generating process to be strictly stationary. The covariances considered in this paper are constant over time, though slow-time-varying covariance matrices are applicable through localization in time (time-domain smoothing). Our consistency result on Σ_u demonstrates that the decomposition (1.3) is identifiable, and hence our results also shed the light of the “surprising phenomenon” of Candès et al. (2011) that one can separate fully a sparse matrix from a low-rank matrix when only the sum of these two components is available.

The rest of the paper is organized as follows. Section 2 gives our estimation procedures and builds the relationship between the principal components analysis and the factor analysis in high-dimensional space. Section 3 provides the asymptotic theory for various estimated quantities. Section 4 illustrates how to choose the thresholds using cross-validation and guarantees the positive definiteness in any finite sample. Specific applications of regularized covariance matrices are given in Section 5. Numerical results are reported in Section 6. Finally, Section 7 presents a real data application on portfolio allocation. All proofs are given in the appendix. Throughout the paper, we use λ_min(A) and λ_max(A) to denote the minimum and maximum eigenvalues of a matrix A. We also denote by ||A||_F, ||A||, ||A||₁ and ||A||_max the Frobenius norm, spectral norm (also called operator norm), L₁-norm, and elementwise norm of a matrix A, defined respectively by ||A||_F = tr^1/2(A′A), $\left|\right|\mathbf{A}\left|\right|={\lambda }_{max}^{1/2}({\mathbf{A}}^{\prime }\mathbf{A})$, ||A||₁ = max_j Σ_i |a_ij| and ||A||_max = max_i,j |a_ij|. Note that when A is a vector, both ||A||_F and ||A|| are equal to the Euclidean norm. Finally, for two sequences, we write a_T ≫ b_T if b_T = o(a_T) and a_T ≍ b_T if a_T = O(b_T) and b_T = O(a_T).

2 Regularized Covariance Matrix via PCA

There are three main objectives of this paper: (i) understand the relationship between principal component analysis (PCA) and the high-dimensional factor analysis; (ii) estimate both covariance matrices Σ and the idiosyncratic Σ_u and their precision matrices in the presence of common factors, and (iii) investigate the impact of estimating the unknown factors on the covariance estimation. The propositions in Section 2.1 below show that the space spanned by the principal components in the population level Σ is close to the space spanned by the columns of the factor loading matrix B.

2.1 High-dimensional PCA and factor model

Consider a factor model

$$y_{it}={\mathbf{b}}_i^{\prime }{\mathbf{f}}_t+u_{it},i\le p,t\le T,$$

where the number of common factors, K = dim(f_t), is small compared to p and T, and thus is assumed to be fixed throughout the paper. In the model, the only observable variable is the data y_it. One of the distinguished features of the factor model is that the principal eigenvalues of Σ are no longer bounded, but growing fast with the dimensionality. We illustrate this in the following example.

Example 2.1

Consider a single-factor model y_it = b_if_t + u_it where b_i ∈ ℝ. Suppose that the factor is pervasive in the sense that it has non-negligible impact on a non-vanishing proportion of outcomes. It is then reasonable to assume ${\sum }_{i=1}^p{b}_i^2>cp$ for some c > 0. Therefore, assuming that λ_max(Σ_u) = o(p), an application of (1.3) yields,

$${\lambda }_{max}(\mathbf{\sum })\ge var(f_t)\sum _{i=1}^p{b}_i^2-{\lambda }_{max}({\mathbf{\sum }}_u)>\frac{c}{2}var(f_t)p$$

for all large p, assuming var(f_t) > 0.

We now elucidate why PCA can be used for the factor analysis in the presence of spiked eigenvalues. Write B = (b₁, …, b_p)′ as the p × K loading matrix. Note that the linear space spanned by the first K principal components of Bcov(f_t)B′ is the same as that spanned by the columns of B when cov(f_t) is non-degenerate. Thus, we can assume without loss of generality that the columns of B are orthogonal and cov(f_t) = I_K, the identity matrix. This canonical form corresponds to the identifiability condition in decomposition (1.3). Let b̃₁, · · ·, b̃_K be the columns of B, ordered such that ${\{|\left|{\stackrel{\sim }{\mathbf{b}}}_j\right||\}}_{j=1}^K$ is in a non-increasing order. Then, ${\{{\stackrel{\sim }{\mathbf{b}}}_j/|\left|{\stackrel{\sim }{\mathbf{b}}}_j\right||\}}_{j=1}^K$ are eigenvectors of the matrix BB′ with eigenvalues ${\{{\left|\right|{\stackrel{\sim }{\mathbf{b}}}_j\left|\right|}^2\}}_{j=1}^K$ and the rest zero. We will impose the pervasiveness assumption that all eigenvalues of the K × K matrix p⁻¹B′B are bounded away from zero, which holds if the factor loadings ${\{{\mathbf{b}}_i\}}_{i=1}^p$ are independent realizations from a non-degenerate population. Since the non-vanishing eigenvalues of the matrix BB′ are the same as those of B′B, from the pervasiveness assumption it follows that ${\{{\left|\right|{\stackrel{\sim }{\mathbf{b}}}_j\left|\right|}^2\}}_{j=1}^K$ are all growing at rate O(p).

Let ${\{{\lambda }_j\}}_{j=1}^p$ be the eigenvalues of Σ in a descending order and ${\{{\mathit{\xi }}_j\}}_{j=1}^p$ be their corresponding eigenvectors. Then, an application of Weyl’s eigenvalue theorem (see the appendix) yields that

Proposition 2.1

Assume that the eigenvalues of p⁻¹B′B are bounded away from zero for all large p. For the factor model (1.3) with the canonical condition

$$cov({\mathbf{f}}_t)={\mathbf{I}}_K\mathit{and}{\mathbf{B}}^{\prime }\mathbf{B}is\mathit{diagonal},$$ (2.1)

we have

$$\mid {\lambda }_j-{\left|\right|{\stackrel{\sim }{\mathbf{b}}}_j\left|\right|}^2\mid \le \left|\right|{\mathbf{\sum }}_u\left|\right|,\mathit{for}j\le K,\mid {\lambda }_j\mid \le \left|\right|{\mathbf{\sum }}_u\left|\right|,\mathit{for}j>K.$$

In addition, for j ≤ K, lim inf_p→∞ ||b̃_j||²/p > 0.

Using Proposition 2.1 and the sin θ theorem of Davis and Kahn (1970, see the appendix), we have the following:

Proposition 2.2

Under the assumptions of Proposition 2.1, if ${\{|\left|{\stackrel{\sim }{\mathbf{b}}}_j\right||\}}_{j=1}^K$ are distinct, then

$$\left|\right|{\mathit{\xi }}_j-{\stackrel{\sim }{\mathbf{b}}}_j/\left|\right|{\stackrel{\sim }{\mathbf{b}}}_j\left|\right|\left|\right|=O(p^{-1}|\left|{\mathbf{\sum }}_u\right||),\mathit{for}j\le K.$$

Propositions 2.1 and 2.2 state that PCA and factor analysis are approximately the same if ||Σ_u|| = o(p). This is assured through a sparsity condition on Σ_u = (σ_u,ij)_p×p, which is frequently measured through

$$m_p=\underset{i\le p}{max}\sum _{j\le p}{\mid {\sigma }_{u,ij}\mid }^q,\text{for}\text{some}q\in [0,1].$$ (2.2)

The intuition is that, after taking out the common factors, many pairs of the cross-sectional units become weakly correlated. This generalized notion of sparsity was used in Bickel and Levina (2008) and Cai and Liu (2011). Under this generalized measure of sparsity, we have

$$\left|\right|{\mathbf{\sum }}_u\left|\right|\le {\left|\right|{\mathbf{\sum }}_u\left|\right|}_1\le \underset{i}{max}\sum _{j=1}^p{\mid {\sigma }_{u,ij}\mid }^q{({\sigma }_{u,ii}{\sigma }_{u,jj})}^{(1-q)/2}=O(m_p),$$

if the noise variances { ${\sigma }_{u,ii}^2$} are bounded. Therefore, when m_p = o(p), Proposition 2.1 implies that we have distinguished eigenvalues between the principal components ${\{{\lambda }_j\}}_{j=1}^K$ and the rest of the components ${\{{\lambda }_j\}}_{j=K+1}^p$ and Proposition 2.2 ensures that the first K principal components are approximately the same as the columns of the factor loadings.

The aforementioned sparsity assumption appears reasonable in empirical applications. Boivin and Ng (2006) conducted an empirical study and showed that imposing zero correlation between weakly correlated idiosyncratic components improves forecast². More recently, Phan (2012) empirically estimated the level of sparsity of the idiosyncratic covariance using the UK market data.

Recent developments on random matrix theory, for example, Johnstone and Lu (2009) and Paul (2007), have shown that when p/T is not negligible, the eigenvalues and eigenvectors of Σ might not be consistently estimated from the sample covariance matrix. A distinguished feature of the covariance considered in this paper is that there are some very spiked eigenvalues. By Propositions 2.1 and 2.2, in the factor model, the pervasiveness condition

$${\lambda }_{min}(p^{-1}{\mathbf{B}}^{\prime }\mathbf{B})>c>0$$ (2.3)

implies that the first K eigenvalues are growing at a rate p. Moreover, when p is large, the principal components ${\{{\mathit{\xi }}_j\}}_{j=1}^K$ are close to the normalized vectors ${\{{\stackrel{\sim }{\mathbf{b}}}_j\}}_{j=1}^K$ when m_p = o(p). This provides the mathematics for using the first K principal components as a proxy of the space spanned by the columns of the factor loading matrix B. In addition, due to (2.3), the signals of the first K eigenvalues are stronger than those of the spiked covariance model considered by Jung and Marron (2009) and Birnbaum et al. (2012). Therefore, our other conditions for the consistency of principal components at the population level are much weaker than those in the spiked covariance literature. On the other hand, this also shows that, under our setting the PCA is a valid approximation to factor analysis only if p → ∞. The fact that the PCA on the sample covariance is inconsistent when p is bounded was also previously demonstrated in the literature (See e.g., Bai (2003)).

With assumption (2.3), the standard literature on approximate factor models has shown that the PCA on the sample covariance matrix Σ̂_sam can consistently estimate the space spanned by the factor loadings (e.g., Stock and Watson (1998), Bai (2003)). Our contribution in Propositions 2.1 and 2.2 is that we connect the high-dimensional factor model to the principal components, and obtain the consistency of the spectrum in the population level Σ instead of the sample level Σ̂_sam. The spectral consistency also enhances the results in Chamberlain and Rothschild (1983). This provides the rationale behind the consistency results in the factor model literature.

2.2 POET

Sparsity assumption directly on Σ is inappropriate in many applications due to the presence of common factors. Instead, we propose a nonparametric estimator of Σ based on the principal component analysis. Let λ̂₁ ≥ λ̂₂ ≥ · · · ≥ λ̂_p be the ordered eigenvalues of the sample covariance matrix Σ̂_sam and ${\{{\widehat{\mathit{\xi }}}_i\}}_{i=1}^p$ be their corresponding eigenvectors. Then the sample covariance has the following spectral decomposition:

$${\widehat{\mathbf{\sum }}}_{\text{sam}}=\sum _{i=1}^K{\widehat{\lambda }}_i{\widehat{\mathit{\xi }}}_i{\widehat{\mathit{\xi }}}_i^{\prime }+{\widehat{\mathbf{R}}}_K,$$ (2.4)

where ${\widehat{\mathbf{R}}}_K={\sum }_{i=K+1}^p{\widehat{\lambda }}_i{\widehat{\mathit{\xi }}}_i{\widehat{\mathit{\xi }}}_i^{\prime }={({\widehat{r}}_{ij})}_{p\times p}$ is the principal orthogonal complement, and K is the number of diverging eigenvalues of Σ. Let us first assume K is known.

Now we apply thresholding on R̂_K. Define

$${\widehat{\mathbf{R}}}_K^{\tau }={({\widehat{r}}_{ij}^{\tau })}_{p\times p},{\widehat{r}}_{ij}^{\tau }=\{\begin{array}{ll}{\widehat{r}}_{ii},\hfill & i=j;\hfill \\ s_{ij}({\widehat{r}}_{ij})I(\mid {\widehat{r}}_{ij}\mid \ge {\tau }_{ij}),\hfill & i\ne j.\hfill \end{array}$$ (2.5)

where s_ij(·) is a generalized shrinkage function of Antoniadis and Fan (2001), employed by Rothman et al. (2009) and Cai and Liu (2011), and τ_ij > 0 is an entry-dependent threshold. In particular, the hard-thresholding rule s_ij(x) = xI(|x| ≥ τ_ij) (Bickel and Levina, 2008) and the constant thresholding parameter τ_ij = δ are allowed. In practice, it is more desirable to have τ_ij be entry-adaptive. An example of the adaptive thresholding is

$${\tau }_{ij}=\tau {({\widehat{r}}_{ii}{\widehat{r}}_{jj})}^{1/2},\text{for}\text{a}\text{given}\tau >0$$ (2.6)

where r̂_ii is the i^th diagonal element of R̂_K. This corresponds to applying the thresholding with parameter τ to the correlation matrix of R̂_K.

The estimator of Σ is then defined as:

$${\widehat{\mathbf{\sum }}}_K=\sum _{i=1}^K{\widehat{\lambda }}_i{\widehat{\mathit{\xi }}}_i{\widehat{\mathit{\xi }}}_i^{\prime }+{\widehat{\mathbf{R}}}_K^{\mathcal{T}}.$$ (2.7)

We will call this estimator the Principal Orthogonal complEment thresholding (POET) estimator. It is obtained by thresholding the remaining components of the sample covariance matrix, after taking out the first K principal components. One of the attractiveness of POET is that it is optimization-free, and hence is computationally appealing. ³

With the choice of τ_ij in (2.6) and the hard thresholding rule, our estimator encompasses many popular estimators as its specific cases. When τ = 0, the estimator is the sample covariance matrix and when τ = 1, the estimator becomes that based on the strict factor model (Fan, Fan, and Lv, 2008). When K = 0, our estimator is the same as the thresholding estimator of Bickel and Levina (2008) and (with a more general thresholding function) Rothman et al. (2009) or the adaptive thresholding estimator of Cai and Liu (2011) with a proper choice of τ_ij.

In practice, the number of diverging eigenvalues (or common factors) can be estimated based on the sample covariance matrix. Determining K in a data-driven way is an important topic, and is well understood in the literature. We will describe the POET with a data-driven K in Section 2.4.

2.3 Least squares point of view

The POET (2.7) has an equivalent representation using a constrained least squares method. The least squares method seeks for ${\widehat{\mathbf{\Lambda }}}_K={({\widehat{\mathbf{b}}}_1^K,\dots ,{\widehat{\mathbf{b}}}_p^K)}^{\prime }$ and ${\widehat{\mathbf{F}}}_K^{\prime }=({\widehat{\mathbf{f}}}_1^K,\dots ,{\widehat{\mathbf{f}}}_T^K)$ such that

$$({\widehat{\mathbf{\Lambda }}}_K,{\widehat{\mathbf{F}}}_K)=arg\underset{{\mathbf{b}}_i\in {\mathbb{R}}^K,{\mathbf{f}}_t\in {\mathbb{R}}^K}{min}\sum _{i=1}^p\sum _{t=1}^T{(y_{it}-{\mathbf{b}}_i^{\prime }{\mathbf{f}}_t)}^2,$$ (2.8)

subject to the normalization

$$\frac{1}{T}\sum _{t=1}^T{\mathbf{f}}_t{\mathbf{f}}_t^{\prime }={\mathbf{I}}_K,\text{and}\frac{1}{p}\sum _{i=1}^p{\mathbf{b}}_i{\mathbf{b}}_i^{\prime }\text{is}\text{diagonal}.$$ (2.9)

The constraints (2.9) correspond to the normalization (2.1). Here we assume that the mean of each variable ${\{y_{it}\}}_{t=1}^T$ has been removed, that is, Ey_it = Ef_jt = 0 for all i ≤ p, j ≤ K and t ≤ T. Putting it in a matrix form, the optimization problem can be written as

$$\begin{array}{l}arg\underset{\mathbf{B},\mathbf{F}}{min}{\left|\right|\mathbf{Y}-{\mathbf{BF}}^{\prime }\left|\right|}_F^2\\ T^{-1}{\mathbf{F}}^{\prime }\mathbf{F}={\mathbf{I}}_K,{\mathbf{B}}^{\prime }\mathbf{B}\text{is}\text{diagonal}.\end{array}$$ (2.10)

where Y = (y₁, …, y_T) and F′ = (f₁, · · ·, f_T). For each given F, the least-squares estimator of B is Λ = T⁻¹YF, using the constraint (2.9) on the factors. Substituting this into (2.10), the objective function now becomes ${\left|\right|\mathbf{Y}-T^{-1}{\mathbf{YFF}}^{\prime }\left|\right|}_F^2=\text{tr}[(I_T-T^{-1}{\mathbf{FF}}^{\prime }){\mathbf{Y}}^{\prime }\mathbf{Y}]$. The minimizer is now clear: the columns of ${\widehat{\mathbf{F}}}_K/\sqrt{T}$ are the eigenvectors corresponding to the K largest eigenvalues of the T × T matrix Y′Y and Λ̂_K = T⁻¹YF̂_K (see e.g., Stock and Watson (2002)).

We will show that under some mild regularity conditions, as p and T → ∞, ${\widehat{\mathbf{b}}}_i^{K^{\prime }}{\widehat{\mathbf{f}}}_t^K$ consistently estimates the true ${\mathbf{b}}_i^{\prime }{\mathbf{f}}_t$ uniformly over i ≤ p and t ≤ T. Since Σ_u is assumed to be sparse, we can construct an estimator of Σ_u using the adaptive thresholding method by Cai and Liu (2011) as follows. Let ${\widehat{u}}_{it}=y_{it}-{\widehat{\mathbf{b}}}_i^{K^{\prime }}{\widehat{\mathbf{f}}}_t^K,{\widehat{\sigma }}_{ij}-\frac{1}{T}{\sum }_{t=1}^T{\widehat{u}}_{it}{\widehat{u}}_{jt}$, and ${\widehat{\theta }}_{ij}=\frac{1}{T}{\sum }_{t=1}^T{({\widehat{u}}_{it}{\widehat{u}}_{jt}-{\widehat{\sigma }}_{ij})}^2$. For some pre-determined decreasing sequence ω_T > 0, and large enough C > 0, define the adaptive threshold parameter as ${\tau }_{ij}=C\sqrt{{\widehat{\theta }}_{ij}}{\omega }_T$. The estimated idiosyncratic covariance estimator is then given by

$${\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}}={({\widehat{\sigma }}_{ij}^{\mathcal{T}})}_{p\times p},{\widehat{\sigma }}_{ij}^{\mathcal{T}}=\{\begin{array}{ll}{\widehat{\sigma }}_{ii},\hfill & i=j\hfill \\ s_{ij}({\widehat{\sigma }}_{ij}),\hfill & i\ne j,\hfill \end{array}$$ (2.11)

where for all z ∈ ℝ (see Antoniadis and Fan, 2001),

$$s_{ij}(z)=0\text{when}\mid z\mid \le {\tau }_{ij},\mid s_{ij}(z)-z\mid \le {\tau }_{ij}.$$

It is easy to verify that s_ij(·) includes many interesting thresholding functions such as the hard thresholding (s_ij(z) = zI_(|z|≥τij)), soft thresholding (s_ij (z) = sign(z)(|z| − τ_ij)₊), SCAD, and adaptive lasso (See Rothman et al. (2009)).

Analogous to the decomposition (1.3), we obtain the following substitution estimators

$${\mathbf{\sum }^{\sim }}_K={\widehat{\mathbf{\Lambda }}}_K{\widehat{\mathbf{\Lambda }}}_K^{\prime }+{\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}},$$ (2.12)

and by the Sherman-Morrison-Woodbury formula, noting that $\frac{1}{T}{\sum }_{t=1}^T{\widehat{\mathbf{f}}}_t^K{\widehat{\mathbf{f}}}_t^{K^{\prime }}={\mathbf{I}}_K$,

$${({\mathbf{\sum }^{\sim }}_K)}^{-1}={({\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}})}^{-1}-{({\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}})}^{-1}{\widehat{\mathbf{\Lambda }}}_K{[{\mathbf{I}}_K+{\widehat{\mathbf{\Lambda }}}_K^{\prime }{({\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}})}^{-1}{\widehat{\mathbf{\Lambda }}}_K]}^{-1}{\widehat{\mathbf{\Lambda }}}_K^{\prime }{({\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}})}^{-1},$$ (2.13)

In practice, the true number of factors K might be unknown to us. However, for any determined K₁ ≤ p, we can always construct either (Σ̂_K1, ${\widehat{\mathbf{R}}}_{K_1}^{\mathcal{T}}$) as in (2.7) or (Σ̃_K1, ${\widehat{\mathbf{\sum }}}_{u,K_1}^{\mathcal{T}}$) as in (2.12) to estimate (Σ, Σ_u). The following theorem shows that for each given K₁, the two estimators based on either regularized PCA or least squares substitution are equivalent. Similar results were obtained by Bai (2003) when K₁ = K and no thresholding was imposed.

Theorem 2.1

Suppose that the entry-dependent threshold in (2.5) is the same as the thresholding parameter used in (2.11). Then for any K₁ ≤ p, the estimator (2.7) is equivalent to the substitution estimator (2.12), that is,

$${\widehat{\mathbf{\sum }}}_{K_1}={\mathbf{\sum }^{\sim }}_{K_1},\mathit{and}{\widehat{\mathbf{\sum }}}_{u,K_1}^{\mathcal{T}}={\widehat{\mathbf{R}}}_{K_1}^{\mathcal{T}}.$$

In this paper, we will use a data-driven K̂ to construct the POET (see Section 2.4 below), which has two equivalent representations according to Theorem 2.1.

2.4 POET with Unknown K

Determining the number of factors in a data-driven way has been an important research topic in the econometric literature. Bai and Ng (2002) proposed a consistent estimator as both p and T diverge. Other recent criteria are proposed by Kapetanios (2010), Onatski (2010), Alessi et al. (2010), etc.

Our method also allows a data-driven K̂ to estimate the covariance matrices. In principle, any procedure that gives a consistent estimate of K can be adopted. In this paper we apply the well-known method in Bai and Ng (2002). It estimates K by

$$\widehat{K}=arg\underset{0\le K_1\le M}{min}log\left\{\frac{1}{pT}{\left|\right|\mathbf{Y}-T^{-1}\mathbf{Y}{\widehat{\mathbf{F}}}_{K_1}{\widehat{\mathbf{F}}}_{K_1}^{\prime }\left|\right|}_F^2\right\}+K_{1}g(T,p),$$ (2.14)

where M is a prescribed upper bound, F̂_K1 is a T × K₁ matrix whose columns are $\sqrt{T}$ times the eigenvectors corresponding to the K₁ largest eigenvalues of the T × T matrix Y′Y; g(T, p) is a penalty function of (p, T) such that g(T, p) = o(1) and min{p, T}g(T, p) → ∞. Two examples suggested by Bai and Ng (2002) are

$$\begin{array}{l}\text{IC}1:g(T,p)=\frac{p+T}{pT}log\left(\frac{pT}{p+T}\right),\\ \text{IC}2:g(T,p)=\frac{p+T}{pT}logmin\{p,T\}.\end{array}$$

Throughout the paper, we let K̂ be the solution to (2.14) using either IC1 or IC2. The asymptotic results are not affected regardless of the specific choice of g(T, p). We define the POET estimator with unknown K as

$${\widehat{\mathbf{\sum }}}_{\widehat{K}}=\sum _{i=1}^{\widehat{K}}{\widehat{\lambda }}_i{\widehat{\mathit{\xi }}}_i{\widehat{\mathit{\xi }}}_i^{\prime }+{\widehat{\mathbf{R}}}_{\widehat{K}}^{\mathcal{T}}.$$ (2.15)

The procedure is as stated in Section 2.2 except that K̂ is now data-driven.

3 Asymptotic Properties

3.1 Assumptions

This section presents the assumptions on the model (1.2), in which only ${\{{\mathbf{y}}_t\}}_{t=1}^T$ are observable. Recall the identifiability condition (2.1).

The first assumption has been one of the most essential ones in the literature of approximate factor models. Under this assumption and other regularity conditions, the number of factors, loadings and common factors can be consistently estimated (e.g., Stock and Watson (1998, 2002), Bai and Ng (2002), Bai (2003), etc.).

Assumption 3.1

All the eigenvalues of the K × K matrix p⁻¹B′B are bounded away from both zero and infinity as p → ∞.

Remark 3.1

1. It implies from Proposition 2.1 in Section 2 that the first K eigenvalues of Σ grow at rate O(p). This unique feature distinguishes our work from most of other low-rank plus sparse covariances considered in the literature, e.g., Luo (2011), Pati et al. (2012), Agarwal et al. (2012), Birnbaum et al. (2012). ⁴

2. Assumption 3.1 requires the factors to be pervasive, that is, to impact a non-vanishing proportion of individual time series. See Example 2.1 for its meaning. ⁵

3. As to be illustrated in Section 3.3 below, due to the fast diverging eigenvalues, one can hardly achieve a good rate of convergence for estimating Σ under either the spectral norm or Frobenius norm when p > T. This phenomenon arises naturally from the characteristics of the high-dimensional factor model, which is another distinguished feature compared to those convergence results in the existing literature.

Assumption 3.2

1. {u_t, f_t}_t≥1 is strictly stationary. In addition, Eu_it = Eu_itf_jt = 0 for all i ≤ p, j ≤ K and t ≤ T.

2. There exist constants c₁, c₂ > 0 such that λ_min(Σ_u) > c₁, ||Σ_u||₁ < c₂, and min_i≤p,j≤p var(u_itu_jt) > c₁.

3. There exist r₁, r₂ > 0 and b₁, b₂ > 0, such that for any s > 0, i ≤ p and j ≤ K,

   $$P(\mid u_{it}\mid >s)\le exp(-{(s/b_1)}^{r_1}),P(\mid f_{jt}\mid >s)\le exp(-{(s/b_2)}^{r_2}).$$

Condition (i) requires strict stationarity as well as the non-correlation between {u_t} and {f_t}. These conditions are slightly stronger than those in the literature, e.g., Bai (2003), but are still standard and simplify our technicalities. Condition (ii) requires that Σ_u be well-conditioned. The condition ||Σ_u||₁ ≤ c₂ instead of a weaker condition λ_max(Σ_u) ≤ c₂ is imposed here in order to consistently estimate K. But it is still standard in the approximate factor model literature as in Bai and Ng (2002), Bai (2003), etc. When K is known, such a condition can be removed. Our working paper⁶ shows that the results continue to hold for a growing (known) K under the weaker condition λ_max(Σ_u) ≤ c₂. Condition (iii) requires exponential-type tails, which allows us to apply the large deviation theory to $\frac{1}{T}{\sum }_{t=1}^T{u}_{it}{u}_{jt}-{\sigma }_{u,ij}$ and $\frac{1}{T}{\sum }_{t=1}^T{f}_{jt}{u}_{it}$.

We impose the strong mixing condition. Let ${\mathcal{F}}_{-\infty }^0$ and ${\mathcal{F}}_T^{\infty }$ denote the σ-algebras generated by {(f_t, u_t) : t ≤ 0} and {(f_t, u_t) : t ≥ T} respectively. In addition, define the mixing coefficient

$$\alpha (T)=\underset{A\in {\mathcal{F}}_{-\infty }^0,B\in {\mathcal{F}}_T^{\infty }}{sup}\mid P(A)P(B)-P(AB)\mid .$$ (3.1)

Assumption 3.3

Strong mixing: There exists r₃ > 0 such that $3r_1^{-1}+1.5r_2^{-1}+r_3^{-1}>1$, and C > 0 satisfying: for all T ∈ ℤ⁺,

$$\alpha (T)\le exp(-{CT}^{r_3}).$$

In addition, we impose the following regularity conditions.

Assumption 3.4

There exists M > 0 such that for all i ≤ p, t ≤ T and s ≤ T,

1. ||b_i||_max < M,

2. $E{[p^{-1/2}({\mathbf{u}}_s^{\prime }{\mathbf{u}}_t-E{\mathbf{u}}_s^{\prime }{\mathbf{u}}_t)]}^4<M$,

3. $E{\left|\right|p^{-1/2}{\sum }_{i=1}^p{\mathbf{b}}_i{u}_{it}\left|\right|}^4<M$.

These conditions are needed to consistently estimate the transformed common factors as well as the factor loadings. Similar conditions were also assumed in Bai (2003), and Bai and Ng (2006). The number of factors is assumed to be fixed. Our conditions in Assumption 3.4 are weaker than those in Bai (2003) as we focus on different aspects of the study.

3.2 Convergence of the idiosyncratic covariance

Estimating the covariance matrix Σ_u of the idiosyncratic components {u_t} is important for many statistical inferences. For example, it is needed for large sample inference of the unknown factors and their loadings, for testing the capital asset pricing model (Sentana, 2009), and large-scale hypothesis testing (Fan, Han and Gu, 2012). See Section 5.

We estimate Σ_u by thresholding the principal orthogonal complements after the first K̂ principal components of the sample covariance are taken out: ${\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}={\widehat{\mathbf{R}}}_{\widehat{K}}^{\mathcal{T}}$. By Theorem 2.1, it also has an equivalent expression given by (2.11), with ${\widehat{u}}_{it}=y_{it}-{({\widehat{\mathbf{b}}}_i^{\widehat{K}})}^{\prime }{\widehat{\mathbf{f}}}_t^{\widehat{K}}$. Throughout the paper, we apply the adaptive threshold

$${\tau }_{ij}=C\sqrt{{\widehat{\theta }}_{ij}}{\omega }_T,{\omega }_T=\frac{1}{\sqrt{p}}+\sqrt{\frac{logp}{T}}$$ (3.2)

where C > 0 is a sufficiently large constant, though the results hold for other types of thresholding. As in Bickel and Levina (2008) and Cai and Liu (2011), the threshold chosen in the current paper is in fact obtained from the optimal uniform rate of convergence of max_i≤p,j≤p |σ̂_ij − σ_u,ij|. When direct observation of u_it is not available, the effect of estimating the unknown factors also contributes to this uniform estimation error, which is why p^−1/2 appears in the threshold.

The following theorem gives the rate of convergence of the estimated idiosyncratic covariance. Let ${\gamma }^{-1}=3r_1^{-1}+1.5r_2^{-1}+r_3^{-1}+1$. In the convergence rate below, recall that m_p and q are defined in the measure of sparsity (2.2).

Theorem 3.1

Suppose log p = o(T^γ/6), T = o(p²), and Assumptions 3.1–3.4 hold. Then for a sufficiently large constant C > 0 in the threshold (3.2), the POET estimator ${\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}$ satisfies

$$\left|\right|{\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}-{\mathbf{\sum }}_u\left|\right|=O_p({\omega }_T^{1-q}{m}_p).$$

If further ${\omega }_T^{1-q}{m}_p=o(1)$, then the eigenvalues of ${\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}$ are all bounded away from zero with probability approaching one, and

$$\left|\right|{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}-{\mathbf{\sum }}_u^{-1}\left|\right|=O_p({\omega }_T^{1-q}{m}_p).$$

When estimating Σ_u, p is allowed to grow exponentially fast in T, and ${\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}$ can be made consistent under the spectral norm. In addition, ${\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}$ is asymptotically invertible while the classical sample covariance matrix based on the residuals is not when p > T.

Remark 3.2

1. Consistent estimation of Σ_u indicates that Σ_u is identifiable in (1.3), namely, the sparse Σ_u can be separated perfectly from the low-rank matrix there. The result here gives another proof (when assuming ${\omega }_T^{1-q}{m}_p=o(1)$ of the “surprising phenomenon” in Candès et al (2011) under different technical conditions.

2. Fan, Liao and Mincheva (2011) recently showed that when ${\{{\mathbf{f}}_t\}}_{t=1}^T$ are observable and q = 0, the rate of convergence of the adaptive thresholding estimator is given by $\left|\right|{\widehat{\mathbf{\sum }}}_u^{\mathcal{T}}-{\mathbf{\sum }}_u\left|\right|=O_p\left(m_p\sqrt{\frac{logp}{T}}\right)=\left|\right|{({\widehat{\mathbf{\sum }}}_u^{\mathcal{T}})}^{-1}-{\mathbf{\sum }}_u^{-1}\left|\right|$. Hence when the common factors are unobservable, the rate of convergence has an additional term $m_p/\sqrt{p}$, coming from the impact of estimating the unknown factors. This impact vanishes when p log p ≫ T, in which case the minimax rate as in Cai and Zhou (2010) is achieved. As p increases, more information about the common factors is collected, which results in more accurate estimation of the common factors ${\{{\mathbf{f}}_t\}}_{t=1}^T$.

3. When K is known and grows with p and T, with slightly weaker assumptions, our working paper (Fan et al. 2011) shows that under the exactly sparse case (that is, q = 0), the result continues to hold with convergence rate $m_p(K^2\sqrt{\frac{logp}{T}}+\frac{K^3}{\sqrt{p}})$.

3.3 Convergence of the POET estimator

Since the first K eigenvalues of Σ grow with p, one can hardly estimate Σ with satisfactory accuracy in the absolute term. This problem arises not from the limitation of any estimation method, but is due to the nature of the high-dimensional factor model. We illustrate this using a simple example.

Example 3.1

Consider an ideal case where we know the spectrum except for the first eigenvector of Σ. Let ${\{{\lambda }_j,{\mathit{\xi }}_j\}}_{j=1}^p$ be the eigenvalues and vectors, and assume that the largest eigenvalue λ₁ ≥ cp for some c > 0. Let ξ̂₁ be the estimated first eigenvector and define the covariance estimator $\widehat{\mathbf{\sum }}={\lambda }_1{\widehat{\mathit{\xi }}}_1{\widehat{\mathit{\xi }}}_1^{\prime }+{\sum }_{j=2}^p{\lambda }_j{\mathit{\xi }}_j{\mathit{\xi }}_j^{\prime }$. Assume that ξ̂₁ is a good estimator in the sense that ||ξ̂₁ − ξ₁||² = O_p(T⁻¹). However,

$$\left|\right|\widehat{\mathbf{\sum }}-\mathbf{\sum }\left|\right|=\left|\right|{\lambda }_1({\widehat{\mathit{\xi }}}_1{\widehat{\mathit{\xi }}}_1^{\prime }-{\mathit{\xi }}_1{\mathit{\xi }}_1^{\prime })\left|\right|={\lambda }_1{O}_p(||\widehat{\mathit{\xi }}-\mathit{\xi }||)=O_p({\lambda }_1{T}^{-1/2}),$$

which can diverge when T = O(p²).

In the presence of very spiked eigenvalues, while the covariance Σ cannot be consistently estimated in absolute term, it can be well estimated in terms of the relative error matrix

$${\mathbf{\sum }}^{-1/2}\widehat{\mathbf{\sum }}{\widehat{\mathbf{\sum }}}^{-1/2}-{\mathbf{I}}_p$$

which is more relevant for many applications (see Example 5.2). The relative error matrix can be measured by either its spectral norm or the normalized Frobenius norm defined by

$$p^{-1/2}{\left|\right|{\mathbf{\sum }}^{-1/2}\widehat{\mathbf{\sum }}{\mathbf{\sum }}^{-1/2}-{\mathbf{I}}_p\left|\right|}_F={\left(p^{-1}tr[{({\mathbf{\sum }}^{-1/2}\widehat{\mathbf{\sum }}{\mathbf{\sum }}^{-1/2}-{\mathbf{I}}_p)}^2]\right)}^{1/2}.$$ (3.3)

In the last equality, there are p terms being added in the trace operation and the factor p⁻¹ plays the role of normalization. The loss (3.3) is closely related to the entropy loss, introduced by James and Stein (1961). Also note that

$$p^{-1/2}{\left|\right|{\mathbf{\sum }}^{-1/2}\widehat{\mathbf{\sum }}{\mathbf{\sum }}^{-1/2}-{\mathbf{I}}_p\left|\right|}_F={\left|\right|\widehat{\mathbf{\sum }}-\mathbf{\sum }\left|\right|}_{\mathrm{\sum }}$$

where ||A||_Σ = p^−1/2||Σ^−1/2AΣ^−1/2||_F is the weighted quadratic norm in Fan et al (2008).

Fan et al. (2008) showed that in a large factor model, the sample covariance is such that ${\left|\right|{\widehat{\mathbf{\sum }}}_{\text{sam}}-\mathbf{\sum }\left|\right|}_{\mathrm{\sum }}=O_p(\sqrt{p/T})$, which does not converge if p > T. On the other hand, Theorem 3.2 below shows that ||Σ̂_K̂ − Σ||_Σ can still be convergent as long as p = o(T²). Technically, the impact of high-dimensionality on the convergence rate of Σ̂_K̂ − Σ is via the number of rows in B. We show in the appendix that B appears in ||Σ̂_K̂ − Σ||_Σ through B′Σ⁻¹B whose eigenvalues are bounded. Therefore it successfully cancels out the curse of high-dimensionality introduced by B.

Compared to estimating Σ, in a large approximate factor model, we can estimate the precision matrix with a satisfactory rate under the spectral norm. The intuition follows from the fact that Σ⁻¹ has bounded eigenvalues.

The following theorem summarizes the rate of convergence under various norms.

Theorem 3.2

Under the assumptions of Theorem 3.1, the POET estimator defined in (2.15) satisfies

$${\left|\right|{\widehat{\mathbf{\sum }}}_{\widehat{K}}-\mathbf{\sum }\left|\right|}_{\mathrm{\sum }}=O_p\left(\frac{\sqrt{p}logp}{T}+m_p{\omega }_T^{1-q}\right),{\left|\right|{\widehat{\mathbf{\sum }}}_{\widehat{K}}-\mathbf{\sum }\left|\right|}_{max}=O_p({\omega }_T).$$

In addition, if $m_p{\omega }_T^{1-q}=o(1)$, then Σ̂_K̂ is nonsingular with probability approaching one, with

$$\left|\right|{\widehat{\mathbf{\sum }}}_{\widehat{K}}^{-1}-{\mathbf{\sum }}^{-1}\left|\right|=O_p(m_p{\omega }_T^{1-q}).$$

Remark 3.3

1. When estimating Σ⁻¹, p is allowed to grow exponentially fast in T, and the estimator has the same rate of convergence as that of the estimator ${\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}$ in Theorem 3.1. When p becomes much larger than T, the precision matrix can be estimated at the same rate as if the factors were observable.

2. As in Remark 3.2, when K > 0 is known and grows with p and T, the working paper Fan et al. (2011) proves the following results (when q = 0) ⁷:

   $$\begin{array}{l}{\left|\right|{\widehat{\mathbf{\sum }}}^{\mathcal{T}}-\mathbf{\sum }\left|\right|}_{\mathrm{\sum }}=O_p\left(\frac{K\sqrt{p}logp}{T}+K^2{m}_p\sqrt{\frac{logp}{T}}+\frac{m_p{K}^3}{\sqrt{p}}\right),\\ {\left|\right|{\widehat{\mathbf{\sum }}}^{\mathcal{T}}-\mathbf{\sum }\left|\right|}_{max}=O_p\left(K^3\sqrt{\frac{logp}{T}}+\frac{K^3}{\sqrt{p}}\right),\\ \left|\right|{({\widehat{\mathbf{\sum }}}^{\mathcal{T}})}^{-1}-{\mathbf{\sum }}^{-1}\left|\right|=O_p\left(K^2{m}_p\sqrt{\frac{logp}{T}}+\frac{K^3{m}_p}{\sqrt{p}}\right),\end{array}$$

   The results state explicitly the dependence of the rate of convergence on the number of factors.

3. The relative error ||Σ^−1/2 Σ̂_K̂ Σ^−1/2 − I_p|| in operator norm can be shown to have the same order as the maximum relative error of estimated eigenvalues. It does not converge to zero nor diverge. It is much smaller than ||Σ̂_K̂ − Σ||, which is of order $p/\sqrt{T}$ (see Example 3.1).

3.4 Convergence of unknown factors and factor loadings

Many applications of the factor model require estimating the unknown factors. In general, factor loadings in B and the common factors f_t are not separably identifiable, as for any matrix H such that H′H = I_K, Bf_t = BH′Hf_t. Hence (B, f_t) cannot be identified from (BH′, Hf_t). Note that the linear space spanned by the rows of B is the same as that by those of BH′. In practice, it often does not matter which one is used.

Let V denote the K̂ × K̂ diagonal matrix of the first K̂ largest eigenvalues of the sample covariance matrix in decreasing order. Recall that F′ = (f₁, …, f_T) and define a K̂ × K̂ matrix $\mathbf{H}=\frac{1}{T}{\mathbf{V}}^{-1}{\widehat{\mathbf{F}}}^{\prime }{\mathbf{FB}}^{\prime }\mathbf{B}$. Then for t ≤ T, Hf_t = T⁻¹V⁻¹F̂′(Bf₁, …, Bf_T)′Bf_t. Note that Hf_t depends only on the data V⁻¹F̂′ and an identifiable part of parameters ${\{{\mathbf{Bf}}_t\}}_{t=1}^T$. Therefore, there is no identifiability issue in Hf_t regardless of the imposed identifiability condition.

Bai (2003) obtained the rate of convergence for both b̂_i and f̂_t for any fixed (i, t). However, the uniform rate of convergence is more relevant for many applications (see Example 5.1). The following theorem extends those results in Bai (2003) in a uniformity sense. In particular, with a more refined technique, we have improved the uniform convergence rate for f̂_t.

Theorem 3.3

Under the assumptions of Theorem 3.1,

$$\underset{i\le p}{max}\left|\right|{\widehat{\mathbf{b}}}_i-{\mathbf{Hb}}_i\left|\right|=O_p({\omega }_T),\underset{t\le T}{max}\left|\right|{\widehat{\mathbf{f}}}_t-{\mathbf{Hf}}_t\left|\right|=O_p\left(\frac{1}{T^{1/2}}+\frac{T^{1/4}}{\sqrt{p}}\right).$$

As a consequence of Theorem 3.3, we obtain the following: (recall that the constant r₂ is defined in Assumption 3.2.)

Corollary 3.1

Under the assumptions of Theorem 3.1,

$$\underset{i\le p,t\le T}{max}\left|\right|{\widehat{\mathbf{b}}}_i^{\prime }{\widehat{\mathbf{f}}}_t-{\mathbf{b}}_i^{\prime }{\mathbf{f}}_t\left|\right|=O_p\left({(logT)}^{1/r_2}\sqrt{\frac{logp}{T}}+\frac{T^{1/4}}{\sqrt{p}}\right).$$

The rates of convergence obtained above also explain the condition T = o(p²) in Theorems 3.1 and 3.2. It is needed in order to estimate the common factors ${\{{\mathbf{f}}_t\}}_{t=1}^T$ uniformly in t ≤ T. When we do not observe ${\{{\mathbf{f}}_t\}}_{t=1}^T$, in addition to the factor loadings, there are KT factors to estimate. Intuitively, the condition T = o(p²) requires the number of parameters introduced by the unknown factors be “not too many”, so that we can consistently estimate them uniformly. Technically, as demonstrated by Bickel and Levina (2008), Cai and Liu (2011) and many other authors, achieving uniform accuracy is essential for large covariance estimations.

4 Choice of Threshold

4.1 Finite-sample positive definiteness

Recall that the threshold value ${\tau }_{ij}=C\sqrt{{\widehat{\theta }}_{ij}}{\omega }_T$, where C is determined by the users. To make POET operational in practice, one has to choose C to maintain the positive definiteness of the estimated covariances for any given finite sample. We write ${\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}(C)={\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}$, where the covariance estimator depends on C via the threshold. We choose C in the range where ${\lambda }_{min}({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})>0$. Define

$$C_{min}=inf\{C>0:{\lambda }_{min}({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}(M))>0,\forall M>C\}$$ (4.1)

When C is sufficiently large, the estimator becomes diagonal, while its minimum eigenvalue must retain strictly positive. Thus, C_min is well defined and for all C > C_min, ${\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}(C)$ is positive definite under finite sample. We can obtain C_min by solving ${\lambda }_{min}({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}(C))=0$, C ≠ 0. We can also approximate C_min by plotting ${\lambda }_{min}({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}(C))$ as a function of C, as illustrated in Figure 1. In practice, we can choose C in the range (C_min + ε, M) for a small ε and large enough M. Choosing the threshold in a range to guarantee the finite-sample positive definiteness has also been previously suggested by Fryzlewicz (2010).

Figure 1.

Minimum eigenvalue of ${\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}(C)$ as a function of C for three choices of thresholding rules. The plot is based on the simulated data set in Section 6.2.

4.2 Multifold Cross-Validation

In practice, C can be data-driven, and chosen through multifold cross-validation. After obtaining the estimated residuals {û_t}_t≤T by the PCA, we divide them randomly into two subsets, which are, for simplicity, denoted by {û_t}_t∈J1 and {û_t}_t∈J2. The sizes of J₁ and J₂, denoted by T(J₁) and T (J₂), are T (J₁) ≍ T and T (J₂) + T (J₁) = T. For example, in sparse matrix estimation, Bickel and Levina (2008) suggested to choose T(J₁) = T (1 − (log T)⁻¹).

We repeat this procedure H times. At the jth split, we denote by ${\widehat{\mathbf{\sum }}}_u^{\mathcal{T},j}(C)$ the POET estimator with the threshold $C\sqrt{{\theta }_{ij}}{\omega }_T$ on the training data set {û_t}_t∈J1. We also denote by ${\widehat{\mathbf{\sum }}}_u^j$ the sample covariance based on the validation set, defined by ${\widehat{\mathbf{\sum }}}_u^j=T{(J_2)}^{-1}{\sum }_{t\in J_2}{\widehat{\mathbf{u}}}_t{\widehat{\mathbf{u}}}_t^{\prime }$. Then we choose the constant C^* by minimizing a cross-validation objective function over a compact interval

$$C^{\ast }=arg\underset{C_{min}+\epsilon \le C\le M}{min}\frac{1}{H}\sum _{j=1}^H{\left|\right|{\widehat{\mathbf{\sum }}}_u^{\mathcal{T},j}(C)-{\widehat{\mathbf{\sum }}}_u^j\left|\right|}_F^2.$$ (4.2)

Here C_min is the minimum constant that guarantees the positive definiteness of ${\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}(C)$ for C > C_min as described in the previous subsection, and M is a large constant such that ${\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}(M)$ is diagonal. The resulting C^* is data-driven, so depends on Y as well as p and T via the data. On the other hand, for each given N × T data matrix Y, C^* is a universal constant in the threshold ${\tau }_{ij}=C^{\ast }\sqrt{{\widehat{\theta }}_{ij}}{\omega }_T$ in the sense that it does not change with respect to the position (i, j). We also note that the cross-validation is based on the estimate of Σ_u rather than Σ because POET thresholds the error covariance matrix. Thus cross-validation improves the performance of thresholding.

It is possible to derive the rate of convergence for ${\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}(C^{\ast })$ under the current model setting, but it ought to be much more technically involved than the regular sparse matrix estimation considered by Bickel and Levina (2008) and Cai and Liu (2011). To keep our presentation simple we do not pursue it in the current paper.

5 Applications of POET

We give four examples to which the results in Theorems 3.1–3.3 can be applied. Detailed pursuits of these are beyond the scope of the paper.

Example 5.1 (Large-scale hypothesis testing)

Controlling the false discovery rate in large-scale hypothesis testing based on correlated test statistics is an important and challenging problem in statistics (Leek and Storey, 2008; Efron, 2010; Fan, et al., 2012). Suppose that the test statistic for each of the hypothesis

$$H_{i0}:{\mu }_i=0\text{vs}.H_{i1}:{\mu }_i\ne 0$$

is Z_i ~ N (μ_i, 1) and these test statistics Z are jointly normal N (μ, Σ) where Σ is unknown. For a given critical value x, the false discovery proportion is then defined as FDP(x) = V (x)/R(x) where V (x) = p⁻¹ Σ_μi=0 I(|Z_i| > x) and $R(x)=p^{-1}{\sum }_{i=1}^{p}I(\mid Z_i\mid >x)$ are the total number of false discoveries and the total number of discoveries, respectively. Our interest is to estimate FDP(x) for each given x. Note that R(x) is an observable quantity. Only V (x) needs to be estimated.

If the covariance Σ admits the approximate factor structure (1.3), then the test statistics can be stochastically decomposed as

$$\mathbf{Z}=\mathit{\mu }+\mathbf{Bf}+\mathbf{u},\text{where}{\mathbf{\sum }}_u\text{is}\text{sparse}.$$ (5.1)

By the principal factor approximation (Theorem 1, Fan, Han, Gu, 2012)

$$V(x)=\sum _{i=1}^p\{\mathrm{\Phi }(a_i(z_{x/2}+{\eta }_i))+\mathrm{\Phi }(a_i(z_{x/2}-{\eta }_i))\}+o_P(p),$$ (5.2)

when m_p = o(p) and the number of true significant hypothesis {i : μ_i ≠ 0} is o(p), where z_x is the upper x-quantile of the standard normal distribution, η_i = (Bf)_i and a_i = var(u_i)⁻¹.

Now suppose that we have n repeated measurements from the model (5.1). Then, by Corollary 3.1, {η_i} can be uniformly consistently estimated, and hence p⁻¹V (x) and FDP(x) can be consistently estimated. Efron (2010) obtained these repeated test statistics based on the bootstrap sample from the original raw data. Our theory (Theorem 3.3) gives a formal justification to the framework of Efron (2007, 2010).

Example 5.2 (Risk management)

The maximum elementwise estimation error ||Σ̂_K̂ − Σ||_max appears in risk assessment as in Fan, Zhang and Yu (2012). For a fixed portfolio allocation vector w, the true portfolio variance and the estimated one are given by w′Σw and w′Σ̂_K̂w respectively. The estimation error is bounded by

$$\mid {\mathbf{w}}^{\prime }{\widehat{\mathbf{\sum }}}_{\widehat{K}}\mathbf{w}-{\mathbf{w}}^{\prime }\mathbf{\sum }\mathbf{w}\mid \le {\left|\right|{\widehat{\mathbf{\sum }}}_{\widehat{K}}-\mathbf{\sum }\left|\right|}_{max}{\left|\right|\mathbf{w}\left|\right|}_1^2,$$

where ||w||₁, the L₁-norm of w, is the gross exposure of the portfolio. Usually a constraint is placed on the total percentage of the short positions, in which case we have a restriction ||w||₁ ≤ c for some c > 0. In particular, c = 1 corresponds to a portfolio with no-short positions (all weights are nonnegative). Theorem 3.2 quantifies the maximum approximation error.

The above compares the absolute error of perceived risk and true risk. The relative error is bounded by

$$\mid {\mathbf{w}}^{\prime }{\widehat{\mathbf{\sum }}}_{\widehat{K}}\mathbf{w}/{\mathbf{w}}^{\prime }\mathbf{\sum }\mathbf{w}-1\mid \le \left|\right|{\mathbf{\sum }}^{-1/2}{\widehat{\mathbf{\sum }}}_{\widehat{K}}{\mathbf{\sum }}^{-1/2}-{\mathbf{I}}_p\left|\right|$$

for any allocation vector w. Theorem 3.2 quantifies this relative error.

Example 5.3 (Panel regression with a factor structure in the errors)

Consider the following panel regression model

$$Y_{it}={\mathbf{x}}_{it}^{\prime }\mathit{\beta }+{\epsilon }_{it},{\epsilon }_{it}={\mathbf{b}}_i^{\prime }{\mathbf{f}}_t+u_{it},i\le p,t\le T,$$

where x_it is a vector of observable regressors with fixed dimension. The regression error ε_it has a factor structure and is assumed to be independent of x_it, but b_i, f_t and u_it are all unobservable. We are interested in the common regression coefficients β. The above panel regression model has been considered by many researchers, such as Ahn, Lee and Schmidt (2001), Pesaran (2006), and has broad applications in social sciences.

Although OLS (ordinary least squares) produces a consistent estimator of β, a more efficient estimation can be obtained by GLS (generalized least squares). The GLS method depends, however, on an estimator of ${\mathbf{\sum }}_{\epsilon }^{-1}$, the inverse of the covariance matrix of ε_t = (ε_1t, …, ε_pt)′. By assuming the covariance matrix of (u_1t, …, u_pt) to be sparse, we can successfully solve this problem by applying Theorem 3.2. Although ε_it is unobservable, it can be replaced by the regression residuals ε̂_it, obtained via first regressing Y_it on x_it. We then apply the POET estimator to $T^{-1}{\sum }_{t=1}^T{\widehat{\epsilon }}_t{\widehat{\epsilon }}_t^{\prime }$. By Theorem 3.2, the inverse of the resulting estimator is a consistent estimator of ${\mathbf{\sum }}_{\epsilon }^{-1}$ under the spectral norm. A slight difference lies in the fact that when we apply POET, $T^{-1}{\sum }_{t=1}^T{\mathit{\epsilon }}_t{\mathit{\epsilon }}_t^{\prime }$ is replaced with $T^{-1}{\sum }_{t=1}^T{\widehat{\mathit{\epsilon }}}_t{\widehat{\mathit{\epsilon }}}_t^{\prime }$, which introduces an additional term $O_p(\sqrt{\frac{logp}{T}})$ in the estimation error.

Example 5.4 (Validating an asset pricing theory)

A celebrated financial economic theory is the capital asset pricing model (CAPM, Sharpe 1964) that makes William Sharpe win the Nobel prize in Economics in 1990, whose extension is the multi-factor model (Ross, 1976, Chamberlain and Rothschild, 1983). It states that in a frictionless market, the excessive return of any financial asset equals the excessive returns of the risk factors times its factor loadings plus noises. In the multi-period model, the excess return y_it of firm i at time t follows model (1.1), in which f_t is the excess returns of the risk factors at time t. To test the null hypothesis (1.2), one embeds the model into the multivariate linear model

$${\mathbf{y}}_t=\mathit{\alpha }+{\mathbf{Bf}}_t+{\mathbf{u}}_t,t=1,\cdots ,T$$ (5.3)

and wishes to test H₀ : α = 0. The F-test statistic involves the estimation of the covariance matrix Σ_u, whose estimates are degenerate without regularization when p ≥ T. Therefore, in the literature (Sentana, 2009, and references therein), one focuses on the case p is relatively small. The typical choices of parameters are T = 60 monthly data and the number of assets p = 5, 10 or 25. However, the CAPM should hold for all tradeable assets, not just a small fraction of assets. With our regularization technique, non-degenerate estimate ${\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}$ can be obtained and the F-test or likelihood-ratio test statistics can be employed even when p ≫ T.

To provide some insights, let α̂ be the least-squares estimator of (5.3). Then, when u_t ~ N (0, Σ_u), α̂ ~ N(α, Σ_u/c_T) for a constant c_T which depends on the observed factors. When Σ_u is known, the Wald test statistic is $W=c_T{\widehat{\mathit{\alpha }}}^{\prime }{\mathbf{\sum }}_u^{-1}\widehat{\mathit{\alpha }}$. When it is unknown and p is large, it is natural to use the F-type of test statistic $\widehat{W}=c_T{\widehat{\mathit{\alpha }}}^{\prime }{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}\widehat{\mathit{\alpha }}$. The difference between these two statistics is bounded by

$$\mid \widehat{W}-W\mid \le c_T\left|\right|{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}-{\mathbf{\sum }}_u^{-1}\left|\right|{\left|\right|\widehat{\mathit{\alpha }}\left|\right|}^2.$$

Since under the null hypothesis α̂ ~ N(0, Σ_u/c_T), we have $c_T{\left|\right|{\mathbf{\sum }}_u^{-1/2}\widehat{\mathit{\alpha }}\left|\right|}^2=O(p)$. Thus, it follows from boundness of ||Σ_u|| that $\mid \widehat{W}-W\mid =O(p)\left|\right|{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}-{\mathbf{\sum }}_u^{-1}\left|\right|$. Theorem 3.1 provides the rate of convergence for the above difference. Detailed development is out of the scope of the current paper, and we will leave it as a separate research project.

6 Monte Carlo Experiments

In this section, we will examine the performance of the POET method in a finite sample. We will also demonstrate the effect of this estimator on the asset allocation and risk assessment. Similarly to Fan, et al. (2008, 2011), we simulated from a standard Fama-French three-factor model, assuming a sparse error covariance matrix and three factors. Throughout this section, the time span is fixed at T = 300, and the dimensionality p increases from 1 to 600. We assume that the excess returns of each of p stocks over the risk-free interest rate follow the following model:

$$y_{it}=b_{i1}{f}_{1t}+b_{i2}{f}_{2t}+b_{i3}{f}_{3t}+u_{it}.$$

The factor loadings are drawn from a trivariate normal distribution b ~ N₃(μ_B, Σ_B), the idiosyncratic errors from u_t ~ N_p(0, Σ_u), and the factor returns f_t follow a VAR(1) model. To make the simulation more realistic, model parameters are calibrated from the financial returns, as detailed in the following section.

6.1 Calibration

To calibrate the model, we use the data on annualized returns of 100 industrial portfolios from the website of Kenneth French, and the data on 3-month Treasury bill rates from the CRSP database. These industrial portfolios are formed as the intersection of 10 portfolios based on size (market equity) and 10 portfolios based on book equity to market equity ratio. Their excess returns (ỹ_t) are computed for the period from January 1^st, 2009 to December 31^st, 2010. Here, we present a short outline of the calibration procedure.

1. Given ${\{{\stackrel{\sim }{\mathbf{y}}}_t\}}_{t=1}^{500}$ as the input data, we fit a Fama-French-three-factor model and calculate a 100 × 3 matrix B̃, and 500 × 3 matrix F̃, using the principal components method described in Section 3.1.

2. We summarize 100 factor loadings (the rows of B̃) by their sample mean vector μ_B and sample covariance matrix Σ_B, which are reported in Table 1. The factor loadings b_i = (b_i1, b_i2, b_i3)^T for i = 1, …, p are drawn from N₃(μ_B, Σ_B).

3. We run the stationary vector autoregressive model f_t = μ + Φf_t−1 + ε_t, a VAR(1) model, to the data F̃ to obtain the multivariate least squares estimator for μ and Φ, and estimate Σ_ε. Note that all eigenvalues of Φ in Table 2 fall within the unit circle, so our model is stationary. The covariance matrix cov(f_t) can be obtained by solving the linear equation cov(f_t) = Φcov(f_t) Φ′ + Σ_ε. The estimated parameters are depicted in Table 2 and are used to generate f_t.

4. For each value of p, we generate a sparse covariance matrix Σ_u of the form:

   $${\mathbf{\sum }}_u=\mathbf{D}{\mathbf{\sum }}_0\mathbf{D}.$$

   Here, Σ₀ is the error correlation matrix, and D is the diagonal matrix of the standard deviations of the errors. We set D = diag(σ₁, …, σ_p), where each σ_i is generated independently from a Gamma distribution G(α, β), and α and β are chosen to match the sample mean and sample standard deviation of the standard deviations of the errors. A similar approach to Fan et al. (2011) has been used in this calibration step. The off-diagonal entries of Σ₀ are generated independently from a normal distribution, with mean and standard deviation equal to the sample mean and sample standard deviation of the sample correlations among the estimated residuals, conditional on their absolute values being no larger than 0.95. We then employ hard thresholding to make Σ₀ sparse, where the threshold is found as the smallest constant that provides the positive definiteness of Σ₀. More precisely, start with threshold value 1, which gives Σ₀ = I_p and then decrease the threshold values in a grid until positive definiteness is violated.

Table 1.

Mean and covariance matrix used to generate b

μB	ΣB
0.0047	0.0767	−0.00004	0.0087
0.0007	−0.00004	0.0841	0.0013
−1.8078	0.0087	0.0013	0.1649

Table 2.

Parameters of f_t generating process

μ	cov(ft)			Φ
−0.0050	1.0037	0.0011	−0.0009	−0.0712	0.0468	0.1413
0.0335	0.0011	0.9999	0.0042	−0.0764	−0.0008	0.0646
−0.0756	−0.0009	0.0042	0.9973	0.0195	−0.0071	−0.0544

6.2 Simulation

For the simulation, we fix T = 300, and let p increase from 1 to 600. For each fixed p, we repeat the following steps N = 200 times, and record the means and the standard deviations of each respective norm.

1. Generate independently ${\{{\mathbf{b}}_i\}}_{i=1}^p~N_3({\mathit{\mu }}_B,{\mathbf{\sum }}_B)$, and set B = (b₁, …, b_p)′.

2. Generate independently ${\{{\mathbf{u}}_t\}}_{t=1}^T~N_p(\mathbf{0},{\mathbf{\sum }}_{\mathbf{u}})$.

3. Generate ${\{{\mathbf{f}}_t\}}_{t=1}^T$ as a vector autoregressive sequence of the form f_t = μ + Φf_t−1 + ε_t.

4. Calculate ${\{{\mathbf{y}}_t\}}_{t=1}^T$ from y_t = Bf_t + u_t.

5. Set hard-thresholding with threshold $0.5\sqrt{{\widehat{\theta }}_{ij}}(\sqrt{\frac{logp}{T}}+\frac{1}{\sqrt{p}})$. Estimate K using Bai and Ng (2002)’s IC1. Calculate covariance estimators using the POET method. Calculate the sample covariance matrix Σ̂_sam.

In the graphs below, we plot the averages and standard deviations of the distance from Σ̂_K̂ and Σ̂_sam to the true covariance matrix Σ, under norms ||.||_Σ, ||.|| and ||.||_max. We also plot the means and standard deviations of the distances from (Σ̂_K̂)⁻¹ and ${\widehat{\mathbf{\sum }}}_{\text{sam}}^{-1}$ to Σ⁻¹ under the spectral norm. The dimensionality p ranges from 20 to 600 in increments of 20. Due to invertibility, the spectral norm for ${\widehat{\mathbf{\sum }}}_{\text{sam}}^{-1}$ is plotted only up to p = 280. Also, we zoom into these graphs by plotting the values of p from 1 to 100, this time in increments of 1. Notice that we also plot the distance from Σ̂_obs to Σ for comparison, where Σ̂_obs is the estimated covariance matrix proposed by Fan et al. (2011), assuming the factors are observable.

6.3 Results

In a factor model, we expect POET to perform as well as Σ̂_obs when p is relatively large, since the effect of estimating the unknown factors should vanish as p increases. This is illustrated in the plots below.

From the simulation results, reported in Figures 2–5, we observe that POET under the unobservable factor model performs just as well as the estimator in Fan et al. (2011) if the factors are known, when p is large enough. The cost of not knowing the factors is approximately of order $O_p(1/\sqrt{p})$. It can be seen in Figures 2 and 3 that this cost vanishes for p ≥ 200. To give a better insight of the impact of estimating the unknown factors for small p, a separate set of simulations is conducted for p ≤ 100. As we can see from Figures 2 (bottom panel) and 3 (middle and bottom panels), the impact decreases quickly. In addition, when estimating Σ⁻¹, it is hard to distinguish the estimators with known and unknown factors, whose performances are quite stable compared to the sample covariance matrix. Also, the maximum absolute elementwise error (Figure 4) of our estimator performs very similarly to that of the sample covariance matrix, which coincides with our asymptotic result. Figure 5 shows that the performances of the three methods are indistinguishable in the spectral norm, as expected.

Figure 2.

Averages (left panel) and standard deviations (right panel) of the relative error p^−1/2||Σ^−1/2Σ̂Σ^−1/2 − I_p||_F with known factors (Σ̂ = Σ̂_obs solid red curve), POET (Σ̂ = Σ̂_K̂ solid blue curve), and sample covariance (Σ̂ = Σ̂_sam dashed curve) over 200 simulations, as a function of the dimensionality p. Top panel: p ranges in 20 to 600 with increment 20; bottom panel: p ranges in 1 to 100 with increment 1.

Figure 5.

Averages of ||Σ̂ − Σ|| (left panel) and ||Σ^−1/2Σ̂Σ^−1/2 − I_p|| with known factors (Σ̂ = Σ̂_obs solid red curve), POET (Σ̂= Σ̂_K̂ ω solid blue curve), and sample covariance (Σ̂ = Σ̂_sam dashed curve) over 200 simulations, as a function of the dimensionality p. The three curves are hardly distinguishable on the left panel.

Figure 3.

Averages (left panel) and standard deviations (right panel) of ||Σ̂⁻¹ − Σ⁻¹|| with known factors (Σ̂ = Σ̂_obs solid red curve), POET (Σ̂ = Σ̂_K̂ solid blue curve), and sample covariance (Σ̂ = Σ̂_sam dashed curve) over 200 simulations, as a function of the dimensionality p. Top panel: p ranges in 20 to 600 with increment 20; middle panel: p ranges in 1 to 100 with increment 1; Bottom panel: the same as the top panel with dashed curve excluded.

Figure 4.

Averages (left panel) and standard deviations (right panel) of ||Σ̂ − Σ||_max with known factors (Σ̂ = Σ̂_obs solid red curve), POET (Σ̂ = Σ̂_K̂ solid blue curve), and sample covariance (Σ̂ = Σ̂_sam dashed curve) over 200 simulations, as a function of the dimensionality p. They are nearly indifferentiable.

6.4 Robustness to the estimation of K

The POET estimator depends on the estimated number of factors. Our theory uses a consistent esimator K̂. To assess the robustness of our procedure to K̂ in finite sample, we calculate ${\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}}$ for K = 1, 2, …, 10. Again, the threshold is fixed to be $0.5\sqrt{{\widehat{\theta }}_{ij}}(\sqrt{\frac{logp}{T}}+\frac{1}{\sqrt{p}})$.

6.4.1 Design 1

The simulation setup is the same as before where the true K₀ = 3. We calculate $\left|\right|{\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}}-{\mathbf{\sum }}_u\left|\right|,\left|\right|{({\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}})}^{-1}-{\mathbf{\sum }}_u^{-1}\left|\right|,\left|\right|{\widehat{\mathbf{\sum }}}_K^{-1}-{\mathbf{\sum }}^{-1}\left|\right|$ and ||Σ̂_K − Σ||_Σ for K = 1, 2, …, 10. Figure 6 plots these norms as p increases but with a fixed T = 300. The results demonstrate a trend that is quite robust when K ≥ 3; especially, the estimation accuracy of the spectral norms for large p are close to each other. When K = 1 or 2, the estimators perform badly due to modeling bias. Therefore, POET is robust to over-estimated K, but not to under-estimation.

Figure 6.

Robustness of K as p increases for various choices of K (Design 1, T = 300). Top left: || ${\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}}-{\mathbf{\sum }}_u$||; top right: || ${({\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}})}^{-1}-{\mathbf{\sum }}_u^{-1}$||; bottom left: ||Σ̂_K − Σ||_Σ; bottom right: || ${\widehat{\mathbf{\sum }}}_K^{-1}-{\mathbf{\sum }}^{-1}$||.

6.4.2 Design 2

We also simulated from a new data generating process for the robustness assessment. Consider a banded idiosyncratic matrix

$${\sigma }_{u,ij}=\{\begin{array}{ll}{0.5}^{\mid i-j\mid },\hfill & \mid i-j\mid \le 9\hfill \\ 0,\hfill & \mid i-j\mid >9\hfill \end{array},({\mathbf{u}}_1,\dots ,{\mathbf{u}}_T){~}^{i.i.d.}{N}_p(0,{\mathbf{\sum }}_u).$$

We still consider a K₀ = 3 factor model, where the factors are independently simulated as

$$f_{it}~N(0,1),b_{ji}~N(0,1)i\le 3,j\le p,t\le T,$$

Table 3 summarizes the average estimation error of covariance matrices across K in the spectral norm. Each simulation is replicated 50 times and T = 200.

Table 3.

Robustness of K. Design 2, estimation errors in spectral norm

		K
		1	2	3	4	5	6	8
p = 100	${\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}}$	10.70	5.23	1.63	1.80	1.91	2.04	2.22
	${({\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}})}^{-1}$	2.71	2.51	1.51	1.50	1.44	1.84	2.82
	${\widehat{\mathbf{\sum }}}_K^{-1}$	2.69	2.48	1.47	1.49	1.41	1.56	2.35
	Σ̂K	94.66	91.36	29.41	31.45	30.91	33.59	33.48
	Σ−1/2 Σ̂K Σ−1/2	17.37	10.04	2.05	2.83	2.94	2.95	2.93
p = 200	${\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}}$	11.34	11.45	1.64	1.71	1.79	1.87	2.01
	${({\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}})}^{-1}$	2.69	3.91	1.57	1.56	1.81	2.26	3.42
	${\widehat{\mathbf{\sum }}}_K^{-1}$	2.67	3.72	1.57	1.55	1.70	2.13	3.19
	Σ̂K	200.82	195.64	57.44	63.09	64.53	60.24	56.20
	Σ−1/2 Σ̂K Σ−1/2	20.86	14.22	3.29	4.52	4.72	4.69	4.76
p = 300	${\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}}$	12.74	15.20	1.66	1.71	1.78	1.84	1.95
	${({\widehat{\mathbf{\sum }}}_{u,K}^{\mathcal{T}})}^{-1}$	7.58	7.80	1.74	2.18	2.58	3.54	5.45
	${\widehat{\mathbf{\sum }}}_K^{-1}$	7.59	7.49	1.70	2.13	2.49	3.37	5.13
	Σ̂K	302.16	274.12	87.92	92.47	91.90	83.21	92.50
	Σ−1/2 Σ̂K Σ−1/2	23.43	16.89	4.38	6.04	6.16	6.14	6.20

Table 3 illustrates some interesting patterns. First of all, the best estimation accuracy is achieved when K = K₀. Second, the estimation is robust for K ≥ K₀. As K increases from K₀, the estimation error becomes larger, but is increasing slowly in general, which indicates the robustness when a slightly larger K has been used. Third, when the number of factors is under-estimated, corresponding to K = 1, 2, all the estimators perform badly, which demonstrates the danger of missing any common factors. Therefore, over-estimating the number of factors, while still maintaining a satisfactory estimation accuracy of the covariance matrices, is much better than under-estimating. The resulting bias caused by under-estimation is more severe than the additional variance introduced by over-estimation. Finally, estimating Σ, the covariance of y_t, does not achieve a good accuracy even when K = K₀ in the absolute term ||Σ̂ − Σ||, but the relative error ||Σ^−1/2 Σ̂_KΣ^−1/2 − I_p|| is much smaller. This is consistent with our discussions in Section 3.3.

6.5 Comparisons with Other Methods

6.5.1 Comparison with related methods

We compare POET with related methods that address low-rank plus sparse covariance estimation, specifically, LOREC proposed by Luo (2012), the strict factor model (SFM) by Fan, Fan and Lv (2008), the Dual Method (Dual) by Lin et al. (2009), and finally, the singular value thresholding (SVT) by Cai, Candès and Shen (2008). In particular, SFM is a special case of POET which employs a large threshold that forces Σ̂_u to be diagonal even when the true Σ_u might not be. Note that Dual, SVT and many others dealing with low-rank plus sparse, such as Candès et al. (2011) and Wright et al. (2009), assume a known Σ and focus on recovering the decomposition. Hence they do not estimate Σ or its inverse, but decompose the sample covariance into two components. The resulting sparse component may not be positive definite, which can lead to large estimation errors for ${\widehat{\mathbf{\sum }}}_u^{-1}$ and Σ̂⁻¹.

Data are generated from the same setup as Design 2 in Section 6.4. Table 4 reports the averaged estimation error of the four comparing methods, calculated based on 50 replications for each simulation. Dual and SVT assume the data matrix has a low-rank plus sparse representation, which is not the case for the sample covariance matrix (though the population Σ has such a representation). The tuning parameters for POET, LOREC, Dual and SVT are chosen to achieve the best performance for each method.⁸

Table 4.

Method Comparison under spectral norm for T = 100. RelE represents the relative error ||Σ^−1/2Σ̂Σ^−1/2 − I_p||

		Σ̂u	${\widehat{\mathrm{\sum }}}_u^{-1}$	RelE	Σ̂−1	Σ̂
p = 100	POET	1.624	1.336	2.080	1.309	29.107
	LOREC	2.274	1.880	2.564	1.511	32.365
	SFM	2.084	2.039	2.707	2.022	34.949
	Dual	2.306	5.654	2.707	4.674	29.000
	SVT	2.59	13.64	2.806	103.1	29.670
p = 200	POET	1.641	1.358	3.295	1.346	58.769
	LOREC	2.179	1.767	3.874	1.543	62.731
	SFM	2.098	2.071	3.758	2.065	60.905
	Dual	2.41	6.554	4.541	5.813	56.264
	SVT	2.930	362.5	4.680	47.21	63.670
p = 300	POET	1.662	1.394	4.337	1.395	65.392
	LOREC	2.364	1.635	4.909	1.742	91.618
	SFM	2.091	2.064	4.874	2.061	88.852
	Dual	2.475	2.602	6.190	2.234	74.059
	SVT	2.681	> 103	6.247	> 103	80.954

6.5.2 Comparison with direct thresholding

This section compares POET with direct thresholding on the sample covariance matrix without taking out common factors (Rothman et al. 2009, Cai and Liu 2011. We denote this method by THR). We also run simulations to demonstrate the finite sample performance when Σ itself is sparse and has bounded eigenvalues, corresponding to the case K = 0. Three models are considered and both POET and THR use the soft thresholding. We fix T = 200. Reported results are the average of 100 replications.

Model 1: one-factor

The factors and loadings are independently generated from N(0, 1). The error covariance is the same banded matrix as Design 2 in Section 6.4. Here Σ has one diverging eigenvalue.

Model 2: sparse covariance

Set K = 0, hence Σ = Σ_u itself is a banded matrix with bounded eigenvalues.

Model 3: cross-sectional AR(1)

Set K = 0, but Σ = Σ_u = (0.85^|i−j|)_p×p. Now Σ is no longer sparse (or banded), but is not too dense either since Σ_ij decreases to zero exponentially fast as |i − j| → ∞. This is the correlation matrix if ${\{y_{it}\}}_{i=1}^p$ follows a cross-sectional AR(1) process: y_it = 0.85y_i−1,t + ε_it.

For each model, POET uses an estimated K̂ based on IC1 of Bai and Ng (2002), while THR thresholds the sample covariance directly. We find that in Model 1, POET performs significantly better than THR as the latter misses the common factor. For Model 2, IC1 estimates K̂ = 0 precisely in each replication, and hence POET is identical to THR. For Model 3, POET still outperforms. The results are summarized in Table 5.

Table 5.

Method Comparison. T = 200

		||Σ̂ − Σ||		||Σ̂−1 − Σ−1||		K̂
		POET	THR	POET	THR
p = 200	Model 1	26.20	240.18	1.31	2.67	1
	Model 2	2.04	2.04	2.07	2.07	0
	Model 3	7.73	11.24	8.48	11.40	6.2
p = 300	Model 1	32.60	314.43	2.18	2.58	1
	Model 2	2.03	2.03	2.08	2.08	0
	Model 3	9.41	11.29	8.81	11.41	5.45

The reported numbers are the averages based on 100 replications.

6.6 Simulated portfolio allocation

We demonstrate the improvement of our method compared to the sample covariance and that based on the strict factor model (SFM), in a problem of portfolio allocation for risk minimization purposes.

Let Σ̂ be a generic estimator of the covariance matrix of the return vector y_t, and w be the allocation vector of a portfolio consisting of the corresponding p financial securities. Then the theoretical and the empirical risk of the given portfolio are R(w) = w′Σw and R̂ (w) = w′Σ̂w, respectively. Now, define

$$\widehat{\mathbf{w}}={\text{argmin}}_{{\mathbf{w}}^{\prime }1=1}{\mathbf{w}}^{\prime }\widehat{\mathbf{\sum }}\mathbf{w},$$

the estimated (minimum variance) portfolio. Then the actual risk of the estimated portfolio is defined as R(ŵ) = ŵ′Σŵ, and the estimated risk (also called empirical risk) is equal to R̂ (ŵ) = ŵ′Σ̂ŵ. In practice, the actual risk is unknown, and only the empirical risk can be calculated.

For each fixed p, the population Σ was generated in the same way as described in Section 6.1, with a sparse but not diagonal error covariance. We use three different methods to estimate Σ and obtain ŵ: strict factor model Σ̂_diag (estimate Σ_u using a diagonal matrix), our POET estimator Σ̂_POET, both are with unknown factors, and sample covariance Σ̂_Sam. We then calculate the corresponding actual and empirical risks.

It is interesting to examine the accuracy and the performance of the actual risk of our portfolio ŵ in comparison to the oracle risk R^* = min_w′1=1 w′Σw, which is the theoretical risk of the portfolio we would have created if we knew the true covariance matrix Σ. We thus compare the regret R(ŵ) − R^*, which is always nonnegative, for three estimators of Σ̂. They are summarized by using the box plots over the 200 simulations. The results are reported in Figure 7. In practice, we are also concerned about the difference between the actual and empirical risk of the chosen portfolio ŵ. Hence, in Figure 8, we also compare the average estimation error |R(ŵ) − R̂ (ŵ)| and the average relative estimation error |R̂ (ŵ)/R(ŵ) − 1| over 200 simulations. When ŵ is obtained based on the strict factor model, both differences - between actual and oracle risk, and between actual and empirical risk, are persistently greater than the corresponding differences for the approximate factor estimator. Also, in terms of the relative estimation error, the factor model based method is negligible, where as the sample covariance does not process such a property.

Figure 7.

Box plots of regrets R(ŵ) − R^* for p = 80 and 140. In each panel, the box plots from left to right correspond to ŵ obtained using Σ̂ based on approximate factor model, strict factor model, and sample covariance, respectively.

Figure 8.

Estimation errors for risk assessments as a function of the portfolio size p. Left panel plots the average absolute error |R(ŵ) − R̂ (ŵ)| and right panel depicts the average relative error |R̂ (ŵ)/R(ŵ) − 1|. Here, ŵ and R̂ are obtained based on three estimators of Σ̂.

7 Real Data Example

We demonstrate the sparsity of the approximate factor model on real data, and present the improvement of the POET estimator over the strict factor model (SFM) in a real-world application of portfolio allocation.

7.1 Sarsity of Idiosyncratic Errors

The data were obtained from the CRSP (The Center for Research in Security Prices) database, and consists of p = 50 stocks and their annualized daily returns for the period January 1^st, 2010-December 31^st, 2010 (T = 252). The stocks are chosen from 5 different industry sectors, (more specifically, Consumer Goods-Textile & Apparel Clothing, Financial-Credit Services, Healthcare-Hospitals, Services-Restaurants, Utilities-Water utilities), with 10 stocks from each sector. We made this selection to demonstrate a block diagonal trend in the sparsity. More specifically, we show that the non-zero elements are clustered mainly within companies in the same industry. We also notice that these are the same groups that show predominantly positive correlation.

The largest eigenvalues of the sample covariance equal 0.0102, 0.0045 and 0.0039, while the rest are bounded by 0.0020. Hence K = 0, 1, 2, 3 are the possible values of the number of factors. Figure 9 shows the heatmap of the thresholded error correlation matrix (for simplicity, we applied hard thresholding). The threshold has been chosen using the cross validation as described in Section 4. We compare the level of sparsity (percentage of non-zero off-diagonal elements) for the 5 diagonal blocks of size 10 × 10, versus the sparsity of the rest of the matrix. For K = 2, our method results in 25.8% non-zero off-diagonal elements in the 5 diagonal blocks, as opposed to 7.3% non-zero elements in the rest of the covariance matrix. Note that, out of the non-zero elements in the central 5 blocks, 100% are positive, as opposed to a distribution of 60.3% positive and 39.7% negative amongst the non-zero elements in off-diagonal blocks. There is a strong positive correlation between the returns of companies in the same industry after the common factors are taken out, and the thresholding has preserved them. The results for K = 1, 2 and 3 show the same characteristics. These provide stark evidence that the strict factor model is not appropriate.

Figure 9.

Heatmap of thresholded error correlation matrix for number of factors K = 0, K = 1, K = 2 and K = 3.

7.2 Portfolio Allocation

We extend our data size by including larger industrial portfolios (p = 100), and longer period (ten years): January 1^st, 2000 to December 31^st, 2010 of annualized daily excess returns. Two portfolios are created at the beginning of each month, based on two different covariance estimates through approximate and strict factor models with unknown factors. At the end of each month, we compare the risks of both portfolios.

The number of factors is determined using the penalty function proposed by Bai and Ng (2002), as defined in (2.14). For calibration, we use the last 100 consecutive business days of the above data, and both IC1 and IC2 give K̂ = 3. On the 1^st of each month, we estimate Σ̂_diag (SFM) and Σ̂_K̂ (POET with soft thresholding) using the historical data of excess daily returns for the proceeding 12 months (T = 252). The value of the threshold is determined using the cross-validation procedure. We minimize the empirical risk of both portfolios to obtain the two respective optimal portfolio allocations ŵ = ŵ₁ and ŵ₂ (based on Σ̂ = Σ̂_diag and Σ̂_K̂): ŵ = arg min_w̃′1=1 w′Σ̂w. At the end of the month (21 trading days), their actual risks are compared, calculated by

$$R_i={\widehat{\mathbf{w}}}_i^{\prime }\frac{1}{21}\sum _{t=1}^{21}{\mathbf{y}}_t{\mathbf{y}}_t^{\prime }{\widehat{\mathbf{w}}}_i,\text{for}i=1,2.$$

We can see from Figure 10 that the minimum-risk portfolio created by the POET estimator performs significantly better, achieving lower variance 76% of the time. Amongst those months, the risk is decreased by 48.63%. On the other hand, during the months that POET produces a higher-risk portfolio, the risk is increased by only 17.66%.

Figure 10.

Risk of portfolios created with POET and SFM (strict factor model)

Next, we demonstrate the impact of the choice of number of factors and threshold on the performance of POET. If cross-validation seems computationally expensive, we can choose a common soft-threshold throughout the whole investment process. The average constant in the cross-validation was 0.53, close to our suggested constant 0.5 used for simulation. We also present the results based on various choices of constant C = 0.5, 0.75, 1 and 1.25, with soft threshold $C\sqrt{{\widehat{\theta }}_{ij}}{\omega }_T$. The results are summarized in Table 6. The performance of POET seems consistent across different choices of these parameters.

Table 6.

Comparisons of the risks of portfolios using POET and SFM: The first number is proportion of the time POET outperforms and the second number is percentage of average risk improvements. C represents the constant in the threshold.

C	K̂ = 1	K̂ = 2	K̂ = 3
0.25	0.58/29.6%	0.68/38%	0.71/33%
0.5	0.66/31.7%	0.70/38.2%	0.75/33.5%
0.75	0.68/29.3%	0.70/29.6%	0.71/25.1%
1	0.66/20.7%	0.62/19.4%	0.69/18%

8 Conclusion and Discussion

We study the problem of estimating a high-dimensional covariance matrix with conditional sparsity. Realizing unconditional sparsity assumption is inappropriate in many applications, we introduce a latent factor model that has a conditional sparsity feature, and propose the POET estimator to take advantage of the structure. This expands considerably the scope of the model based on the strict factor model, which assumes independent idiosyncratic noise and is too restrictive in practice. By assuming sparse error covariance matrix, we allow for the presence of the cross-sectional correlation even after taking out the common factors. The sparse covariance is estimated by the adaptive thresholding technique.

It is found that the rates of convergence of the estimators have an extra term approximately O_p(p^−1/2) in addition to the results based on observable factors by Fan et al. (2008, 2011), which arises from the effect of estimating the unobservable factors. As we can see, this effect vanishes as the dimensionality increases, as more information about the common factors becomes available. When p gets large enough, the effect of estimating the unknown factors is negligible, and we estimate the covariance matrices as if we knew the factors.

The proposed POET also has wide applicability in statistical genomics. For example, Carvalho et al. (2008) applied a Bayesian sparse factor model to study the breast cancer hormonal pathways. Their real-data results have identified about two common factors that have highly loaded genes (about half of 250 genes). As a result, these factors should be treated as “pervasive” (see the explanation in Example 2.1), which will result in one or two very spiked eigenvalues of the gene expressions’ covariance matrix. The POET can be applied to estimate such a covariance matrix and its network model.

APPENDIX

A Estimating a sparse covariance with contaminated data

We estimate Σ_u by applying the adaptive thresholding given by (2.11). However, the task here is slightly different from the standard problem of estimating a sparse covariance matrix in the literature, as no direct observations for ${\{{\mathbf{u}}_t\}}_{t=1}^T$ are available. In many cases the original data are contaminated, including any type of estimate of the data when direct observations are not available. This typically happens when ${\{{\mathbf{u}}_t\}}_{t=1}^T$ represent the error terms in regression models or when data is subject to measurement of errors. Instead, we may observe ${\{{\widehat{\mathbf{u}}}_t\}}_{t=1}^T$. For instance, in the approximate factor models, ${\widehat{\mathbf{u}}}_t={\mathbf{y}}_t-{\widehat{\mathbf{b}}}_i^{\prime }{\widehat{\mathbf{f}}}_t$.

We can estimate Σ_u using the adaptive thresholding proposed by Cai and Liu (2011): for the threshold ${\tau }_{ij}=C\sqrt{{\widehat{\theta }}_{ij}}{\omega }_T$, define

$${\widehat{\sigma }}_{ij}=\frac{1}{T}\sum _{t=1}^T{\widehat{u}}_{it}{\widehat{u}}_{jt},\text{and}{\widehat{\theta }}_{ij}=\frac{1}{T}\sum _{t=1}^T{({\widehat{u}}_{it}{\widehat{u}}_{jt}-{\widehat{\sigma }}_{ij})}^2.$$ (A.1)

$${\widehat{\mathbf{\sum }}}_u^{\mathcal{T}}={(s_{ij}({\widehat{\sigma }}_{ij}))}_{p\times p},$$

where s_ij(.) satisfies: for all z ∈ ℝ, s_ij(z) = 0, when |z| ≤ τ_ij; |s_ij(z) − z| ≤ τ_ij.

When ${\{{\widehat{\mathbf{u}}}_t\}}_{t=1}^T$ is close enough to ${\{{\mathbf{u}}_t\}}_{t=1}^T$, we can show that ${\widehat{\mathbf{\sum }}}_u^{\mathcal{T}}$ is also consistent. The following theorem extends the standard thresholding results in Bickel and Levina (2008) and Cai and Liu (2011) to the case when no direct observations are available, or the original data are contaminated. For the tail and mixing parameters r₁ and r₃ defined in Assumptions 3.2 and 3.3, let $\alpha =3r_1^{-1}+r_3^{-1}+1$.

Theorem A.1

Suppose (log p)^6α = o(T), and Assumptions 3.2 and 3.3 hold. In addition, suppose there is a sequence a_T = o(1) so that ${max}_{i\le p}\frac{1}{T}{\sum }_{t=1}^T{\mid u_{it}-{\widehat{u}}_{it}\mid }^2=O_p(a_T^2)$, and max_i≤p,t≤T |u_it − û_it| = o_p(1); Then there is a constant C > 0 in the adaptive thresholding estimator (A.1) with

$${\omega }_T=\sqrt{\frac{logp}{T}}+a_T$$

such that

$$\left|\right|{\widehat{\mathbf{\sum }}}_u^{\mathcal{T}}-{\mathbf{\sum }}_u\left|\right|=O_p({\omega }_T^{1-q}{m}_p).$$

If further ω_T m_p = o(1), then ${\widehat{\mathbf{\sum }}}_u^{\mathcal{T}}$ is invertible with probability approaching one, and

$$\left|\right|{({\widehat{\mathbf{\sum }}}_u^{\mathcal{T}})}^{-1}-{\mathbf{\sum }}_u^{-1}\left|\right|=O_p({\omega }_T^{1-q}{m}_p).$$

Proof

By Assumptions 3.2 and 3.3, the conditions of Lemmas A.3 and A.4 of Fan, Liao and Mincheva (2011, Ann. Statist, 39, 3320–3356) are satisfied. Hence for any ε > 0, there are positive constants M, θ₁ and θ₂ such that each of the events

$$\begin{array}{l}{A}_1=\{\underset{i\le p,j\le p}{max}\mid {\widehat{\sigma }}_{ij}-{\sigma }_{u,ij}\mid <M{\omega }_T\}\\ A_2=\{{\theta }_1>\sqrt{{\widehat{\theta }}_{ij}}>{\theta }_2,\text{all}i\le p,j\le p\}.\end{array}$$

occurs with probability at least 1 − ε. By the condition of threshold function, $s_{ij}(t)=s_{ij}(t)I_{\mid t\mid >C{\omega }_T\sqrt{{\widehat{\theta }}_{ij}}}$. Now for $C={\theta }_2^{-1}2M$, under the event A₁ ∩ A₂,

$$\begin{array}{l}\left|\right|{\widehat{\mathbf{\sum }}}_u^{\mathcal{T}}-{\mathbf{\sum }}_u\left|\right|\le \underset{i\le p}{max}\sum _{j=1}^p\mid s_{ij}({\widehat{\sigma }}_{ij})-{\sigma }_{u,ij}\mid \\ =\underset{i\le p}{max}\sum _{j=1}^p\mid s_{ij}({\widehat{\sigma }}_{ij})I_{(\mid {\widehat{\sigma }}_{ij}\mid >C{\omega }_T\sqrt{{\widehat{\theta }}_{ij}})}-{\sigma }_{u,ij}{I}_{(\mid {\widehat{\sigma }}_{ij}\mid >C{\omega }_T\sqrt{{\widehat{\theta }}_{ij}})}-{\sigma }_{u,ij}{I}_{(\mid {\widehat{\sigma }}_{ij}\mid \le C{\omega }_T\sqrt{{\widehat{\theta }}_{ij}})}\mid \\ \le \underset{i\le p}{max}\sum _{j=1}^p\mid s_{ij}({\widehat{\sigma }}_{ij})-{\widehat{\sigma }}_{ij}\mid I_{(\mid {\widehat{\sigma }}_{ij}\mid >C{\omega }_T\sqrt{{\widehat{\theta }}_{ij}})}+\sum _{j=1}^p\mid {\widehat{\sigma }}_{ij}-{\sigma }_{u,ij}\mid I_{(\mid {\widehat{\sigma }}_{ij}\mid >C{\omega }_T\sqrt{{\widehat{\theta }}_{ij}})}+\sum _{j=1}^p\mid {\sigma }_{u,ij}\mid I_{(\mid {\widehat{\sigma }}_{ij}\mid \le C{\omega }_T\sqrt{{\widehat{\theta }}_{ij}})}\\ \le \underset{i\le p}{max}\sum _{j=1}^{p}C{\omega }_T\sqrt{{\widehat{\theta }}_{ij}}{I}_{(\mid {\widehat{\sigma }}_{ij}\mid >C{\omega }_T{\theta }_2)}+M{\omega }_T\sum _{j=1}^p{I}_{(\mid {\widehat{\sigma }}_{ij}\mid >C{\omega }_T{\theta }_2)}+\sum _{j=1}^p\mid {\sigma }_{u,ij}\mid I_{(\mid {\widehat{\sigma }}_{ij}\mid \le C{\omega }_T{\theta }_1)}\\ \le (C{\theta }_1+M){\omega }_T\underset{i\le p}{max}\sum _{j=1}^p{I}_{(\mid {\sigma }_{u,ij}\mid >M{\omega }_T)}+\underset{i\le p}{max}\sum _{j=1}^p\mid {\sigma }_{u,ij}\mid I_{(\mid {\sigma }_{u,ij}\mid \le C{\omega }_T{\theta }_1+M{\omega }_T)}\\ \le (C{\theta }_1+M){\omega }_T\underset{i\le p}{max}\sum _{j=1}^p\frac{{\mid {\sigma }_{u,ij}\mid }^q}{M^q{\omega }_T^q}{I}_{(\mid {\sigma }_{u,ij}\mid >M{\omega }_T)}+\underset{i\le p}{max}\sum _{j=1}^p\mid {\sigma }_{u,ij}\mid \frac{{(C{\theta }_1+M)}^{1-q}{\omega }_T^{1-q}}{{\mid {\sigma }_{u,ij}\mid }^{1-q}}{I}_{(\mid {\sigma }_{u,ij}\mid \le (C{\theta }_1+M){\omega }_T)}\\ \le \frac{C{\theta }_1+M}{M^q}{\omega }_T^{1-q}\underset{i\le p}{max}\sum _{j=1}^p{\mid {\sigma }_{u,ij}\mid }^q+\underset{i\le p}{max}\sum _{j=1}^p{\mid {\sigma }_{u,ij}\mid }^q{(C{\theta }_1+M)}^{1-q}{\omega }_T^{1-q}\\ =m_p{\omega }_T^{1-q}(C{\theta }_1+M)(M^{-q}+{(C{\theta }_1+M)}^{-q}).\end{array}$$

Let M₁ = (Cθ₁+M)(M^−q+(Cθ₁+M)^−q). Then with probability at least 1 − 2ε, $\left|\right|{\widehat{\mathbf{\sum }}}_u^{\mathcal{T}}-{\mathbf{\sum }}_u\left|\right|\le m_p{\omega }_T^{1-q}{M}_1$. Since ε is arbitrary, we have $\left|\right|{\widehat{\mathbf{\sum }}}_u^{\mathcal{T}}-{\mathbf{\sum }}_u\left|\right|=O_p({\omega }_T^{1-q}{m}_p)$. If in addition, ω_T m_p = o(1), then the minimum eigenvalue of ${\widehat{\mathbf{\sum }}}_u^{\mathcal{T}}$ is bounded away from zero with probability approaching one since λ_min(Σ_u) > c₁. This then implies $\left|\right|{({\widehat{\mathbf{\sum }}}_u^{\mathcal{T}})}^{-1}-{\mathbf{\sum }}_u^{-1}\left|\right|=O_p\left({\omega }_T^{1-q}{m}_p\right)$.

B Proofs for Section 2

We first cite two useful theorems, which are needed to prove propositions 2.1 and 2.2. In Lemma B.1 below, let ${\{{\lambda }_i\}}_{i=1}^p$ be the eigenvalues of Σ in descending order and ${\{{\mathit{\xi }}_i\}}_{i=1}^p$ be their associated eigenvectors. Correspondingly, let ${\{{\widehat{\lambda }}_i\}}_{i=1}^p$ be the eigenvalues of Σ̂ in descending order and ${\{{\widehat{\mathit{\xi }}}_i\}}_{i=1}^p$ be their associated eigenvectors.

Lemma B.1

1. (Weyl’s Theorem) |λ̂_i − λ_i| ≤ ||Σ̂ − Σ||.

2. (sin θ Theorem, Davis and Kahan, 1970)

   $$\left|\right|{\widehat{\mathit{\xi }}}_i-{\mathit{\xi }}_i\left|\right|\le \frac{\sqrt{2}\left|\right|\widehat{\mathbf{\sum }}-\mathbf{\sum }\left|\right|}{min(\mid {\widehat{\lambda }}_{i-1}-{\lambda }_i\mid ,\mid {\lambda }_i-{\widehat{\lambda }}_{i+1}\mid )}.$$

Proof of Proposition 2.1

Proof

Since ${\{{\lambda }_j\}}_{j=1}^p$ are the eigenvalue of Σ and ${\{{\left|\right|{\stackrel{\sim }{\mathbf{b}}}_j\left|\right|}^2\}}_{j=1}^K$ are the first K eigenvalues of BB′ (the remaining p − K eigenvalues are zero), then by the Weyl’s theorem, for each j ≤ K,

$$\mid {\lambda }_j-{\left|\right|{\stackrel{\sim }{\mathbf{b}}}_j\left|\right|}^2\mid \le \left|\right|\mathbf{\sum }-{\mathbf{BB}}^{\prime }\left|\right|=\left|\right|{\mathbf{\sum }}_u\left|\right|.$$

For j > K, |λ_j| = |λ_j − 0| ≤ ||Σ_u||. On the other hand, the first K eigenvalues of BB are also the eigenvalues of B′B. By the assumption, the eigenvalues of p⁻¹B′B are bounded away from zero. Thus when j ≤ K, ||b̃_j||²/p are bounded away from zero for all large p.

Proof of Proposition 2.2

Proof

Applying the sin θ theorem yields

$$\left|\right|{\xi }_j-{\stackrel{\sim }{\mathbf{b}}}_j/\left|\right|{\stackrel{\sim }{\mathbf{b}}}_j\left|\right|\left|\right|\le \frac{\sqrt{2}\left|\right|{\mathbf{\sum }}_u\left|\right|}{min(\mid {\lambda }_{j-1}-{\left|\right|{\stackrel{\sim }{\mathbf{b}}}_j\left|\right|}^2\mid ,\mid {\left|\right|{\stackrel{\sim }{\mathbf{b}}}_j\left|\right|}^2-{\lambda }_{j+1}\mid )}$$

For a generic constant c > 0, |λ_j−1 − ||b̃_j||²| ≥ |||b̃_j−1||² − ||b̃_j||²| − |λ_j−1 − ||b̃_j−1||²| ≥ cp for all large p, since |||b̃_j−1||² − ||b̃_j||²| ≥ cp but |λ_j−1 − ||b̃_j−1||²| is bounded by Prosposition 2.1. On the other hand, if j < K, the same argument implies |||b̃_j||² − λ_j+1| ≥ cp. If j = K, |||b̃_j||² − λ_j+1| = p|||b̃_K||²/p − λ_K+1/p|, where ||b̃_K||²/p is bounded away from zero, but λ_K+1/p = O(p⁻¹). Hence again, |||b̃_j||² − λ_j+1| ≥ cp.

Proof of Theorem 2.1

Proof

The sample covariance matrix of the residuals using least squares method is given by ${\widehat{\mathbf{\sum }}}_u\frac{1}{T}(\mathbf{Y}-\widehat{\mathbf{\Lambda }}{\widehat{\mathbf{F}}}^{\prime })({\mathbf{Y}}^{\prime }-\widehat{\mathbf{F}}{\widehat{\mathbf{\Lambda }}}^{\prime })=\frac{1}{T}{\mathbf{YY}}^{\prime }-\widehat{\mathbf{\Lambda }}{\widehat{\mathbf{\Lambda }}}^{\prime }$. where we used the normalization condition F̂′F̂ = TI_K and Λ̂ = YF̂/T. If we show that $\widehat{\mathbf{\Lambda }}{\widehat{\mathbf{\Lambda }}}^{\prime }={\sum }_{i=1}^K{\widehat{\lambda }}_i{\widehat{\mathit{\xi }}}_i{\widehat{\mathit{\xi }}}_i^{\prime }$, then from the decompositions of the sample covariance

$$\frac{1}{T}{\mathbf{YY}}^{\prime }=\widehat{\mathbf{\Lambda }}{\widehat{\mathbf{\Lambda }}}^{\prime }+{\widehat{\mathbf{\sum }}}_u=\sum _{i=1}^K{\widehat{\lambda }}_i{\widehat{\mathit{\xi }}}_i{\widehat{\mathit{\xi }}}_i^{\prime }+\widehat{\mathbf{R}},$$

we have R̂ = Σ̂_u. Consequently, applying thresholding on Σ̂_u is equivalent to applying thresholding on R̂, which gives the desired result.

We now show $\widehat{\mathbf{\Lambda }}{\widehat{\mathbf{\Lambda }}}^{\prime }={\sum }_{i=1}^K{\widehat{\lambda }}_i{\widehat{\mathit{\xi }}}_i{\widehat{\mathit{\xi }}}_i^{\prime }$ indeed holds. Consider again the least squares problem (2.8) but with the following alternative normalization constraints: $\frac{1}{p}{\sum }_{i=1}^p{\mathbf{b}}_i{\mathbf{b}}_i^{\prime }={\mathbf{I}}_K$, and $\frac{1}{T}{\sum }_{t=1}^T{\mathbf{f}}_t{\mathbf{f}}_t^{\prime }$ is diagonal. Let (Λ̃, F̃) be the solution to the new optimization problem. Switching the roles of B and F, then the solution of (2.10) is Λ̃ = (ξ̂₁, · · ·, ξ̂_K) and F̃ = p⁻¹Y′Λ̃. In addition, T⁻¹F̃′F̃= diag(λ̂₁, · · ·, λ̂_K). From Λ̂F̂′ = Λ̃F̃′, it follows that $\widehat{\mathbf{\Lambda }}{\widehat{\mathbf{\Lambda }}}^{\prime }=\frac{1}{T}\widehat{\mathbf{\Lambda }}{\widehat{\mathbf{F}}}^{\prime }\widehat{\mathbf{F}}{\widehat{\mathbf{\Lambda }}}^{\prime }=\frac{1}{T}\stackrel{\sim }{\mathbf{\Lambda }}{\stackrel{\sim }{\mathbf{F}}}^{\prime }\stackrel{\sim }{\mathbf{F}}{\stackrel{\sim }{\mathbf{\Lambda }}}^{\prime }={\sum }_{i=1}^K{\widehat{\lambda }}_i{\widehat{\mathit{\xi }}}_i{\widehat{\mathit{\xi }}}_i^{\prime }$.

C Proofs for Section 3

We will proceed by subsequently showing Theorems 3.3, 3.1 and 3.2.

C.1 Preliminary lemmas

The following results are to be used subsequently. The proofs of Lemmas C.1, C.2 and C.3 are found in Fan, Liao and Mincheva (2011).

Lemma C.1

Suppose A, B are symmetric semi-positive definite matrices, and λ_min(B) > c_T for a sequence c_T > 0. If ||A − B|| = o_p(c_T ), then λ_min(A) > c_T/2, and

$$\left|\right|{\mathbf{A}}^{-1}-{\mathbf{B}}^{-1}\left|\right|=O_p(c_T^{-2})\left|\right|\mathbf{A}-\mathbf{B}\left|\right|.$$

Lemma C.2

Suppose that the random variables Z₁, Z₂ both satisfy the exponential-type tail condition: There exist r₁, r₂ ∈ (0, 1) and b₁, b₂ > 0, such that ∀s > 0,

$$P(\mid Z_i\mid >s)\le exp(-{(s/b_i)}^{r_i}),i=1,2.$$

Then for some r₃ and b₃ > 0, and any s > 0,

$$P(\mid Z_1{Z}_2\mid >s)\le exp(1-{(s/b_3)}^{r_3}).$$ (C.1)

Lemma C.3

Under the assumptions of Theorem 3.1,

1. ${max}_{i,j\le K}\mid \frac{1}{T}{\sum }_{t=1}^T{f}_{it}{f}_{jt}-E{f}_{it}{f}_{jt}\mid =O_p(\sqrt{1/T})$.

2. ${max}_{i,j\le p}\mid \frac{1}{T}{\sum }_{t=1}^T{u}_{it}{u}_{jt}-{Eu}_{it}{u}_{jt}\mid =O_p(\sqrt{(logp)/T})$

3. ${max}_{i\le K,j\le p}\mid \frac{1}{T}{\sum }_{t=1}^T{f}_{it}{u}_{jt}\mid =O_p(\sqrt{(logp)/T})$

Lemma C.4

Let λ̂_K denote the Kth largest eigenvalue of ${\widehat{\mathbf{\sum }}}_{\mathit{sam}}=\frac{1}{T}{\sum }_{t=1}^T{\mathbf{y}}_t{\mathbf{y}}_t^{\prime }$, then λ̂_K > C₁p with probability approaching one for some C₁ > 0.

Proof

First of all, by Proposition 2.1, under Assumption 3.1, the Kth largest eigenvalue λ_K of Σ satisfies: for some c > 0,

$${\lambda }_K\ge {\left|\right|{\stackrel{\sim }{\mathbf{b}}}_K\left|\right|}^2-\mid {\lambda }_K-{\left|\right|{\stackrel{\sim }{\mathbf{b}}}_K\left|\right|}^2\mid \ge cp\left|\right|{\mathbf{\sum }}_u\left|\right|\ge cp/2$$

for sufficiently large p. Using Weyl’s theorem, we need only to prove that ||Σ̂_sam − Σ|| = o_p(p). Without loss of generality, we prove the result under the identifiability condition (2.1). Using model (1.2), ${\widehat{\mathbf{\sum }}}_{\text{sam}}=T^{-1}{\sum }_{t=1}^T({\mathbf{Bf}}_t+{\mathbf{u}}_t){({\mathbf{Bf}}_t+{\mathbf{u}}_t)}^{\prime }$. Using this and (1.3), Σ̂_sam − Σ can be decomposed as the sum of the four terms:

$$\begin{array}{l}{\mathbf{D}}_1=(T^{-1}\mathbf{B}\sum _{t=1}^T{\mathbf{f}}_t{\mathbf{f}}_t^{\prime }-{\mathbf{I}}_K){\mathbf{B}}^{\prime },{\mathbf{D}}_2=T^{-1}\sum _{t=1}^T({\mathbf{u}}_t{\mathbf{u}}_t^{\prime }-{\mathbf{\sum }}_u),\\ {\mathbf{D}}_3=\mathbf{B}{T}^{-1}\sum _{t=1}^T{\mathbf{f}}_t{\mathbf{u}}_t^{\prime },{\mathbf{D}}_4={\mathbf{D}}_3^{\prime }\end{array}$$

We now deal them term by term. We will repeatedly use the fact that for a p × p matrix A,

$$\left|\right|\mathbf{A}\left|\right|\le p{\left|\right|\mathbf{A}\left|\right|}_{max}.$$

First of all, by Lemma C.3, $\left|\right|T^{-1}{\sum }_{t=1}^T{\mathbf{f}}_t{\mathbf{f}}_t^{\prime }-{\mathbf{I}}_K\left|\right|\le K{\left|\right|T^{-1}{\sum }_{t=1}^T{\mathbf{f}}_t{\mathbf{f}}_t^{\prime }-{\mathbf{I}}_K\left|\right|}_{max}=O_p(\sqrt{1/T})$, which is o_p(p) if K log p = o(T). Consequently, by Assumption 3.1, we have

$$\left|\right|{\mathbf{D}}_1\left|\right|\le O_p(K\sqrt{(logK)/T})\left|\right|{\mathbf{BB}}^{\prime }\left|\right|=O_p(p\sqrt{1/T}).$$

We now deal with D₂. It follows from Lemma C.3 that

$$\left|\right|{\mathbf{D}}_2\left|\right|\le p{\left|\right|T^{-1}\sum _{t=1}^T({\mathbf{u}}_t{\mathbf{u}}_t^{\prime }-{\mathbf{\sum }}_u)\left|\right|}_{max}=O_p(p\sqrt{(logp)/T}).$$

Since ||D₄|| = ||D₃||, it remains to deal with D₃, which is bounded by

$$\left|\right|{\mathbf{D}}_3\left|\right|\le \left|\right|T^{-1}\sum _{t=1}^T{\mathbf{f}}_t{\mathbf{u}}_t^{\prime }\left|\right|\left|\right|\mathbf{B}\left|\right|=O_p(p\sqrt{(logp)/T}),$$

which is o_p(p) since log p = o(T).

Lemma C.5

Under Assumption 3.3, ${max}_{t\le T}{\sum }_{s=1}^T\mid E{\mathbf{u}}_s^{\prime }{\mathbf{u}}_t\mid /p=O(1)$.

Proof

Since ${\{{\mathbf{u}}_t\}}_{t=1}^T$ is weakly stationary, ${max}_{t\le T}{\sum }_{s=1}^T\mid E{\mathbf{u}}_s^{\prime }{\mathbf{u}}_t\mid /p\le 2{\sum }_{t=1}^{\infty }\mid E{\mathbf{u}}_1^{\prime }{\mathbf{u}}_t\mid /p$. In addition, E|u_it|⁴ < M for some constant M and any i, t since u_it has exponential tail. Hence by Davydov’s inequality (Corollary 16.2.4 in Athreya and Lahiri 2006), there is a constant C > 0, for all i ≤ p, t ≤ T, $\mid {Eu}_{i1}{u}_{it}\mid \le C\sqrt{\alpha (t)}$, where α (t) is the α-mixing coefficient. By Assumption 3.3, ${\sum }_{t=1}^{\infty }\sqrt{\alpha (t)}<\infty$. Thus uniformly in T,

$$\underset{t\le T}{max}\sum _{s=1}^T\mid E{\mathbf{u}}_s^{\prime }{\mathbf{u}}_t\mid /p\le 2\sum _{t=1}^{\infty }\mid E{\mathbf{u}}_1^{\prime }{\mathbf{u}}_t\mid /p\le 2\sum _{t=1}^{\infty }\underset{i\le p}{max}\mid {Eu}_{i1}{u}_{it}\mid \le 2C\sum _{t=1}^{\infty }\sqrt{\alpha (t)}<\infty .$$

C.2 Proof of Theorem 3.3

Our derivation below relies on a result obtained by Bai and Ng (2002), which showed that the estimated number of factors is consistent, in the sense that K̂ equals the true K with probability approaching one. Note that under our Assumptions 3.1–3.4, all the assumptions in Bai and Ng (2002) are satisfied. Thus immediately we have the following Lemma.

Lemma C.6 (Theorem 2 in Bai and Ng (2002))

For K̂ defined in (2.14),

$$P(\widehat{K}=K)\to 1.$$

Proof

See Bai and Ng (2002).

Using (A.1) in Bai (2003), we have the following identity:

$${\widehat{\mathbf{f}}}_t-{\mathbf{Hf}}_t={(\mathbf{V}/p)}^{-1}\left(\frac{1}{T}\sum _{s=1}^T{\widehat{\mathbf{f}}}_{s}E({\mathbf{u}}_s^{\prime }{\mathbf{u}}_t)/p+\frac{1}{T}\sum _{s=1}^T{\widehat{\mathbf{f}}}_s{\zeta }_{st}+\frac{1}{T}\sum _{s=1}^T{\widehat{\mathbf{f}}}_s{\eta }_{st}+\frac{1}{T}\sum _{s=1}^T{\widehat{\mathbf{f}}}_s{\xi }_{st}\right)$$ (C.2)

where ${\zeta }_{st}={\mathbf{u}}_s^{\prime }{\mathbf{u}}_t/p-E({\mathbf{u}}_s^{\prime }{\mathbf{u}}_t)/p,{\eta }_{st}={\mathbf{f}}_s^{\prime }{\sum }_{i=1}^p{\mathbf{b}}_i{u}_{it}/p$, and ${\xi }_{st}={\mathbf{f}}_t^{\prime }{\sum }_{i=1}^p{\mathbf{b}}_i{u}_{is}/p$.

We first prove some preliminary results in the following Lemmas. Denote by f̂_t = (f̂_1t, …, f̂ _K̂t)′.

Lemma C.7

For all i ≤ K̂,

1. $\frac{1}{T}{\sum }_{t=1}^T{(\frac{1}{T}{\sum }_{s=1}^T{\widehat{f}}_{is}E({\mathbf{u}}_s^{\prime }{\mathbf{u}}_t)/p)}^2=O_p(T^{-1})$,

2. $\frac{1}{T}{\sum }_{t=1}^T{(\frac{1}{T}{\sum }_{s=1}^T{\widehat{f}}_{is}{\zeta }_{st})}^2=O_p(p^{-1})$,

3. $\frac{1}{T}{\sum }_{t=1}^T{(\frac{1}{T}{\sum }_{s=1}^T{\widehat{f}}_{is}{\eta }_{st})}^2=O_p(p^{-1})$,

4. $\frac{1}{T}{\sum }_{t=1}^T{(\frac{1}{T}{\sum }_{s=1}^T{\widehat{f}}_{is}{\xi }_{st})}^2=O_p(p^{-1})$.

Proof

1. We have ∀i, ${\sum }_{s=1}^T{\widehat{f}}_{is}^2=T$. By the Cauchy-Schwarz inequality,

   $$\begin{array}{l}\frac{1}{T}\sum _{t=1}^T{(\frac{1}{T}\sum _{s=1}^T{\widehat{f}}_{is}E({\mathbf{u}}_s^{\prime }{\mathbf{u}}_t)/p)}^2\le \frac{1}{T}\sum _{t=1}^T\frac{1}{T}\sum _{s=1}^T{(E{\mathbf{u}}_s^{\prime }{\mathbf{u}}_t/p)}^2\\ \le \underset{t\le T}{max}\frac{1}{T}\sum _{s=1}^T{(E{\mathbf{u}}_s^{\prime }{\mathbf{u}}_t/p)}^2\le \underset{s,t}{max}\mid E{\mathbf{u}}_s^{\prime }{\mathbf{u}}_t/p\mid \underset{t\le T}{max}\frac{1}{T}\sum _{s=1}^T\mid E{\mathbf{u}}_s^{\prime }{\mathbf{u}}_t/p\mid \end{array}$$

   By Lemma C.5, ${max}_{t\le T}{\sum }_{s=1}^T\mid E{\mathbf{u}}_s^{\prime }{\mathbf{u}}_t/p\mid =O(1)$, which then yields the result.

2. By the Cauchy-Schwarz inequality,

   $$\begin{array}{l}\frac{1}{T}\sum _{t=1}^T{(\frac{1}{T}\sum _{s=1}^T{\widehat{f}}_{is}{\zeta }_{st})}^2=\frac{1}{T^3}\sum _{s=1}^T\sum _{l=1}^T{\widehat{f}}_{is}{\widehat{f}}_{il}(\sum _{t=1}^T{\zeta }_{st}{\zeta }_{lt})\le \frac{1}{T^3}{\left(\sum _{sl}{({\widehat{f}}_{is}{\widehat{f}}_{il})}^2\sum _{sl}{(\sum _{t=1}^T{\zeta }_{st}{\zeta }_{lt})}^2\right)}^{1/2}\\ \le \frac{1}{T^3}\sum _{s=1}^T{\widehat{f}}_{is}^2{\left(\sum _{sl}{(\sum _{t=1}^T{\zeta }_{st}{\zeta }_{lt})}^2\right)}^{1/2}=\frac{1}{T^2}{\left(\sum _{s=1}^T\sum _{l=1}^T{(\sum _{t=1}^T{\zeta }_{st}{\zeta }_{lt})}^2\right)}^{1/2}.\end{array}$$

   Note that $E({\sum }_{s=1}^T{\sum }_{l=1}^T{({\sum }_{t=1}^T{\zeta }_{st}{\zeta }_{lt})}^2)=T^{2}E{({\sum }_{t=1}^T{\zeta }_{st}{\zeta }_{lt})}^2\le T^4{max}_{st}E{\mid {\zeta }_{st}\mid }^4$. By Assumption 3.4, ${max}_{st}E{\zeta }_{st}^4=O(p^{-2})$, which implies that ${\sum }_{s,l}{({\sum }_{t=1}^T{\zeta }_{st}{\zeta }_{lt})}^2=O_p(T^4/p^2)$, and yields the result.

3. By definition, ${\eta }_{st}={\mathbf{f}}_s^{\prime }{\sum }_{i=1}^p{\mathbf{b}}_i{u}_{it}/p$. We first bound || ${\sum }_{i=1}^p{\mathbf{b}}_i{u}_{it}$||. Assumption 3.4 implies $E\frac{1}{T}{\sum }_{t=1}^T{\left|\right|{\sum }_{i=1}^p{\mathbf{b}}_i{u}_{it}\left|\right|}^2=E{\left|\right|{\sum }_{i=1}^p{\mathbf{b}}_i{u}_{it}\left|\right|}^2=O(p)$. Therefore, by the Cauchy-Schwarz inequality,

   $$\begin{array}{l}\frac{1}{T}\sum _{t=1}^T{(\frac{1}{T}\sum _{s=1}^T{\widehat{f}}_{is}{\eta }_{st})}^2\le {\left|\right|\frac{1}{T}\sum _{s=1}^T{\widehat{f}}_{is}{\mathbf{f}}_s^{\prime }\left|\right|}^2\frac{1}{T}\sum _{t=1}^T{\left|\right|\sum _{j=1}^p{\mathbf{b}}_j{u}_{jt}\frac{1}{p}\left|\right|}^2\\ \le \frac{1}{{Tp}^2}\sum _{t=1}^T{\left|\right|\sum _{j=1}^p{\mathbf{b}}_j{u}_{jt}\left|\right|}^2\left(\frac{1}{T}\sum _{s=1}^T{\widehat{f}}_{is}^2\frac{1}{T}\sum _{s=1}^T{\left|\right|{\mathbf{f}}_s\left|\right|}^2\right)=O_p\left(\frac{1}{p}\right).\end{array}$$
4. Similar to part (iii), noting that ξ_st is a scalar, we have:

   $$\begin{array}{l}\frac{1}{T}\sum _{t=1}^T{(\frac{1}{T}\sum _{s=1}^T{\widehat{f}}_{is}{\xi }_{st})}^2=\frac{1}{T}\sum _{t=1}^T{\left|\frac{1}{T}\sum _{s=1}^T{\mathbf{f}}_t^{\prime }\sum _{j=1}^p{\mathbf{b}}_j{u}_{js}\frac{1}{p}{\widehat{f}}_{is}\right|}^2\le \frac{1}{T}\sum _{t=1}^T{\left|\right|{\mathbf{f}}_t\left|\right|}^2·{\Vert \frac{1}{T}\sum _{s=1}^T\sum _{j=1}^p{\mathbf{b}}_j{u}_{js}\frac{1}{p}{\widehat{f}}_{is}\Vert }^2\\ \le O_p(1)\frac{1}{T}\sum _{s=1}^T{\Vert \sum _{j=1}^p{\mathbf{b}}_j{u}_{js}\frac{1}{p}\Vert }^2·\frac{1}{T}\sum _{s=1}^T{\widehat{f}}_{is}^2\le O_p\left(\frac{1}{p}\right),\end{array}$$

   where the third line follows from the Cauchy-Schwarz inequality.

Lemma C.8

1. ${max}_{t\le T}\left|\right|\frac{1}{Tp}{\sum }_{s=1}^T{\widehat{\mathbf{f}}}_{s}E({\mathbf{u}}_s^{\prime }{\mathbf{u}}_t)\left|\right|=O_p(\sqrt{1/T})$,

2. ${max}_{t\le T}\left|\right|\frac{1}{T}{\sum }_{s=1}^T{\widehat{\mathbf{f}}}_s{\zeta }_{st}\left|\right|=O_p(T^{1/4}/\sqrt{p})$,

3. ${max}_{t\le T}\left|\right|\frac{1}{T}{\sum }_{s=1}^T{\widehat{\mathbf{f}}}_s{\eta }_{st}\left|\right|=O_p(T^{1/4}/\sqrt{p})$,

4. ${max}_{t\le T}\left|\right|\frac{1}{T}{\sum }_{s=1}^T{\widehat{\mathbf{f}}}_s{\xi }_{st}\left|\right|=O_p(T^{1/4}/\sqrt{p})$.

Proof

1. By the Cauchy-Schwarz inequality and the fact that $\frac{1}{T}{\sum }_{t=1}^T{\left|\right|{\widehat{\mathbf{f}}}_t\left|\right|}^2=O_p(1)$,

   $$\begin{array}{l}\underset{t\le T}{max}\left|\right|\frac{1}{Tp}\sum _{s=1}^T{\widehat{\mathbf{f}}}_{s}E({\mathbf{u}}_s^{\prime }{\mathbf{u}}_t)\left|\right|\le \underset{t\le T}{max}{\left(\frac{1}{T}\sum _{s=1}^T{\left|\right|{\widehat{\mathbf{f}}}_s\left|\right|}^2\frac{1}{T}\sum _{s=1}^T{(E{\mathbf{u}}_s^{\prime }{\mathbf{u}}_t/p)}^2\right)}^{1/2}\\ \le O_p(1)\underset{t\le T}{max}{\left(\frac{1}{T}\sum _{s=1}^T{(E{\mathbf{u}}_s^{\prime }{\mathbf{u}}_t/p)}^2\right)}^{1/2}\le O_p(1)\underset{s,t}{max}\sqrt{\mid E{\mathbf{u}}_s^{\prime }{\mathbf{u}}_t/p\mid }\underset{t\le T}{max}{\left(\frac{1}{T}\sum _{s=1}^T\mid E{\mathbf{u}}_s^{\prime }{\mathbf{u}}_t/p\mid \right)}^{1/2}.\end{array}$$

   The result then follows from Assumption 3.3.

2. By the Cauchy-Schwarz inequality,

   $$\underset{t\le T}{max}\left|\right|\frac{1}{T}\sum _{s=1}^T{\widehat{\mathbf{f}}}_s{\zeta }_{st}\left|\right|\le \underset{t\le T}{max}\frac{1}{T}{\left(\sum _{s=1}^T{\left|\right|{\widehat{\mathbf{f}}}_s\left|\right|}^2\sum _{s=1}^T{\zeta }_{st}^2\right)}^{1/2}\le {\left(O_p(1)\underset{t}{max}\frac{1}{T}\sum _{s=1}^T{\zeta }_{st}^2\right)}^{1/2}.$$

   It follows from Assumption 3.4 that $E{(\frac{1}{T}{\sum }_{s=1}^T{\zeta }_{st}^2)}^2\le {max}_{s,t\le T}E{\zeta }_{st}^4=O(\frac{1}{p^2})$. It then follows from the Chebyshev’s inequality and Bonferroni’s method that $\frac{1}{T}{\sum }_{s=1}^T{\zeta }_{st}^2=O_p(\sqrt{T}/p)$.

3. By Assumption 3.4, $E{\left|\right|\frac{1}{\sqrt{p}}{\sum }_{i=1}^p{\mathbf{b}}_i{u}_{it}\left|\right|}^4\le K^{2}M$. Chebyshev’s inequality and Bonferroni’s method yield ${max}_{t\le T}\left|\right|{\sum }_{i=1}^p{\mathbf{b}}_i{u}_{it}\left|\right|=O_p(T^{1/4}\sqrt{p})$ with probability one, which then implies: ${max}_{t\le T}\left|\right|\frac{1}{T}{\sum }_{s=1}^T{\widehat{\mathbf{f}}}_s{\eta }_{st}\left|\right|\le \left|\right|\frac{1}{T}{\sum }_{s=1}^T{\widehat{\mathbf{f}}}_s{\mathbf{f}}_s^{\prime }\left|\right|{max}_t\left|\right|\frac{1}{p}{\sum }_{i=1}^p{\mathbf{b}}_i{u}_{it}\left|\right|=o_p(T^{1/4}/p^{1/2})$.

4. By the Cauchy-Schwarz inequality and Assumption 3.4, we have demonstrated that $\left|\right|\frac{1}{T}{\sum }_{s=1}^T{\sum }_{i=1}^p{\mathbf{b}}_i{u}_{is}\frac{1}{p}{\widehat{\mathbf{f}}}_s\left|\right|=O_p(p^{-1/2})$. In addition, since E||K⁻² f_t||⁴ < M, max_t≤T ||f_t|| = O_p(T^1/4). It follows that ${max}_{t\le T}\left|\right|\frac{1}{T}{\sum }_{s=1}^T{\widehat{\mathbf{f}}}_s{\xi }_{st}\left|\right|\le {max}_{t\le T}\left|\right|{\mathbf{f}}_t\left|\right|·\left|\right|\frac{1}{T}{\sum }_{s=1}^T{\sum }_{i=1}^p{\mathbf{b}}_i{u}_{is}\frac{1}{p}{\widehat{\mathbf{f}}}_s\left|\right|=O_p(T^{1/4}/p^{1/2})$.

Lemma C.9

1. ${max}_{i\le K}\frac{1}{T}{\sum }_{t=1}^T{({\widehat{\mathbf{f}}}_t-{\mathbf{Hf}}_t)}_i^2=O_p(1/T+1/p)$.

2. $\frac{1}{T}{\sum }_{t=1}^T{\left|\right|{\widehat{\mathbf{f}}}_t-{\mathbf{Hf}}_t\left|\right|}^2+O_p(1/T+1/p)$.

3. ${max}_{t\le T}\left|\right|{\widehat{\mathbf{f}}}_t-{\mathbf{Hf}}_t\left|\right|=O_p(\sqrt{1/T}+T^{1/4}/\sqrt{p})$.

Proof

We prove this lemma conditioning on the event K̂= K. Once this is done, due to P(K̂ ≠ K) = o(1), it then implies the unconditional arguments.

1. When K̂ = K, by Lemma C.4, all the eigenvalues of V/p are bounded away from zero. Using the inequality (a + b + c + d)² ≤ 4(a² + b² + c² + d²) and the identity (C.2), we have, for some constant C > 0,

   $$\begin{array}{l}\underset{i\le K}{max}\frac{1}{T}\sum _{t=1}^T{({\widehat{\mathbf{f}}}_t-{\mathbf{Hf}}_t)}_i^2\le C\underset{i\le K}{max}\frac{1}{T}\sum _{t=1}^T{(\frac{1}{T}\sum _{s=1}^T{\widehat{f}}_{is}E({\mathbf{u}}_s^{\prime }{\mathbf{u}}_t)/p)}^2+C\underset{i\le K}{max}\frac{1}{T}\sum _{t=1}^T{(\frac{1}{T}\sum _{s=1}^T{\widehat{f}}_{is}{\zeta }_{st})}^2\\ +C\underset{i\le K}{max}\frac{1}{T}\sum _{t=1}^T{(\frac{1}{T}\sum _{s=1}^T{\widehat{f}}_{is}{\eta }_{st})}^2+C\underset{i\le K}{max}\frac{1}{T}\sum _{t=1}^T{(\frac{1}{T}\sum _{s=1}^T{\widehat{f}}_{is}{\xi }_{st})}^2.\end{array}$$

   Each of the four terms on the right hand side above are bounded in Lemma C.7, which then yields the desired result.

2. It follows from part (i) and that $\frac{1}{T}{\sum }_{t=1}^T{\left|\right|{\widehat{\mathbf{f}}}_t-{\mathbf{Hf}}_t\left|\right|}^2\le K{max}_{i\le K}\frac{1}{T}{\sum }_{t=1}^T{({\widehat{\mathbf{f}}}_t-{\mathbf{Hf}}_t)}_i^2$. Part (iii) is implied by (C.2) and Lemma C.8.

Lemma C.10

1. ${\mathbf{HH}}^{\prime }={\mathbf{I}}_{\widehat{K}}+O_p(1/\sqrt{T}+1/\sqrt{p})$.

2. ${\mathbf{H}}^{\prime }\mathbf{H}={\mathbf{I}}_K+O_p(1/\sqrt{T}+1/\sqrt{p})$.

Proof

We first condition on K̂ = K. (i) Lemma C.4 implies ||V⁻¹|| = O_p(p⁻¹). Also $\left|\right|\mathbf{F}\left|\right|={\lambda }_{max}^{1/2}({\mathbf{FF}}^{\prime })={\lambda }_{max}^{1/2}({\sum }_{t=1}^T{\mathbf{f}}_t{\mathbf{f}}_t^{\prime })=O_p(\sqrt{T})$. In addition, $\left|\right|\widehat{\mathbf{F}}\left|\right|=\sqrt{T}$. It then follows from the definition of H that ||H|| = O_p(1). Define $\widehat{cov}({\mathbf{Hf}}_t)=\frac{1}{T}{\sum }_{t=1}^T{\mathbf{Hf}}_t{({\mathbf{Hf}}_t)}^{\prime }$. Applying the triangular inequality gives:

$${\left|\right|{\mathbf{HH}}^{\prime }-{\mathbf{I}}_{\widehat{K}}\left|\right|}_F\le {\left|\right|{\mathbf{HH}}^{\prime }-\widehat{cov}({\mathbf{Hf}}_t)\left|\right|}_F+{\left|\right|\widehat{cov}({\mathbf{Hf}}_t)-{\mathbf{I}}_{\widehat{K}}\left|\right|}_F$$ (C.3)

By Lemma C.3, the first term in (C.3) is ${\left|\right|{\mathbf{HH}}^{\prime }-\widehat{cov}({\mathbf{Hf}}_t)\left|\right|}_F\le {\left|\right|\mathbf{H}\left|\right|}^2{\left|\right|{\mathbf{I}}_K-\widehat{cov}({\mathbf{f}}_t)\left|\right|}_F=O_p\left(\frac{1}{T}\right)$. The second term of (C.3) can be bounded, by the Cauchy-Schwarz inequality and Lemma C.9, as follows:

$$\begin{array}{l}{\Vert \frac{1}{T}\sum _{t=1}^T{\mathbf{Hf}}_t{({\mathbf{Hf}}_t)}^{\prime }-\frac{1}{T}\sum _{t=1}^T{\widehat{\mathbf{f}}}_t{\widehat{\mathbf{f}}}_t^{\prime }\Vert }_F\le {\Vert \frac{1}{T}\sum _t({\mathbf{Hf}}_t-{\widehat{\mathbf{f}}}_t){({\mathbf{Hf}}_t)}^{\prime }\Vert }_F+{\Vert \frac{1}{T}\sum _t{\widehat{\mathbf{f}}}_t({\widehat{\mathbf{f}}}_t^{\prime }-{({\mathbf{Hf}}_t)}^{\prime })\Vert }_F\\ \le \left(\frac{1}{T}\sum _t{\left|\right|{\mathbf{Hf}}_t-{\widehat{\mathbf{f}}}_t\left|\right|}^2\frac{1}{T}\sum _t{\left|\right|{\mathbf{Hf}}_t\left|\right|}^2\right)+{\left(\frac{1}{T}\sum _t{\left|\right|{\mathbf{Hf}}_t-{\widehat{\mathbf{f}}}_t\left|\right|}^2\frac{1}{T}\sum _t{\left|\right|{\widehat{\mathbf{f}}}_t\left|\right|}^2\right)}^{1/2}\\ =O_p\left(\frac{1}{\sqrt{T}}+\frac{1}{\sqrt{p}}\right).\end{array}$$

(ii) Still conditioning on K̂ = K, since ${\mathbf{HH}}^{\prime }={\mathbf{I}}_K+O_p(1/\sqrt{T}+1/\sqrt{p})$ and ||H|| = O_p(1), right multiplying H gives ${\mathbf{HH}}^{\prime }\mathbf{H}=\mathbf{H}+O_p(1/\sqrt{T}+1/\sqrt{p})$. Part (i) also gives, conditioning on K̂ = K, ||H⁻¹|| = O_p(1). Hence further left multiplying H⁻¹ yields ${\mathbf{H}}^{\prime }\mathbf{H}={\mathbf{I}}_K+O_p(1/\sqrt{T}+\sqrt{p})$. Due to P(K̂ = K) → 1, we reach the desired result.

Proof of Theorem 3.3

Proof

The second part of this theorem was proved in Lemma C.9. We now derive the convergence rate of max_i≤p ||b̂_i − Hb_i||.

Using the facts that ${\widehat{\mathbf{b}}}_i=\frac{1}{T}{\sum }_{t=1}^T{y}_{it}{\widehat{\mathbf{f}}}_t$, and that $\frac{1}{T}{\sum }_{t=1}^T{\widehat{\mathbf{f}}}_t{\widehat{\mathbf{f}}}_t^{\prime }=I_k$, we have

$${\widehat{\mathbf{b}}}_i-{\mathbf{Hb}}_i=\frac{1}{T}\sum _{t=1}^T{\mathbf{Hf}}_t{u}_{it}+\frac{1}{T}\sum _{t=1}^T{y}_{it}({\widehat{\mathbf{f}}}_t-{\mathbf{Hf}}_t)+\mathbf{H}(\frac{1}{T}\sum _{t=1}^T{\mathbf{f}}_t{\mathbf{f}}_t^{\prime }-{\mathbf{I}}_K){\mathbf{b}}_i.$$ (C.4)

We bound the three terms on the right hand side respectively. It follows from Lemmas C.3 and C.10 that ${max}_{i\le p}\left|\right|\frac{1}{T}{\sum }_{t=1}^T{\mathbf{Hf}}_t{u}_{it}\left|\right|\le \left|\right|\mathbf{H}\left|\right|{max}_i\sqrt{{\sum }_{k=1}^K{(\frac{1}{T}{\sum }_{t=1}^T{f}_{kt}{u}_{it})}^2}=O_p\left(\sqrt{\frac{logp}{T}}\right)$. For the second term, $E{y}_{it}^2=O(1)$. Therefore, ${max}_i{T}^{-1}{\sum }_{t=1}^T{y}_{it}^2=O_p(1)$. The Cauchy-Schwarz inequality and Lemma C.9 imply

$$\underset{i}{max}\left|\right|\frac{1}{T}\sum _{t=1}^T{y}_{it}({\widehat{\mathbf{f}}}_t-{\mathbf{Hf}}_t)\left|\right|\le \underset{i}{max}{\left(\frac{1}{T}\sum _{t=1}^T{y}_{it}^2\sum _{t=1}^T{\left|\right|{\widehat{\mathbf{f}}}_t-{\mathbf{Hf}}_t\left|\right|}^2\right)}^{1/2}=O_p(\frac{1}{\sqrt{T}}+\frac{1}{\sqrt{p}}).$$

Finally, $\left|\right|\frac{1}{T}{\sum }_{t=1}^T{\mathbf{f}}_t{\mathbf{f}}_t^{\prime }-{\mathbf{I}}_K\left|\right|=O_p(T^{-1/2})$ and max_i ||b_i|| = O(1) imply that the third term is O_p(T^−1/2).

Proof of Corollary 3.1

Under Assumption 3.3, it can be shown by Bonferroni’s method that max_t≤T ||f_t|| = O_p((log T )^1/r2 ). By Theorem 3.3, uniformly in i and t,

$$\begin{array}{l}\left|\right|{\widehat{\mathbf{b}}}_i^{\prime }{\widehat{\mathbf{f}}}_t-{\mathbf{b}}_i^{\prime }{\mathbf{f}}_t\left|\right|\le \left|\right|{\widehat{\mathbf{b}}}_i-{\mathbf{Hb}}_i\left|\right|\left|\right|{\widehat{\mathbf{f}}}_t-{\mathbf{Hf}}_t\left|\right|+\left|\right|{\mathbf{Hb}}_i\left|\right|\left|\right|{\widehat{\mathbf{f}}}_t-{\mathbf{Hf}}_t\left|\right|+\left|\right|{\widehat{\mathbf{b}}}_i-{\mathbf{Hb}}_i\left|\right|\left|\right|{\mathbf{Hf}}_t\left|\right|+\left|\right|{\mathbf{b}}_i\left|\right|{\mathbf{f}}_t\left|\right|\left|\right|{\mathbf{H}}^{\prime }\mathbf{H}-{\mathbf{I}}_K\left|\right|\\ =O_p\left({(logT)}^{1/r_2}\sqrt{\frac{logp}{T}}+\frac{T^{1/4}}{\sqrt{p}}\right).\end{array}$$

C.3 Proof of Theorem 3.1

Lemma C.11

${max}_{i\le p}\frac{1}{T}{\sum }_{t=1}^T{\mid u_{it}-{\widehat{u}}_{it}\mid }^2=O_p({\omega }_T^2)$, and max_i,t |u_it − û_it| = o_p(1).

Proof

We have, $u_{it}-{\widehat{u}}_{it}={\mathbf{b}}_i^{\prime }{\mathbf{H}}^{\prime }({\widehat{\mathbf{f}}}_t-{\mathbf{Hf}}_t)+({\widehat{\mathbf{b}}}_i^{\prime }-{\mathbf{b}}_i^{\prime }{\mathbf{H}}^{\prime }){\widehat{\mathbf{f}}}_t+{\mathbf{b}}_i^{\prime }({\mathbf{H}}^{\prime }\mathbf{H}-{\mathbf{I}}_K){\mathbf{f}}_t$. Therefore, using the inequality (a + b + c)² ≤ 4a² + 4b² + 4c², we have:

$$\begin{array}{l}\underset{i\le p}{max}\frac{1}{T}\sum _{t=1}^T{(u_{it}-{\widehat{u}}_{it})}^2\le 4\underset{i}{max}{\left|\right|{\mathbf{b}}_i^{\prime }{\mathbf{H}}^{\prime }\left|\right|}^2\frac{1}{T}\sum _{t=1}^T{\left|\right|{\widehat{\mathbf{f}}}_t-{\mathbf{Hf}}_t\left|\right|}^2\\ +4\underset{i}{max}{\left|\right|{\widehat{\mathbf{b}}}_i^{\prime }-{\mathbf{b}}_i^{\prime }{\mathbf{H}}^{\prime }\left|\right|}^2\frac{1}{T}\sum _{t=1}^T{\left|\right|{\widehat{\mathbf{f}}}_t\left|\right|}^2+4\underset{i}{max}{\left|\right|{\mathbf{b}}_i\left|\right|}^2\frac{1}{T}\sum _{t=1}^T{\left|\right|{\mathbf{f}}_t\left|\right|}^2{\left|\right|{\mathbf{H}}^{\prime }\mathbf{H}-{\mathbf{I}}_K\left|\right|}_F^2\end{array}$$

The first part of the lemma then follows from Theorem 3.3 and Lemma C.9. The second part follows from Corollary 3.1.

Proof of Theorem 3.1

The theorem follows immediately from Theorem A.1 and Lemma C.11.

C.4 Proof of Theorem 3.2

Define

$${\mathbf{C}}_T=\widehat{\mathbf{\Lambda }}-{\mathbf{BH}}^{\prime }.$$

Lemma C.12

1. ${\left|\right|{\mathbf{C}}_T\left|\right|}_F^2=O_p({\omega }_T^{2}p)$, and ${\left|\right|{\mathbf{C}}_T^{\prime }{\mathbf{C}}_T\left|\right|}_{\mathrm{\sum }}^2=O_p({\omega }_T^{4}p)$.

2. ${\left|\right|{\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}-{\mathbf{\sum }}_u\left|\right|}_{\mathrm{\sum }}^2=O_p({\omega }_T^{2-2q}{m}_p^2)$.

3. ${\left|\right|{\mathbf{BH}}^{\prime }{\mathbf{C}}_T^{\prime }\left|\right|}_{\mathrm{\sum }}^2=O_p({\omega }_T^2)$.

4. ${\left|\right|\mathbf{B}({\mathbf{H}}^{\prime }\mathbf{H}-{\mathbf{I}}_K){\mathbf{B}}^{\prime }\left|\right|}_{\mathrm{\sum }}^2=O_p(p^{-2}+{(pT)}^{-1})$.

Proof

1. We have ${\left|\right|{\mathbf{C}}_T\left|\right|}_F^2\le {max}_{i\le p}{\left|\right|{\widehat{\mathbf{b}}}_i-{\mathbf{Hb}}_i\left|\right|}^{2}p=O_p({\omega }_T^{2}p)$. Moreover, since all the eigenvalues of Σ are bounded away from zero, for any matrix A, ${\left|\right|\mathbf{A}\left|\right|}_{\mathrm{\sum }}^2=O_p(p^{-1}){\left|\right|\mathbf{A}\left|\right|}_F^2$. Hence ${\left|\right|{\mathbf{C}}_T^{\prime }{\mathbf{C}}_T\left|\right|}_{\mathrm{\sum }}^2=O_p(p^{-1}{\left|\right|{\mathbf{C}}_T\left|\right|}_F^4)=O_p(p{\omega }_T^4)$.

2. By Theorem 3.1, ${\left|\right|{\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}-{\mathbf{\sum }}_u\left|\right|}_{\mathrm{\sum }}^2=O_p(p^{-1}{\left|\right|{\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}-{\mathbf{\sum }}_u\left|\right|}_F^2)=O_p({\left|\right|{\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}-{\mathbf{\sum }}_u\left|\right|}^2)=O_p({\omega }_T^{2-2q}{m}_p^2)$.

3. The same argument of the proof of Theorem 2 in Fan, Fan and Lv (2008) implies that ||B′Σ⁻¹B|| = O(1). Thus, ${\left|\right|{\mathbf{BH}}^{\prime }{\mathbf{C}}_T^{\prime }\left|\right|}_{\mathrm{\sum }}^2=p^{-1}\text{tr}({\mathbf{H}}^{\prime }{\mathbf{C}}_T^{\prime }{\mathbf{\sum }}^{-1}{\mathbf{C}}_T{\mathbf{HB}}^{\prime }{\mathbf{\sum }}^{-1}\mathbf{B})$ is upper bounded by $p^{-1}{\left|\right|\mathbf{H}\left|\right|}^2\left|\right|{\mathbf{B}}^{\prime }{\mathbf{\sum }}^{-1}\mathbf{B}\left|\right|\left|\right|{\mathbf{\sum }}^{-1}\left|\right|{\left|\right|{\mathbf{C}}_T\left|\right|}_F^2=O_p(p^{-1}{\left|\right|{\mathbf{C}}_T\left|\right|}_F^2)=O_p({\omega }_T^2)$.

4. Again, by ||B′Σ⁻¹B|| = O(1), and Lemma C.10,

   $$\begin{array}{l}{\left|\right|\mathbf{B}({\mathbf{H}}^{\prime }\mathbf{H}-{\mathbf{I}}_K){\mathbf{B}}^{\prime }\left|\right|}_{\mathrm{\sum }}^2=p^{-1}\text{tr}(({\mathbf{H}}^{\prime }\mathbf{H}-{\mathbf{I}}_K){\mathbf{B}}^{\prime }{\mathbf{\sum }}^{-1}\mathbf{B}({\mathbf{H}}^{\prime }\mathbf{H}-{\mathbf{I}}_K){\mathbf{B}}^{\prime }{\mathbf{\sum }}^{-1}\mathbf{B})\\ \le p^{-1}{\left|\right|{\mathbf{H}}^{\prime }\mathbf{H}-{\mathbf{I}}_K\left|\right|}_F^2{\left|\right|{\mathbf{B}}^{\prime }{\mathbf{\sum }}^{-1}\mathbf{B}\left|\right|}^2=O_p(p^{-2}+{(pT)}^{-1}).\end{array}$$ (C.5)

Proof of Theorem 3.2 (i)

Proof

By Lemma C.12, ${\left|\right|\mathbf{B}({\mathbf{H}}^{\prime }\mathbf{H}-{\mathbf{I}}_K){\mathbf{B}}^{\prime }\left|\right|}_{\mathrm{\sum }}^2+{\left|\right|{\mathbf{BH}}^{\prime }{{\mathbf{C}}_T}^{\prime }\left|\right|}_{\mathrm{\sum }}^2+{\left|\right|{\mathbf{C}}_T{{\mathbf{C}}_T}^{\prime }\left|\right|}_{\mathrm{\sum }}^2=O_p({\omega }_T^2+\frac{p{log}^{2}p}{T^2})$. Hence for a generic constant C > 0,

$$\begin{array}{l}{\left|\right|{\widehat{\mathbf{\sum }}}_K-\mathbf{\sum }\left|\right|}_{\mathrm{\sum }}^2\le C{\left|\right|\widehat{\mathbf{\Lambda }}{\widehat{\mathbf{\Lambda }}}^{\prime }-{\mathbf{BB}}^{\prime }\left|\right|}_{\mathrm{\sum }}^2+C{\left|\right|{\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}-{\mathbf{\sum }}_u\left|\right|}_{\mathrm{\sum }}^2\\ \le C[{\left|\right|\mathbf{B}({\mathbf{H}}^{\prime }\mathbf{H}-{\mathbf{I}}_K){\mathbf{B}}^{\prime }\left|\right|}_{\mathrm{\sum }}^2+{\left|\right|{\mathbf{BH}}^{\prime }{{\mathbf{C}}_T}^{\prime }\left|\right|}_{\mathrm{\sum }}^2+{\left|\right|{\mathbf{C}}_T{{\mathbf{C}}_T}^{\prime }\left|\right|}_{\mathrm{\sum }}^2+{\left|\right|{\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}-{\mathbf{\sum }}_u\left|\right|}_{\mathrm{\sum }}^2]\\ =O_p({\omega }_T^{2-2q}{m}_p^2+\frac{p{log}^{2}p}{T^2}).\end{array}$$

Lemma C.13

$\left|\right|{\widehat{\mathbf{\Lambda }}}^{\prime }{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}\widehat{\mathbf{\Lambda }}-{({\mathbf{BH}}^{\prime })}^{\prime }{\mathbf{\sum }}_u^{-1}{\mathbf{BH}}^{\prime }\left|\right|=O_p(p{\omega }_T^{1-q}{m}_p)$.

Proof

${\left|\right|{\mathbf{C}}_T\left|\right|}_F^2=O_p({\omega }_T^{2}p)$. Hence

$$\begin{array}{l}\left|\right|{\widehat{\mathbf{\Lambda }}}^{\prime }{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}\widehat{\mathbf{\Lambda }}-{({\mathbf{BH}}^{\prime })}^{\prime }{\mathbf{\sum }}_u^{-1}{\mathbf{BH}}^{\prime }\left|\right|\le \left|\right|{\mathbf{C}}_T^{\prime }{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}{\mathbf{C}}_T\left|\right|\\ +2\left|\right|{\mathbf{C}}_T^{\prime }{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}{\mathbf{BH}}^{\prime }\left|\right|+\left|\right|{\mathbf{BH}}^{\prime }({({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}-{\mathbf{\sum }}_u^{-1}){\mathbf{BH}}^{\prime }\left|\right|=O_p(p{\omega }_T^{1-q}{m}_p)\end{array}$$ (C.6)

Lemma C.14

If ${\omega }_T^{1-q}{m}_p=o(1)$, then with probability approaching one, for some c > 0,

1. ${\lambda }_{min}({\mathbf{I}}_K+{({\mathbf{BH}}^{\prime })}^{\prime }{\mathbf{\sum }}_u^{-1}{\mathbf{BH}}^{\prime })\ge cp$.

2. ${\lambda }_{min}({\mathbf{I}}_K+{\widehat{\mathbf{\Lambda }}}^{\prime }{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}\widehat{\mathbf{\Lambda }})\ge cp$.

3. ${\lambda }_{min}({\mathbf{I}}_K+{\mathbf{B}}^{\prime }{\mathbf{\sum }}_u^{-1}\mathbf{B})\ge cp$.

4. ${\lambda }_{min}({({\mathbf{HH}}^{\prime })}^{-1}+{\mathbf{B}}^{\prime }{\mathbf{\sum }}_u^{-1}\mathbf{B})\ge cp$.

Proof

(i) By Lemma C.10, with probability approaching one, λ_min(HH′) is bounded away from zero. Hence,

$$\begin{array}{l}{\lambda }_{min}({\mathbf{I}}_K+{({\mathbf{BH}}^{\prime })}^{\prime }{\mathbf{\sum }}_u^{-1}{\mathbf{BH}}^{\prime })\ge {\lambda }_{min}({({\mathbf{BH}}^{\prime })}^{\prime }{\mathbf{\sum }}_u^{-1}{\mathbf{BH}}^{\prime }))\\ \ge {\lambda }_{min}({\mathbf{\sum }}_u^{-1}){\lambda }_{min}({\mathbf{HB}}^{\prime }{\mathbf{BH}}^{\prime })\ge {\lambda }_{min}({\mathbf{\sum }}_u^{-1}){\lambda }_{min}({\mathbf{B}}^{\prime }\mathbf{B}){\lambda }_{min}({\mathbf{HH}}^{\prime })\ge cp.\end{array}$$

(ii) The result follows from part (i) and Lemma C.13. Part (iii) and (iv) follow from a similar argument of part (i) and Lemma C.10.

Proof of Theorem 3.2

Proof

We derive the rate for $\left|\right|{\widehat{\mathbf{\sum }}}_{\widehat{K}}^{-1}-{\mathbf{\sum }}^{-1}\left|\right|$. Define

$$\mathbf{\sum }^{\sim }={\mathbf{BH}}^{\prime }{\mathbf{HB}}^{\prime }+{\mathbf{\sum }}_u.$$

Note that ${\widehat{\mathbf{\sum }}}_{\widehat{K}}=\widehat{\mathbf{\Lambda }}{\widehat{\mathbf{\Lambda }}}^{\prime }+{\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}$ and Σ = BB′ + Σ_u. The triangular inequality gives

$$\left|\right|{\widehat{\mathbf{\sum }}}_{\widehat{K}}^{-1}-{\mathbf{\sum }}^{-1}\left|\right|\le \left|\right|{\widehat{\mathbf{\sum }}}_{\widehat{K}}^{-1}-{\mathbf{\sum }^{\sim }}^{-1}\left|\right|+\left|\right|{\mathbf{\sum }^{\sim }}^{-1}-{\mathbf{\sum }}^{-1}\left|\right|.$$

Using the Sherman-Morrison-Woodbury formula, we have $\left|\right|{\widehat{\mathbf{\sum }}}_{\widehat{K}}^{-1}-{\mathbf{\sum }^{\sim }}^{-1}\left|\right|\le {\sum }_{i=1}^6{L}_i$, where

$$\begin{array}{l}{L}_1=\left|\right|{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}-{\mathbf{\sum }}_u^{-1}\left|\right|\\ L_2=\left|\right|({({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}-{\mathbf{\sum }}_u^{-1})\widehat{\mathbf{\Lambda }}{[{\mathbf{I}}_K+{\widehat{\mathbf{\Lambda }}}^{\prime }{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}\widehat{\mathbf{\Lambda }}]}^{-1}{\widehat{\mathbf{\Lambda }}}^{\prime }{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}\left|\right|\\ L_3=\left|\right|({({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}-{\mathbf{\sum }}_u^{-1})\widehat{\mathbf{\Lambda }}{[{\mathbf{I}}_K+{\widehat{\mathbf{\Lambda }}}^{\prime }{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}\widehat{\mathbf{\Lambda }}]}^{-1}{\widehat{\mathbf{\Lambda }}}^{\prime }{\mathbf{\sum }}_u^{-1}\left|\right|\\ L_4=\left|\right|{\mathbf{\sum }}_u^{-1}(\widehat{\mathbf{\Lambda }}-{\mathbf{BH}}^{\prime }){[{\mathbf{I}}_K+{\widehat{\mathbf{\Lambda }}}^{\prime }{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}\widehat{\mathbf{\Lambda }}]}^{-1}{\widehat{\mathbf{\Lambda }}}^{\prime }{\mathbf{\sum }}_u^{-1}\left|\right|\\ L_5=\left|\right|{\mathbf{\sum }}_u^{-1}(\widehat{\mathbf{\Lambda }}-{\mathbf{BH}}^{\prime }){[{\mathbf{I}}_K+{\widehat{\mathbf{\Lambda }}}^{\prime }{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}\widehat{\mathbf{\Lambda }}]}^{-1}{\mathbf{HB}}^{\prime }{\mathbf{\sum }}_u^{-1}\left|\right|\\ L_6=\left|\right|{\mathbf{\sum }}_u^{-1}{\mathbf{BH}}^{\prime }({[{\mathbf{I}}_K+{\widehat{\mathbf{\Lambda }}}^{\prime }{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}\widehat{\mathbf{\Lambda }}]}^{-1}-{[{\mathbf{I}}_K+{\mathbf{HB}}^{\prime }{\mathbf{\sum }}_u^{-1}{\mathbf{BH}}^{\prime }]}^{-1}){\mathbf{HB}}^{\prime }{\mathbf{\sum }}_u^{-1}\left|\right|.\end{array}$$ (C.7)

We bound each of the six terms respectively. First of all, L₁ is bounded by Theorem 3.1. Let $\mathbf{G}={[{\mathbf{I}}_K+{\widehat{\mathbf{\Lambda }}}^{\prime }{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}\widehat{\mathbf{\Lambda }}]}^{-1}$, then

$$L_2\le \left|\right|{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}-{\mathbf{\sum }}_u^{-1}\left|\right|·\left|\right|\widehat{\mathbf{\Lambda }}\mathbf{G}{\widehat{\mathbf{\Lambda }}}^{\prime }\left|\right|·\left|\right|{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}\left|\right|.$$

Note that Theorem 3.1 implies $\left|\right|{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}\left|\right|=O_p(1)$. Lemma C.14 then implies ||G|| = O_p(p⁻¹). This shows that L₂ = O_p(L₁). Similarly L₃ = O_p(L₁). In addition, since ${\left|\right|{\mathbf{C}}_T\left|\right|}_F^2={\left|\right|\widehat{\mathbf{\Lambda }}-{\mathbf{BH}}^{\prime }\left|\right|}_F^2=O_p({\omega }_T^{2}p),L_4\le \left|\right|{\mathbf{\sum }}_u^{-1}(\widehat{\mathbf{\Lambda }}-{\mathbf{BH}}^{\prime })\left|\right|\left|\right|\mathbf{G}\left|\right|\left|\right|\widehat{\mathbf{\Lambda }}{\mathbf{\sum }}_u^{-1}\left|\right|=O_p({\omega }_T)$. Similarly L₅ = O_p(L₄). Finally, let ${\mathbf{G}}_1={[{\mathbf{I}}_K+{({\mathbf{BH}}^{\prime })}^{\prime }{\mathbf{\sum }}_u^{-1}{\mathbf{BH}}^{\prime }]}^{-1}$. By Lemma C.14, ||G₁|| = O_p(p⁻¹). Then by Lemma C.13,

$$\begin{array}{l}\left|\right|\mathbf{G}-{\mathbf{G}}_1\left|\right|=\left|\right|\mathbf{G}({\mathbf{G}}^{-1}-{\mathbf{G}}_1^{-1}){\mathbf{G}}_1\left|\right|\le O_p(p^{-2})\left|\right|{({\mathbf{BH}}^{\prime })}^{\prime }{\mathbf{\sum }}_u^{-1}{\mathbf{BH}}^{\prime }-{\widehat{\mathbf{\Lambda }}}^{\prime }{({\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}})}^{-1}\widehat{\mathbf{\Lambda }}\left|\right|\\ =O_p(p^{-1}{\omega }_T^{1-q}{m}_p).\end{array}$$

Consequently, $L_6\le {\left|\right|{\mathbf{\sum }}_u^{-1}{\mathbf{BH}}^{\prime }\left|\right|}^2\left|\right|\mathbf{G}-{\mathbf{G}}_1\left|\right|=O_p({\omega }_T^{1-q}{m}_p)$. Adding up L₁–L₆ gives

$$\left|\right|{\widehat{\mathbf{\sum }}}_{\widehat{K}}^{-1}-{\mathbf{\sum }^{\sim }}^{-1}\left|\right|=O_p({\omega }_T^{1-q}{m}_p).$$

One the other hand, using Sherman-Morrison-Woodbury formula again implies

$$\begin{array}{l}\left|\right|{\mathbf{\sum }^{\sim }}^{-1}-{\mathbf{\sum }}^{-1}\left|\right|\le \left|\right|{\mathbf{\sum }}_u^{-1}\mathbf{B}({[{({\mathbf{H}}^{\prime }\mathbf{H})}^{-1}+{\mathbf{B}}^{\prime }{\mathbf{\sum }}_u^{-1}\mathbf{B}]}^{-1}-{[{\mathbf{I}}_K+{\mathbf{B}}^{\prime }{\mathbf{\sum }}_u^{-1}\mathbf{B}]}^{-1}){\mathbf{B}}^{\prime }{\mathbf{\sum }}_u^{-1}\left|\right|\\ \le O(p)\left|\right|{[{({\mathbf{H}}^{\prime }\mathbf{H})}^{-1}+{\mathbf{B}}^{\prime }{\mathbf{\sum }}_u^{-1}\mathbf{B}]}^{-1}-{[{\mathbf{I}}_K+{\mathbf{B}}^{\prime }{\mathbf{\sum }}_u^{-1}\mathbf{B}]}^{-1}\left|\right|\\ =O_p(p^{-1})\left|\right|{({\mathbf{H}}^{\prime }\mathbf{H})}^{-1}-{\mathbf{I}}_K\left|\right|=o_p({\omega }_T^{1-q}{m}_p).\end{array}$$

Proof of Theorem 3.2

||Σ̂^τ − Σ||_max

Proof

We first bound ||Λ̂Λ̂′ − BB′||_max. Repeatedly using the triangular inequality yields

$$\begin{array}{l}{\left|\right|\widehat{\mathbf{\Lambda }}{\widehat{\mathbf{\Lambda }}}^{\prime }-{\mathbf{BB}}^{\prime }\left|\right|}_{max}=\underset{i,j\le p}{max}\mid {\widehat{\mathbf{b}}}_i^{\prime }{\widehat{\mathbf{b}}}_j-{\mathbf{b}}_i^{\prime }{\mathbf{b}}_j\mid \\ \le \underset{ij}{max}[\mid {({\widehat{\mathbf{b}}}_i-{\mathbf{Hb}}_i)}^{\prime }{\widehat{\mathbf{b}}}_j\mid +\mid {\mathbf{b}}_i^{\prime }{\mathbf{H}}^{\prime }({\widehat{\mathbf{b}}}_j-{\mathbf{Hb}}_j)\mid +\mid {\mathbf{b}}_i^{\prime }({\mathbf{H}}^{\prime }\mathbf{H}-{\mathbf{I}}_K){\mathbf{b}}_j\mid ]\\ \le {(\underset{i}{max}||{\widehat{\mathbf{b}}}_i-{\mathbf{Hb}}_i||)}^2+2\underset{ij}{max}\left|\right|{\widehat{\mathbf{b}}}_i-{\mathbf{Hb}}_i\left|\right|\left|\right|{\mathbf{Hb}}_j\left|\right|+\underset{i}{max}{\left|\right|{\mathbf{b}}_i\left|\right|}^2{\mathbf{H}}^{\prime }\mathbf{H}-{\mathbf{I}}_K\left|\right|\\ =O_p({\omega }_T).\end{array}$$

On the other hand, let σ_u,ij be the (i, j) entry of Σ_u. Then max_ij |σ̂_ij − σ_u,ij| = O_p(ω_T).

$$\underset{ij}{max}\mid s_{ij}({\widehat{\sigma }}_{ij})-{\sigma }_{u,ij}\mid \le \underset{ij}{max}\mid s_{ij}({\widehat{\sigma }}_{ij})-{\widehat{\sigma }}_{ij}\mid +\mid {\widehat{\sigma }}_{ij}-{\sigma }_{u,ij}\mid \le \underset{ij}{max}{\tau }_{ij}+O_p({\omega }_T)=O_p({\omega }_T).$$

Hence ${\left|\right|{\widehat{\mathbf{\sum }}}_{u,\widehat{K}}^{\mathcal{T}}-{\mathbf{\sum }}_u\left|\right|}_{max}=O_p({\omega }_T)$. The result then follows immediately.