Optimal Estimation and Rank Detection for Sparse Spiked Covariance Matrices

Abstract

This paper considers a sparse spiked covariancematrix model in the high-dimensional setting and studies the minimax estimation of the covariance matrix and the principal subspace as well as the minimax rank detection. The optimal rate of convergence for estimating the spiked covariance matrix under the spectral norm is established, which requires significantly different techniques from those for estimating other structured covariance matrices such as bandable or sparse covariance matrices. We also establish the minimax rate under the spectral norm for estimating the principal subspace, the primary object of interest in principal component analysis. In addition, the optimal rate for the rank detection boundary is obtained. This result also resolves the gap in a recent paper by Berthet and Rigollet [2] where the special case of rank one is considered.

1 Introduction

Covariance matrix plays a fundamental role in multivariate analysis. Many methodologies, including discriminant analysis, principal component analysis and clustering analysis, rely critically on the knowledge of the covariance structure. Driven by a wide range of contemporary applications in many fields including genomics, signal processing, and financial econometrics, estimation of covariance matrices in the high-dimensional setting is of particular interest.

There have been significant recent advances on the estimation of a large covariance matrix and its inverse, the precision matrix. A variety of regularization methods, including banding, tapering, thresholding and penalization, have been introduced for estimating several classes of covariance and precision matrices with different structures. See, for example, [3, 4, 8, 12, 13, 17, 22, 26, 36, 41], among many others.

1.1 Sparse spiked covariance matrix model

In the present paper, we consider spiked covariance matrix models in the high-dimensional setting, which arise naturally from factor models with homoscedastic noise. To be concrete, suppose that we observe an n × p data matrix X with the rows X_1*, …, X_n* i.i.d. following a multivariate normal distribution with mean 0 and covariance matrix Σ, denoted by by N(0, Σ), where the covariance matrix Σ is given by

$$\mathbf{\sum }=\mathrm{Cov}({\mathbf{X}}_{i\ast })=\mathbf{V}\mathbf{\Lambda }\mathbf{V}\prime +{\sigma }^2{\mathbf{I}}_p,$$ (1)

where Λ = diag(λ₁, …, λ_r) with λ₁ ≥ ⋯ ≥ λ_r > 0, and V = [v₁, …, v_r] is p × r with orthonormal columns. The r largest eigenvalues of Σ are λ_i + σ², i = 1, …, r, and the rest are all equal to σ². The r leading eigenvectors of Σ are given by the column vectors of V. Since the spectrum of Σ has r spikes, (1) is termed by [19] as the spiked covariance matrix model. This covariance structure and its variations have been widely used in signal processing, chemometrics, econometrics, population genetics, and many other fields. See, for instance, [16, 24, 32, 34]. In the high-dimensional setting, various aspects of this model have been studied by several recent papers, including but not limited to [2, 5, 10, 20, 21, 31, 33, 35]. For simplicity, we assume σ is known. Since σ can always be factored out by scaling X, without loss of generality, we assume σ = 1. Data-based estimation of σ will be discussed in Section 6.

The primary focus of this paper is on the setting where V and Σ are sparse, and our goal is threefold. First, we consider the minimax estimation of the spiked covariance matrix Σ under the spectral norm. The method as well as the optimal rates of convergence in this problem are considerably different from those for estimating other recently studied structured covariance matrices, such as bandable and sparse covariance matrices. Second, we are interested in rank detection. The rank r plays an important role in principal component analysis (PCA) and is also of significant interest in signal processing and other applications. Last but not least, we consider optimal estimation of the principal subspace span(V) under the spectral norm, which is the main object of interest in PCA. Each of these three problems is important in its own right.

We now explain the sparsity model of V and Σ. The difficulty of estimation and rank detection depends on the joint sparsity of the columns of V. Let V_j* denote the j^th row of V. The row support of V is defined by

$$\mathrm{supp}(\mathbf{V})=\{j\in [p]:{\mathbf{V}}_{j\ast }\ne \mathbf{0}\},$$ (2)

whose cardinality is denoted by |supp(V)|. Let the collection of p × r matrices with orthonormal columns be O(p, r) = {V ∈ ℝ^{p × r} : V′V = I_r}. Define the following parameter spaces for Σ,

$$\begin{array}{ll}{\Theta }_0(k,p,r,\lambda ,\tau )=\{\mathbf{\sum }=\mathbf{V\Lambda }{\mathbf{V}}^{\prime }+{\mathbf{I}}_p& :\lambda /\tau \le {\lambda }_r\le \cdots \le {\lambda }_1\le \lambda ,\\ & \mathbf{V}\in O(p,r),|\mathrm{supp}(\mathbf{V})|\le k\},\end{array}$$ (3)

where τ ≥ 1 is a constant and r ≤ k ≤ p is assumed throughout the paper. Note that the condition number of Λ is at most τ. Moreover, for each covariance matrix in Θ₀ (k, p, r, λ, τ), the leading r singular vectors (columns of V) are jointly k-sparse in the sense that the row support size of V is upper bounded by k. The structure of group sparsity has proved useful for high-dimensional regression; See, for example, [30]. In addition to (3), we also define the following parameter spaces by dropping the dependence on τ and r, respectively:

$$\begin{array}{ll}{\Theta }_1(k,p,r,\lambda )& =\mathbf{cl}\underset{\tau \ge 1}{\cup }{\mathit{\Theta }}_0(k,p,r,\lambda ,\tau )\\ & \begin{array}{ll}=\{\mathbf{\sum }=\mathbf{V}\mathbf{\Lambda }{\mathbf{V}}^{\prime }+{\mathbf{I}}_p:& 0\le {\lambda }_r\le \cdots \le {\lambda }_1\le \lambda ,\\ & \mathbf{V}\in O(p,r),|\mathrm{supp}(\mathbf{V})|\le k\}\end{array}\end{array}$$ (4)

and

$${\mathit{\Theta }}_2(k,p,\lambda ,\tau )=\underset{r=0}{\overset{k}{\cup }}{\mathit{\Theta }}_0(k,p,r,\lambda ,\tau ).$$ (5)

As a consequence of the group sparsity in V, a covariance matrix Σ in any of the above parameter spaces has at most k rows and k columns containing nonzero off-diagonal entries. We note that the matrix is more structured than the so-called “k-sparse” matrices considered in [3, 8, 13], where each row (or column) has at most k nonzero off-diagonals.

1.2 Main contributions

In statistical decision theory, the minimax rate quantifies the difficulty of an inference problem and is frequently used as a benchmark for the performance of inference procedures. The main contributions of this paper include the sharp non-asymptotic minimax rates for estimating the covariance matrix Σ and the principal subspace span(V) under the squared spectral norm loss, as well as for detecting the rank r of the principal subspace. In addition, we also establish the minimax rates for estimating the precision matrix Ω = Σ⁻¹ as well as the eigenvalues of Σ under the spiked covariance matrix model (1).

We establish the minimax rate for estimating the spiked covariance matrix Σ in (1) under the spectral norm

$$L(\mathbf{\sum },\widehat{\mathbf{\sum }})={\Vert \mathbf{\sum }-\widehat{\mathbf{\sum }}\Vert }^2,$$ (6)

where for a matrix A its spectral norm is defined as ∥A∥ = sup_∥x∥2=1 ∥Ax∥₂ with ∥·∥₂ the vector ℓ₂ norm. The minimax upper and lower bounds developed in Sections 3 and 4 yield the following optimal rate for estimating sparse spiked covariance matrices under the spectral norm

$$\underset{\widehat{\mathbf{\sum }}}{\inf }\underset{\mathbf{\sum }\in {\mathit{\Theta }}_1(k,p,r,\lambda )}{\sup }\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert }^2\asymp \left[\frac{(\lambda +1)k}{n}\log \frac{\mathrm{e}p}{k}+\frac{{\lambda }^{2}r}{n}\right]\wedge {\lambda }^2,$$ (7)

subject to certain mild regularity conditions.¹ The two terms in the squared bracket are contributed by the estimation error of the eigenvectors V and the eigenvalues, respectively. Note that the second term can be dominant if λ is large.

An important quantity of the spiked model is the rank r of the principal subspace span(V), or equivalently, the number of spikes in the spectrum of Σ, which is of significant interest in chemometrics [24], signal array processing [25], and other applications. Our second goal is the minimax estimation of the rank r under the zero–one loss, or equivalently, the minimax detection of the rank r. It is intuitively clear that the difficulty in estimating the rank r depends crucially on the magnitude of the minimum spike λ_r. Results in Sections 3 and 4 show that the optimal rank detection boundary over the parameter space Θ₂(k, p, λ, τ) is of order $\sqrt{\frac{k}{n}\log \frac{\mathrm{e}p}{k}}$. Equivalently, the rank r can be exactly recovered with high probability if ${\lambda }_r\ge \beta \sqrt{\frac{k}{n}\log \frac{\mathit{ep}}{k}}$ for a sufficiently large constant β; On the other hand, reliable detection becomes impossible by any method if ${\lambda }_r\le {\beta }_0\sqrt{\frac{k}{n}\log \frac{\mathit{ep}}{k}}$ for some positive constant β₀. Lying in the heart of the arguments is a careful analysis of the moment generating function of a squared symmetric random walk stopped at a hypergeometrically distributed time, which is summarized in Lemma 1. It is worth noting that the optimal rate for rank detection obtained in the current paper resolves a gap left open in a recent paper by Berthet and Rigollet [2], where the authors obtained the optimal detection rate $\sqrt{\frac{k}{n}\log \frac{\mathit{ep}}{k}}$ for the rank-one case in the regime of $k\ll \sqrt{p}$, but the lower bound deteriorates to $\sqrt{\frac{p}{\mathit{kn}}}$ when $k\gg \sqrt{p}$ which is strictly suboptimal.

In many statistical applications, instead of the covariance matrix itself, the object of direct interest is often a lower dimensional functional of the covariance matrix, e.g., the principal subspace span(V). This problem is known in the literature as sparse PCA [5, 10, 20, 31]. The third goal of the paper is the minimax estimation of the principal subspace span(V). To this end, we note that the principal subspace can be uniquely identified with the associated projection matrix VV′. Moreover, any estimator can be identified with a projection matrix V̂V̂′, where the columns of V̂ constitute an orthonormal basis for the subspace estimator. Thus, estimating span(V) is equivalent to estimating VV′. We aim to optimally estimate span(V) under the loss [37, Section II.4]

$$L(\mathbf{V},\widehat{\mathbf{V}})={\Vert \mathbf{V}{\mathbf{V}}^{\prime }-\widehat{\mathbf{V}}{\widehat{\mathbf{V}}}^{\prime }\Vert }^2,$$ (8)

which equals the squared sine of the largest canonical angle between the respective linear spans. In the sparse PCA literature, the loss (8) was first used in [31] for multi-dimensional subspaces. For this problem, we shall show that, under certain regularity conditions, the minimax rate of convergence is

$$\underset{\widehat{\mathbf{V}}}{\inf }\underset{\mathbf{\sum }\in {\mathit{\Theta }}_0(k,p,r,\lambda ,\tau )}{\sup }\mathrm{{\rm E}}{\Vert \widehat{\mathbf{V}}{\widehat{\mathbf{V}}}^{\prime }-\mathbf{V}{\mathbf{V}}^{\prime }\Vert }^2\asymp \frac{(\lambda +1)k}{n{\lambda }^2}\log \frac{\mathrm{e}p}{k}\wedge 1.$$ (9)

In the present paper we considered estimation of the principal subspace span(V) under the spectral norm loss (8). It is interesting to compare the results with those for optimal estimation under the Frobenius norm loss [10, 40] ${\Vert \mathbf{V}{\mathbf{V}}^{\prime }-\widehat{\mathbf{V}}\widehat{\mathbf{V}}\prime \Vert }_{\mathrm{F}}^2$, whose ratio to (8) is between 1 and 2r. The optimal rate under the spectral norm loss given in (9) does not depend on the rank r, whereas the optimal rate under the Frobenius norm loss has an extra term $\frac{\lambda +1}{n{\lambda }^2}r\left(k-r\right)$, which depends on the rank r quadratically through r(k − r) [10]. Therefore the rate under the Frobenius norm far exceeds (9) when $r\gg \log \frac{\mathrm{e}p}{k}$. When r = 1, both norms lead to the same rate and the result in (9) recovers earlier results on estimating the leading eigenvector obtained in [5, 39, 29].

In addition to the optimal rates for estimating the covariance matrix Σ, the rank r and the principal subspace span(V), the minimax rates for estimating the precision matrix Ω = Σ⁻¹ as well as the eigenvalues of Σ are also established.

1.3 Other related work

Apart from the spiked covariance matrix model studied in this paper, other covariance matrix models have been considered in the literature. The most commonly imposed structural assumptions include “Toeplitz”, where each descending diagonal from left to right is constant, “bandable”, where the entries of the covariance matrix decay as they move away from the diagonal, and “sparse”, where only a small number of entries in each row/column are nonzero. The optimal rates of convergence were established in [11], [12] and [13] for estimating Toeplitz, bandable, and sparse covariance matrices, respectively. Estimation of sparse precision matrices has also been actively studied due to its close connection to Gaussian graphical models [9, 36, 41]. In addition, our work is also connected to the estimation of effective low-rank covariance matrices. See, for example, [28, 7] and the reference therein.

1.4 Organization

The rest of the paper is organized as the following. Section 2 introduces basic notation and then gives a detailed description of the procedures for estimating the spiked covariance matrix Σ, the rank r and the principal subspace span(V). The rates of convergence of these estimators are given in Section 3. Section 4 presents the minimax lower bounds that match the upper bounds in Section 3 in terms of the convergence rates, thereby establishing the minimax rates of convergence and rate-optimality of the estimators constructed in Section 2. The minimax rates for estimating the eigenvalues and the precision matrix are given in Section 5. Section 6 discusses computational and other related issues. The proofs are given in Section 7.

2 Estimation Procedure

We give a detailed description of the estimation procedure in this section and study its properties in Section 3. Throughout, we shall focus on minimax estimation and assume the sparsity k is known, while the rank r will be selected based on data. Adaptation to k will be discussed in Section 6.

Notation

We first introduce some notation. For any matrix X = (x_ij) and any vector u, denote by ∥X∥ the spectral norm, ∥X∥_F the Frobenius norm, and ∥u∥ the vector ℓ₂ norm. Moreover, the i^th row of X is denoted by X_i* and the j^th column by X_*j. Let supp(X) = {i : X_i* ≠ 0} denote the row support of X. For a positive integer p, [p] denotes the index set {1, 2, …, p}. For any set A, |A| denotes its cardinality, and A^c its complement. For two subsets I and J of indices, denote by X_IJ the |I| × |J| submatrices formed by x_ij with (i, j) ∈ I × J. Let X_I* = X_I[n] and X_*J = X_[p]J. For any square matrix A = (a_ij), we let Tr(A) = Σ_i a_ii be its trace. Define the inner product of any two matrices B and C of the same size by 〈B, C〉 = Tr(B′C). For any matrix A, we use σ_i(A) to denote its i^th largest singular value. When A is positive semi-definite, σ_i(A) is also the i^th largest eigenvalue of A. For any real number a and b, set a ∨ b = max{a, b} and a ∧ b = min{a, b}. Let S^p−1 denote the unit sphere in ℝ^p. For any event E, we write 1_{E} as its indicator function.

For any set B ⊂ [p], let B^c be its complement. For any symmetric matrix A ∈ ℝ^{p × p}, we use A_B to denote the p × p matrix whose B × B block is A_BB, the remaining diagonal elements are all ones and the remaining off-diagonal elements are all zeros, i.e.,

$${({\mathbf{A}}_B)}_{\mathit{ij}}=a_{\mathit{ij}}{\mathbf{1}}_{\{i\in B,j\in B\}}+{\mathbf{1}}_{\{i=j\in B^{\mathrm{c}}\}}.$$ (10)

In other word, after proper reordering of rows and columns, we have

$${\mathbf{A}}_B=\left[\begin{array}{cc}{\mathbf{A}}_{\mathit{BB}}& \mathbf{0}\\ \mathbf{0}& {\mathbf{I}}_{B^{\mathrm{c}}{B}^{\mathrm{c}}}\end{array}\right].$$

Let P ⊗ Q denote the product measure of P and Q and P^⊗n the n-fold product of P. For random variables X and Y, we write $X\stackrel{(\mathrm{d})}{=}Y$ if they follow the same distribution, and $X\stackrel{\mathrm{s}.\mathrm{t}.}{\le }Y$ if Ρ(X > t) ≤ Ρ(Y > t) for all t ∈ ℝ. Throughout the paper, we use C to denote a generic positive absolute constant, whose actual value may depend on the context. For any two sequences {a_n} and {b_n} of positive numbers, we write a_n ≲ b_n when a_n ≤ Cb_n for some numeric constant C, and a_n ≳ b_n when b_n ≲ a_n, and a_n ≍ b_n when both a_n ≳ b_n and a_n ≲ b_n hold.

Estimators

We are now ready to present the procedure for estimating the spiked covariance matrix Σ, the rank r, and the principal subspace span(V).

Let

$$A=\mathrm{supp}(\mathbf{V})$$ (11)

be the row support of V. For any m ∈ [p], let

$$\eta (m,n,p,{\gamma }_1)=2\left(\sqrt{\frac{m}{n}}+\sqrt{\frac{{\gamma }_1}{n}m\log \frac{\mathrm{e}p}{m}}\right)+{\left(\sqrt{\frac{m}{n}}+\sqrt{\frac{{\gamma }_1}{n}m\log \frac{\mathrm{e}p}{m}}\right)}^2.$$ (12)

Recall that the observed matrix X has i.i.d. rows X_i* ~ N(0, Σ). We define $\mathbf{S}=\frac{1}{n}{\mathbf{X}}^{\prime }\mathbf{X}$ as the sample covariance matrix. Also recall that we assume knowledge of the sparsity level k which is an upper bound for the support size |A|. The first step in the estimation scheme is to select a subset  of k features based on the data. To this end, let

$$\begin{array}{lll}{\mathrm{B}}_k& =& \{B\subset [p]:|B|=k,\mathrm{and\; for\; all}D\subset B^{\mathrm{c}}\mathrm{with}|D|\le k,\\ & & \Vert {\mathbf{S}}_D-\mathbf{I}\Vert \le \eta (|D|,n,p,{\gamma }_1),\Vert {\mathbf{S}}_{\mathit{DB}}\Vert \le 2\sqrt{\Vert {\mathbf{S}}_B\Vert }\eta (k,n,p,{\gamma }_1)\}.\end{array}$$ (13)

The appropriate value of γ₁ will be specified later in the statement of the theorems. Intuitively speaking, the requirements in (13) aim to ensure that for any B ∈ B_k, there is no evidence in data suggesting that B^c overlaps with the row support A of V. If B_k ≠ 0̸, denote by  an arbitrary element of B_k (or we can let  = argmax_B∈Bk Tr(S_BB) for concreteness). As we will show later, B_k is non-empty with high probability; See Lemma 3 in Section 7.1. The set  represents the collection of selected features, which turns out to be instrumental in constructing optimal estimators for the three objects we are interested in: the covariance matrix, the rank of the spiked model, and the principle subspace. The estimator  of the support set A is obtained through searching over all subsets of size k. Such a global search, though computationally expensive, appears to be necessary in order for our procedure to optimally estimate Σ and V under the spectral norm. For example, estimating row-by-row is not guaranteed to always yield optimal results. Whether there exist computationally efficient procedures attaining the optimal rate is currently unknown. See Section 6 for more discussions.

Given Â, the estimators for the above three objects are defined as follows. Recalling the notation in (10), we define the covariance matrix estimator as

$$\widehat{\mathbf{\sum }}={\mathbf{S}}_{\widehat{A}}{\mathbf{1}}_{\{{\mathrm{B}}_k\ne \varnothing \}}+{\mathbf{I}}_p{\mathbf{1}}_{\{{\mathrm{B}}_k=\varnothing \}},$$ (14)

The estimator for the rank is

$$\widehat{r}=\max \left\{l:{\sigma }_l({\mathbf{S}}_{\widehat{A}\widehat{A}})\ge 1+{\gamma }_2\sqrt{\Vert {\mathbf{S}}_{\widehat{A}}\Vert }\eta (k,n,p,{\gamma }_1)\right\}.$$ (15)

The appropriate value of γ₂ will be specified later in the statement of the theorems. Last but not least, the estimator for the principle subspace is span(V̂), Where

$$\widehat{\mathbf{V}}=[{\widehat{\mathbf{v}}}_1,\dots ,{\widehat{\mathbf{v}}}_{\widehat{r}}]\in O(p,\widehat{r})$$ (16)

with v̂_l the l^th eigenvector of Σ̂. When B_k = 0̸, we set r̂ = 0 and V̂ = 0 since Σ̂ = I_p. Note that the estimator V̂ is based on the estimated rank r̂. Whenever r̂ ≠ r, the value of the loss function (8) equals 1.

For a brief discussion on the comparison between the foregoing estimation procedure and that in [40], see Section 6.

3 Minimax Upper Bounds

We now investigate the properties of the estimation procedure introduced in Section 2. Rates of convergence for estimating the whole covariance matrix and its principal subspace under the spectral norm as well as for rank detection are established. The minimax lower bounds given in Section 4 will show that these rates are optimal and together they yield the minimax rates of convergence.

We begin with the estimation error of the covariance matrix estimator (14). Note that here we do not require the ratio λ₁/λ_r to be bounded.

Theorem 1

Let Σ̂ be defined in (14) with B_k given by (13) for some γ₁ ≥ 10. If $\frac{k}{n}\log \frac{\mathrm{e}p}{k}\le c_0$ for a sufficiently small constant c₀ > 0, then

$$\underset{\mathbf{\sum }\in {\mathit{\Theta }}_1(k,p,r,\lambda )}{\sup }\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert }^2\lesssim \frac{(\lambda +1)k}{n}\log \frac{\mathrm{e}p}{k}+\frac{{\lambda }^{2}r}{n},$$ (17)

where the parameter space Θ₁ (k, p, r, λ) is defined in (4).

As we shall show later in Theorem 4, the rates in (17) are optimal with respect to all the model parameters, namely k, p, r and λ.

Remark 1

Since the parameter space Θ₁ in (4) is contained in the set of k-sparse covariance matrices, it is of interest to compare the minimax rates for covariance matrix estimation in these two nested parameter spaces. For simplicity, only consider the case where the spectral norms of covariance matrices are uniformly bounded by a constant. Cai and Zhou [13] showed that, under certain regularity conditions, the minimax rate of convergence for estimating k-sparse matrices is $k^2\frac{\log p}{n}$, while the rate over Θ₁ in (7) reduces to $\frac{k}{n}\log \frac{\mathrm{e}p}{k}$ when the spectral norm of the matrix and hence λ is bounded. Thus, ignoring the logarithmic terms, the rate over the smaller parameter space Θ₁ is faster by a factor of k. This faster rate can be achieved because the group k-sparsity considered in our parameter space imposes much more structure than the row-wise k-sparsity does for the general k-sparse matrices.

The next result concerns with the detection rate of the rank estimator (15) under the extra assumption that the ratio of the largest spike to the smallest, i.e., λ₁/λ_r, is bounded.

Theorem 2

Let r̂ = r̂(γ₁, γ₂) be defined in (15) for some constants γ₁ ≥ 10 and ${\gamma }_2\ge 8\sqrt{{\gamma }_1}+34$. Assume that

$$\frac{\mathit{\tau k}}{n}\log \frac{\mathrm{e}p}{k}\le c_0$$ (18)

for a sufficiently small constant c₀ ∈ (0, 1) which depends on γ₁. If $\lambda \ge \beta \sqrt{\frac{k}{n}\log \frac{\mathrm{e}p}{k}}$ for some sufficiently large β depending only on γ₁, γ₂ and τ, then

$$\underset{\mathbf{\sum }\in {\mathit{\Theta }}_2(k,p,\lambda ,\tau )}{\sup }\mathrm{{\rm P}}\{\widehat{r}\ne \mathrm{rank}\left(\mathbf{\Lambda }\right)\}\le C{p}^{1-{\gamma }_1/2}$$ (19)

where the parameter space Θ₂(k, p, λ, τ) is defined in (5).

By Theorem 5 to be introduced later, the detection rate of $\sqrt{\frac{k}{n}\log \frac{\mathrm{e}p}{k}}$ is optimal. For more details, see the discussion in Section 4.

Finally, we turn to the risk of the principal subspace estimator. As in Theorem 2, we require λ₁/λ_r to be bounded.

Theorem 3

Suppose

$$M_1\log p\ge \log n\ge M_0\log \lambda$$ (20)

holds for some absolute constants M₀, M₁ > 0. Let V̂ be defined in (16) with constants ${\gamma }_1\ge 10\vee \left(1+\frac{2}{M_0}\right)M_1$ and ${\gamma }_2\ge 8\sqrt{{\gamma }_1}+34$ in (13) and (15). If (18) holds for a sufficiently small constant c₀ depending on γ₁, then

$$\underset{\mathbf{\sum }\in {\mathit{\Theta }}_0(k,p,r,\lambda ,\tau )}{\sup }\mathrm{{\rm E}}{\Vert \widehat{\mathbf{V}}{\widehat{\mathbf{V}}}^{\prime }-\mathbf{V}{\mathbf{V}}^{\prime }\Vert }^2\lesssim \frac{k(\lambda +1)}{n{\lambda }^2}\log \frac{\mathrm{e}p}{k}\wedge 1,$$ (21)

where the parameter space Θ₀(k, p, r, λ, τ) is defined in (3).

Remark 2

To ensure that the choice of γ₁ for achieving (21) is data-driven, we only need an under-estimate for M₀ = log n/ log λ, or equivalently an overestimate for λ. (Note that M₁ = log n/ log p can be obtained directly given the data matrix.) To this end, we first estimate Σ by Σ̂ in (14) with an initial γ₁ = 10 in (13). Then we control λ by 2∥Σ̂∥ − 1. By the proof of Theorem 2, and in particular (75), this is an over-estimate of λ with high probability. The upper bound in (21) remains valid if we compute V̂ with a (possibly) new γ₁ = 10 ∧ (1 + 2/M̂₀)M₁ in (13), where M̂₀ = log n/ log(2∥ Σ̂ ∥ − 1).

It is worth noting that the rate in (21) does not depend on r, and is optimal, by the lower bound given in Theorem 4 later.

The problems of estimating Σ and V are clearly related, but they are also distinct from each other. To discuss their relationship, we first note the following result (proved in Appendix 7.5) which is a variation of the well-known sin-theta theorem [15]:

Proposition 1

Let Σ and Σ̂ be p × p symmetric matrices. Let r ∈ [p − 1] be arbitrary and let V, V̂ ∈ O(p, r) be formed by the r leading singular vectors of Σ and Σ̂, respectively. Then

$$\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert \ge \frac{1}{2}({\sigma }_r(\mathbf{\sum })-{\sigma }_{r+1}(\mathbf{\sum }))\Vert \widehat{\mathbf{V}}{\widehat{\mathbf{V}}}^{\prime }-\mathbf{V}{\mathbf{V}}^{\prime }\Vert .$$ (22)

In view of Proposition 1, the minimax risks for estimating the spiked co-variance matrix Σ and the principle subspace V under the spectral norm can be tied as follows:

$$\underset{\widehat{\mathbf{\sum }}}{\inf }\underset{\mathbf{\sum }\in \mathit{\Theta }}{\sup }\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert }^2\gtrsim {\lambda }_r^2\underset{\widehat{\mathbf{V}}}{\inf }\underset{\mathbf{\sum }\in \mathit{\Theta }}{\sup }\mathrm{{\rm E}}{\Vert \widehat{\mathbf{V}}{\widehat{\mathbf{V}}}^{\prime }-\mathbf{V}{\mathbf{V}}^{\prime }\Vert }^2,$$ (23)

where Θ = Θ₀(k, p, r, λ, τ).

The results of Theorems 1 and 3 suggest, however, that the above inequality is not tight when λ is large. The optimal rate for estimating V is not equivalent to that for estimating Σ divided by λ² when $\frac{\lambda +1}{{\lambda }^2}\log \frac{\mathrm{e}p}{k}\ll \frac{r}{k}$. Consequently, Theorem 3 cannot be directly deduced from Theorem 1 but requires a different analysis by introducing an intermediate matrix S₀ defined later in (50). This is because the estimation of Σ needs to take into account the extra error in estimating the eigenvalues in addition to those in estimating V. On the other hand, in proving Theorem 1 we need to contend with the difficulty that the loss function is unbounded.

4 Minimax Lower Bounds and Optimal Rates of Convergence

In this section we derive minimax lower bounds for estimating the spiked covariance matrix Σ and the principal subspace span(V) as well as for the rank detection. These lower bounds hold for all parameters and are non-asymptotic. The lower bounds together with the upper bounds given in Section 3 establish the optimal rates of convergence for the three problems.

The technical analysis heavily relies on a careful study of a rank-one testing problem and analyzing the moment generating function of a squared symmetric random walk stopped at a hypergeometrically distributed time. This lower bound technique is of independent interest and can be useful for other related matrix estimation and testing problems.

4.1 Lower bounds and minimax rates for matrix and subspace estimation

We first consider the lower bounds for estimating the spiked covariance matrix Σ and the principal subspace V under the spectral norm.

Theorem 4

For any 1 ≤ r ≤ k ≤ p and n ∈ ℕ,

$$\underset{\mathbf{\sum }^{\sim }}{\inf }\underset{\mathbf{\sum }\in {\mathit{\Theta }}_1(k,p,r,\lambda )}{\sup }\mathrm{{\rm E}}{\Vert \mathbf{\sum }^{\sim }-\mathbf{\sum }\Vert }^2\gtrsim \left(\frac{(\lambda +1)k}{n}\log \frac{\mathrm{e}p}{k}+\frac{{\lambda }^{2}r}{n}\right)\wedge {\lambda }^2,$$ (24)

and

$$\underset{\stackrel{\sim }{\mathbf{V}}}{\inf }\underset{\mathbf{\sum }\in {\mathit{\Theta }}_0(k,p,r,\lambda ,\tau )}{\sup }\mathrm{{\rm E}}{\Vert \stackrel{\sim }{\mathbf{V}}{\stackrel{\sim }{\mathbf{V}}}^{\prime }-\mathbf{V}{\mathbf{V}}^{\prime }\Vert }^2\gtrsim \frac{(\lambda +1)(k-r)}{n{\lambda }^2}\log \frac{\mathrm{e}(p-r)}{k-r}\wedge 1,$$ (25)

where the parameter spaces Θ₁ (k, p, r, λ) and Θ₀ (k, p, r, λ, τ) are defined in (4) and (3), respectively.

To better understand the lower bound (24), it is helpful to write it equivalently as

$$\left(\frac{(\lambda +1)k}{n}\log \frac{\mathrm{e}p}{k}\right)\wedge {\lambda }^2+\frac{{\lambda }^{2}r}{n},$$

which can be proved by showing that the minimax risk is lower bounded by each of these two terms. The first term does not depend on r and is the same as the lower bound in the rank-one case. The second term is the oracle risk when the true support of V is known. The key to the proof is the analysis of the rank-one case which will be discussed in more detail in Section 4.3. The proof of (25) is relatively straightforward by using known results on rank-one estimation.

In view of the upper bounds given in Theorems 1 and 3 and the lower bounds given in Theorem 4, we establish the following minimax rates of convergence for estimating the spiked covariance matrix Σ and the principal subspace V, subject to the constraints on the parameters given in Theorems 1 and 3:

$$\underset{\mathbf{\sum }^{\sim }}{\inf }\underset{\mathbf{\sum }\in {\mathit{\Theta }}_1(k,p,r,\lambda ,\tau )}{\sup }\mathrm{{\rm E}}{\Vert \mathbf{\sum }^{\sim }-\mathbf{\sum }\Vert }^2\asymp \left(\frac{(\lambda +1)k}{n}\log \frac{\mathrm{e}p}{k}+\frac{{\lambda }^{2}r}{n}\right)\wedge {\lambda }^2,$$ (26)

$$\underset{\stackrel{\sim }{\mathbf{V}}}{\inf }\underset{\mathbf{\sum }\in {\mathit{\Theta }}_0(k,p,r,\lambda ,\tau )}{\sup }\mathrm{{\rm E}}{\Vert \stackrel{\sim }{\mathbf{V}}{\stackrel{\sim }{\mathbf{V}}}^{\prime }-\mathbf{V}{\mathbf{V}}^{\prime }\Vert }^2\asymp \frac{(\lambda +1)k}{n{\lambda }^2}\log \frac{\mathrm{e}p}{k}\wedge 1,$$ (27)

where (27) holds under the addition condition that k − r ≳ k. Therefore, the estimators of Σ and V given in Section 2 are rate optimal. In (26), the trivial upper bound of λ² can always be achieved by using the identity matrix as the estimator.

4.2 Lower bound and minimax rate for rank detection

We now turn to the lower bound and minimax rate for the rank detection problem.

Theorem 5

Let ${\beta }_0<\frac{1}{36}$ be a constant. For any 1 ≤ r ≤ k ≤ p and n ∈ ℕ, if $\lambda \le 1\wedge \sqrt{\frac{{\beta }_{0}k}{n}\log \frac{\mathit{ep}}{k}}$, then

$$\underset{\stackrel{\sim }{r}}{\inf }\underset{\mathbf{\sum }\in {\mathit{\Theta }}_2(k,p,\lambda ,\tau )}{\sup }\mathrm{{\rm P}}\{\stackrel{\sim }{r}\ne \mathrm{rank}(\mathbf{\sum })\}\ge w({\beta }_0),$$ (28)

where the function w : (0, $\frac{1}{36}$) → (0, 1) satisfies w(0+) = 1.

The upper and lower bounds given in Theorems 2 and 5 show that the optimal detection boundary for the rank r is $\sqrt{\frac{k}{n}\log \frac{\mathit{ep}}{k}}$. That is, the rank r can be estimated with an arbitrarily small error probability when ${\lambda }_r\ge \beta \sqrt{\frac{k}{n}\log \frac{\mathit{ep}}{k}}$ for a sufficiently large constant β, whereas this is impossible to achieve by any method if ${\lambda }_r\le \sqrt{\frac{{\beta }_{0}k}{n}\log \frac{\mathit{ep}}{k}}$ for some small positive constant β₀. Note that Theorem 5 applies to the full range of sparsity including the non-sparse case k = p, which requires ${\lambda }_r\ge \sqrt{\frac{p}{n}}$. This observation turns out to be useful in proving the “parametric term” in the minimax lower bound for estimating Σ in Theorem 4.

The rank detection lower bound in Theorem 5 is in fact a direct consequence of the next proposition concerning testing the identity covariance matrix against rank-one alternatives,

$$H_0:\mathbf{\sum }={\mathbf{I}}_p,\mathrm{versus}{H}_1:\mathbf{\sum }={\mathbf{I}}_p+\lambda \mathbf{v}{\mathbf{v}}^{\prime },\mathbf{v}\in {\mathrm{S}}^{p-1}\cap B_0(k),$$ (29)

where B₀(k), ≜ {x ∈ ℝ^p : |supp(x)| ≤ k}. Note that Σ is in the parameter space Θ₂ under both the null and the alternative hypotheses. The rank-one testing problem (29) has been studied in [2], where there is a gap between the lower and upper bounds when $k\gtrsim \sqrt{p}$. The following result show that their lower bound is in fact sub-optimal in this case. We shall give below a dimension-free lower bound for the optimal probability of error and determine the optimal rate of separation. The proof is deferred to Section 7.2.2.

Proposition 2

Let ${\beta }_0<\frac{1}{36}$ be a constant. Let X be an n × p random matrix whose rows are independently drawn from N(0, Σ). For any k ∈ [p] and n ∈ ℕ, if $\lambda \le 1\wedge \sqrt{\frac{{\beta }_{0}k}{n}\log \frac{\mathrm{e}p}{k}}$, the minimax sum of Type-I and Type-II error probabilities for the testing problem (29) satisfies

$${\epsilon }_n(k,p,\lambda )\triangleq \underset{\varphi :{ℝ}^{n\times p}\to \{0,1\}}{\inf }\left({\mathrm{{\rm P}}}_{{\mathbf{I}}_p}\{\varphi (\mathbf{X})=1\}+\underset{\mathbf{\sum }\in H_1}{\sup }{\mathrm{{\rm P}}}_{\mathbf{\sum }}\{\varphi (\mathbf{X})=0\}\right)\ge w({\beta }_0)$$

where the function w : (0, $\frac{1}{36}$) → (0, 1) satisfies w(0+) = 1.

Proposition 2 shows that testing independence in the rank-one spiked co-variance model can be achieved reliably only if the effective signal-to-noise-ratio

$${\beta }_0=\frac{{\lambda }^2}{\frac{k}{n}\log \frac{\mathrm{e}p}{k}}\to \infty .$$

Furthermore, the lower bound in Proposition 2 also captures the following phenomenon: if β₀ vanishes, then the optimal probability of error converges to one. In fact, the lower bound in Proposition 2 is optimal in the sense that the following test succeeds with vanishing probability of error if β₀ → ∞:

$$\mathrm{reject}{H}_0\mathrm{if\; and\; only\; if}\underset{\underset{|J|=k}{J\subset [p]}}{\max }\Vert {\mathbf{S}}_{\mathit{JJ}}-{\mathbf{I}}_{\mathit{JJ}}\Vert \ge c\sqrt{\frac{k}{n}\log \frac{\mathrm{e}p}{k}}$$

for some c only depends on β₀. See, e.g., [2, Section 4]. However, the above test has high computational complexity since one needs to enumerate all k × k submatrices of S. It remains an open problem to construct tests that are both computationally feasible and minimax rate-optimal.

4.3 Testing rank-one spiked model

As mentioned earlier, a careful study of the rank-one testing problem (29) provides a major tool for the lower bound arguments. A key step in this study is the analysis of the moment generating function of a squared symmetric random walk stopped at a hypergeometrically distributed time. We present the main ideas in this section as the techniques can also be useful for other related matrix estimation and testing problems.

It is well-known that the minimax risk is given by the least-favorable Bayesian risk under mild regularity conditions on the model [27]. For the composite testing problem (29), it turns out that the rate-optimal least-favorable prior for v is given by the distribution of the following random vector:

$$\mathbf{u}=\frac{1}{\sqrt{k}}{\mathbf{J}}_I\mathbf{w},$$ (30)

where w = (w₁, …, w_p) consists of iid Rademacher entries, and J_I is a diagonal matrix given by (J_I)_ii = 1_{i∈I} with I uniformly chosen from all subsets of [p] of size k. In other words, u is uniformly distributed on the collection of k-sparse vectors of unit length with equal-magnitude non-zeros. Hence u ∈ S^p−1 ∩ B₀(k). We set

$${\lambda }^2=\frac{{\beta }_{0}k}{n}\log \frac{\mathrm{e}p}{k},$$

where β₀ > 0 is a sufficiently small absolute constant.² The desired lower bound then follows if we establish that the following (Bayesian) hypotheses

$$H_0:\mathbf{\sum }={\mathbf{I}}_p\mathrm{v.s.}{H}_1:\mathbf{\sum }={\mathbf{I}}_p+\lambda \mathbf{u}{\mathbf{u}}^{\prime }$$ (31)

cannot be separated with vanishing probability of error.

Remark 3

The composite testing problem (31) has also been considered in [2]. In particular, the following suboptimal lower bound is given in [2, Theorem 5.1]: If

$$\lambda \le 1\wedge \sqrt{\frac{\mathit{\upsilon k}}{n}\log \left(1+\frac{p}{k^2}\right)}$$ (32)

then the optimal error probability satisfies ε_n(k, p, λ) ≥ C(υ), where C(υ) → 1 as υ → 0. This result is established based on the following prior:

$$\mathbf{u}=\frac{1}{\sqrt{k}}{\mathbf{J}}_I\mathbf{1},$$ (33)

which is a binary sparse vector with uniformly chosen support.

Compared to the result in Proposition 2, (32) is rate-optimal in the very sparse regime of $k=o(\sqrt{p})$. However, since log(1 + x) ≍ x when x ≲ 1, in the moderately sparse regime of $k\gtrsim \sqrt{p}$, $\log (1+\frac{p}{k^2})\asymp \frac{p}{k^2}$ and so the lower bound in (32) is substantially smaller than the optimal rate in Proposition 2 by a factor of $\sqrt{\frac{k^2}{p}\log \frac{\mathrm{e}p}{k}}$, which is a polynomial factor in k when k ≳ p^α for any α > 1/2. In fact, by strengthening the proof in [2], one can show that the optimal separation for discriminating (31) using the binary prior (33) is $1\wedge \sqrt{\frac{k}{n}\log \frac{\mathrm{e}p}{k}}\wedge \frac{p}{k\sqrt{n}}$. Therefore the prior (33) is rate-optimal only in the regime of $k=o(p^{\frac{2}{3}})$, while (30) is rate-optimal for all k. Examining the role of the prior (30) in the proof of Theorem 5, we see that it is necessary to randomize the signs of the singular vector in order to take advantage of the central limit theorem and Gaussian approximation. When $k\gtrsim p^{\frac{2}{3}}$, the fact that the singular vector u is positive componentwise reduces the difficulty of the testing problem.

The main technical tool for establishing the rank-detection lower bound in Proposition 2 is the following lemma, which can be of independent interest. It deals with the behavior of a symmetric random walk stopped after a hypergeometrically distributed number of steps. Moreover, note that Lemma 1 also incorporates the non-sparse case (k = p and H = k), which proves to be useful in establishing the minimax lower bound for estimating Σ in Theorem 4. The proof of Lemma 1 is deferred to Section 7.6.

Lemma 1

Let p ∈ ℕ and k ∈ [p]. Let B₁, …,B_k be independently Rademacher distributed. Denote the symmetric random walk on ℤ stopped at the m^th step by

$$G_m\triangleq \mathbf{\sum }_{i=1}^m{B}_i.$$ (34)

Let H ~ Hypergeometric(p, k, k) with $\mathrm{{\rm P}}\{H=i\}=\frac{\left(\begin{array}{c}k\\ i\end{array}\right)\left(\begin{array}{c}p-k\\ k-i\end{array}\right)}{\left(\begin{array}{c}p\\ k\end{array}\right)}$, i = 0, …, k. Then there exists a function g : (0, $\frac{1}{36}$) → (1, ∞) with g(0+) = 1, such that for any $a<\frac{1}{36}$,

$$\mathrm{{\rm E}}\exp (t{G}_H^2)\le g(a),$$ (35)

where $t=\frac{a}{k}\log \frac{\mathrm{e}p}{k}$.

Remark 4 (Tightness of Lemma 1)

The purpose of Lemma 1 is to seek the largest t, as a function of (p, k), such that Ε exp ( $t{G}_H^2$) is upper bounded by a constant non-asymptotically. The condition that $t\asymp \frac{1}{k}\log \frac{\mathit{ep}}{k}$ is in fact both necessary and sufficient. To see the necessity, note that Ρ {G_H = H∣H = i} = 2⁻ⁱ. Therefore

$$\mathrm{{\rm E}}\exp (t{G}_H^2)\ge \mathrm{{\rm E}}\exp (t{H}^2)2^{-H}\ge \exp (t{k}^2)2^{-k}\mathrm{{\rm P}}\{H=k\}\ge \exp \left(t{k}^2-k\log \frac{2p}{k}\right),$$

which cannot be upper bounded by an absolutely constant unless $t\lesssim \frac{1}{k}\log \frac{\mathrm{e}p}{k}$.

5 Estimation of Precision Matrix and Eigenvalues

We have so far focused on the optimal rates for estimating the spiked covariance matrix Σ, the rank r and the principal subspace span(V). The technical results and tools developed in the earlier sections turn out to be readily useful for establishing the optimal rates of convergence for estimating the precision matrix Ω = Σ⁻¹ as well as the eigenvalues of Σ under the spiked covariance matrix model (1).

Besides the covariance matrix Σ, it is often of significant interest to estimate the precision matrix Ω, which is closely connected to the Gaussian graphical model as the support of the precision matrix Ω coincides with the edges in the corresponding Gaussian graph. Let Σ̂ be defined in (14) and let σ_i (Σ̂) denote its i^th largest eigenvalue value for all i ∈ [p]. Define the precision matrix estimator as

$$\widehat{\mathbf{\Omega }}=\{\begin{array}{ll}{(\widehat{\mathbf{\sum }})}^{-1},& {\sigma }_p(\widehat{\mathbf{\sum }})\ge \frac{1}{2},\\ {\mathbf{I}}_p,& {\sigma }_p(\widehat{\mathbf{\sum }})<\frac{1}{2}.\end{array}$$ (36)

The following result gives the optimal rates for estimating Ω under the spectral norm.

Proposition 3 (Precision matrix estimation)

Assume that λ ≍ 1. If $\frac{k}{n}\log \frac{\mathrm{e}p}{k}\le c_0$ for a sufficiently small constant c₀ > 0, then

$$\underset{\widehat{\mathbf{\Omega }}}{\inf }\underset{\mathbf{\sum }\in {\mathit{\Theta }}_1(k,p,r,\lambda )}{\sup }\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\Omega }}-\mathbf{\Omega }\Vert }^2\asymp \frac{k}{n}\log \frac{\mathrm{e}p}{k},$$ (37)

where the upper bound is attained by the estimator (36) with γ₁ ≥ 10 in obtaining Σ̂.

The upper bound follows from the lines in the proof of Theorem 1 after we control the smallest eigenvalue of Σ̂ as in (36). Proposition 2 can be readily applied to yield the desired lower bound.

Note that the optimal rate in (37) is quite different from the minimax rate of convergence $M^2{k}^2\frac{\log p}{n}$ for estimating k-sparse precision matrices where each row/column has at most k nonzero entries. Here M is the ℓ₁ norm bound for the precision matrices. See [9]. So the sparsity in the principal eigenvectors and the sparsity in the precision matrix itself have significantly different implications for estimation of Ω under the spectral norm.

We now turn to estimation of the eigenvalues. Since σ is assumed to be equal to one, it suffices to estimate the eigenvalue matrix E = diag(λ_i) where λ_i, ≜ 0 for i > r. For any estimator Ẽ = diag(λ̃_i), we quantify the estimator error by the loss function ∥Ẽ − E∥² = max_i∈[p] |λ̃_i − λ_i|². The following result gives the optimal rate of convergence for this estimation problem.

Proposition 4 (Uniform estimation of spectra)

Under the same conditions of Proposition 3,

$$\underset{\widehat{\mathbf{E}}}{\inf }\underset{\mathbf{\sum }\in {\mathit{\Theta }}_1(k,p,r,\lambda )}{\sup }\mathrm{{\rm E}}{\Vert \widehat{\mathbf{E}}-\mathbf{E}\Vert }^2\asymp \frac{k}{n}\log \frac{\mathrm{e}p}{k},$$ (38)

where the upper bound is attained by the estimator Ê = diag(σ_i(Σ̂)) − I_p with Σ̂ defined in (14) with γ₁ ≥ 10.

Hence, the spikes and the eigenvalues can be estimated uniformly at the rate of $\frac{k}{n}\log \frac{\mathrm{e}p}{k}$ when k ≪ n/ log p. Proposition 4 is a direct consequence of Theorem 1 and Proposition 2. The proofs of Propositions 3 and 4 are deferred to Section 7.4.

6 Discussions

We have assumed the knowledge of the noise level σ = 1 and the support size k. For a given value of σ², one can always rescale the data and reduce to the case of unit variance. As a consequence of rescaling, the results in this paper remain valid for a general σ² by replacing each λ with λ/σ² in both the expressions of the rates and the definitions of the parameter spaces. When σ² is unknown, it can be easily estimated based on the data. Under the sparsity models (3)-(5), when k < p/2, σ² can be well estimated by σ̂² = median(s_jj) as suggested in [20], where s_jj is j^th diagonal element of the sample covariance matrix.

The knowledge of the support size k is much more crucial for our procedure. An interesting topic for future research is the construction of adaptive estimators which could achieve the minimax rates in Theorems 1–3 without knowing k. One possibility is to define B_k in (13) for all k ∈ [p], find the smallest k such that B_k is non-empty, and then define estimators for that particular k similar to those in Section 2 with necessary adjustments accounting for the extra multiplicity in the support selection procedure.

For ease of exposition, we have assumed that the data are normally distributed and so various non-asymptotic tail bounds used in the proof follow. Since these bounds typically only rely on sub-Gaussianity assumptions on the data, we expect the results in the present paper readily extendable to data generated from distributions with appropriate sub-Gaussian tail conditions. The estimation procedure in Section 2 is different from that in [40]. Although both are based on enumerating all possible support sets of size k, Vu and Lei [40] proposed to pick the support set which maximizes a quadratic form, while ours is based on picking the set satisfying certain deviation bounds.

A more important issue is the computational complexity required to obtain the minimax rate optimal estimators. The procedure described in Section 2 entails a global search for the support set A, which can be computationally intensive. In many cases, this seems unavoidable since the spectral norm is not separable in terms of the entries/rows/columns. However, in some other cases, there are estimators that are computationally more efficient and can attain the same rates of convergence. For instance, in the low rank cases where $r\lesssim \log \frac{\mathrm{e}p}{k}$, the minimax rates for estimating span(V) under the spectral norm and under the Frobenius norm coincide with each other. See the discussion following Theorem 3 in Section 3. Therefore the procedure introduced in [10, Section 3] attains the optimal rates under both norms simultaneously. As shown in [10], this procedure is not only computationally efficient, but also adaptive to the sparsity k. Finding the general complexity-theoretic limits for attaining the minimax rates under the spectral norm is an interesting and challenging topic for future research. Following the initial post of the current paper, Berthet and Rigollet [1] showed in a closely related sparse principal component detection problem that the minimax detection rate cannot be achieved by any computationally efficient algorithm in the highly sparse regime.

7 Proofs

We first collect a few useful technical lemmas in Section 7.1 before proving the main theorems in Section 7.2 in the order of Theorems 1 - 4. We then give the proofs of the propositions in the order of Propositions 2, 3, 4, and 1. As mentioned in Section 4, Theorem 5 on the rank detection lower bound is a direct consequence of Proposition 2. We complete this section with the proof of Lemma 1.

Recall that the row vector of X are i.i.d. samples from the N(0, Σ) distribution with Σ specified by (1). Equivalently, one can think of X as an n × p data matrix generated as

$$\mathbf{X}=\mathbf{UD}{\mathbf{V}}^{\prime }+\mathbf{Z},$$ (39)

where U is an n × r random effects matrix with iid N(0, 1) entries, $\mathbf{D}=\mathrm{diag}({\lambda }_1^{1/2},\dots ,{\lambda }_r^{1/2})$, V is p × r orthonormal, and Z has iid N(0, 1) entries which are independent of U.

7.1 Technical Lemmas

Lemma 2

Let $\mathbf{S}=\frac{1}{n}\mathbf{X}\prime \mathbf{X}$ be the sample covariance matrix, then

$$\mathrm{{\rm P}}\left(\underset{|B|=k}{\max }\Vert {\mathbf{S}}_{\mathit{BB}}\Vert \le ({\lambda }_1+1){\left(1+\sqrt{\frac{k}{n}}+\frac{t}{\sqrt{n}}\right)}^2\right)\le \left(\begin{array}{c}p\\ k\end{array}\right){\mathrm{e}}^{-t^2/2}.$$

Proof

Note that

$$\begin{array}{lll}\underset{|B|=k}{\max }\Vert {\mathbf{S}}_{\mathit{BB}}\Vert & \le & \underset{|B|=k}{\max }\Vert {\mathbf{\sum }}_{\mathit{BB}}\Vert ·\Vert {\mathbf{\sum }}_{\mathit{BB}}^{-1/2}{\mathbf{S}}_{\mathit{BB}}{\mathbf{\sum }}_{\mathit{BB}}^{-1/2}\Vert \\ & \le & ({\lambda }_1+1)\underset{|B|=k}{\max }\Vert {\mathbf{\sum }}_{\mathit{BB}}^{-1/2}{\mathbf{S}}_{\mathit{BB}}{\mathbf{\sum }}_{\mathit{BB}}^{-1/2}\Vert \stackrel{(\mathrm{d})}{=}({\lambda }_1+1)\underset{|B|=k}{\max }{\Vert {\mathbf{Z}}_{\ast B}\Vert }^2.\end{array}$$

The result follows from the Davidson–Szarek bound [14, Theorem II.7] and the union bound.

Lemma 3

Let B_k be defined as in (13) with γ₁ ≥ 3. Then Ρ(A ∉ B_k) ≤ 5(ep)^1−γ₁/2.

Proof

Note that by union bound

$$\begin{array}{ll}\mathrm{{\rm P}}(A\notin B_k)\le & \mathrm{{\rm P}}(\exists D\subset A^{\mathrm{c}},|D|\le k,\Vert {\mathbf{S}}_D-\mathbf{I}\Vert >\eta (|D|,n,p,{\gamma }_1))\\ & +\mathrm{{\rm P}}(\exists D\subset A^{\mathrm{c}},|D|\le k,\Vert {\mathbf{S}}_{\mathit{DA}}\Vert >2\sqrt{\Vert {\mathbf{S}}_A\Vert }\eta (k,n,p,{\gamma }_1)).\end{array}$$

We now bound the two terms on the right-hand side separately.

For the first term, note that A = supp(V). Then for any D ⊂ A^c, ${\mathbf{S}}_D=\frac{1}{n}{\mathbf{Z}}_{\ast D}^{\prime }{\mathbf{Z}}_{\ast D}$. Hence

$$\begin{array}{l}\mathrm{{\rm P}}(\exists D\subset A^{\mathrm{c}},|D|\le k,\Vert {\mathbf{S}}_D-\mathbf{I}\Vert >\eta (|D|,n,p,{\gamma }_1))\\ \le \mathbf{\sum }_{D\subset A^{\mathrm{c}},|D|\le k}\mathrm{{\rm P}}(\Vert {\mathbf{S}}_D-\mathbf{I}\Vert >\eta (|D|,n,p,{\gamma }_1))\\ \le \mathbf{\sum }_{l=1}^k\left(\begin{array}{c}p-k\\ l\end{array}\right)2\exp \left(-\frac{{\gamma }_1}{2}l\log \frac{\mathrm{e}p}{l}\right)\\ \le 2\mathbf{\sum }_{l=1}^k{\left(\frac{\mathrm{e}p}{l}\right)}^{l(1-{\gamma }_1/2)}\le 4{(\mathrm{e}p)}^{1-{\gamma }_1/2},\end{array}$$

where the second inequality is [10, Proposition 4], and the last inequality holds for all γ₁ ≥ 3 and p ≥ 2.

For the second term, note that for any fixed D ⊂ A^c, ${\mathbf{S}}_{\mathit{DA}}=\frac{1}{n}{\mathbf{Z}}_{\ast D}^{\prime }{\mathbf{X}}_{\ast A}$, where Z_*D and X_*A are independent. Thus, let W be the left singular vector matrix of X_*A, we obtain that³

$$\Vert {\mathbf{S}}_{\mathit{DA}}\Vert \le \frac{1}{n}\Vert {\mathbf{Z}}_{\ast D}^{\prime }\mathbf{W}\Vert \Vert {\mathbf{X}}_{\ast A}\Vert \stackrel{(\mathrm{d})}{=}\frac{1}{\sqrt{n}}\Vert \mathbf{Y}\Vert \sqrt{\Vert {\mathbf{S}}_A\Vert },$$

where Y is a |D| × k matrix with i.i.d. N(0, 1) entries. Thus, we have

$$\begin{array}{l}\mathrm{{\rm P}}(\exists D\subset A^{\mathrm{c}},|D|\le k,\Vert {\mathbf{S}}_{\mathit{DA}}\Vert >2\sqrt{\Vert {\mathbf{S}}_A\Vert }\eta (k,n,p,{\gamma }_1))\\ \le \mathbf{\sum }_{D\subset A^{\mathrm{c}},|D|\le k}\mathrm{{\rm P}}(\Vert {\mathbf{S}}_{\mathit{DA}}\Vert >2\sqrt{\Vert {\mathbf{S}}_A\Vert }\eta (k,n,p,{\gamma }_1))\\ \le \mathbf{\sum }_{D\subset A^{\mathrm{c}},|D|\le k}\mathrm{{\rm P}}(\Vert \mathbf{Y}\Vert >2\sqrt{n}\eta (k,n,p,{\gamma }_1))\\ \le \sum _{l=1}^k\left(\begin{array}{c}p-k\\ l\end{array}\right)\exp \left(-2{\gamma }_{1}k\log \frac{\mathrm{e}p}{k}\right)\\ \le k{\left(\mathrm{e}\frac{p}{k}\right)}^{k(1-2{\gamma }_1)}\le {(\mathrm{e}p)}^{1-{\gamma }_1/2},\end{array}$$

where the third inequality is due to the Davidson-Szarek bound. Combining the two bounds completes the proof.

Lemma 4

Let γ₁ ≥ 3. Suppose that $\frac{k}{n}\log \frac{\mathrm{e}p}{k}\le C_0$ for a sufficiently small constant c₀ > 0. Then with probability at least 1 − 12(ep)^1−γ₁/2, B_k ≠ 0̸ and

$$\Vert {\mathbf{S}}_{\widehat{A}}-{\mathbf{S}}_A\Vert \le C_0({\gamma }_1)\sqrt{{\lambda }_1+1}\eta (k,n,p,{\gamma }_1),$$ (40)

where $C_0({\gamma }_1)=14+4\sqrt{{\gamma }_1}$.

Proof

We focus on the event B_k ≠ 0̸. Define the sets

$$G=A\cap \widehat{A},M=A\cap {\widehat{A}}^{\mathrm{c}},O=A^{\mathrm{c}}\cap \widehat{A},$$ (41)

which correspond to the sets of correctly identified, missing, and overly identified features by the selected support set Â, respectively. By the triangle inequality, we have

$$\Vert {\mathbf{S}}_{\widehat{A}}-{\mathbf{S}}_A\Vert \le \Vert {\mathbf{S}}_{\mathit{MM}}-{\mathbf{I}}_{\mathit{MM}}\Vert +\Vert {\mathbf{S}}_{\mathit{OO}}-{\mathbf{I}}_{\mathit{OO}}\Vert +2\Vert {\mathbf{S}}_{\mathit{GM}}\Vert +2\Vert {\mathbf{S}}_{\mathit{GO}}\Vert .$$ (42)

We now provide high probability bounds for each term on the right-hand side.

For the first term, recall that  ∈ B_k which is defined in (13). Since M ⊂ Â^c, we have

$$\Vert {\mathbf{S}}_{\mathit{MM}}-{\mathbf{I}}_{\mathit{MM}}\Vert \le \eta (|M|,n,p,{\gamma }_1)\le \eta (k,n,p,{\gamma }_1).$$ (43)

For the second term, by similar calculation to that in the proof of Lemma 3, we have that when γ₁ ≥ 3,

$$\Vert {\mathbf{S}}_{\mathit{OO}}-{\mathrm{I}}_{\mathit{OO}}\Vert \le \eta (k,n,p,{\gamma }_1)$$ (44)

with probability at least 1 − 4(ep)^1−γ₁/2. For the third term, we turn to the definition of B_k in (13) again to obtain

$$\Vert {\mathbf{S}}_{\mathit{GM}}\Vert \le \Vert {\mathbf{S}}_{\widehat{A}M}\Vert \le 2\sqrt{\Vert {\mathbf{S}}_{\widehat{A}\widehat{A}}\Vert }\eta (k,n,p,{\gamma }_1).$$

By Lemma 2, with probability at least 1 − (ep)^1−γ₁/2,

$$\begin{array}{lll}\Vert {\mathbf{S}}_{\widehat{A}\widehat{A}}\Vert ,\Vert {\mathbf{S}}_{\mathit{AA}}\Vert & \le & \underset{|B|=k}{\max }\Vert S_{\mathit{BB}}\Vert \le ({\lambda }_1+1){\left(1+\sqrt{\frac{k}{n}}+\sqrt{\frac{{\gamma }_{1}k}{n}\log \frac{\mathrm{e}p}{k}}\right)}^2\\ & \le & {(1+(\sqrt{{\gamma }_1}+1)\sqrt{c_0})}^2({\lambda }_1+1)\\ & \le & {(\frac{1}{2}\sqrt{{\gamma }_1}+\frac{3}{2})}^2({\lambda }_1+1),\end{array}$$ (45)

where the second inequality holds when $\frac{k}{n}\log \frac{\mathrm{e}p}{k}\le c_0$ and the last inequality holds for sufficiently small constant c₀. Moreover, the last two displays jointly imply

$$\Vert {\mathbf{S}}_{\mathit{GM}}\Vert \le (\sqrt{{\gamma }_1}+3)\sqrt{{\lambda }_1+1}\eta (k,n,p,{\gamma }_1).$$ (46)

For the fourth term, we obtain by similar arguments that with probability at least 1 − (ep)^{1 − γ₁/2},

$$\Vert {\mathbf{S}}_{\mathit{GO}}\Vert \le \Vert {\mathbf{S}}_{\mathit{AO}}\Vert \le (\sqrt{{\gamma }_1}+3)\sqrt{{\lambda }_1+1}\eta (k,n,p,{\gamma }_1).$$ (47)

Note that Ρ(B_k = 0̸) ≤ Ρ(A ∉ B_k), and by Lemma 3, the latter is bounded above by 5(ep)^{1 −γ₁/2}. So the union bound implies that the intersection of the event {B_k ≠ 0̸} and the event that (43)-(47) all hold has probability at least 1 − 12(ep)^{1 −γ₁/2}. On this event, we assemble (42)-(47) to obtain

$$\begin{array}{lll}\Vert {\mathbf{S}}_{\widehat{A}}-{\mathbf{S}}_A\Vert & \le & [2+4(\sqrt{{\gamma }_1}+3)\sqrt{{\lambda }_1+1}]\eta (k,n,p,{\gamma }_1)\\ & \le & [2+4(\sqrt{{\gamma }_1}+3)]\sqrt{{\lambda }_1+1}\eta (k,n,p,{\gamma }_1).\end{array}$$

This completes the proof.

Lemma 5

Let Y ∈ ℝ^{n × k} and Z ∈ ℝ^{n × m} be two independent matrices with i.i.d. N (0, 1) entries. Then there exists an absolute constant C > 0, such that

$$\mathrm{{\rm E}}{\Vert \frac{1}{n}{\mathbf{Y}}^{\prime }\mathbf{Y}-{\mathbf{I}}_k\Vert }^2\le C\left(\frac{k}{n}+\frac{k^2}{n^2}\right),\mathit{and}$$ (48)

$$\mathrm{{\rm E}}{\Vert {\mathbf{Y}}^{\prime }\mathbf{Z}\Vert }^2\le C(n(k+m)+\mathit{km}).$$ (49)

Proof

The inequality (48) follows directly from integrating the high-probability upper bound in [10, Proposition 4].

Let the SVD of Y be Y = ACB′, where A,B are n × (n ∧ k) and k × (n ∧ k) uniformly Haar distributed, and C is an (n ∧ k) × (n ∧ k) diagonal matrix with ∥C∥ = ∥Y∥. Since A and Z are independent, A′Z has the same law as a (n ∧ k) × m iid Gaussian matrix. Therefore

$$\mathrm{{\rm E}}{\Vert {\mathbf{Y}}^{\prime }\mathbf{Z}\Vert }^2\le \mathrm{{\rm E}}{\Vert \mathbf{Y}\Vert }^2\mathrm{{\rm E}}{\Vert {\mathbf{A}}^{\prime }\mathbf{Z}\Vert }^2\le C_0(n+k)(n\wedge k+m).$$

for some absolute constant C₀, where the last inequality follows from the Davidson-Szarek theorem [14, Theorem II.7]. Exchanging the role of Y and Z, we have E∥Y′Z∥² ≤ C₀(n + m)(n ∧ m + k). Consequently,

$$\mathrm{{\rm E}}{\Vert {\mathbf{Y}}^{\prime }\mathbf{Z}\Vert }^2\le C_0((n+k)(n\wedge k+m)\wedge (n+m)(n\wedge m+k))\le 2C_0(n(k+m)+\mathit{km}).$$

This completes the proof.

In the proofs, the following intermediate matrix

$${\mathbf{S}}_0=\frac{1}{n}\mathbf{VD}{\mathbf{U}}^{\prime }\mathbf{UD}{\mathbf{V}}^{\prime }+{\mathbf{I}}_p$$ (50)

plays a key role. In particular, the following results on S₀ will be used repeatedly.

Lemma 6

Suppose λ₁/λ_r ≤ τ for some constant τ ≥ 1. If $\Vert \frac{1}{n}{\mathbf{U}}^{\prime }\mathbf{U}-{\mathbf{I}}_r\Vert <1/\tau$, then

$${\sigma }_l({\mathbf{S}}_0)>1,\forall l\in [r],\mathit{and}{\sigma }_l({\mathbf{S}}_0)=1,\forall l>r.$$ (51)

Moreover, VV′ is the projection matrix onto the rank r principal subspace of S₀.

Proof

It is straightforward to verify that

$${\mathbf{S}}_0-\mathbf{\sum }=\mathbf{VD}\left(\frac{1}{n}{\mathbf{U}}^{\prime }\mathbf{U}-{\mathbf{I}}_r\right)\mathbf{D}{\mathbf{V}}^{\prime }.$$ (52)

When $\Vert \frac{1}{n}{\mathbf{U}}^{\prime }\mathbf{U}-{\mathbf{I}}_r\Vert <1$, $\Vert {\mathbf{S}}_0-\sum \Vert \le {\Vert \mathbf{D}\Vert }^2\Vert \frac{1}{n}{\mathbf{U}}^{\prime }\mathbf{U}-{\mathbf{I}}_r\Vert <{\lambda }_1/\tau \le {\lambda }_r$. So Weyl’s inequality [18, Theorem 4.3.1] leads to

$${\sigma }_r({\mathbf{S}}_0)\ge {\sigma }_r(\mathbf{\sum })-\Vert {\mathbf{S}}_0-\mathbf{\sum }\Vert >{\lambda }_r+1-{\lambda }_r=1.$$

Note that S₀ always has 1 as its eigenvalue with multiplicity at least p − r. We thus obtain (51).

When (51) holds, (50) shows that the rank r principal subspace of S₀ is equal to that of $\frac{1}{n}\mathbf{VD}{\mathbf{U}}^{\prime }\mathbf{UD}{\mathbf{V}}^{\prime }$. Therefore, the subspace is spanned by the column vectors of V, and VV′ is the projection matrix onto it since V ∈ O(p, r).

To prove the lower bound for rank detection, we need the following lemma concerning the the χ²-divergences in covariance models. Recall that the χ²-divergence between two probability measures is defined as

$${\chi }^2(P\Vert Q)\triangleq {\mathit{\int }\left(\frac{\mathrm{d}P}{\mathrm{d}Q}-1\right)}^2\mathrm{d}Q.$$

For a distribution F, we use F^⊗n to denote the product distribution of n independent copies of F.

Lemma 7

Let ν be a probability distribution on the space of p × p symmetric random matrix M such that ∥M∥ ≤ 1 almost surely. Consider the scale mixture distribution Ε [N(0, I_p + M)^⊗n] = ∫ N(0, I_p + M)^⊗n ν(dM). Then

$${\chi }^2(\mathrm{{\rm E}}[N{(0,{\mathbf{I}}_p+\mathbf{M})}^{\otimes n}]\Vert N{(0,{\mathbf{I}}_p)}^{\otimes n})+1=\mathrm{{\rm E}}\det {({\mathbf{I}}_p-{\mathbf{M}}_1{\mathbf{M}}_2)}^{-\frac{n}{2}}$$ (53)

$$\ge \mathrm{{\rm E}}\exp \left(\frac{n}{2}\langle {\mathbf{M}}_1,{\mathbf{M}}_2\rangle \right),$$ (54)

where M₁ and M₂ are independently drawn from ν. Moreover, if rank(M) = 1 a.s., then (54) holds with equality.

Proof

Denote by g_i the probability density function of N (0, Σ_i) for i = 0, 1 and 2, respectively. Then it is straightforward to verify that

$$\mathit{\int }\frac{g_1{g}_2}{g_0}=\det {({\mathbf{\sum }}_0)}^{-1}{[\det ({\mathbf{\sum }}_0({\mathbf{\sum }}_1+{\mathbf{\sum }}_2)-{\mathbf{\sum }}_1{\mathbf{\sum }}_2)]}^{-\frac{1}{2}}$$ (55)

if Σ₀(Σ₁ + Σ₂) ≥ Σ₁Σ₂; otherwise, the integral on the left-hand side of (55) is infinite. Applying (55) to Σ₀ = I_p and Σ_i = I_p + M_i and using Fubini’s theorem, we have

$$\begin{array}{c}{\chi }^2(\mathrm{{\rm E}}[N{(0,{\mathbf{I}}_p+\mathbf{M})}^{\otimes n}]\Vert N{(0,{\mathbf{I}}_p)}^{\otimes n})+1\\ =\mathrm{{\rm E}}\det {({\mathbf{I}}_p-{\mathbf{M}}_1{\mathbf{M}}_2)}^{-\frac{n}{2}}\end{array}$$ (56)

$$=\mathrm{{\rm E}}\exp \left(-\frac{n}{2}\log \det ({\mathbf{I}}_p-{\mathbf{M}}_1{\mathbf{M}}_2)\right)$$ (57)

$$\ge \mathrm{{\rm E}}\exp \left(\frac{n}{2}\mathrm{Tr}({\mathbf{M}}_1{\mathbf{M}}_2)\right),$$ (58)

where (56) is due to ∥M₁M₂∥ ≤ ∥M₁∥∥M₂∥ ≤ 1 a.s., and (58) is due to log det(I + A) ≤ Tr(A), with equality if and only if rank(A) = 1.

7.2 Proofs of Main Results

7.2.1 Proofs of the Upper Bounds

Proof of Theorem 1

Let γ₁ ≥ 10 be a constant. Denote by E the event that (40) holds, which, in particular, contains the event {B_k ≠ 0̸}. By triangle inequality,

$$\begin{array}{lll}\mathrm{{\rm E}}{\Vert {\mathbf{S}}_{\widehat{A}}-\mathbf{\sum }\Vert }^2{\mathbf{1}}_E& \le & 2\mathrm{{\rm E}}{\Vert {\mathbf{S}}_{\widehat{A}}-{\mathbf{S}}_A\Vert }^2{\mathbf{1}}_E+2\mathrm{{\rm E}}{\Vert {\mathbf{S}}_A-\mathbf{\sum }\Vert }^2{\mathbf{1}}_E\\ & \lesssim & 2({\lambda }_1+1){\eta }^2(k,n,p,{\gamma }_1)+2\mathrm{{\rm E}}{\Vert {\mathbf{S}}_A-\mathbf{\sum }\Vert }^2\\ & \asymp & ({\lambda }_1+1)\frac{k}{n}\log \frac{\mathrm{e}p}{k}+\mathrm{{\rm E}}{\Vert {\mathbf{S}}_A-\mathbf{\sum }\Vert }^2,\end{array}$$ (59)

where the last step holds because $\frac{k}{n}\log \frac{\mathrm{e}p}{k}\le c_0$. In view of the definition of η in (12), we have ${\eta }^2(k,n,p,{\gamma }_1)\asymp \frac{k}{n}\log \frac{\mathrm{e}p}{k}$.

To bound the second term in (59), define J_B as the diagonal matrix given by

$${({\mathbf{J}}_B)}_{\mathit{ii}}={\mathbf{1}}_{\{i\in B\}}.$$ (60)

Then, for S₀ in (50),

$$\begin{array}{lll}{\mathbf{S}}_A& =& {\mathbf{J}}_A(\mathbf{S}-\mathbf{I}){\mathbf{J}}_A+\mathbf{I}={\mathbf{J}}_A\mathbf{S}{\mathbf{J}}_A-{\mathbf{I}}_{\mathit{AA}}+\mathbf{I}\\ & =& {\mathbf{S}}_0+\frac{1}{n}\mathbf{VD}{\mathbf{U}}^{\prime }\mathbf{Z}{\mathbf{J}}_A+\frac{1}{n}{\mathbf{J}}_A{\mathbf{Z}}^{\prime }\mathbf{UD}{\mathbf{V}}^{\prime }+{\mathbf{J}}_A\left(\frac{1}{n}{\mathbf{Z}}^{\prime }\mathbf{Z}-\mathbf{I}\right){\mathbf{J}}_A.\end{array}$$ (61)

Therefore

$$\Vert {\mathbf{S}}_A-\mathbf{\sum }\Vert \le \Vert {\mathbf{S}}_0-\mathbf{\sum }\Vert +\frac{2}{n}\Vert \mathbf{VD}{\mathbf{U}}^{\prime }\mathbf{Z}{\mathbf{J}}_A\Vert +\frac{1}{n}\Vert {\mathbf{J}}_A\left(\frac{1}{n}{\mathbf{Z}}^{\prime }\mathbf{Z}-\mathbf{I}\right){\mathbf{J}}_A\Vert$$ (62)

In view of (52) and Lemma 5, we have

$$\mathrm{{\rm E}}{\Vert {\mathbf{S}}_0-\mathbf{\sum }\Vert }^2\le {\Vert \mathbf{D}\Vert }^4\mathrm{{\rm E}}{\Vert \frac{1}{n}{\mathbf{U}}^{\prime }\mathbf{U}-{\mathbf{I}}_r\Vert }^2\lesssim {\lambda }_1^2(\frac{r}{n}+\frac{r^2}{n^2})\asymp \frac{{\lambda }_1^{2}r}{n},$$ (63)

where the last step is due to $n\ge c_{0}k\log \frac{\mathrm{e}p}{k}\ge c_{0}k$ by assumption. Similarly,

$$\mathrm{{\rm E}}{\Vert {\mathbf{J}}_A\left(\frac{1}{n}{\mathbf{Z}}^{\prime }\mathbf{Z}-\mathbf{I}\right){\mathbf{J}}_A\Vert }^2\lesssim \frac{k}{n},$$ (64)

Again by (49) in Lemma 5,

$$\frac{1}{n^2}\mathrm{{\rm E}}{\Vert \mathbf{VD}{\mathbf{U}}^{\prime }\mathbf{Z}{\mathbf{J}}_A\Vert }^2\le \frac{{\lambda }_1}{n^2}\mathrm{{\rm E}}{\Vert {\mathbf{U}}^{\prime }\mathbf{Z}{\mathbf{J}}_A\Vert }^2\lesssim \frac{{\lambda }_1(n(k+r)+\mathit{kr})}{n^2}\asymp \frac{{\lambda }_{1}k}{n}.$$ (65)

Assembling (62) - (65), we have

$$\mathrm{{\rm E}}{\Vert {\mathbf{S}}_A-\mathbf{\sum }\Vert }^2\lesssim \frac{{\lambda }_1^{2}r}{n}+\frac{({\lambda }_1+1)k}{n}.$$ (66)

Combining (59) and (66) yields

$$\mathrm{{\rm E}}{\Vert {\mathbf{S}}_{\widehat{A}}-\mathbf{\sum }\Vert }^2{\mathbf{1}}_E\lesssim ({\lambda }_1+1)\frac{k}{n}\log \frac{\mathrm{e}p}{k}+\frac{{\lambda }_1^{2}r}{n}.$$ (67)

Next we control the estimation error conditioned on the complement of the event E. First note that ${\mathbf{S}}_{\widehat{A}}-\mathbf{\sum }=\left[\begin{array}{cc}{\mathbf{S}}_{\widehat{A}\widehat{A}}& \mathbf{0}\\ \mathbf{0}& {\mathbf{I}}_{{\widehat{A}}^{\mathrm{c}}{\widehat{A}}^{\mathrm{c}}}\end{array}\right]-\mathbf{\sum }$. Then ∥S_Â − Σ∥ ≤ ∥S∥ + ∥Σ∥ + 1 ≤ ∥Σ∥ (∥W_p∥ + 1), where W_p is equal to $\frac{1}{n}$ times a p × p Wishart matrix with n degrees of freedom. Also, ∥Σ − I∥ = λ₁. In view of (14), we have ∥Σ̂ − Σ∥ ≤ (1 + λ₁)(∥W_p∥ + 2). Using Cauchy-Schwartz inequality, we have

$$\begin{array}{lll}\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert }^2{\mathbf{1}}_{E^{\mathrm{c}}}& \le & {(1+{\lambda }_1)}^2\mathrm{{\rm E}}{(\Vert {\mathbf{W}}_p\Vert +2)}^2\mathbf{1}{E}^{\mathrm{c}}\\ & \le & {(1+{\lambda }_1)}^2\sqrt{\mathrm{{\rm E}}{(\Vert {\mathbf{W}}_p\Vert +2)}^4}\sqrt{\mathrm{{\rm P}}\{{\mathbf{1}}_{E^{\mathrm{c}}}\}}.\end{array}$$

Note that $\Vert {\mathbf{W}}_p\Vert \stackrel{(\mathrm{d})}{=}\frac{1}{n}{\sigma }_1^2(\mathbf{Z})$. By [14, Theorem II.7], $\mathrm{{\rm P}}\{{\sigma }_1(\mathbf{Z})\ge 1+\sqrt{\frac{p}{n}}+t\}\le \exp (-n{t}^2/2)$. Then $\mathrm{{\rm E}}{\Vert {\mathbf{W}}_p\Vert }^4\lesssim \frac{1}{n^4}(1+\frac{p^4}{n^4})$. By Lemma 4, we have Ρ {E^c} ≤ 12(ep)^1−γ₁/2. Therefore

$$\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert }^2{\mathbf{1}}_{E^{\mathrm{c}}}\lesssim {(1+{\lambda }_1)}^2\frac{1}{n^2}\left(1+\frac{p^2}{n^2}\right)p^{\frac{1}{2}-\frac{{\gamma }_1}{4}}$$ (68)

Choose a fixed γ₁ ≥ 10. Assembling (67) and (68), we have

$$\begin{array}{lll}\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert }^2& =& \mathrm{{\rm E}}{\Vert {\mathbf{S}}_{\widehat{A}}-\mathbf{\sum }\Vert }^2{\mathbf{1}}_E+\mathrm{{\rm E}}{\Vert {\mathbf{S}}_{\widehat{A}}-\mathbf{\sum }\Vert }^2{\mathbf{1}}_{E^{\mathrm{c}}}\\ & \lesssim & (1+{\lambda }_1)\frac{k}{n}\log \frac{\mathrm{e}p}{k}+\frac{{\lambda }_1^{2}r}{n}+{\left(1+{\lambda }_1\right)}^2\frac{1}{n^2}\asymp (1+{\lambda }_1)\frac{k}{n}\log \frac{\mathrm{e}p}{k}+\frac{{\lambda }_1^{2}r}{n}.\end{array}$$

Proof of Theorem 2

To prove the theorem, it suffices to show that with the desired probability,

$${\sigma }_r({\mathbf{S}}_{\widehat{A}})>1+{\gamma }_2\sqrt{\Vert {\mathbf{S}}_{\widehat{A}}\Vert }\eta (k,n,p,{\gamma }_1)>{\sigma }_{r+1}({\mathbf{S}}_{\widehat{A}}).$$ (69)

Recall (50) and (52). By [10, Proposition 4], with probability at least 1 − 2(ep)^−γ₁/2, $\Vert \frac{1}{n}\mathbf{U}\prime \mathbf{U}-{\mathbf{I}}_r\Vert \le \eta (k,n,p,{\gamma }_1)$. Under (18), $\sqrt{\frac{k}{n}\log \frac{\mathrm{e}p}{k}}\asymp \eta (k,n,p,{\gamma }_1)\le 1/(2\tau )$ when c₀ is sufficiently small, and so η(k, n, p, γ₁) ≤ 1/(2τ). Thus,

$$\Vert {\mathbf{S}}_0-\mathbf{\sum }\Vert \le {\Vert \mathbf{D}\Vert }^2\Vert \frac{1}{n}{\mathbf{U}}^{\prime }\mathbf{U}-{\mathbf{I}}_r\Vert \le {\lambda }_r/2.$$ (70)

Therefore, Lemma 6 leads to (51). Moreover, Weyl’s inequality leads to

$${\sigma }_r({\mathbf{S}}_0)\ge {\sigma }_r(\mathbf{\sum })-|{\sigma }_r({\mathbf{S}}_0)-{\sigma }_r(\mathbf{\sum })|\ge {\lambda }_r+1-{\lambda }_1\eta (k,n,p,{\gamma }_1)\ge \frac{1}{2}{\lambda }_r+1>1.$$ (71)

Next, we consider S_A − S₀. By (61), we have

$$\Vert {\mathbf{S}}_A-{\mathbf{S}}_0\Vert \le \frac{2}{n}\Vert \mathbf{VD}{\mathbf{U}}^{\prime }\mathbf{Z}{\mathbf{J}}_A\Vert +\frac{1}{n}\Vert {\mathbf{J}}_A({\mathbf{Z}}^{\prime }\mathbf{Z}-\mathbf{I}){\mathbf{J}}_A\Vert .$$

By [10, Proposition 4] and (18), when c₀ is sufficiently small, with probability at least 1 − (ep + 1)(ep)^−γ₁/2,

$$\begin{array}{lll}\Vert \mathbf{VD}{\mathbf{U}}^{\prime }\mathbf{Z}{\mathbf{J}}_A\Vert & \le & \Vert \mathbf{D}\Vert \Vert {\mathbf{U}}^{\prime }\mathbf{Z}{\mathbf{J}}_A\Vert \\ & \le & \sqrt{{\lambda }_1}n\sqrt{1+\frac{8{\gamma }_1\log p}{3n}}\left(\sqrt{\frac{r}{n}}+\sqrt{\frac{k}{n}}+\sqrt{{\gamma }_1\log \frac{\mathrm{e}p}{n}}\right)\\ & \le & n\sqrt{{\lambda }_1}\eta (k,p,n,{\gamma }_1).\end{array}$$

Moreover, [10, Proposition 3] implies that with probability at least 1−2(ep)^−γ₁/2,

$$\frac{1}{n}\Vert {\mathbf{J}}_A({\mathbf{Z}}^{\prime }\mathbf{Z}-\mathbf{I}){\mathbf{J}}_A\Vert \le \eta (k,n,p,{\gamma }_1).$$

Assembling the last three displays, we obtain that with probability at least 1 − (ep + 3)(ep)^−γ₁/2,

$$\Vert {\mathbf{S}}_A-{\mathbf{S}}_0\Vert \le (2\sqrt{{\lambda }_1}+1)\eta (k,n,p,{\gamma }_1).$$ (72)

Last but not least, Lemma 4 implies that, under the condition of Theorem 2, with probability at least 1 − 12(ep)^1−γ₁/2, (40) holds. Together with (72), it implies that

$$\Vert {\mathbf{S}}_{\widehat{A}}-{\mathbf{S}}_0\Vert \le [C_0({\gamma }_1)+3]\sqrt{{\lambda }_1+1}\eta (k,n,p,{\gamma }_1),$$ (73)

where $C_0({\gamma }_1)=14+4\sqrt{{\gamma }_1}$. By (18), we could further upper bound the right-hand side by λ₁/4 ∨ 1/2. When λ₁ ≤ 1, $\sqrt{{\lambda }_1+1}\le \sqrt{2{\lambda }_1}<\sqrt{2}{\lambda }_1$, so for sufficiently small c₀, (18) implies that the right side of (73) is further bounded by λ₁/4. When λ₁ ≤ 1, $\sqrt{{\lambda }_1+1}\le \sqrt{2}$, and so the right side of (73) is further bounded by 1/2 for sufficiently small c₀.

Thus, the last display, together with (70), implies

$$\begin{array}{ll}\Vert {\mathbf{S}}_{\widehat{A}}-\mathbf{\sum }\Vert & \le {\lambda }_1\eta (k,n,p,{\gamma }_1)+[C_0({\gamma }_1)+3]\sqrt{{\lambda }_1+1}\eta (k,n,p,{\gamma }_1)\\ & <\frac{1}{2}({\lambda }_1+1).\end{array}$$ (74)

Here, the last inequality comes from the above discussion, and the fact that η(k, n, p, γ₁) < 1/4 for small c₀. The triangle inequality further leads to

$$\frac{1}{2}({\lambda }_1+1)<\Vert \mathbf{\sum }\Vert -\Vert {\mathbf{S}}_{\widehat{A}}-\mathbf{\sum }\Vert \le \Vert {\mathbf{S}}_{\widehat{A}}\Vert \le \Vert \mathbf{\sum }\Vert +\Vert {\mathbf{S}}_{\widehat{A}}-\mathbf{\sum }\Vert <\frac{3}{2}({\lambda }_1+1).$$ (75)

Set ${\gamma }_2\ge 2[C_0({\gamma }_1)+3]=8\sqrt{{\gamma }_1}+34$. Then (51), (73) and (75) jointly imply that, with probability at least 1−(13ep+5)(ep)^−γ₁/2, the second inequality in (69) holds. Moreover, (74) and the triangle inequality implies that, with the same probability,

$$\begin{array}{lll}{\sigma }_r({\mathbf{S}}_{\widehat{A}})& \ge & {\sigma }_r(\mathbf{\sum })-\Vert {\mathbf{S}}_{\widehat{A}}-\mathbf{\sum }\Vert \\ & \ge & 1+{\lambda }_r-{\lambda }_1\eta (k,n,p,{\gamma }_1)-[C_0({\gamma }_1)+3]\sqrt{{\lambda }_1+1}\eta (k,n,p,{\gamma }_1)\\ & \ge & 1+{\lambda }_r\left[1-\mathit{\tau \eta }(k,n,p,{\gamma }_1)-\frac{\sqrt{{\lambda }_1+1}}{2{\lambda }_1}{\gamma }_2\mathit{\tau \eta }(k,n,p,{\gamma }_1)\right]\\ & \ge & 1+\frac{3}{2}{\gamma }_2({\lambda }_1+1)\eta (k,n,p,{\gamma }_1).\end{array}$$

Here, the last inequality holds when ${\lambda }_r\ge \beta \sqrt{\frac{k}{n}\log \frac{\mathrm{e}p}{k}}$ for a sufficiently large β which depends only on γ₁, γ₂ and τ. In view of (75), the last display implies the first inequality of (69). This completes the proof of the upper bound.

Proof of Theorem 3

Let E be the event such that Lemma 3, Lemma 4, the upper bound in Theorem 2, and (71) hold. Then Ρ(E^c) ≤ C(ep)^1−γ₁/2.

On the event E, Σ̂ = S_Â. Moreover, Lemma 6 shows that VV′ is the projection matrix onto the principal subspace of S₀, and Theorem 2 ensures V̂ ∈ ℝ^{p × r}. Thus, Proposition 1 leads to

$$\begin{array}{lll}\Vert \widehat{\mathbf{\sum }}-{\mathbf{S}}_0\Vert {\mathbf{1}}_{\{E\}}& \ge & \frac{1}{2}({\sigma }_r({\mathbf{S}}_0)-{\sigma }_{r+1}({\mathbf{S}}_0))\Vert \widehat{\mathbf{V}}{\widehat{\mathbf{V}}}^{\prime }-\mathbf{V}{\mathbf{V}}^{\prime }\Vert {\mathbf{1}}_{\{E\}}\\ & \ge & \frac{{\lambda }_r}{4}\Vert \widehat{\mathbf{V}}{\widehat{\mathbf{V}}}^{\prime }-\mathbf{V}{\mathbf{V}}^{\prime }\Vert {\mathbf{1}}_{\{E\}},\end{array}$$

and so

$$\mathrm{{\rm E}}{\Vert \widehat{\mathbf{V}}{\widehat{\mathbf{V}}}^{\prime }-\mathbf{V}{\mathbf{V}}^{\prime }\Vert }^2{\mathbf{1}}_{\{E\}}\le \frac{16}{{\lambda }_r^2}\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\sum }}-{\mathbf{S}}_0\Vert }^2{\mathbf{1}}_{\{E\}}.$$

To further bound the right-hand side of the last display, we apply (61), (64), (65), Lemma 3 and Lemma 4 to obtain

$$\begin{array}{lll}\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\sum }}-{\mathbf{S}}_0\Vert }^2{\mathbf{1}}_{\{E\}}& \le & \mathrm{{\rm E}}{\Vert {\mathbf{S}}_{\widehat{A}}-{\mathbf{S}}_0\Vert }^2{\mathbf{1}}_{\{E\}}\\ & \lesssim & \mathrm{{\rm E}}{\Vert {\mathbf{S}}_{\widehat{A}}-{\mathbf{S}}_A\Vert }^2{\mathbf{1}}_{\{E\}}+\mathrm{{\rm E}}{\Vert {\mathbf{S}}_A-{\mathbf{S}}_0\Vert }^2\lesssim ({\lambda }_1+1)\frac{k}{n}\log \frac{\mathrm{e}p}{k}.\end{array}$$

Together with the second last display, this implies

$$\mathrm{{\rm E}}{\Vert \widehat{\mathbf{V}}{\widehat{\mathbf{V}}}^{\prime }-\mathbf{V}{\mathbf{V}}^{\prime }\Vert }^2{\mathbf{1}}_{\{E\}}\le C_{{\tau }^2}\frac{k({\lambda }_1+1)}{n{\lambda }_1^2}\log \frac{\mathrm{e}p}{k}.$$ (76)

Now consider the event E^c. Note that ∥V̂V̂′ − VV′∥ ≤ 1 always holds. Thus,

$$\mathrm{{\rm E}}{\Vert \widehat{\mathbf{V}}{\widehat{\mathbf{V}}}^{\prime }-\mathbf{V}{\mathbf{V}}^{\prime }\Vert }^2{\mathbf{1}}_{\{E^{\mathrm{c}}\}}\le \mathrm{{\rm P}}(E^{\mathrm{c}})\le C{(\mathrm{e}p)}^{1-{\gamma }_1/2}\lesssim \frac{{\lambda }_1+1}{n{\lambda }_1^2},$$ (77)

where the last inequality holds under condition (20) for all ${\gamma }_1\ge (1+\frac{2}{M_o})M_1$. Assembling (76) and (77), we obtain the upper bounds.

7.2.2 Proofs of the Lower Bounds

Proof of Theorem 4

1° The minimax lower bound for estimating span(V) follows straightforwardly from previous results on estimating the leading singular vector, i.e., the rank-one case (see, e.g., [5, 39]). The desired lower bound (25) can be found in [10, Eq. (58)] in the proof of [10, Proof of Theorem 3].

2° Next we establish the minimax lower bound for estimating the spiked covariance matrix Σ under the spectral norm, which is considerably more involved. Let Θ₁ = Θ₁(k, p, r, λ). In view of the fact that a ∧ (b + c) ≤ a ∧ b + a ∧ c ≤ 2(a ∧ (b + c)) for all a, b, c ≥ 0, it is convenient to prove the following equivalent lower bound

$$\underset{\widehat{\mathbf{\sum }}}{\inf }\underset{\mathbf{\sum }\in \mathit{\Theta }}{\sup }\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert }^2\gtrsim {\lambda }^2\wedge \left(\frac{(\lambda +1)k}{n}\log \frac{\mathit{ep}}{k}\right)+{\lambda }^2\left(1\wedge \frac{r}{n}\right).$$ (78)

To this end, we show that the minimax risk is lower bounded by the two terms on the right-hand side of (78) separately. In fact, the first term is the minimax rate in the rank-one case and the second term is the rate of the oracle risk when the estimator knows the true support of V.

2.1° Consider the following rank-one subset of the parameter space

$${\mathit{\Theta }}^{\prime }=\{\mathbf{\sum }={\mathbf{I}}_p+\lambda \mathbf{v}{\mathbf{v}}^{\prime }:\mathbf{v}\in B_0(k)\cap {\mathrm{S}}^{p-1}\}\subset {\mathit{\Theta }}_1.$$

Then σ₁(Σ) = 1 + λ and σ₂(Σ) = 1. For any estimator Σ̂ denote by v̂ its leading singular vector. Applying Proposition 1 yields

$$\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert \ge \frac{\lambda }{2}\Vert \widehat{\mathbf{v}}{\widehat{\mathbf{v}}}^{\prime }-\mathbf{v}{\mathbf{v}}^{\prime }\Vert .$$

Then

$$\begin{array}{lll}\underset{\widehat{\mathbf{\sum }}}{\inf }\underset{\mathbf{\sum }\in {\mathit{\Theta }}^{\prime }}{\sup }\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert }^2& \ge & \frac{{\lambda }^2}{4}\underset{\widehat{\mathbf{v}}}{\inf }\underset{\mathbf{v}\in B_0(k)\cap {\mathrm{S}}^{p-1}}{\sup }\mathrm{{\rm E}}{\Vert \widehat{\mathbf{v}}{\widehat{\mathbf{v}}}^{\prime }-\mathbf{v}{\mathbf{v}}^{\prime }\Vert }^2\\ & \ge & \frac{{\lambda }^2}{8}\underset{\widehat{\mathbf{v}}}{\inf }\underset{\mathbf{v}\in B_0(k)\cap {\mathrm{S}}^{p-1}}{\sup }\mathrm{{\rm E}}{\Vert \widehat{\mathbf{v}}{\widehat{\mathbf{v}}}^{\prime }-\mathbf{v}{\mathbf{v}}^{\prime }\Vert }_{\mathrm{F}}^2\end{array}$$ (79)

$$\gtrsim {\lambda }^2\left(1\wedge \frac{(\lambda +1)k}{n{\lambda }^2}\log \frac{\mathit{ep}}{k}\right),$$ (80)

where (79) follows from rank(v̂v̂′− vv′) ≤ 2 and (80) follows from the minimax lower bound in [39, Theorem 2.1] (see also [5, Theorem 2]) for estimating the leading singular vector.

2.2° To prove the lower bound ${\lambda }^2(1\wedge \frac{r}{n})$, consider the following (composite) hypotheses testing problem:

$$H_0:\mathbf{\sum }=\left[\begin{array}{cc}(1+\frac{\lambda }{2}){\mathbf{I}}_r& \mathbf{0}\\ \mathbf{0}& {\mathbf{I}}_{p-r}\end{array}\right],\mathrm{vs}.H_1:\mathbf{\sum }=\left[\begin{array}{cc}(1+\frac{\lambda }{2})({\mathbf{I}}_r+\rho \mathbf{v}{\mathbf{v}}^{\prime })& \mathbf{0}\\ \mathbf{0}& {\mathbf{I}}_{p-r}\end{array}\right],\mathbf{v}\in {\mathrm{S}}^{r-1},$$ (81)

where $\rho =\frac{b\lambda }{\lambda +2}(1\wedge \sqrt{\frac{r}{n}})$ with a sufficiently small absolute constant b > 0. Since r ∈ [k] and $\rho \left(1+\frac{\lambda }{2}\right)\le \lambda$, both the null and the alternative hypotheses belong to the parameter set Θ₁ defined in (4). Following the Le Cam’s twopoint argument [38, Section 2.3], next we show that the minimal sum of Type-I and Type-II error probabilities of testing (81) is non-vanishing. Since any pair of covariance matrices in H₀ and H₁ differ in operator norm by at least $\rho (1+\frac{\lambda }{2})=\frac{b\lambda }{2}\left(1\wedge \sqrt{\frac{r}{n}}\right)$, we obtain a lower bound of rate ${\lambda }^2(1\wedge \frac{r}{n})$.

To this end, let X consist of n iid rows drawn from N(0, Σ), where Σ is either from H₀ or H₁. Since under both the null and the alternative, the last p−r columns of X are standard normal and independent of the first r columns, we conclude that the first r columns form a sufficient statistic. Therefore the minimal Type-I+II error probability testing (81), denoted by ε_n, is equal to that of the following testing problem of dimension r and sample size n:

$$H_0:\mathbf{\sum }={\mathbf{I}}_r,\mathrm{versus}{H}_1:\mathbf{\sum }={\mathbf{I}}_r+\rho \mathbf{v}{\mathbf{v}}^{\prime },\mathbf{v}\in {\mathrm{S}}^{r-1}.$$ (82)

Recall that the minimax risk is lower bounded by the Bayesian risk. For any random vector u taking values in S^r−1, denote by the Ε[N(0, I_r + ρuu′)^⊗n] the mixture alternative distribution with a prior equal to the distribution of u. Applying [38, Theorem 2.2 (iii)] we obtain the following lower bound in terms of the χ²-divergence from the mixture alternative to the null:

$${\epsilon }_n\ge 1-{({\chi }^2(\mathrm{{\rm E}}[N{(0,{\mathbf{I}}_r+\rho \mathbf{u}{\mathbf{u}}^{\prime })}^{\otimes n}]\Vert N{(0,{\mathbf{I}}_r)}^{\otimes n})/2)}^{1/2}$$ (83)

Consider the unit random vector u with iid coordinates taking values in $\frac{1}{\sqrt{r}}\left\{\pm 1\right\}$ uniformly. Since ρ ≤ 1, applying the equality case of Lemma 7 yields

$$\begin{array}{lll}1+{\chi }^2{(\mathrm{{\rm E}}[N(0,{\mathbf{I}}_r+\rho \mathbf{u}{\mathbf{u}}^{\prime })}^{\otimes n}]\Vert N{(0,{\mathbf{I}}_r)}^{\otimes n)}& =& \mathrm{{\rm E}}\exp (n{\rho }^2{\langle \mathbf{u}\mathbf{u},\stackrel{\sim }{\mathbf{u}}\stackrel{\sim }{\mathbf{u}}\rangle }^2)\\ & =& \mathrm{{\rm E}}\exp (n{\rho }^2{G}_r^2),\end{array}$$

where in the last step we recall that G_r is the symmetric random walk on ℤ at r^th step defined in (34). Since $n{\rho }^2\le b^2\frac{r}{n}$, choosing $b^2\le \frac{1}{36}$ as a fixed constant and applying Lemma 1 with p = k = r (the non-sparse case), we conclude that

$${\chi }^2{(\mathrm{{\rm E}}[N(0,{\mathbf{I}}_p+\lambda \mathbf{u}{\mathbf{u}}^{\prime })}^{\otimes n}]\Vert N{(0,{\mathbf{I}}_p)}^{\otimes n})\le g(b^2)-1,$$ (84)

where g is given by in Lemma 1 and satisfies g > 1. Combining (83) - (84), we obtain the following lower bound for estimating Σ:

$$\underset{\widehat{\mathbf{\sum }}}{\inf }\underset{\mathbf{\sum }\in \mathit{\Theta }}{\sup }\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert }^2\gtrsim {\left(1+\frac{\lambda }{2}\right)}^2{\rho }^2\asymp {\lambda }^2\left(1\wedge \frac{r}{n}\right).$$ (85)

As we mentioned in Section 4, the rank-detection lower bound in Theorem 5 is a direct consequence of Proposition 2 concerning testing rank-zero versus rank-one perturbation, which we prove below.

7.3 Proof of Proposition 2

Proof

Analogous to (83), any random vector u taking vales in S^p−1 ∩ B₀(k) gives the Bayesian lower bound

$${\epsilon }_n(k,p,\lambda )\ge 1-({\chi }^2(\mathrm{{\rm E}}N{(0,{\mathbf{I}}_p+\lambda \mathbf{u}{\mathbf{u}}^{\prime })}^{\otimes n}\Vert N{(0,{\mathbf{I}}_p)}^{\otimes n})/2).$$ (86)

Put

$$t\triangleq \frac{n{\lambda }^2}{k^2}=\frac{{\beta }_0}{k}\log \frac{\mathrm{e}p}{k}.$$

Let the random sparse vector u be defined in (30). In view of Lemma 7 as well as the facts that rank(λvv′) = 1 and ∥λvv′∥ = λ ≤ 1, we have

$$\begin{array}{lll}1+{\chi }^2{(\mathrm{{\rm E}}[N(0,{\mathbf{I}}_p+\lambda \mathbf{u}{\mathbf{u}}^{\prime })}^{\otimes n}& ]& \Vert N{(0,{\mathbf{I}}_p)}^{\otimes n})\\ & =& \mathrm{{\rm E}}\exp (n{\lambda }^2{\langle \mathbf{u}{\mathbf{u}}^{\prime },\stackrel{\sim }{\mathbf{u}}{\stackrel{\sim }{\mathbf{u}}}^{\prime }\rangle }^2)\\ & =& \mathrm{{\rm E}}\exp \left(n{\lambda }^2{\langle \frac{1}{\sqrt{k}}{\mathbf{J}}_I\mathbf{w},\frac{1}{\sqrt{k}}{\mathbf{J}}_{\stackrel{\sim }{I}}\stackrel{\sim }{\mathbf{w}}\rangle }^2\right)\\ & =& \mathrm{{\rm E}}\exp (t{({\mathbf{w}}^{\prime }{\mathbf{J}}_{I\cap \stackrel{\sim }{I}}{\stackrel{\sim }{\mathbf{w}}}^{\prime })}^2)\\ & =& \mathrm{{\rm E}}\exp (t{G}_H^2),\end{array}$$

where in the last step we have defined H ≜ |I ∩ Ĩ| ~ Hypergeometric(p, k, k) and {G_m} is the symmetric random walk on ℤ defined in (34). Now applying Lemma 1, we conclude that

$${\chi }^2{(\mathrm{{\rm E}}[N(0,{\mathbf{I}}_p+\lambda \mathbf{u}{\mathbf{u}}^{\prime })}^{\otimes n}]\Vert N{(0,{\mathbf{I}}_p)}^{\otimes n})\le g({\beta }_0)-1$$

where g is given by in Lemma 1 satisfying g(0+) = 1. In view of (86), we conclude that

$${\epsilon }_n(k,p,\lambda )\ge \max \{0,1-\sqrt{(g({\beta }_0)-1)/2\}}\triangleq w({\beta }_0).$$

Note that the function w satisfies w(0+) = 1.

7.4 Proof of Propositions 3 and 4

We give here a joint proof of Propositions 3 and 4.

Proof

1° (Upper bounds) By assumption, we have λ ≍ 1 and $\frac{k}{n}\log \frac{\mathrm{e}p}{k}\le c_0$. Since r ∈ [k], applying Theorem 1 yields

$$\underset{\mathbf{\sum }\in {\mathit{\Theta }}_1(k,p,r,\lambda )}{\sup }\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert }^2\lesssim \frac{k}{n}\log \frac{\mathrm{e}p}{k}.$$ (87)

Note that

$$\Vert {\widehat{\mathbf{\sum }}}^{-1}-{\mathbf{\sum }}^{-1}\Vert \le \Vert {\widehat{\mathbf{\sum }}}^{-1}\Vert \Vert {\mathbf{\sum }}^{-1}\Vert \Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert \le \frac{\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert }{1-\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert },$$ (88)

where the last inequality follows from σ_p (Σ) = 1 and Weyl’s inequality. By Chebyshev’s inequality, $\mathrm{{\rm P}}\left\{\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert \ge t\right\}\le \frac{1}{t^2}\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert }^2$. Let $E=\{\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert \le \frac{1}{2}\}$. Then

$$\mathrm{{\rm P}}\{E^{\mathrm{c}}\}\lesssim \frac{k}{n}\log \frac{\mathrm{e}p}{k}.$$ (89)

Moreover, again by Weyl’s inequality, we have $E\subset \{{\sigma }_p\left(\widehat{\mathbf{\sum }}\right)\ge \frac{1}{2}\}$ hence $\widehat{\mathbf{\Omega }}{\mathbf{1}}_{\left\{E\right\}}={\left(\widehat{\mathbf{\sum }}\right)}^{-1}{\mathbf{1}}_{\left\{E\right\}}$ in view of (36). On the other hand, by definition we always have ∥Ω̂∥ ≤ 2. Therefore

$$\begin{array}{lll}\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\Omega }}-\mathbf{\Omega }\Vert }^2& =& \mathrm{{\rm E}}{\Vert {\widehat{\mathbf{\sum }}}^{-1}-{\mathbf{\sum }}^{-\mathbf{1}}\Vert }^2{\mathbf{1}}_{\{E\}}+\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\Omega }}-\mathbf{\Omega }\Vert }^2{\mathbf{1}}_{\{E^{\mathrm{c}}\}}\\ & \le & \mathrm{{\rm E}}{\Vert {\widehat{\mathbf{\sum }}}^{-1}-{\mathbf{\sum }}^{-1}\Vert }^2{\mathbf{1}}_{\{E\}}+3\mathrm{{\rm P}}\{E^{\mathrm{c}}\}\\ & \le & 4\mathrm{{\rm E}}{\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert }^2+3\mathrm{{\rm P}}\{E^{\mathrm{c}}\}\end{array}$$ (90)

$$\lesssim \frac{k}{n}\log \frac{\mathrm{e}p}{k}.$$ (91)

where (90) follows from (88) and (91) is due to (87) and (89).

Next we prove upper bound for estimating E. Recall that E+I_p and Ê+I_p give the diagonal matrices formed by the ordered singular values of Σ and Σ̂, respectively. Similar to the proof of Proposition 4, Weyl’s inequality implies that ∥Ê−E∥ = ∥Ê+I_p−(E+I_p)∥ = max_i |σ_i(Σ̂)−σ_i(Σ)| ≤ ∥Σ̂−Σ∥, where for any i ∈ [p], σ_i(Σ) is the i^th largest eigenvalue of Σ. In view of Theorem 1, we have $\mathrm{{\rm E}}{\Vert \widehat{\mathbf{E}}-\mathbf{E}\Vert }^2\le \frac{k}{n}\log \frac{\mathrm{e}p}{k}$.

2° (Lower bounds) The lower bound follows from the testing result in Proposition 2. Consider the testing problem (29), then both the null (Σ = I_p) and alternatives (Σ = I_p+λvv′) are contained in the parameter space Θ₁. By Proposition 2, they cannot be distinguished with probability 1 if ${\lambda }^2\asymp \frac{k}{n}\log \frac{\mathrm{e}p}{k}$. The spectra differs at least |σ₁(I_p) − σ₁(I_p + λvv′)| = λ. By the Woodbury identity, ${\left({\mathbf{I}}_p+\lambda \mathbf{vv}\prime \right)}^{-1}={\mathbf{I}}_p-\frac{\lambda }{\lambda +1}\mathbf{v}{\mathbf{v}}^{\prime }$, hence $\Vert {\mathbf{I}}_p-{\left({\mathbf{I}}_p+\lambda \mathbf{v}{\mathbf{v}}^{\prime }\right)}^{-1}\Vert =\frac{\lambda }{1+\lambda }\asymp \lambda$. The lower bound $\frac{k}{n}\log \frac{\mathrm{e}p}{k}$ is now completed by way of the usual two-point argument.

7.5 Proof of Proposition 1

Proof

Recall that σ₁ (Σ) ≥ … ≥ σ_p(Σ) ≥ 0 denote the ordered singular values of Σ. If ${\sigma }_r\left(\widehat{\mathbf{\sum }}\right)\le \frac{{\sigma }_r\left(\mathbf{\sum }\right)+{\sigma }_{r+1}\left(\mathbf{\sum }\right)}{2}$, then by Weyl’s inequality, we have $\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert \ge \frac{1}{2}\left({\sigma }_r\left(\mathbf{\sum }\right)-{\sigma }_{r+1}\left(\mathbf{\sum }\right)\right)$. If ${\sigma }_r\left(\widehat{\mathbf{\sum }}\right)\ge \frac{{\sigma }_r\left(\mathbf{\sum }\right)+{\sigma }_{r+1}\left(\mathbf{\sum }\right)}{2}$, then by David–Kahn’s sin-theta theorem [15] (see also [10, Theorem 10] we have

$$\Vert \widehat{\mathbf{V}}{\widehat{\mathbf{V}}}^{\prime }-\mathbf{V}{\mathbf{V}}^{\prime }\Vert \le \frac{\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert }{|{\sigma }_r(\widehat{\mathbf{\sum }})-{\sigma }_{r+1}(\mathbf{\sum })|}\le \frac{2\Vert \widehat{\mathbf{\sum }}-\mathbf{\sum }\Vert }{{\sigma }_r(\mathbf{\sum })-{\sigma }_{r+1}(\mathbf{\sum })},$$

completing the proof of (22) in view of the fact that ∥V̂V̂′ − VV′∥ ≤ 1.

7.6 Proof of Lemma 1

Proof

First of all, we can safely assume that p ≥ 5, for otherwise the expectation on the right-hand side of (35) is obviously upper bounded by an absolutely constant. In the sequel we shall assume that

$$0<a\le \frac{1}{36}.$$ (92)

We use normal approximation of the random walk G_m for small m and use truncation argument to deal with large m. To this end, we divide [p], the whole range of k, into three regimes.

Case I: Large k

Assume that $\frac{p}{2}\le k\le p$. Then $t\le \frac{2a\log \left(2\mathrm{e}\right)}{p}$. By the following non-asymptotic version of the Tusnády’s coupling lemma (see, for example, [6, Lemma 4, p. 242]), for each m, there exists Z_m ~ N(0, m), such that

$$|G_m|\stackrel{\mathrm{s}.\mathrm{t}.}{\le }|Z_m|+2.$$ (93)

Since H ≤ p, in view of (92), we have

$$\begin{array}{lll}\mathrm{{\rm E}}\exp (t{G}_H^2)& \le & \mathrm{{\rm E}}\exp (t{G}_p^2)\le \mathrm{{\rm E}}\exp (t{(|Z_p|+2)}^2)\\ & \le & \exp (8t)\mathrm{{\rm E}}\exp (2t{Z}_p^2)\end{array}$$ (94)

$$=\frac{\exp (8t)}{\sqrt{1-4\mathit{pt}}}\le \frac{\exp (7a)}{\sqrt{1-8\log (2\mathrm{e})a}}$$ (95)

Case II: Small k

Assume that $1\le k\le \log \frac{\mathrm{e}p}{k}$, which, in particular, implies that $k\le p^{\frac{1}{3}}\wedge \frac{p}{2}$ since p ≥ 5. Using $G_H^2\le H^2\le kH$, we have

$$\mathrm{{\rm E}}\exp (t{G}_H^2)\le \mathrm{{\rm E}}{\left(\frac{\mathrm{e}p}{k}\right)}^{\mathit{aH}}\le {\left(1+\frac{k}{p-k}\left({\left(\frac{\mathrm{e}p}{k}\right)}^a-1\right)\right)}^k$$ (96)

$$\le \exp \left\{\frac{2\sqrt{k}}{\sqrt{p}}\left({\left(\frac{\mathrm{e}p}{k}\right)}^a-1\right)\right\}$$ (97)

$$\le \exp (8a),$$ (98)

where (96) follows from the stochastic dominance of hypergeometric distributions by binomial distributions (see, e.g., [23, Theorem 1.1(d)])

$$\mathrm{Hypergeometric}(p,k,k)\stackrel{\mathrm{s}.\mathrm{t}.}{\le }\mathrm{Bin}\left(k,\frac{k}{p-k}\right),k\le \frac{p}{2}$$ (99)

and the moment generating function of binomial distributions, (97) is due to (1 + x)^k ≤ exp(kx) for x ≥ 0 and $\frac{k^2}{p}\le \sqrt{\frac{k}{p}}$, (98) is due to

$$\underset{y\ge 1}{\sup }\frac{{(\mathrm{e}y)}^a-1}{\sqrt{y}}=2\sqrt{\mathrm{e}}a{((1-2a))}^{\frac{1}{2a}-1}\le 4a,\forall a\le \frac{1}{2},$$

and

Case III: Moderate k

Assume that $\log \frac{\mathrm{e}p}{k}\le k\le \frac{p}{2}$. Define

$$A\triangleq \frac{k}{\sqrt{a}\log \frac{\mathrm{e}p}{k}}\wedge k$$ (100)

and write

$$\mathrm{{\rm E}}\exp (t{G}_H^2)=\mathrm{{\rm E}}\exp (t{G}_H^2){\mathbf{1}}_{\{H\le A\}}+\mathrm{{\rm E}}\exp (t{G}_H^2){\mathbf{1}}_{\{H>A\}}.$$ (101)

Next we bound the two terms in (101) separately: For the first term, we use normal approximation of G_m. By (93) - (94), for each fixed m ≤ A, we have

$$\mathrm{{\rm E}}\exp (t{G}_m^2)\le \frac{\exp (8t)}{\sqrt{1-4\mathit{mt}}}\le \frac{\exp (8a)}{\sqrt{1-4\sqrt{a}}},$$ (102)

where the last inequality follows from $k\ge \log \frac{\mathrm{e}p}{k}$ and (100).

To control the second term in (101), without loss of generality, we assume that A ≤ k, i.e., $A=\frac{k}{\sqrt{a}\log \frac{\mathrm{e}p}{k}}\ge \frac{1}{\sqrt{a}}$. we proceed similarly as in (96) - (98):

$$\begin{array}{ll}& \mathrm{{\rm E}}\exp (t{G}_H^2){\mathbf{1}}_{\{H>A\}}\\ \le & \mathrm{{\rm E}}\exp (\mathit{tkH}){\mathbf{1}}_{\{H>A\}}\\ \le & \mathbf{\sum }_{m>A}\exp (\mathit{tkm}){\left(\frac{k}{p-k}\right)}^m{\left(\frac{p-2k}{p-k}\right)}^{k-m}\left(\begin{array}{c}k\\ m\end{array}\right)\end{array}$$ (103)

$$\le \mathbf{\sum }_{m>A}\exp \left(\mathit{am}\log \frac{p}{k}-m\log \frac{p}{2k}+m\log \left(\sqrt{a}\mathrm{e}\log \frac{\mathrm{e}p}{k}\right)\right)$$ (104)

$$\le \mathbf{\sum }_{m>A}\exp \left\{-\left(1-a-\frac{1}{\mathrm{e}}\right)m\log 2+m\log (2\sqrt{a}\mathrm{e})+\frac{m}{\mathrm{e}}\right\}$$ (105)

$$\le 7\exp \left(-\frac{1}{\sqrt{a}}\right),$$ (106)

where (103) follows from (99), (104) follows from that $\left(\begin{array}{l}k\\ m\end{array}\right)\le {\left(\frac{\mathrm{e}k}{m}\right)}^m$, p ≥ 2k and m ≤ A, (105) follows from that e log x ≤ x for all x ≥ 1, and (106) is due to (100) and our choice of a in (92). Plugging (102) and (106) into (101), we obtain

$$\mathrm{{\rm E}}\exp (t{G}_H^2)\le \frac{\exp (8a)}{\sqrt{1-4\sqrt{a}}}+7\exp \left(-\frac{1}{\sqrt{a}}\right).$$ (107)

We complete the proof of (35), with g(a) defined as the maxima of the righthand sides of (95), (98) and (107).