ESTIMATION OF FUNCTIONALS OF SPARSE COVARIANCE MATRICES

Abstract

High-dimensional statistical tests often ignore correlations to gain simplicity and stability leading to null distributions that depend on functionals of correlation matrices such as their Frobenius norm and other ℓ_r norms. Motivated by the computation of critical values of such tests, we investigate the difficulty of estimation the functionals of sparse correlation matrices. Specifically, we show that simple plug-in procedures based on thresholded estimators of correlation matrices are sparsity-adaptive and minimax optimal over a large class of correlation matrices. Akin to previous results on functional estimation, the minimax rates exhibit an elbow phenomenon. Our results are further illustrated in simulated data as well as an empirical study of data arising in financial econometrics.

1. Introduction

Covariance matrices are at the core of many statistical procedures such as principal component analysis or linear discriminant analysis. Moreover, not only do they arise as natural quantities to capture interactions between variables but, as we illustrate below, they often characterize the asymptotic variance of commonly used estimators. Following the original papers of Bickel and Levina (2008a, 2008b), much work has focused on the inference of high-dimensional covariance matrices under sparsity [Cai and Liu (2011), Cai, Ren and Zhou (2013), Cai and Yuan (2012), Cai, Zhang and Zhou (2010), Cai and Zhou (2012), El Karoui (2008), Lam and Fan (2009), Ravikumar et al. (2011)] and other structural assumptions related to sparse principal component analysis [Amini and Wainwright (2009), Berthet and Rigollet (2013a, 2013b), Birnbaum et al. (2013), Cai, Ma and Wu (2013, 2015), Fan, Fan and Lv (2008), Johnstone and Lu (2009), Levina and Vershynin (2012), Ma (2013), Onatski, Moreira and Hallin (2013), Paul and Johnstone (2012), Rothman, Levina and Zhu (2009), Fan, Liao and Mincheva (2011, 2013), Jung and Marron (2009), Vu and Lei (2012), Zou, Hastie and Tibshirani (2006)]. This area of research is very active and, as a result, this list of references is illustrative rather than comprehensive. This line of work can be split into two main themes: estimation and detection. The former is the main focus of the present paper. However, while most of the literature has focused on estimating the covariance matrix itself, under various performance measures, we depart from this line of work by focusing on functionals of the covariance matrix rather than the covariance matrix itself.

Estimation of functionals of unknown signals such as regression functions or densities is known to be different in nature from estimation of the signal itself. This problem has received most attention in nonparametric estimation, originally in the Gaussian white noise model [Efromovich and Low (1996), Fan (1991), Ibragimov, Nemirovskiĭ and Khas'minskiĭ (1987), Nemirovskiĭ and Khas'minskiĭ (1987)] [see also Nemirovski (2000) for a survey of results in the Gaussian white noise model] and later extended to density estimation [Bickel and Ritov (1988), Hall and Marron (1987)] and various other models such as regression [Donoho and Nussbaum (1990), Cai and Low (2005, 2006), Klemelä (2006)] and inverse problems [Butucea (2007), Butucea and Meziani (2011)]. Most of these papers study the estimation of quadratic functionals and, interestingly, exhibit an elbow in the rates of convergence: there exists a critical regularity parameter below which the rate of estimation is nonparametric and above which, it becomes parametric. As we will see below the phenomenon also arises when regularity is measured by sparsity.

Over the past decade, sparsity has become the prime measure of regularity, both for its flexibility and generality. In particular, smooth functions can be viewed as functions with a sparse expansion in an appropriate basis. At a high level, sparsity assumes that many of the unknown parameters are equal to zero or nearly so, so that the few nonzero parameters can be consistently estimated using a small number of observations relative to the apparent dimensionality of the problem. Moreover, sparsity acts not only as a regularity parameter that stabilizes statistical procedures but also as key feature for interpretability. Indeed, it is often the case that setting many parameters to zero simply corresponds to a simpler sub-model. The main idea is to let data select the correct sub-model. This is the case in particular for covariance matrix estimation where zeros in the matrix correspond to uncor-related variables. Yet, while the value of sparsity for covariance matrix estimation has been well established, to the best of our knowledge, this paper provides the first analysis for the estimation of functionals of sparse covariance matrix. Indeed, the actual performance of many estimators critically depends on such functionals. Therefore, accurate functional estimation leads to a better understanding the performance of many estimators and can ultimately serve as a guide to selecting the best estimator. Applications of our results are illustrated in Section 2.

Our work is not only motivated by real applications, but also by a natural extension of the theoretical analysis carried out in the sparse Gaussian sequence model [Cai and Low (2005)]. In that paper, Cai and Low assume that the unknown parameter θ belong to an ℓ_q-ball, where q > 0 can be arbitrarily close to 0. Such balls are known to emulate sparsity and actually correspond to a more accurate notion of sparsity for signal θ that is encountered in applications [see, e.g., Foucart and Rauhut (2013)]. They also show that a nonquadratic estimator can be fully efficient to estimate quadratic functionals. We extend some of these results to covariance matrix estimation. Such an extension is not trivial since, unlike the Gaussian sequence model, covariance matrix lies at high-dimensional manifolds and its estimation exhibits complicated dependencies in the structure of the noise.

We also compare our results for optimal rates of estimating matrix functionals with that of estimating matrix itself. Many methods have been proposed to estimate covariance matrix in different sense of sparsity using different techniques including thresholding [Bickel and Levina (2008a)], tapering [Bickel and Levina (2008b), Cai, Zhang and Zhou (2010), Cai and Zhou (2012)] and penalized likelihood [Lam and Fan (2009)] to name only a few. These methods often lead to minimax optimal rates in various classes and under several metrics [Cai, Zhang and Zhou (2010), Cai and Zhou (2012), Rigollet and Tsybakov (2012)]. However, the optimal rates of estimating matrix functionals have not yet been covered by much literature. Intuitively, it should have faster rates of convergence on estimating a matrix functional than itself since it is just a one-dimensional estimating problem and the estimating error cancel with each other when we sum those elements together. We will see this is indeed the case when we compare the minimax rates of estimating matrix functionals with those of estimating matrices.

The rest of the paper is organized as follows. We begin in Section 2 by two motivating examples of high-dimensional hypothesis testing problems: a two-sample testing problem of Gaussian means that arises in genomics and validating the efficiency of markets based on the Capital Asset Pricing Model (CAPM). Next, in Section 3, we introduce an estimator of the quadratic functional of interest that is based on the thresholding estimator introduced in Bickel and Levina (2008a). We also prove its optimality in a minimax sense over a large class of sparse covariance matrices. The study is further extended to estimating other measures of sparsity of covariance matrix. Finally, we study the numerical performance of our estimator in Section 5 on simulated experiments as well as in the framework of the two applications described in Section 2. Due to space restrictions, the proofs for the upper bounds are relegated to the Appendix in the supplementary material [Fan, Rigollet and Wang (2015)].

Notation: Let d be a positive integer. The space of d × d positive semi-definite matrices is denoted by ${\mathbf{S}}_d^{+}$. For any two integers c < d, define [c : d] = {c, c + 1, …, d} to be the sequence of contiguous integers between c and d, and we simply write [d] = {1, …, d}. I_d denotes the identity matrix of ${\mathbb{R}}^d$. Moreover, for any subset S ⊂ [d], denote by 1_S ∈ {0, 1}^d the column vector with jth coordinate equal to one iff j ∈ S. In particular, 1_[d] denotes the d dimensional vector of all ones.

We denote by tr the trace operator on square matrices and by diag (resp., off) the linear operator that sets to 0 all the off diagonal (resp., diagonal) elements of a square matrix. The Frobenius norm of a real matrix M is denoted by ∥M∥_F and is defined by ${\Vert M\Vert }_{\mathrm{F}}=\sqrt{\text{tr}\left(M^{⊺}M\right)}$. Note that ∥M∥_F is a the Hilbert–Schimdt norm associated with inner product 〈A, B〉 = tr(A^⊺B) defined on the space of real rectangular matrices of the same size. Moreover, |A| denotes the determinant of a square matrix A. The variance of a random variable X is denote by var(X).

In the proofs, we often employ C to denote a generic positive constant that may change from line to line.

2. Two motivating examples

In this section, we describe our main motivation for estimating quadratic functionals of a high-dimensional covariance matrix in the light of two applications to high-dimensional testing problems. The first one is a high-dimensional two-sample hypothesis testing with applications in gene-set testing. The second example is about testing the validity of the capital asset pricing model (CAPM) from financial economics.

2.1. Two-sample hypothesis testing in high-dimensions

In various statistical applications, in particular in genomics, the dimensionality of the problems is so large that statistical procedures involving inverse covariance matrices are not viable due to its lack of stability both from a statistical and numerical point of view. This limitation can be well illustrated on a showcase example: two-sample hypothesis testing [Bai and Saranadasa (1996)] in high-dimensions.

Suppose that we observe two independent samples $X_1^{\left(1\right)},\cdots ,X_{n_1}^{\left(1\right)}\in {\mathbb{R}}^p$ that are i.i.d. $\mathcal{N}({\mu }_1,{\Sigma }_1)$ and $X_1^{\left(2\right)},\cdots ,X_{n_2}^{\left(2\right)}\in {\mathbb{R}}^p$ that are i.i.d. $\mathcal{N}({\mu }_2,{\Sigma }_2)$. Let n = n₁ + N₂. The goal is to test H₀ : μ₁ = μ₂ vs. H₁ : μ₁ ≠ μ₂.

Assume first that Σ₁ = Σ₂ = Σ. In this case, Hotelling's test is commonly employed when p is small. Nevertheless, when p is large, Bai and Saranadasa (1996) showed that the test based on Hotelling's T ² has low power and suggest a new statistics M for the random matrix asymptotic regime where n, p → ∞, $\frac{n}{p}\to \gamma >0$, $\frac{n_1}{n_1+n_2}\to \kappa \in (0,1)$. The statistics, implementing the naive Bayes rule, is defined as

$$M={({\stackrel{‒}{X}}^{\left(1\right)}-{\stackrel{‒}{X}}^{\left(2\right)})}^{⊺}({\stackrel{‒}{X}}^{\left(1\right)}-{\stackrel{‒}{X}}^{\left(2\right)})-\frac{n}{n_1{n}_2}\mathrm{tr}\left(\widehat{\Sigma }\right),$$

and is proved to be asymptotically normal under the null hypothesis with

$$\mathrm{var}\left(M\right)=2\frac{n(n-1)}{{\left(n_1{n}_2\right)}^2}{\Vert \Sigma \Vert }_{\mathrm{F}}^2(1+o(1\left)\right).$$

Clearly, the asymptotic variance of M depends on the unknown covariance matrix Σ through its quadratic functional, and in order to compute the critical value of the test, Bai and Saranadasa suggest to estimate ${\Vert \Sigma \Vert }_{\mathrm{F}}^2$ by the quantity

$$B^2=\frac{n^2}{(n+2)(n-1)}\left[{\Vert \widehat{\Sigma }\Vert }_{\mathrm{F}}^2-\frac{1}{n}{\left(\mathrm{tr}\right(\widehat{\Sigma }\left)\right)}^2\right].$$

They show that B² is a ratio-consistent estimator of ${\Vert \Sigma \Vert }_{\mathrm{F}}^2$ in the sense that $B^2=(1+o_P(1\left)\right){\Vert \Sigma \Vert }_F^2$. Clearly, this solution does not leverage any sparsity assumption and may suffer from power deficiency if the matrix Σ is indeed sparse. Rather, if the covariance matrix Σ is believed to be sparse, one may prefer to use a thresholded estimator for Σ as in Bickel and Levina (2008a) rather than the empirical covariance matrix $\widehat{\Sigma }$. In this case, we estimate ${\Vert \Sigma \Vert }_F^2$ by $\widehat{{\Vert \Sigma \Vert }_F^2}={\sum }_{i,j=1}^p{\widehat{\sigma }}_{ij}^{2}1\{\mid {\widehat{\sigma }}_{ij}\mid >\tau \}$, where $\{{\widehat{\sigma }}_{ij},i,j\in [p\left]\right\}$ could be any consistent estimator of σ_ij and τ > 0 is a threshold parameter.

More recently, Chen and Qin (2010) took into account the case Σ₁ ≠ Σ₂ and proposed a test statistic based on an unbiased estimate of each of the three quantities in ${\Vert {\mu }_1-{\mu }_2\Vert }^2={\Vert {\mu }_1\Vert }^2+{\Vert {\mu }_2\Vert }^2-2{\mu }_1^{⊺}{\mu }_2$. In this case, the quantities ${\Vert {\Sigma }_i\Vert }_F^2$ and 〈Σ₁, Σ₂〉 appear in the asymptotic variance. The detailed formulation assumptions of this statistic, as well as discussions about other testing methods such as Srivastava and Du (2008), are provided in the supplementary material [Fan, Rigollet and Wang (2015)] for completeness. If Σ₁ and Σ₂ are indeed sparse, akin to the above reasoning, we can also estimate ${\Vert {\Sigma }_i\Vert }_F^2,i=1,2$ and 〈Σ₁, Σ₂〉 using thresholding to leverage sparsity assumption. It is not hard to derive a theory for estimating quadratic functionals involving two covariance matrices but the details of this procedure are beyond the scope of the present paper.

2.2. Testing high-dimensional CAPM model

The capital asset pricing model (CAPM) is a simple financial model that postulates how individual asset returns are related to the market risks. Specifically, the individual excessive return $Y_t^{\left(i\right)}$ of asset i ∈ [N] over the risk-free rate at time t ∈ [T] can be expressed as an affine function of a vector of K risk factors $f_t\in {\mathbb{R}}^K$:

$$Y_t^{\left(i\right)}={\alpha }_i+{\beta }_i^{⊺}{f}_t+{\epsilon }_t^{\left(i\right)},$$ (2.1)

where we assume for any t ∈ [T], $f_t\in {\mathbb{R}}^K$ are observed. The case K = 1 with f_t being the excessive return of the market portfolio corresponds to the CAPM [Lintner (1965), Mossin (1966), Sharpe (1964)]. It is nowadays more common to employ the Fama–French three-factor model [see Fama and French (1993) for a definition] for the US equity market, corresponding to K = 3.

For simplicity, let us rewrite the model (2.1) in the vectorial form

$$Y_t=\alpha +B{f}_t+{\epsilon }_t,t\in \left[T\right].$$

The multi-factor pricing model postulates α = 0. Namely, all returns are fully compensated by their risks: no extra returns are possible and the market is efficient. This leads us to naturally consider the hypothesis testing problem H₀ : α = 0 vs. H₁ : α ≠ 0.

Let $\widehat{\alpha }$ and $\widehat{B}$ be the least-squares estimate and ${\widehat{\epsilon }}_t=Y_t-\widehat{\alpha }-\widehat{B}{f}_t$ be a residual vector. Then an unbiased estimator of Σ = var(ε_t) is

$$\tilde{\Sigma }=\frac{1}{T-K-1}\sum _{t=1}^T{\widehat{\epsilon }}_t{\widehat{\epsilon }}_t^{⊺}.$$

Let $\widehat{D}=\text{diag}\left(\tilde{\Sigma }\right)$ and M_F = I_T − F(F^⊺ F)⁻¹F^⊺ where F = (f₁, …, f_T)^⊺ . Define $W_d=\left(1_{\left[T\right]}^{⊺}{M}_F{1}_{\left[T\right]}\right){\widehat{\alpha }}^{⊺}{\widehat{D}}^{-1}\widehat{\alpha }$ the Wald-type of test statistics with correlation ignored, whose normalized version is given by

$$J_{\alpha }=\frac{W_d-\mathbb{E}\left(W_d\right)}{\sqrt{\mathrm{var}\left(W_d\right)}}.$$ (2.2)

Under some conditions, it was shown by Pesaran and Yamagata (2012) that, under H₀, $J_{\alpha }\to \mathcal{N}(0,1)$ as N → ∞. Moreover, if ${\epsilon }_t^{\left(i\right)}$'s are i.i.d. Gaussian, it holds that $\mathbb{E}\left(W_d\right)=\nu N∕(\nu -2)$ and

$$\mathrm{var}\left(W_d\right)=\frac{2N(v-1)}{v-4}{\left(\frac{v}{v-2}\right)}^2[1+(N-1){\stackrel{‒}{\rho }}^2+O(v^{-1∕2}\left)\right],$$

where ν = T − K − 1 is the degrees of freedom and

$${\stackrel{‒}{\rho }}^2=\frac{2}{N(N-1)}\sum _{i=2}^N\sum _{j=1}^{i-1}{\rho }_{ij}^2,$$

where ρ = D^−1/2ΣD^−1/2 with D = diag(Σ) is the correlation matrix of the stationary process (ε_t)_t∈[T]. The authors go on to propose an estimator of the quadratic functional ρ⁻² by replacing the correlation coefficients ρ_ij in the above expression by ${\widehat{\rho }}_{i,j}1(\mid {\widehat{\rho }}_{ij}\mid >\tau )$ where ${\left({\widehat{\rho }}_{ij}\right)}_{i,j\in \left[N\right]}={\widehat{D}}^{-1∕2}\tilde{\Sigma }{\widehat{D}}^{-1∕2}$ and τ > 0 is a threshold parameter. However, they did not provide any analysis of this method, nor any guidance to chose τ.

3. Optimal estimation of quadratic functionals

In the previous section, we have described rather general questions involving the estimation of quadratic functions of covariance or correlation matrices. We begin by observing that consistent estimation of ${\Vert \Sigma \Vert }_{\mathrm{F}}^2$ is impossible unless p = o(n). This precludes in particular the high-dimensional framework that motivates our study.

Our goal is to estimate the Frobenius norm ${\Vert \Sigma \Vert }_{\mathrm{F}}^2$ of a sparse p × p covariance matrix Σ using n i.i.d. observations $X_1,\dots ,X_n\sim \mathcal{N}(0,\Sigma )$. Observe that $\Vert \Sigma {\Vert }_{\mathrm{F}}^2$ can be decomposed as ${\Vert \Sigma \Vert }_{\mathrm{F}}^2=Q\left(\Sigma \right)+D\left(\Sigma \right)$ where $Q\left(\Sigma \right)={\sum }_{i\ne j}{\sigma }_{ij}^2$ corresponds to the off-diagonal elements and $D\left(\Sigma \right)={\sum }_j{\sigma }_{jj}^2$ corresponds to the diagonal elements. The following theorem, implies that even if Σ = diag(Σ) is diagonal, the quadratic functional ${\Vert \Sigma \Vert }_{\mathrm{F}}^2$ cannot be estimated consistently in absolute error if p ≥ n. Note that the situation is quite different when it comes to relative error. Indeed, the estimator of Bai and Saranadasa (1996) is consistent in relative error with no sparsity assumption even in the high-dimensional regime. Study of the relative error in the presence of sparsity is an interesting question that deserves further developments. This makes sense intuitively as the diagonal of Σ consists of p unknown parameters while we have only n observations.

Proposition 3.1. Fix n, p ≥ 1 and let

$${\mathcal{D}}_p=\{\Sigma \in {\mathbf{S}}_p^{+}:\Sigma =\text{diag}(\Sigma ),{\Sigma }_{ii}\le 1\}$$

be the class of diagonal covariance matrices with diagonal elements bounded by 1. Then there exists a universal constant C > 0 such that

$$\underset{\widehat{D}}{\inf }\underset{\Sigma \in {\mathcal{D}}_p}{\sup }\mathbb{E}{[\widehat{D}-D(\Sigma \left)\right]}^2\ge C\frac{p}{n}.$$

In particular, it implies that

$$\underset{\widehat{F}}{\inf }\underset{\Sigma \in {\mathcal{D}}_p}{\sup }\mathbb{E}{[\widehat{F}-{\Vert \Sigma \Vert }_{\mathrm{F}}^2]}^2\ge C\frac{p}{n},$$

where the infima are taken with over all real valued measurable functions of the observations.

Proof

Our lower bounds rely on standard arguments from minimax theory. We refer to Chapter 2 of Tsybakov (2009) for more details. In the sequel, let $\mathrm{KL}(P,\stackrel{‒}{P})$ denote the Kullback–Leibler divergence between two distributions P and $\stackrel{‒}{P}$, where $P⪡\stackrel{‒}{P}$. It is defined by

$$\mathrm{KL}(P,\stackrel{‒}{P})=\int \log \left(\frac{\mathrm{d}P}{\mathrm{d}\stackrel{‒}{P}}\right)\mathrm{d}P.$$

We are going to employ a simple two-point lower bound. Fix ε ∈ (0, 1/2) and let $P_p^n$ (resp., $P_p^n$) denote the distribution of a sample X₁, …, X_n where $X_1~\mathcal{N}(0,I_p)$ [resp., $X_1~\mathcal{N}(0,(1-\epsilon \left)I_p\right)$]. Next, observe that I_p, $(1-\epsilon )I_p\subset {\mathcal{D}}_p$ so that

$$\underset{\Sigma \in {\mathcal{D}}_p}{\sup }\mathbb{E}\mid \widehat{D}-D\left(\Sigma \right)\mid \ge \underset{\Sigma \in \{I_p,(1-\epsilon \left)I_p\right\}}{\max }\mathbb{E}\mid \widehat{D}-D\left(\Sigma \right)\mid .$$ (3.1)

Moreover, |D(I_p) − D((1 − ε)I_p)| = p(2ε − ε²) > pε. Then it follows from the Markov inequality that

$$\begin{array}{cc}\hfill \frac{1}{p\epsilon }\underset{\Sigma \in \{I_p,(1-\epsilon \left)I_p\right\}}{\max }\mathbb{E}\mid \widehat{D}-D\left(\Sigma \right)\mid & \ge \underset{\Sigma \in \{I_p,(1-\epsilon \left)I_p\right\}}{\max }\mathbb{P}[\mid \widehat{D}-D(\Sigma )\mid >p\epsilon ]\hfill \\ \hfill & \ge \frac{1}{4}\exp [-\mathrm{KL}(P_p^n,{\stackrel{‒}{P}}_p^n\left)\right],\hfill \end{array}$$ (3.2)

where the last inequality follows from Theorem 2.2(iii) of Tsybakov (2009).

Completion of the proof requires an upper bound on $\mathrm{KL}(P_p^n,{\stackrel{‒}{P}}_p^n)$. To that end, note that it follows from the chain rule and simple algebra that

$$\mathrm{KL}(P_p^n,{\stackrel{‒}{P}}_p^n)=np\mathrm{KL}(P_1^1,{\stackrel{‒}{P}}_1^1)=\frac{np}{2}\left[\log (1-\epsilon )+\frac{\epsilon }{1-\epsilon }\right]\le \frac{np}{2}\frac{{\epsilon }^2}{1-\epsilon }\le np{\epsilon }^2.$$

Taking now $\epsilon =1∕\left(2\sqrt{np}\right)\le 1∕2$ yields $\mathrm{KL}(P_p^n,Q_p^n)\le 1∕4$. Together with (3.1) and (3.2), it yields

$$\underset{\widehat{D}}{\inf }\underset{\Sigma \in {\mathcal{D}}_p}{\sup }\mathbb{E}\mid \widehat{D}-D\left(\Sigma \right)\mid \ge \frac{1}{8e^{1∕4}}\sqrt{\frac{p}{n}}.$$

To complete the proof, we square the above inequality and employ Jensen's inequality.

To overcome the above limitation, we consider the following class of sparse covariance matrices (indeed correlation matrices). For any q ∈ [0, 2), R > 0 let ${\mathcal{F}}_q\left(R\right)$ denote the set of p × p covariance matrices defined by

$${\mathcal{F}}_q\left(R\right)=\left\{\Sigma \in {\mathbf{S}}_p^{+}:\sum _{i\ne j}\mid {\sigma }_{ij}{\mid }^q\le R,\text{diag}\left(\Sigma \right)=I_p\right\}.$$ (3.3)

Note that for this class of functions, we assume that the variance along each coordinate is normalized to 1. This normalization is frequently obtained by sample estimates, as shown in the previous section. This simplified assumption is motivated also by Proposition 3.1 above which implies that ${\Vert \Sigma \Vert }_{\mathrm{F}}^2$ for general covariance matrix cannot be estimated accurately in absolute error in the large p small n regime since sparsity assumptions on the diagonal elements are implausible. Note that the condition diag(Σ) = I_p implies that diagonal elements D(Σ) of matrices in ${\mathcal{F}}_q\left(R\right)$ can be estimated without error so that we could possibly achieve consistency even if the case of large p small n.

Matrices in ${\mathcal{F}}_q\left(R\right)$ have many small coefficients for small values of q and R. In particular, when q = 0, there are no more than R entries of nonvanishing correlations. Following a major trend in the estimation of sparse covariance matrices [Bickel and Levina (2008a, 2008b), Cai and Liu (2011), Cai and Yuan (2012), Cai, Zhang and Zhou (2010), Cai and Zhou (2012), El Karoui (2008), Lam and Fan (2009)], we employ a thresholding estimator of the covariance matrix as a running horse to estimate the quadratic functionals. From the n i.i.d. observations $X_1,\dots ,X_n~\mathcal{N}(0,\Sigma )$, we form the empirical covariance matrix $\widehat{\Sigma }$ that is defined by

$$\widehat{\Sigma }=\frac{1}{n}\sum _{k=1}^n{X}_k{X}_k^{⊺}$$ (3.4)

with elements $\widehat{\Sigma }={\left\{{\widehat{\sigma }}_{ij}\right\}}_{ij}$ and for any threshold τ > 0, let ${\tilde{\Sigma }}_{\tau }={\left\{{\tilde{\sigma }}_{ij}\right\}}_{ij}$ denote the thresholding estimator of Σ defined by ${\tilde{\sigma }}_{ij}={\widehat{\sigma }}_{ij}1\{\mid {\widehat{\sigma }}_{ij}\mid >\tau \}$ if i ≠ j and ${\tilde{\sigma }}_{ii}={\widehat{\sigma }}_{ii}$.

Next, we employ a simple plug-in estimator for Q(Σ):

$$\widehat{Q\left(\Sigma \right)}=Q\left({\tilde{\Sigma }}_{\tau }\right)=\sum _{i\ne j}{\widehat{\sigma }}_{ij}^{2}1\{\mid {\widehat{\sigma }}_{ij}\mid >\tau \}.$$ (3.5)

Note that no value of the diagonal elements is used to estimate Q(Σ).

In the rest of this section, we establish that $\widehat{Q\left(\Sigma \right)}$ is minimax adaptive over the scale {${\mathcal{F}}_q\left(R\right)$, q ∈ [0, 2), R > 0}. Interestingly, we will see that the minimax rate presents an elbow as often in quadratic functional estimation.

Theorem 3.1

Assume that γ log(p) < n for some constant γ > 8 and fix C₀ ≥ 4. Consider the threshold

$$\tau =2C_0\sqrt{\frac{\gamma \log p}{n}},$$

and assume that τ ≤ 1. Then, for any q ∈ [0, 2), R > 0, the plug-in estimator $Q\left({\tilde{\Sigma }}_{\tau }\right)$ satisfies

$$\mathbb{E}\left[{\left(Q\right({\tilde{\Sigma }}_{\tau })-Q(\Sigma \left)\right)}^2\right]\le C_1{\psi }_{n,p}(q,R)+C_2{p}^{4-\gamma ∕2},$$

where

$${\psi }_{n,p}(q,R)=\frac{R^2}{n}\vee R^2{\left(\frac{\log p}{n}\right)}^{2-q},$$

and C₁, C₂ are positive constants depending on γ, C₀, q.

The proof is postponed to the supplementary material.

Note that the rates ψ_n,p(q, R) present an elbow at q = 1 − log log p/log n as usually the case in functional estimation. We now argue that the rates ψ_n,p(q, R) are optimal in a minimax sense for a wide range of settings. In particular, the elbow effect arising from the maximum in the definition of ψ is not an artifact. In the following theorem, we emphasize the dependence on Σ by using the notation ${\mathbb{E}}_{\Sigma }$ for the expectation with respect to the distribution of the sample X₁, …, X_n, where $X_i~\mathcal{N}(0,\Sigma )$.

Theorem 3.2

Fix q ∈ [0, 2), R > 0 and assume 2 log p < n and R² < (p − 1)n^−q/2. Then there exists a positive constant C₃ > 0 such that

$$\underset{\widehat{Q}}{\inf }\underset{\Sigma \in {\mathcal{F}}_q\left(R\right)}{\sup }{\mathbb{E}}_{\Sigma }\left[{(\widehat{Q}-Q(\Sigma \left)\right)}^2\right]\ge C_3{\varphi }_{n,p}(q,R),$$

where ϕ_n,p(q, R) is defined by

$${\varphi }_{n,p}(q,R)=\frac{R^2}{n}\vee \left\{R^2{\left(\frac{\log \left(\right(p-1)-∕(R^2{n}^q)+1)}{2n}\right)}^{2-q}\wedge R^{4∕q}\wedge 1\right\}$$ (3.6)

and the infimum is taken over all measurable functions $\widehat{Q}$ of the sample X₁, …, X_n.

Before proceeding to the proof, a few remarks are in order.

1. The additional term of order p^4−γ/2 in Theorem 3.1 can be made negligible by taking γ large enough. To show this tradeoff explicitly, we decided keep this term.

2. When 1 ≤ R² < p^αn^−q for some constant α < 1, a slightly stronger requirement than Theorem 3.2, the lower bound there can be written as

   $${\varphi }_{n,p}(q,R)=\frac{R^2}{n}\vee \left\{R^2{\left(\frac{\log p}{n}\right)}^{2-q}\wedge 1\right\}.$$ (3.7)

   Observe that the above lower bound matches the upper bound presented in Theorem 3.1 when R^2/(2−q) log p ≤ n. Arguably, this is the most interesting range as it characterizes rates of convergence (to zero) rather than rates of divergence, that may be of different nature [see, e.g., Verzelen (2012)]. In other words, the rates given in (3.7) are minimax adaptive with respect to n, R, p and q. In our formulation, we allow R = R_n,p to depend on other parameters of the problem. We choose here to keep the notation light.

3. The reason we choose correlation matrix class to present the elbow effect is just for simplicity. Actually, we can replace the constraint diag(Σ) = I_p in the definition of ${\mathcal{F}}_q\left(R\right)$ by boundedness of diagonal elements of Σ. Then for estimating off-diagonal elements Q(Σ), following exactly the same derivation, the same elbow phenomenon has been noticed. Meanwhile, the optimal rate for estimating diagonal elements D(Σ) is again of the order p/n. This optimal rate can be attained by the estimator

   $$\widehat{D\left(\Sigma \right)}=\frac{1}{n(n-1)}\sum _{i=1}^p\sum _{k\ne j}{X}_{k,i}^2{X}_{j,i}^2.$$ (3.8)

   We omitted the proof here. Thus, if we do not have prior information about diagonal elements, we could still estimate optimally the quadratic functional of a covariance matrix by applying the thresholding method (3.5) for off-diagonal elements, together with (3.8) for diagonal elements.

4. The rate ϕ_n,p(q, R) presents the same elbow phenomenon at q = 1 observed in the estimation of functionals, starting independently with work of Bickel and Ritov (1988) and Fan (1991). Closer to the present setup is the work of Cai and Low (2005) who study the estimation of functionals of “sparse” sequences in the infinite Gaussian sequence model. There, a parameter controls the speed of decay of the unknown coefficients. Note that while smaller values q lead to sparser matrices Σ, no estimator can benefit further from sparsity below q = 1 [the estimator has a rate of convergence O(R²/n) for any q < 1], unlike in the case of estimation of Σ. Again, this is inherent to estimating functionals.

5. The condition R² < (p − 1)n^−q/2 corresponds to the high-dimensional regime and allows us to keep clean terms in the logarithm. Similar assumptions are made in related literature [see, e.g., Cai and Zhou (2012)].

6. The optimal rates obtained here cannot be implied by existing ones for estimating sparse covariance matrices. In particular, the latter do not admit an elbow phenomenon. Specifically, Rigollet and Tsybakov (2012) showed the optimal rate for estimating Σ for $\Sigma \in {\mathcal{F}}_q\left(R\right)$ under the Frobenius norm is $\sqrt{R}{(\log p∕n)}^{1∕2-q∕4}$ for 0 ≤ q < 2. Using this, it is not hard to derive with high probability,

   $$\mid Q\left(\widehat{\Sigma }\right)-Q\left(\Sigma \right)\mid \le C_{1}R{\left(\frac{\log p}{n}\right)}^{1∕2-q∕4}+C_{2}R{\left(\frac{\log p}{n}\right)}^{1-q∕2},$$

   since ${\Vert Q\left(\Sigma \right)\Vert }_F=O\left(\sqrt{R}\right)$ if nonvanishing correlations are bounded away from zero. On one hand, when q < 2 the first term always dominates so that we do not observe the elbow effect. In addition, the rate so obtained is not optimal.

We now turn to the proof of Theorem 3.2

Proof of Theorem 3.2

To prove minimax lower bounds, we employ a standard technique that consists of reducing the estimation problem to a testing problem. We split this proof into two parts and begin by proving

$$\underset{\widehat{Q}}{\inf }\underset{\Sigma \in {\mathcal{F}}_q\left(R\right)}{\sup }{\mathbb{E}}_{\Sigma }{[\widehat{Q}-Q(\Sigma \left)\right]}^2\ge C\frac{R^2}{n},$$

for some positive constant C > 0. To that end, for any $A\in {\mathbf{S}}_p^{+}$, let ${\mathbb{P}}_A$ denote the distribution of $X~\mathcal{N}(0,A)$. It is not hard to show if |A| > 0 and |B| > 0, $A,B\in {\mathbf{S}}_p^{+}$, then the Kullback–Leibler divergence between ${\mathbb{P}}_A$ and ${\mathbb{P}}_B$ is given by

$$\mathrm{KL}({\mathbb{P}}_A,{\mathbb{P}}_B)=\frac{1}{2}\left[\log \left(\frac{\mid B\mid }{\mid A\mid }\right)+\mathrm{tr}\left(B^{-1}A\right)-p\right].$$ (3.9)

Next, take A and B to be of the form

$$A^{\left(k\right)}=\left(\begin{array}{ccc}\hfill {11}^{⊺}\hfill & \hfill a{11}^{⊺}\hfill & \hfill 0\hfill \\ \hfill a{11}^{⊺}\hfill & \hfill {11}^{⊺}\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill I_{p-k}\hfill \end{array}\right),B^{\left(k\right)}=\left(\begin{array}{ccc}\hfill {11}^{⊺}\hfill & \hfill b{11}^{⊺}\hfill & \hfill 0\hfill \\ \hfill b{11}^{⊺}\hfill & \hfill {11}^{⊺}\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill I_{p-k}\hfill \end{array}\right),$$

where a, b ∈ (0, 1/2), 0 is a generic symbol to indicate that the missing space is filled with zeros, and 1 denotes a vector of ones of length k/2. Note that if we have random variables (X, Y, Z₁, …, Z_p−2) chosen from distribution $\mathcal{N}(0,A^{\left(2\right)})$ meaning that Z_k's are independent with X, Y but the correlation between X and Y is a, then random vector (X, …, X, Y, …, Y, Z₁, …, Z_p−k) with k/2 X's and Y's in it follows $\mathcal{N}(0,A^{\left(k\right)})$. It is obvious that these two matrices are degenerate and comes from perfectly correlated random variables. Since perfectly correlated random variables do not add new information, for such matrices, an application of (3.9) yields

$$\mathrm{KL}({\mathbb{P}}_{A^{\left(k\right)}},{\mathbb{P}}_{B^{\left(k\right)}})=\mathrm{KL}({\mathbb{P}}_{A^{\left(2\right)}},{\mathbb{P}}_{B^{\left(2\right)}})=\frac{1-ab}{1-b^2}-\frac{1}{2}\log \left(\frac{1-a^2}{1-b^2}\right)-1.$$

Next, using the convexity inequality log(1 + x) ≥ x − x²/2 for all x > 0, we get that

$$\mathrm{KL}({\mathbb{P}}_{A^{\left(k\right)}},{\mathbb{P}}_{B^{\left(k\right)}})\le \frac{{(a-b)}^2}{2(1-b^2)}\left[1+\frac{{(a+b)}^2}{2(1-b^2)}\right]\le 2{(a-b)}^2,$$

using the fact that a, b ∈ (0, 1/2). Take now if R > 4

$$a=\frac{1}{4},b=a+\frac{1}{4\sqrt{n}},k=\sqrt{R}$$

so that we indeed have a, b ∈ (0, 1/2) and also A^(k), $B^{\left(k\right)}\in {\mathcal{F}}_q\left(R\right)$ obviously. If R < 4, take k = 2, $a=\sqrt{R}∕8$, $b=a+\sqrt{R∕64n}$ instead. Moreover, this choice leads to $n\mathrm{KL}({\mathbb{P}}_A,{\mathbb{P}}_B)\le 1∕5$. Using standard techniques to reduce estimation problems to testing problems [see, e.g., Theorem 2.5 of Tsybakov (2009)], we find that

$$\underset{\widehat{Q}}{\inf }\underset{\Sigma \in \{A,B\}}{\max }{\mathbb{E}}_{\Sigma }\left[{(\widehat{Q}-Q(\Sigma \left)\right)}^2\right]\ge C{\left(Q\right(A)-Q(B\left)\right)}^2.$$

For the above choice of A and B, we have

$${\left(Q\right(A^{\left(k\right)})-Q(B^{\left(k\right)}\left)\right)}^2=\frac{k^4}{4}{(a^2-b^2)}^2\ge C\frac{R^2}{n}.$$

Since A^(k), $B^{\left(k\right)}\in {\mathcal{F}}_q\left(R\right)$, the above two displays imply that

$$\underset{\widehat{Q}}{\inf }\underset{\Sigma \in {\mathcal{F}}_q\left(R\right)}{\max }{\mathbb{E}}_{\Sigma }\left[{(\widehat{Q}-Q(\Sigma \left)\right)}^2\right]\ge C\frac{R^2}{n},$$

which completes the proof of the first part of the lower bound.

For the second part of the lower bound, we reduce our problem to a testing problem of the same flavor as Arias-Castro, Bubeck and Lugosi (2015), Berthet and Rigollet (2013b). Note, however, that our construction is different because the covariance matrices considered in these papers do not yield large enough lower bounds. We use the following construction.

Fix an integer k ∈ [p − 1] and let $\mathcal{S}=\{S\subset [p-1]:\mid S\mid =k\}$ denote the set of subsets of [p − 1] that have cardinality k. Fix a ∈ (0, 1) to be chosen later and for any $S\in \mathcal{S}$, recall that 1_S is the column vector in {0, 1}^p−1 with support given by S. For each $S\in \mathcal{S}$, we define the following p × p covariance matrix:

$${\Sigma }_S=\left(\begin{array}{cc}\hfill 1\hfill & \hfill a{1}_S^{⊺}\hfill \\ \hfill a{1}_S\hfill & \hfill I_{p-1}\hfill \end{array}\right).$$ (3.10)

Let ${\mathbb{P}}_0$ denote the distribution of $X\sim {\mathcal{N}}_p(0,I_p)$ and ${\mathbb{P}}_S$ denote the distribution of $X\sim {\mathcal{N}}_p(0,{\Sigma }_S)$. Let ${\mathbb{P}}_0^n$ (resp., ${\mathbb{P}}_S^n$) denote the distribution of X = (X₁, …, X_n) of a collection n i.i.d. random variables drawn from ${\mathbb{P}}_0$ (resp., ${\mathbb{P}}_S$). Moreover, let ${\stackrel{‒}{\mathbb{P}}}^n$ denote the distribution of X where the X_i 's are drawn as follows: first draw S uniformly at random from $\mathcal{S}$ and then, conditionally on S, draw X₁, …, X_n independently from ${\mathbb{P}}_S$. Note that ${\stackrel{‒}{\mathbb{P}}}^n$ is the mixture of n independent samples rather the distribution of n independent random vectors drawn from a mixture distribution. Consider the following testing problem:

$$H_0:\mathbf{X}~{\mathbb{P}}_0^n\text{vs}.H_1:\mathbf{X}~{\stackrel{‒}{\mathbb{P}}}^n.$$

Using Theorem 2.2, part (iii) of Tsybakov (2009), we get that for any test ψ = ψ(X), we have

$${\mathbb{P}}_0^n(\psi =0)\vee \underset{S\in \mathcal{S}}{\max }{\mathbb{P}}_S^n(\psi =1)\ge {\mathbb{P}}_0^n(\psi =0)\vee {\stackrel{‒}{\mathbb{P}}}^n(\psi =1)\ge \frac{1}{4}\exp (-{\chi }^2({\stackrel{‒}{\mathbb{P}}}^n,{\mathbb{P}}_0\left)\right),$$

where we recall that the χ²-divergence between two probability distributions P and Q is defined by

$${\chi }^2(P,Q)=\{\begin{array}{cc}\int {\left(\frac{\mathrm{d}P}{\mathrm{d}Q}-1\right)}^{2}dQ,\hfill & \text{if}P\ll Q,\hfill \\ \infty ,\hfill & \text{otherwise}.\hfill \end{array}\phantom{\}}$$

Lemma A.1 implies that for suitable choices of the parameters a and k, we have ${\chi }^2({\stackrel{‒}{\mathbb{P}}}^n,{\mathbb{P}}_0)\le 2$ so that the test errors are bounded below by a constant C = e⁻²/4. Since Q(Σ_S) = 2ka² for any $S\in \mathcal{S}$, it follows from a standard reduction from hypothesis testing to estimation [see, e.g., Theorem 2.5 of Tsybakov (2009)] that the above result implies the following lower bound:

$$\underset{\widehat{Q}}{\inf }\underset{\Sigma \in \mathcal{H}}{\max }{\mathbb{E}}_{\Sigma }\left[{(\widehat{Q}-Q(\Sigma \left)\right)}^2\right]\ge C{k}^2{a}^4,$$ (3.11)

for some positive constant C, where the infimum is taken over all estimators $\widehat{Q}$ of Q(Σ) based on n observations and $\mathcal{H}$ is the class of covariance matrices defined by

$$\mathcal{H}=\left\{I_p\right\}\cup \{{\Sigma }_S:S\in \mathcal{S}\}.$$

To complete the proof, observe that the values of a and k prescribed in Lemma A.1 imply that $\mathcal{H}\subset {\mathcal{F}}_q\left(R\right)$ and give the desired lower bound. Note first that, for any choice of a and k, the following holds trivially: $I_p\in {\mathcal{F}}_q\left(R\right)$ and diag(Σ_S) = I_p for any $S\in \mathcal{S}$. Write Σ_S = (σ_ij) and observe that

$$\sum _{i\ne j}{\mid {\sigma }_{ij}\mid }^q=2k{a}^q.$$

Next, we treat each case of Lemma A.1 separately.

• Case 1. Note first that 2ka^q = R/2 < R so that ${\Sigma }_S\in {\mathcal{F}}_q\left(R\right)$. Moreover, k²a⁴ = CR^4/q.

• Case 2. Note first that 2ka^q ≤ R/2 < R so that ${\Sigma }_S\in {\mathcal{F}}_q\left(R\right)$. Since k ≥ 2 and k² ≤ R²n^q, we have

  $$k\ge \frac{R}{4}{\left(\frac{\log \left(\right(p-1)∕k^2+1)}{2n}\right)}^{-q∕2}.$$

  Therefore,

  $$k^2{a}^4\ge \frac{R^2}{16}{\left(\frac{\log \left(\right(p-1)∕(R^2{n}^q)+1)}{2n}\right)}^{2-q}\wedge \frac{1}{4}.$$

  Combining the two cases, we get

  $$k^2{a}^4\ge C\left[R^2{\left(\frac{\log \left(\right(p-1)∕(R^2{n}^q)+1)}{2n}\right)}^{2-q}\wedge R^{4∕q}\wedge 1\right].$$

  Together with (3.11), this completes the proof of the second part of the lower bound.

4. Extension to nonquadratic functionals

Closely related to quadratic functional is the ℓ_r functional of covariance matrices, which is defined by

$${\ell }_r\left(\Sigma \right)=\underset{i\le p}{\max }\sum _{j\le p}{\mid {\sigma }_{ij}\mid }^r.$$ (4.1)

It is often used to measure the sparsity of a covariance matrix and plays an important role in estimating sparse covariance matrix. This along the theoretical interest on the difficulty of estimating such a functional give rise to this study. Note that ℓ₁(Σ) functional is indeed the ℓ₁-norm of the covariance matrix Σ, whereas when r = 2, ℓ_r functional is the maximal row-wise quadratic functional. Thus, the non-quadratic ℓ_r functional is just a natural extension of such a maximal quadratic functional, whose optimal estimation problem will be the main focus of this section.

4.1. Optimal estimation of ℓ_r functionals

We consider a class of matrix with row-wise sparsity structure as follows:

$${\mathcal{G}}_q\left(R\right)=\left\{\Sigma \in {\mathbf{S}}_p^{+}:\underset{i\le p}{\max }\sum _{j\le p}{\mid {\sigma }_{ij}\mid }^q\le R,\text{diag}\left(\Sigma \right)=I_p\right\},$$ (4.2)

for q ∈ [0, r) and R > 0 which can depend on n and p. A similar class of covariance matrices has been considered by Bickel and Levina (2008a) and Cai and Zhou (2012).

Theorem 4.1. Fix q ∈ [0, r), R > 0 and assume that 2 log p < n and R² < (p − 1)n^−q/2. Then there exists a positive constant C₄ > 0 such that,

$$\underset{\widehat{L}}{\inf }\underset{\Sigma \in {\mathcal{G}}_q\left(R\right)}{\sup }{\mathbb{E}}_{\Sigma }\left[{(\widehat{L}-{\ell }_r(\Sigma \left)\right)}^2\right]\ge C_4{\tilde{\varphi }}_{n,p}(q,R),$$

where ${\tilde{\varphi }}_{n,p}(q,R)$ is defined by

$${\tilde{\varphi }}_{n,p}(q,R)=R^2\frac{\log p}{n}\vee \left\{R^2{\left(\frac{\log \left(\right(p-1)∕(R^2{n}^q)+1)}{2n}\right)}^{r-q}\wedge R^{2r∕q}\wedge 1\right\}$$ (4.3)

and the infimum is taken over all measurable functions $\widehat{L}$ of the sample X₁, …, X_n.

The proof is similar to that of Theorem 3.2 and is relegated to the Appendix.

As in (3.7), when 1 < R² < p^αn^−q for some α < 1, the lower bound in Theorem 4.1 can be written as

$${\tilde{\varphi }}_{n,p}(q,R)=R^2\frac{\log p}{n}\vee \left\{R^2{\left(\frac{\log p}{n}\right)}^{r-q}\wedge 1\right\}.$$ (4.4)

To establish the upper bound, we consider again a thresholding estimator. Naturally, we estimate ℓ_r functional of each single row, denoted by ${\ell }_r^{\left(i\right)}\left(\Sigma \right)={\Sigma }_j{\mid {\sigma }_{ij}\mid }^r$, using the thresholding technique. Following the same notation as the previous section, the estimator is defined by

$$\widehat{{\ell }_r\left(\Sigma \right)}={\ell }_r\left({\tilde{\Sigma }}_{\tau }\right)=\underset{i}{\max }{\ell }_r^{\left(i\right)}\left({\tilde{\Sigma }}_{\tau }\right)=\underset{i}{\max }\sum _{j\le p}{\mid {\widehat{\sigma }}_{ij}\mid }^{r}1\{\mid {\widehat{\sigma }}_{ij}\mid >\tau \},$$ (4.5)

for a threshold τ > 0. We will see in the next theorem that this estimator achieves the adaptive minimax optimal rate.

Theorem 4.2

Assume that γ log(p) < n for some constant γ > 8 and fix C₀ ≥ 4. Consider the threshold

$$\tau =2C_0\sqrt{\frac{\gamma \log p}{n}}$$

and assume that τ ≤ 1. Then, for any q ∈ [0, r), R > 0, the plug-in estimator ${\ell }_r\left({\tilde{\Sigma }}_{\tau }\right)$ satisfies

$$\mathbb{E}\left[{\left({\ell }_r\right({\tilde{\Sigma }}_{\tau })-{\ell }_r(\Sigma \left)\right)}^2\right]\le C_5{\tilde{\psi }}_{n,p}(q,R)+C_6{p}^{4-\gamma ∕2},$$

where

$${\tilde{\psi }}_{n,p}(q,R)=\{\begin{array}{cc}\frac{R^2\log p}{n},\hfill & \mathit{if}q<\max \{r-1,0\},\hfill \\ R^2{\left(\frac{\gamma \log p}{n}\right)}^{r-q},\hfill & \mathit{if}q\ge \max \{r-1,0\}\hfill \end{array}\phantom{\}}$$

and C₅ and C₆ are positive constants.

The proof of this theorem is a generalization of the proof of Theorem 3.1 but some aspects that have independent value are presented here. In the proof of Theorem 3.1, we used the decomposition

$${\widehat{\sigma }}_{ij}^2-{\sigma }_{ij}^2=2{\sigma }_{ij}({\widehat{\sigma }}_{ij}-{\sigma }_{ij})+{({\widehat{\sigma }}_{ij}-{\sigma }_{ij})}^2,$$

which is actually the Taylor expansion of ${\widehat{\sigma }}_{i,j}^2$ at σ_i,j. Carefully scrutinizing the proof, we find that the first term has the parametric rate O(R²/n) whereas the second term contributes to the rate O(R²(log p/n)^2−q). This phenomenon can be generalized to the ℓ_r-functional. In the latter case, we will apply the Taylor expansion of ${\mid {\widehat{\sigma }}_{ij}\mid }^r$ at |σ_ij|, and the first-order term will contribute to the parametric rate of O(R² log p/n) while the second-order term has the rate O(R²(log p/n)^r−q). The elbow effect stems from the dominance of estimation errors of the first- and second-order terms of Taylor's expansion. We relegate the complete proof to the supplementary material.

A few remarks should be mentioned:

1. The combination of the two theorems imply that the estimator ${\ell }_r\left({\tilde{\Sigma }}_{\tau }\right)$ is minimax adaptive over the space $\left\{{\mathcal{G}}_q\right(R),q\in [0,r),R>0\}$ under very mild conditions. The adaptive minimax optimal rate of convergence is given by (4.4). The term p^4−γ/2 can be made arbitrarily small by choosing large enough γ.

2. The ℓ_r functional involves the maxima of the row sums. Compared it with estimating the quadratic functional, we need to pay the price of an extra log p term in the parametric rate.

3. The rate ${\tilde{\varphi }}_{n,p}(q,R)$ presents the elbow phenomenon at q = r − 1 if r > 1. So quadratic row-wise functional ${\ell }_2\left({\tilde{\Sigma }}_{\tau }\right)$ bears the same elbow behavior as the quadratic functional $Q\left({\tilde{\Sigma }}_{\tau }\right)$.

4.2. Optimal detection of correlations

In this subsection, we illustrate the intrinsic link between functional estimation and hypothesis testing. To that end, consider the following hypothesis testing problem:

$$H_0:X~\mathcal{N}(0,I_p),$$

$$H_{1}X~\mathcal{N}(0,I_p+\kappa \cdot \text{off}(\Sigma \left)\right),\Sigma \in \bigcup _{q\in [0,r)}\left\{{\mathcal{G}}_q\right(R):{\ell }_r(\text{off}\left(\Sigma \right))=1\}.$$

This problem is intimately linked to sparse principal component analysis [Berthet and Rigollet (2013a, 2013b)]. A natural question associated with this problem is to find the minimal signal strength κ such that these hypotheses can be tested with high accuracy.

The previous subsection provides the optimal estimate for ℓ_r(off(Σ)). However, we need a result with high probability rather than in expectation. Using Lemma 4.2 in the supplementary material [Fan, Rigollet and Wang (2015)] and arguments similar to those employed to prove Theorem 4.2, it is not hard to show that

$$\mid {\ell }_r\left({\tilde{\Sigma }}_{\tau }\right)-{\ell }_r\left(\Sigma \right)\mid \le CR{\left(\frac{\gamma \log p}{n}\right)}^{(r-q)∕2}=CR{\left(\frac{2\log p+\log (4∕\delta )}{n}\right)}^{(r-q)∕2},$$

with probability larger than 1 − 4p^−(γ−2) =: 1 − δ. Therefore, letting

$$s_0=1+CR{\left(\frac{2\log p+\log (4∕\delta )}{n}\right)}^{(r-q)∕2},$$

$$s_1=1+{\kappa }^r-CR{\left(\frac{2\log p+\log (4∕\delta )}{n}\right)}^{(r-q)∕2},$$

we get ${\mathbb{P}}_{H_0}\left({\ell }_r\right({\tilde{\Sigma }}_{\tau })\le s_0)\ge 1-\delta$ and ${\mathbb{P}}_{H_1}\left({\ell }_r\right({\tilde{\Sigma }}_{\tau })\ge s_1)\ge 1-\delta$. Here, ${\mathbb{P}}_{H_0}$ denotes the probability under the null hypothesis and ${\mathbb{P}}_{H_1}$ denotes the largest probability over the composite alternative. To build a hypothesis test, note that if s₁ > s₀, then for any s ∈ [s₀, s₁], the test $\psi =1\left\{{\ell }_r\right({\tilde{\Sigma }}_{\tau })\ge s\}$ satisfies ${\mathbb{P}}_{H_0}(\psi =1)\vee {\mathbb{P}}_{H_1}(\psi =0)\le \delta$. We say that the test ψ discriminates between H₀ and H₁ with accuracy δ.

Theorem 4.3

Assume that n, p, R, q, r and δ are such that $\stackrel{‒}{\kappa }<1$ where

$$\stackrel{‒}{\kappa }≔2C{R}^{1∕r}{\left(\frac{2\log p+\log (4∕\delta )}{n}\right)}^{(r-q)∕\left(2r\right)}.$$

Then, for any $\kappa >\stackrel{‒}{\kappa }$ and for any s ∈ [s₀, s₁], the test $\psi =1\left\{{\ell }_r\right({\tilde{\Sigma }}_{\tau })\ge s\}$ discriminates between H₀ and H₁ with accuracy δ.

The minimax risk for the correlation detection is given in the next theorem, which will be proved in the Appendix.

Theorem 4.4

For fixed ν > 0, define $\underset{‒}{\kappa }>0$ by

$$\underset{¯}{\kappa }≔R^{1∕r}{\left(\frac{\log (vp∕(R^2{n}^q\left)\right)}{2n}\right)}^{(r-q)∕\left(2r\right)}.$$

Then for any $\kappa <\underset{‒}{\kappa }$,

$$\underset{\psi }{\inf }\left\{{\mathbb{P}}_{H_0}\right(\psi =1)\vee {\mathbb{P}}_{H_1}(\psi =0\left)\right\}\ge C_v,$$

where the infimum is taken over all possible tests and C_ν > 0 is a continuous function of ν that tends to 1/2 as ν → 0.

If we assume the high-dimensional regime R² < p^αn^−q for some α < 1 as discussed before, then the lower bound matches the upper bound. So the theorem concludes that no test has asymptotic power for correlation detection unless κ is of higher order than R^1/r(log p/n)^(r−q)/(2r) and the detection method based on optimal ℓ_r(Σ) estimation is also optimal for testing existence of correlation.

5. Numerical experiments

Simulations are conducted in this section to evaluate the numerical performance of our plug-in estimator for quadratic functionals. Then the proposed method is applied to two high-dimensional testing problems: simulated two-sample data and real financial equity market data.

5.1. Quadratic functional estimation

We first study the behavior of estimators $\widehat{Q\left(\Sigma \right)}+\widehat{D\left(\Sigma \right)}$ for the total quadratic functional and $\widehat{Q\left(\Sigma \right)}=Q\left({\tilde{\Sigma }}_{\tau }\right)$ for its off-diagonal part. To that end, four sparse covariance matrix structures were used in the simulations:

• (M1) auto-correlation AR(1) covariance matrix σ_ij = 0.25^|i−j|;

• (M2) banded correlation matrix with σ_ij = 0.3 if |i − j| = 1 and 0 otherwise;

• (M3) sparse matrix with a block, size p/20 by p/20, of correlation 0.3;

• (M4) identity matrix (it attains the maximal level of sparsity).

We chose p = 500 and let n vary from 30 to 100. For estimating the total quadratic functional, our proposed thresholding estimator, BS [Bai and Saranadasa (1996)] estimator and CQ [Chen and Qin (2010)] estimator were applied to each setting for repetition of 500 times. Their mean absolute estimation errors were reported in log scale (base 2) in Figure 1 with their standard deviations omitted here. BS and CQ cannot be directly used for off-diagonal quadratic functional estimation, so we deducted $\widehat{D\left(\Sigma \right)}$ from both of them to serve as an estimator for only the off-diagonal part. The mean absolute estimation errors, compared with our proposed estimator $Q\left({\tilde{\Sigma }}_{\tau }\right)$, are depicted in log scale (base 2) in Figure 2.

Fig. 1.

Performance of estimating ${\Vert \Sigma \Vert }_{\mathrm{F}}^2$ using thresholded estimator $\widehat{Q}+\widehat{D}$ (dotted), CQ (solid) and BS (dashed). The mean of absolute errors over 500 repetitions in log scale (base 2) versus the sample size were reported for matrix M1 (top left), M2 (top right), M3 (bottom left), M4 (bottom right).

Fig. 2.

Performance of estimating Q(Σ) using thresholded estimator $\widehat{Q}$ (dotted), $CQ-\widehat{D}$ (solid) and $BS-\widehat{D}$ (dashed). The mean of absolute errors over 500 repetitions in log scale (base 2) versus the sample size were reported for matrix M1 (top left), M2 (top right), M3 (bottom left), M4 (bottom right).

The four plots correspond to the aforementioned four covariance structures. We did not report the estimation error of directly using the naive plug-in which is an obvious disaster. In all the four cases, the BS (dashed line) method does not perform well in the “large p small n” regime. The method CQ (solid line) exhibits a relatively small estimation error in general, but it can still be improved using the thresholding method. As theory shows, the method CQ is ratio-consistent [Chen and Qin (2010)], so our method (dotted line) is better only to a second order, which was captured by the small gap between dotted and solid curves. When estimating only off-diagonal quadratic functionals (Figure 2), the advantage of the thresholding method is even sharper since the error caused by nonsparse diagonal elements has been eliminated. The improved performance comes from the prior knowledge of sparsity, thus our method works best for very sparse matrix, especially well for identity matrix as seen in Figure 1.

A practical question is how to choose a proper threshold, as this is important to the performance of the thresholding estimator. In the above simulations, we chose $\tau =C\sqrt{\log p∕n}$ with constant C slightly different for the four cases but all close to 1.5. In the next two applications to hypothesis testing, we employ the cross validation to choose a proper thresholding. The procedure consists of the following steps:

• (1)The data is split into training data $D_{\mathcal{S}}^{\left(v\right)}$ of sample size n₁ and testing data $D_{{\mathcal{S}}^c}^{\left(v\right)}$ of sample size n − n₁ for m times, v = 1, 2, …, m.
• (2)The training data $D_{\mathcal{S}}^{\left(v\right)}$ is used to construct the thresholding estimator $Q\left({\tilde{\Sigma }}_{\tau }^{\left(v\right)}\right)$ under a sequence of thresholds while the testing data $D_{{\mathcal{S}}^c}^{\left(v\right)}$ constructs the nonthresholded ratio-consistent estimator ${\widehat{Q}}^{\left(v\right)}$, for example, using CQ estimator of ${\Vert \Sigma \Vert }_{\mathrm{F}}^2$.
• (3)The candidates of thresholds are ${\tau }_j=j\Delta \sqrt{\log \left(p\right)∕n_1}$ for j = 1, 2, …, J where J is chosen to be a reasonably large number, say 50, and Δ is such that $J\Delta \sqrt{\log \left(p\right)∕n_1}\le \widehat{M}≔{\max }_i{\widehat{\sigma }}_{ii}$.
• (4)
  The final j* is taken to be the minimizer of the following problem:

  $$\underset{j\in \{1,2,\dots ,J\}}{\min }\frac{1}{m}\sum _{v=1}^m\mid Q\left({\tilde{\Sigma }}_{{\tau }_j}^{\left(v\right)}\right)-{\widehat{Q}}^{\left(v\right)}\mid .$$
• (5)The final estimator $Q\left({\tilde{\Sigma }}_{{\tau }_{j^{\ast }}}\right)$ is obtained by applying threshold $j^{\ast }\Delta \sqrt{\log \left(p\right)∕n}$ to the empirical covariance matrix of the entire n data.

Bickel and Levina (2008a) suggested to use n₁ = n/ log n for estimating covariance matrices. This is consistent with our experience for estimating functionals when no prior knowledge about the covariance matrix structure is provided. We will apply this splitting rule in the later simulation studies on high-dimensional hypothesis testing.

5.2. Application to high-dimensional two-sample testing

In this section, we apply the thresholding estimator of quadratic functionals to the high-dimensional two-sample testing problem. Two groups of data are simulated from the Gaussian models:

$$X_{i,j}~\mathcal{N}({\mu }_i,\Sigma )\text{for}i=1,2\text{and}j=1,\dots ,n∕2.$$

The dimensions considered for this problem are (p, n) ∈ {(500, 100), (1000, 150), (2000, 200)}. For simplicity, we choose Σ to be a correlation matrix and choose the sparse covariance structure to be 2 by 2 block diagonal matrices with 250 of them having correlations 0.3 and the rest having correlations 0. So the off-diagonal quadratic functional is always 45, which does not increase with p in our setting. The mean vectors μ₁ and μ₂ are chosen as follows. Let μ₁ = 0 and the percentage of μ_1,k = μ_2,k to be in {0%, 50%, 95%, 100%}. The 100% proportion corresponds to the case where the two groups are identical, thus gives information about accuracy of the size of the tests. The 95% proportion represents the situation where the alternative hypotheses are sparse. For those k such that μ_1,k ≠ μ_2,k, we simply chose the value of each μ_2,k equally. To make the power comparable among different configurations, we use a constant signal-to-noise ratio $\eta =\Vert {\mu }_1-{\mu }_2\Vert ∕\sqrt{\mathrm{tr}\left({\Sigma }^2\right)}=0.1$ across experiments.

Table 1 reports the empirical power and size of six testing methods based on 500 repetitions.

Table 1.

Empirical testing power and size of 6 testing methods based on 500 simulations

Prop. of equalities	BS	newBS	CQ	newCQ	Bonf	BH
	p = 500, n = 100
0%	0.408	0.422	0.428	0.432	0.104	0.110
50%	0.396	0.422	0.418	0.428	0.110	0.116
95%	0.422	0.440	0.438	0.442	0.208	0.214
100% (size)	0.030	0.036	0.036	0.038	0.042	0.042
	p = 1000, n = 150
0%	0.696	0.710	0.718	0.718	0.082	0.086
50%	0.698	0.712	0.712	0.714	0.106	0.112
95%	0.702	0.716	0.718	0.722	0.308	0.328
100% (size)	0.040	0.044	0.048	0.046	0.050	0.050
	p = 2000, n = 200
0%	0.930	0.938	0.940	0.940	0.138	0.146
50%	0.918	0.922	0.924	0.928	0.104	0.106
95%	0.922	0.928	0.930	0.930	0.324	0.338
100% (size)	0.046	0.050	0.050	0.050	0.046	0.046

(BS) Bai and Saranadasa's original test.

(newBS) Bai and Saranadasa's modified test where tr(Σ²) is estimated by thresholding the sample covariance matrix.

(CQ) Chen and Qin's original test.

(newCQ) Chen and Qin's modified test where $\mathrm{tr}\left({\Sigma }_i^2\right)$ and tr(Σ₁Σ₂) are estimated by thresholding their empirical counterparts.

(Bonf) Bonferroni correction: This method regards the high-dimensional testing problem as p univariate testing problems. If there is a p-value that is less than 0.05/p, the null hypothesis is rejected.

(BH) Benjamini–Hochberg method. The method is similar to the Bonferroni correction, but employs the Benjamini–Hochberg method in decision making.

For estimating quadratic functionals, the cross-validation is employed using n/log(n) splitting rule. The first four methods are evaluated at the 5% significance level while Bonferroni correction and Benjamini–Hochberg correction are evaluated at 5% family-wise error rate or FDR. We also list the average relative estimation errors for the quadratic functionals of the first four methods in Table 2. Here, the average is taken over four different proportions of equalities and the average for CQ and newCQ is also taken over errors in estimating $\mathrm{tr}\left({\Sigma }_1^2\right)$ and $\mathrm{tr}\left({\Sigma }_2^2\right)$.

Table 2.

Mean and SD of relative errors for estimating quadratic functionals (in percentage)

	p = 500, n = 100	p = 1000, n = 150	p = 2000, n = 200
BS	4.93 (2.48)	4.47 (1.56)	5.05 (1.10)
newBS	2.12 (1.43)	0.74 (0.56)	0.54 (0.40)
CQ	3.72 (1.97)	2.32 (1.24)	1.70 (0.91)
newCQ	2.77 (1.38)	1.27 (0.64)	0.62 (0.33)

Several comments are in order. First, the first four methods based on Wald-type of statistic with correlation ignored perform much better, in terms of the power, than the last two methods which combines individual tests. Even in the case that proportional of equalities is 95% where the individual difference is large for nonidentical means, aggregating the signals together in the Wald-type of statistic still outperforms. However, in the case of 0% identical means, the power of Bonferroni or FDR method is extremely small, due to small individual differences. Second, the method newCQ, which combines CQ and thresholding estimator of the quadratic functional, has the highest power and performs the best among all methods. The corrected BS method also improves the performance by estimating the quadratic functionals better compared with original BS. CQ indeed is more powerful than BS as claimed by Chen and Qin (2010), but we can even improve the performance of those two methods more by leveraging the sparsity structure of covariance matrices.

5.3. Estimation of ℓ_r functional and correlation detection

In order to check the effectiveness of using ℓ_r norm of the thresholded sample matrix to detect correlation, let us take one simple matrix structure as an example and use r = 1. Under H₀, assume $X~\mathcal{N}(0,I_p)$; while under $H_1,X~\mathcal{N}(0,\Sigma )$, where Σ_ij = 0.8 if i, $j\in \mathcal{S}$ and $\mathcal{S}$ is a random subset of size p/20 in {1, 2, …, n}. We chose to use p = 500 and generated n = 100 independent random vectors under both H₀ and H₁. The whole simulation was done for N = 1000 times.

We compare the ℓ₁ norm estimates based on empirical covariance matrix ${\ell }_1\left(\widehat{\Sigma }\right)$ and thresholded empirical covariance matrix ${\ell }_1\left({\tilde{\Sigma }}_{\tau }\right)$. The threshold is decided by cross validation with n/log(n) splitting. The simulations yielded N estimates for both null and alternative hypotheses, which were plotted in Figure 3. The optimal estimator ${\ell }_1\left({\tilde{\Sigma }}_{\tau }\right)$ perfectly discriminates the null and alternative hypotheses while ${\ell }_1\left(\widehat{\Sigma }\right)$ overestimates ℓ₁ functional and blurs the difference of the two hypotheses.

Fig. 3.

Histogram of 1000 ℓ₁ functional estimates for H₀ and _H1 by ${\ell }_r\left(\widehat{\Sigma }\right)$ (left) and optimal estimator ${\ell }_r\left({\tilde{\Sigma }}_{\tau }\right)$ (right).

5.4. Application to testing multifactor pricing model

In this section, we test the validity of the Capital Asset Pricing Model (CAPM) and Fama–French models using Pesaran and Yamagata's method (2.2) for the securities in the Standard & Poor 500 (S&P 500) index. Following the literature, we used 60 monthly stock returns to construct test statistics since monthly returns are nearly independent. The composition of index keeps changing annually, so we selected only 276 large stocks. The monthly returns (adjusted for dividend) between January 1990 and December 2012 are downloaded from the Wharton Research Data Services (WRDS) database. The time series on the risk-free rates and Fama–French three factors are obtained from Ken French's data library. If only the first factor, that is, the excessive return of the market portfolio is used, the Fama–French model reduces to the CAPM model. We tested the null hypothesis H₀ : α = 0 for both models. The p-values of the tests are depicted in Figure 4, which are computed based on running windows of previous 60 months.

Fig. 4.

P -values of testing H₀ : α = 0 in the CAMP and Fama–French 3 factor models based on S&P 500 monthly returns from January 1995 to December 2012.

The results suggest that market efficiency is time dependent and the Fama–French model are rejected less frequently than the CAPM. Before 1998, the evidence that α ≠ 0 is very strong. After 1998, the Fama–French 3-factor model holds most of the time except the period 2007–2009 that contains the financial crisis. On the other hand, the CAPM is rejected for an extended period of time during this period.

APPENDIX A: A TECHNICAL LEMMA ON χ² DIVERGENCES

Lemma A.1

Consider a mixture of Gaussian product distributions

$${\tilde{\mathbb{P}}}^n=\frac{1}{m}\sum _{j=1}^m{\mathbb{P}}_j^n,$$

where ${\mathbb{P}}_j~{\mathcal{N}}_p(0,{\Sigma }_j)$ such that ${\mathbb{P}}_j⪡{\mathbb{P}}_0$. Then

$${\chi }^2({\tilde{\mathbb{P}}}^n,{\mathbb{P}}_0^n)=\frac{1}{m^2}\sum _{j,k=1}^m{\mid I_p-({\Sigma }_j-I)({\Sigma }_k-I)\mid }^{-n∕2}-1.$$ (A.1)

Furthermore, assume 2(log p) ≥ n. Consider the mixture ${\stackrel{‒}{\mathbb{P}}}^n$ defined in the proof of Theorem 3.2 where k and a are defined as follows:

1. If $R<4{\left(\frac{\log p}{n}\right)}^{q∕2}$, then take k = 1 and a = (R/4)^1/q.

2. If $R\ge 4{\left(\frac{\log p}{n}\right)}^{q∕2}$, then take k to be the largest integer such that

$$k\le \frac{R}{2}{\left(\frac{\log \left(\right(p-1)∕k^2+1)}{2n}\right)}^{-q∕2}$$ (A.2)

and

$$a={\left(\frac{\log \left(\right(p-1)∕k^2+1)}{2n}\right)}^{1∕2}\wedge {\left(2k\right)}^{-1∕2}.$$ (A.3)

Such choices yield in both cases

$${\chi }^2({\stackrel{‒}{\mathbb{P}}}^n,{\mathbb{P}}_0^n)\le e-1.$$ (A.4)

Moreover, in case 2 we have that (i) k ≥ 2 and (ii) under the assumption that R² < (p − 1)n^−q/2, we also have k² ≤ R²n^q < (p − 1)/2.

Proof

To unify the notation, we will work directly with ${\mathbb{P}}_S$, $S\in \mathcal{S}$ rather than ${\mathbb{P}}_j$, j ∈ [m]. However, in the first part of the proof, we will not use the specific form Σ_S nor that of $\mathcal{S}$. For now, we simply assume that Σ_S is invertible (we will check this later on). Recall that

$${\chi }^2({\stackrel{‒}{\mathbb{P}}}^n,{\mathbb{P}}_0^n)={\mathbb{E}}_0\left[{\left(\frac{\mathrm{d}{\stackrel{‒}{\mathbb{P}}}^n}{\mathrm{d}{\mathbb{P}}_0^n}-1\right)}^2\right]=\frac{1}{{\mid \mathcal{S}\mid }^2}\sum _{S,T\in \mathcal{S}}{\left({\mathbb{E}}_0\left[\frac{\mathrm{d}{\mathbb{P}}_S}{\mathrm{d}{\mathbb{P}}_0}\frac{\mathrm{d}{\mathbb{P}}_T}{\mathrm{d}{\mathbb{P}}_0}\right]\right)}^n-1,$$

where ${\mathbb{E}}_0$ denotes the expectation with respect to ${\mathbb{P}}_0$. Furthermore,

$${\mathbb{E}}_0\left[\frac{\mathrm{d}{\mathbb{P}}_S}{\mathrm{d}{\mathbb{P}}_0}\frac{\mathrm{d}{\mathbb{P}}_T}{\mathrm{d}{\mathbb{P}}_0}\right]=\frac{1}{{(\mid {\Sigma }_S\mid \mid {\Sigma }_T\mid )}^{1∕2}}{\mathbb{E}}_0\left[\exp \left(-\frac{1}{2}{X}^{⊺}({\Sigma }_S^{-1}+{\Sigma }_T^{-1}-2I_p)X\right)\right].$$

Consider the spectral decomposition of ${\Sigma }_S^{-1}+{\Sigma }_T^{-1}-2I_p=U\Lambda U^{⊺}$, where U is an orthogonal matrix and Λ is a diagonal matrix with eigenvalues λ₁, …, λ_p on its diagonal. Then, by rotational invariance of the Gaussian distribution, it holds

$$\begin{array}{cc}\hfill {\mathbb{E}}_0& \left[\exp \left(-\frac{1}{2}{X}^{⊺}({\Sigma }_S^{-1}+{\Sigma }_T^{-1}-2I_p)X\right)\right]\hfill \\ \hfill & ={\mathbb{E}}_0\left[\exp \left(-\frac{1}{2}{X}^{⊺}\Lambda X\right)\right]\hfill \\ \hfill & =\prod _{j=1}^p{\mathbb{E}}_0\left[\exp \left(-\frac{1}{2}{\lambda }_j{X}_j^2\right)\right]\hfill \\ \hfill & =\{\begin{array}{cc}\prod _{j=1}^p{(1+{\lambda }_j)}^{-1∕2}={\mid I+\Lambda \mid }^{-1∕2},\hfill & \text{if}{\max }_j{\lambda }_j<1,\hfill \\ \infty ,\hfill & \text{otherwise}.\hfill \end{array}\phantom{\}}\hfill \end{array}$$

To ensure that the above expression is finite, note that the Cauchy–Schwarz inequality yields

$$\begin{array}{cc}\hfill {\left({\mathbb{E}}_0\left[\frac{\mathrm{d}{\mathbb{P}}_S}{\mathrm{d}{\mathbb{P}}_0}\frac{\mathrm{d}{\mathbb{P}}_T}{\mathrm{d}{\mathbb{P}}_0}\right]\right)}^2& \le {\mathbb{E}}_0\left[{\left(\frac{\mathrm{d}{\mathbb{P}}_S}{\mathrm{d}{\mathbb{P}}_0}\right)}^2\right]{\mathbb{E}}_0\left[{\left(\frac{\mathrm{d}{\mathbb{P}}_T}{\mathrm{d}{\mathbb{P}}_0}\right)}^2\right]\hfill \\ \hfill & =\left({\chi }^2\right({\mathbb{P}}_S,{\mathbb{P}}_0)+1)\left({\chi }^2\right({\mathbb{P}}_T,{\mathbb{P}}_0)+1)<\infty ,\hfill \end{array}$$

where the two χ² divergences are finite because ${\mathbb{P}}_S⪡{\mathbb{P}}_0$ for any $S\in \mathcal{S}$. Therefore,

$${\mathbb{E}}_0\left[\frac{\mathrm{d}{\mathbb{P}}_S}{\mathrm{d}{\mathbb{P}}_0}\frac{\mathrm{d}{\mathbb{P}}_T}{\mathrm{d}{\mathbb{P}}_0}\right]=\frac{{\mid I+\Lambda \mid }^{-1∕2}}{{(\mid {\Sigma }_S\mid \mid {\Sigma }_T\mid )}^{1∕2}}=\frac{{\mid {\Sigma }_S^{-1}+{\Sigma }_T^{-1}-I_p\mid }^{-1∕2}}{{(\mid {\Sigma }_S\mid \mid {\Sigma }_T\mid )}^{1∕2}}.$$

Next, observe that

$$\begin{array}{cc}\hfill {(\mid {\Sigma }_S\mid \mid {\Sigma }_T\mid \mid {\Sigma }_S^{-1}+{\Sigma }_T^{-1}-I_p\mid )}^{-1∕2}& ={\left(\right(\mid {\Sigma }_S({\Sigma }_S^{-1}+{\Sigma }_T^{-1}-I_p){\Sigma }_T)\mid )}^{-1∕2}\hfill \\ \hfill & ={\mid I-({\Sigma }_S-I)({\Sigma }_T-I)\mid }^{-1∕2}.\hfill \end{array}$$

Since we have not used the specific form of Σ_S, $S\in \mathcal{S}$, this bound is valid for any mixture and completes the proof of (A.1).

Next, we apply this bound to the specific choice for Σ_S of (3.10). Note that the minimal eigenvalue of the matrices Σ_S, $S\in \mathcal{S}$ is $1-\sqrt{k{a}^2}$. Later we will show 2a²k ≤ 1, which implies that Σ_S is always positive definite. In particular, this implies that ${\mathbb{P}}_S⪡{\mathbb{P}}_0$ for any $S\in \mathcal{S}$. Moreover, it follows from definition (3.10) that

$$I-({\Sigma }_S-I)({\Sigma }_T-I)=\left(\begin{array}{cc}\hfill 1-a^2{1}_S^{⊺}{1}_T\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill I-a^2{1}_S{1}_T^{⊺}\hfill \end{array}\right),$$

where 0 is a generic symbol to indicate space filled by zeros. Expanding the determinant along the first row (or column), we get

$$\begin{array}{cc}\hfill {\mid I-({\Sigma }_S-I)({\Sigma }_T-I)\mid }^{-1∕2}& ={(1-a^2{1}_S^{⊺}{1}_T)}^{-1∕2}{\mid I-a^2{1}_S{1}_T^{⊺}\mid }^{-1∕2}\hfill \\ \hfill & ={(1-a^2{1}_S^{⊺}{1}_T)}^{-1},\hfill \end{array}$$

where in the second equality, we used Sylvester's determinant theorem. By (A.1), we have

$${\chi }^2({\stackrel{‒}{\mathbb{P}}}^n,{\mathbb{P}}_0^n)=\frac{1}{{\mid \mathcal{S}\mid }^2}\sum _{S,T\in \mathcal{S}}{(1-a^2{1}_S^{⊺}{1}_T)}^{-n}-1.$$

As to be verified later, 2a²k ≤ 1. Using the fact that (1 − x)⁻¹ ≤ exp(2x) for x ∈ [0, 1/2] and the symmetry, we have

$${\chi }^2({\stackrel{‒}{\mathbb{P}}}^n,{\mathbb{P}}_0^n)\le \frac{1}{\mid \mathcal{S}\mid }\sum _{S\in \mathcal{S}}\exp (2n{a}^2\mid S\cap [k]\mid )-1=\epsilon \left[\exp \right(2n{a}^2\mid S\cap \left[k\right]\mid )-1],$$

where $\mathcal{E}$ denotes the expectation with respect to the distribution of S randomly chosen from $\mathcal{S}$. In particular, $\mid S\cap \left[k\right]\mid ={\Sigma }_{i=1}^{k}1(i\in S)$ is the sum of k negatively associated random variables. Therefore, using negative association, the above expectation is further bounded by

$$\prod _{i=1}^k\epsilon \left[e^{2n{a}^{2}1(i\in S)}\right].$$

Next, for a given by (A.3), we have

$$\prod _{i=1}^k\epsilon \left[e^{2n{a}^{2}1(i\in S)}\right]={\left[(e^{2n{a}^2}-1)\frac{k}{p-1}+1\right]}^k\le {\left[1+\frac{1}{k}\right]}^k\le e.$$

We now show for both cases of the lemma, we have 2a²k ≤ 1. Indeed for case 1, we get 2a²k = 2(R/4)^2/q < 2(log p)/n ≤ 1. For case 2, 2a²k ≤ 1 follows trivially from the definition of a. Also observe that k ≥ 2 since

$$\frac{R}{2}{\left(\frac{\log \left(\right(p-1)∕4+1)}{n}\right)}^{-q∕2}\ge 2{\left(\frac{\log p}{n}\right)}^{q∕2}{\left(\frac{\log \left(\right(p-1)∕4+1)}{n}\right)}^{-q∕2}\ge 2.$$

This proves part (i) of the statement on k. To prove part (ii), observe that R² < (p − 1)n^−q/2 implies that

$$2^{1-2∕q}<1<\log \left(\frac{p-1}{R^2{n}^q}+1\right),$$

which is equivalent to

$$R{n}^{q∕2}>\frac{R}{2}{\left(\frac{\log \left(\right(p-1)∕(R^2{n}^q)+1)}{2n}\right)}^{-q∕2}.$$

Therefore, k² ≤ R²n^q < (p − 1)/2.

APPENDIX B: PROOF OF THEOREM 4.1

The proof follows a similar idea to that of Theorem 3.2. For the second part of the lower bound, we use exactly the same construction of two hypotheses as in Lemma A.1. Then it follows that for the ℓ_r functional,

$$\underset{\widehat{L}}{\inf }\underset{\Sigma \in \mathcal{H}}{\max }{\mathbb{E}}_{\Sigma }\left[{(\widehat{L}-{\ell }_r(\Sigma \left)\right)}^2\right]\ge C{k}^2{a}^{2r},$$

for some positive constant C, where the infimum is taken over all estimators $\widehat{L}$ of ℓ_r(Σ) based on n observations. In case 1, k²a^2r = CR^2r/q while in case 2, following the same arguments as before,

$$k^2{a}^{2r}\ge \frac{R^2}{16}{\left(\frac{\log \left(\right(p-1)∕(R^2{n}^q)+1)}{2n}\right)}^{r-q}\wedge \frac{1}{2}.$$

This completes the second part of the lower bound.

The first part of the result is a little bit more complicated than the construction of A^(k) and B^(k) in the proof of Theorem 3.2 due to the extra log p term in the lower bound. We need to consider a mixture of measures in order to capture the complexity of the problem. With a slight abuse of notation, we redefine (2k) × (2k) matrices A^(k), B^(k) as follows:

$$A^{\left(k\right)}=\left(\begin{array}{cc}\hfill {11}^{⊺}\hfill & \hfill a{11}^{⊺}\hfill \\ \hfill a{11}^{⊺}\hfill & \hfill {11}^{⊺}\hfill \end{array}\right),B^{\left(k\right)}=\left(\begin{array}{cc}\hfill {11}^{⊺}\hfill & \hfill b{11}^{⊺}\hfill \\ \hfill b{11}^{⊺}\hfill & \hfill {11}^{⊺}\hfill \end{array}\right),$$

where a, b ∈ (0, 1/2) and 1 denotes a vector of ones of length k. Since R² < p, we now construct the block diagonal covariance matrices

$${\Sigma }_m^{\left(k\right)}=\text{diag}(C_1,C_2,\dots ,C_M,I_{p-2kM}),m=1,2,\dots ,M,$$

where the mth diagonal block is chosen to be C_m = B^(k) while others are C_i = A^(k) for i ≠ m and M = ⌊p/R⌋. Also define ${\Sigma }_0^{\left(k\right)}$ to be of the same structure with C_i = A^(k) for all i. Then we have ${\Sigma }_m^{(R∕2)}\in {\mathcal{G}}_q\left(R\right)$ for m = 0, 1, …, M, since each row of ${\Sigma }_m^{(R∕2)}$ only contains at most R nonzero elements that are bounded by 1.

Let ${\mathbb{P}}_0$ denote the distribution of $X\sim {\mathcal{N}}_p(0,{\Sigma }_0^{(R∕2)})$ and ${\mathbb{P}}_m$ denote the distribution of $X\sim {\mathcal{N}}_p(0,{\Sigma }_m^{(R∕2)})$. Let ${\mathbb{P}}_0^n$ (resp., ${\mathbb{P}}_m^n$) denote the distribution of X = (X₁, …, X_n) of n i.i.d. random variables drawn from ${\mathbb{P}}_0$ (resp., ${\mathbb{P}}_m$). Moreover, let ${\stackrel{‒}{\mathbb{P}}}^n$ denote the uniform mixture of ${\mathbb{P}}_m^n$ over m ∈ [M]. Consider the testing problem

$$H_0:\mathbf{X}~{\mathbb{P}}_0^n\text{vs}.H_1:\mathbf{X}~{\stackrel{‒}{\mathbb{P}}}^n.$$

Using Theorem 2.2, part (iii) of Tsybakov (2009) as before, we need to show χ²-divergence can be bounded by a constant. By the same calculation as in Lemma A.1, we have

$${\chi }^2({\stackrel{‒}{\mathbb{P}}}^n,{\stackrel{‒}{\mathbb{P}}}_0^n)=\frac{1}{M^2}\sum _{1\le i,j\le M}{\mid I-[{\left({\Sigma }_0^{\left(1\right)}\right)}^{-1}{\Sigma }_i^{\left(1\right)}-I][{\left({\Sigma }_0^{\left(1\right)}\right)}^{-1}{\Sigma }_j^{\left(1\right)}-I]\mid }^{-n∕2}-1.$$

Note that χ²-divergence here depends on ${\Sigma }_m^{\left(1\right)}$ instead of ${\Sigma }_m^{(R∕2)}$ since perfectly correlated random variables do not add additional information and hence do not affect χ²-divergence (see the proof of Theorem 3.2). Using the definition of ${\Sigma }_m^{\left(1\right)}$'s, we obtain

$$\begin{array}{cc}\hfill \mid I-& [{\left({\Sigma }_0^{\left(1\right)}\right)}^{-1}{\Sigma }_i^{\left(1\right)}-I][{\left({\Sigma }_0^{\left(1\right)}\right)}^{-1}{\Sigma }_j^{\left(1\right)}-I]\mid \hfill \\ \hfill & =\{\begin{array}{cc}1-2(1+a^2){\left(\frac{a-b}{1-a^2}\right)}^2+\frac{{(a-b)}^4}{{(1-a^2)}^2},\hfill & \text{if}i=j,\hfill \\ 1,\hfill & \text{otherwise}.\hfill \end{array}\phantom{\}}\hfill \end{array}$$

Therefore,

$${\chi }^2({\stackrel{‒}{\mathbb{P}}}^n,{\mathbb{P}}_0^n)=\frac{1}{M}\left\{{\left(1-2(1+a^2){\left(\frac{a-b}{1-a^2}\right)}^2+\frac{{(a-b)}^4}{{(1-a^2)}^2}\right)}^{-n∕2}-1\right\},$$

which is bounded by ((1 − 5(a − b)²)^−n/2 − 1)/M due to the fact 2(1 + a²)/(1 − a²)² ≤ 5 for a ≤ 1/2. Now choose

$$a=\frac{1}{4},b=a+\frac{1}{4}\sqrt{\frac{\log p}{n}}.$$

By assumption, there exists a constant c₀ > 1 such that R² ≤ c₀p. Thus,

$${\chi }^2({\stackrel{‒}{\mathbb{P}}}^n,{\mathbb{P}}_0^n)\le \frac{1}{M}\left\{{\left(1-\frac{\log p}{2n}\right)}^{-n∕2}-1\right\}\le \frac{R}{p}{e}^{\log p∕2}\le \sqrt{c_0}.$$

Using standard techniques to reduce estimation problems to testing problems as before, we find

$$\underset{\widehat{L}}{\inf }\underset{\Sigma \in \{{\Sigma }_m^{(R∕2)}:m=0,\dots ,M\}}{\max }{\mathbb{E}}_{\Sigma }\left[{(\widehat{L}-{\ell }_r(\Sigma \left)\right)}^2\right]\ge C{\left({\ell }_r\right({\Sigma }_0^{(R∕2)})-{\ell }_r({\Sigma }_1^{(R∕2)}\left)\right)}^2.$$

For the above choice of ${\Sigma }_m^{\left(k\right)}$, we have

$${\left({\ell }_r\right({\Sigma }_0^{(R∕2)})-{\ell }_r({\Sigma }_1^{(R∕2)}\left)\right)}^2=\frac{R^2}{4}{(b^r-a^r)}^2\ge C{R}^2\frac{\log p}{n}.$$

Since ${\Sigma }_m^{(R∕2)}\in {\mathcal{G}}_q\left(R\right)$, the above two displays imply that

$$\underset{\widehat{L}}{\inf }\underset{\Sigma \in {\mathcal{G}}_q\left(R\right)}{\max }{\mathbb{E}}_{\Sigma }\left[{(\widehat{L}-{\ell }_r(\Sigma \left)\right)}^2\right]\ge C\frac{R^2\log p}{n},$$

which together with the other part of the lower bound, completes the proof of the theorem.

APPENDIX C: PROOF OF THEOREM 4.4

The proof is similar to that of Theorem 3.2, but simpler since r ≤ 1 where no elbow effect exists. Consider hypothesis construction (3.10) with ${\Sigma }_S=I_p+\kappa \stackrel{‒}{\Sigma }$ and

$$a=\kappa k^{-1∕r}\text{and}k=\lceil R{\left(\frac{\log (\nu p∕(R^2{n}^q\left)\right)}{2n}\right)}^{-q∕2}\rceil .$$

Choose ν sufficiently small so that $R{\left(\frac{\log (\nu p∕(R^2{n}^q\left)\right)}{2n}\right)}^{1-q∕2}\le 1∕2$, which implies 2ka² ≤ 1 and guarantees the positive semi-definiteness of Σ_S. Furthermore, ka^q ≤ R holds, so ${\Sigma }_S\in {\mathcal{G}}_q\left(R\right)$. By the same derivation as in Theorem 3.2, we are able to show

$${\chi }^2({\stackrel{‒}{\mathbb{P}}}^n,{\mathbb{P}}_0^n)\le e^{\nu }-1,$$

which by Theorem 2.2(iii) of Tsybakov (2009) leads to the final conclusion.