Sparsistency and Rates of Convergence in Large Covariance Matrix Estimation

Abstract

This paper studies the sparsistency and rates of convergence for estimating sparse covariance and precision matrices based on penalized likelihood with nonconvex penalty functions. Here, sparsistency refers to the property that all parameters that are zero are actually estimated as zero with probability tending to one. Depending on the case of applications, sparsity priori may occur on the covariance matrix, its inverse or its Cholesky decomposition. We study these three sparsity exploration problems under a unified framework with a general penalty function. We show that the rates of convergence for these problems under the Frobenius norm are of order (s_n log p_n/n)^1/2, where s_n is the number of nonzero elements, p_n is the size of the covariance matrix and n is the sample size. This explicitly spells out the contribution of high-dimensionality is merely of a logarithmic factor. The conditions on the rate with which the tuning parameter λ_n goes to 0 have been made explicit and compared under different penalties. As a result, for the L₁-penalty, to guarantee the sparsistency and optimal rate of convergence, the number of nonzero elements should be small: $s_n^{\prime }=O\left(p_n\right)$ at most, among $O\left(p_n^2\right)$ parameters, for estimating sparse covariance or correlation matrix, sparse precision or inverse correlation matrix or sparse Cholesky factor, where $s_n^{\prime }$ is the number of the nonzero elements on the off-diagonal entries. On the other hand, using the SCAD or hard-thresholding penalty functions, there is no such a restriction.

1 Introduction

Covariance matrix estimation is a common statistical problem in many scientific applications. For example, in financial risk assessment or longitudinal study, an input of covariance matrix Σ is needed, whereas an inverse of the covariance matrix, the precision matrix Σ⁻¹, is required for optimal portfolio selection, linear discriminant analysis or graphical network models. Yet, the number of parameters in the covariance matrix grows quickly with dimensionality. Depending on the applications, the sparsity of the covariance matrix or precision matrix is frequently imposed to strike a balance between biases and variances. For example, in longitudinal data analysis [see e.g., Diggle and Verbyla (1998), or Bickel and Levina (2008b)], it is reasonable to assume that remote data in time are weakly correlated, whereas in Gaussian graphical models, the sparsity of the precision matrix is a reasonable assumption (Dempster (1972)).

This initiates a series of researches focusing on the parsimony of a covariance matrix. Smith and Kohn (2002) used priors which admit zeros on the off-diagonal elements of the Cholesky factor of the precision matrix Ω = Σ⁻¹, while Wong, Carter and Kohn (2003) used zero-admitting prior directly on the off-diagonal elements of Ω to achieve parsimony. Wu and Pourahmadi (2003) used the Modified Cholesky Decomposition (MCD) to find a banded structure for Ω nonparametrically for longitudinal data. Bickel and Levina (2008b) developed consistency theories on banding methods for longitudinal data, for both Σ and Ω.

Various authors have used penalized likelihood methods to achieve parsimony on covariance selection. Fan and Peng (2004) has laid down a general framework for penalized likelihood with diverging dimensionality, with general conditions for the oracle property stated and proved. However, it is not clear whether it is applicable to the specific case of covariance matrix estimation. In particular, they did not link the dimensionality p_n with the number of nonzero elements s_n in the true covariance matrix Σ₀, or the precision matrix Ω₀. A direct application of their results to our setting can only handle a relatively small covariance matrix of size p_n = o(n^1/10).

Recently, there is a surge of interest on the estimation of sparse covariance matrix or precision matrix using penalized likelihood method. Huang, Liu, Pourahmadi and Liu (2006) used the LASSO on the off-diagonal elements of the Cholesky factor from MCD, while Meinshausen and Bühlmann (2006), d’Aspremont, Banerjee, and El Ghaoui (2008) and Yuan and Lin (2007) used different LASSO algorithms to select zero elements in the precision matrix. A novel penalty called the nested Lasso was constructed in Levina, Rothman and Zhu (2008) to penalize off-diagonal elements. Thresholding the sample covariance matrix in high-dimensional setting was thoroughly studied by El Karoui (2008) and Bickel and Levina (2008a) and Cai, Zhang and Zhou (2009) with remarkable results for high dimensional applications. However, it is not directly applicable to estimating sparse precision matrix when the dimensionality p_n is greater than the sample size n. Wagaman and Levina (2008) proposed an Isomap method for discovering meaningful orderings of variables based on their correlations that result in block-diagonal or banded correlation structure, resulting in an Isoband estimator. A permutation invariant estimator, called SPICE, was proposed in Rothman, Bickel, Levina and Zhu (2008) based on penalized likelihood with L₁-penalty on the off-diagonal elements for the precision matrix. They obtained remarkable results on the rates of convergence. The rate for estimating Ω under the Frobenius norm is of order (s_n log p_n/n)^1/2, with dimensionality cost only a logarithmic factor in the overall mean-square error, where s_n = p_n + s_n1, p_n is the number of the diagonal elements and s_n1 is the number of the nonzero off-diagonal entries. However, such a rate of convergence neither addresses explicitly the issues of sparsistency such as those in Fan and Li (2001) and Zhao and Yu (2006), nor the bias issues due to the L₁-penalty and the sampling distribution of the estimated nonzero elements. These are the core issues of the study. By sparsistency, we mean the property that all parameters that are zero are actually estimated as zero with probability tending to one, a weaker requirement than that of Ravikumar, Lafferty, Liu and Wasserman (2008).

In this paper, we investigate the aforementioned problems using the penalized pseudo-likelihood method. Assume a random sample {y_i}_1≤i≤n with mean zero and covariance matrix Σ₀, satisfying some sub-Gaussian tails conditions as specified in Lemma 2 (see Section 5). The sparsity of the true precision matrix Ω₀ can be explored by maximizing the Gaussian quasi-likelihood or equivalently minimizing

$$q_1\left(\Omega \right)=\mathrm{tr}\left(\mathrm{S}\Omega \right)-\log \mid \Omega \mid +\sum _{i\ne j}{p}_{{\lambda }_{n1}}\left(\mid {\omega }_{ij}\mid \right),$$ (1.1)

which is the penalized negative log-likelihood if the data is Gaussian. The matrix $\mathrm{S}=n^{-1}{\sum }_{i=1}^n{\mathrm{y}}_i{\mathrm{y}}_i^T$ is the sample covariance matrix, Ω = (ω_ij), and p_λn1(·) is a penalty function, depending on a regularization parameter λ_n1, which can be nonconvex. For instance, the L₁-penalty p_λ(θ) = λ|θ| is convex, while the hard-thresholding penalty defined by p_λ(θ) = λ² − (|θ| − λ)²1_{|θ|<λ}, and the SCAD penalty defined by

$$p_{\lambda }^{\prime }\left(\theta \right)=\lambda 1_{\{\theta \le \lambda \}}+{(a\lambda -\theta )}_{+}{1}_{\{\theta >\lambda \}}∕(a-1),\text{for some}a>2,$$ (1.2)

are folded-concave. Nonconvex penalty is introduced to reduce bias when the true parameter has a relatively large magnitude. For example, the SCAD penalty remains constant when θ is large, while the L₁-penalty grows linearly with θ. See Fan and Li (2001) for a detailed account of this and other advantages of such a penalty function. The computation can be done via the local linear approximation (Zhou and Li, 2008, Fan et al. 2009); see Section 2.1 for additional details.

Similarly, the sparsity of the true covariance matrix Σ₀ can be explored by minimizing

$$q_2\left(\Sigma \right)=\mathrm{tr}\left(\mathrm{S}{\Sigma }^{-1}\right)+\log \mid \Sigma \mid +\sum _{i\ne j}{p}_{{\lambda }_{n2}}\left(\mid {\sigma }_{ij}\mid \right),$$ (1.3)

where Σ = (σ_ij). Note that we only penalize the off-diagonal elements of Σ or Ω in the aforementioned two methods, since the diagonal elements of Σ₀ and Ω₀ do not vanish.

In studying a sparse covariance or precision matrix, it is important to distinguish between the diagonal and off-diagonal elements, since the diagonal elements are always positive and they contribute to the overall mean-squares errors. For example, the true correlation matrix, denoted by Γ₀, has the same sparsity structure as Σ₀ without the need to estimating its diagonal elements. In view of this fact, we introduce a revised method (3.2) to take this advantage. It turns out that the correlation matrix can be estimated with a faster rate of convergence, at (s_n1 log p_n/n)^1/2 instead of ((p_n + s_n1) log p_n/n)^1/2, where s_n1 is the number of nonzero correlation coefficients. We can take similar advantages over the estimation of the true inverse correlation matrix, denoted by Ψ₀. See Section 2.5. This is an extension of the work of Rothman et al. (2008) using the L₁-penalty. Such an extension is important since the nonconcave penalized likelihood ameliorates the bias problem for the L₁-penalized likelihood.

The bias issues of the commonly used L₁-penalty, or LASSO, can be seen from our theoretical results. In fact, due to the bias of LASSO, an upper bounded of λ_ni is needed in order to achieve fast rate of convergence. On the other hand, a lower bound is required in order to achieve sparsity of estimated precision or covariance matrices. This is in fact one of the motivations for introducing nonconvex penalty functions in Fan and Li (2001) and Fan and Peng (2004), but we state and prove the explicit rates in the current context. In particular, we demonstrate that the L₁-penalized estimator can achieve simultaneously the optimal rate of convergence and sparsistency for estimation of Σ₀ or Ω₀ when the number of nonzero elements in the off-diagonal entries are no larger than O(p_n), but not guaranteed so otherwise. On the other hand, using the nonconvex penalties like the SCAD or hard-thresholding penalty, such an extra restriction is not needed.

We also compare two different formulations of penalized likelihood using the modified Cholesky decomposition, exploring their respective rates of convergence and sparsity properties.

Throughout this paper, we use λ_min(A), λ_max(A) and tr(A) to denote the minimum eigenvalue, maximum eigenvalue, and trace of a symmetric matrix A respectively. For a matrix B, we define the operator norm and the Frobenius norm, respectively, as $\parallel B\parallel ={\lambda }_{\max }^{1∕2}\left(B^{T}B\right)$ and ∥B∥_F = tr^1/2(B^TB).

2 Estimation of sparse precision matrix

In this section, we present the analysis of (1.1) for estimating a sparse precision matrix. Before this, let us first present an algorithm for computing the nonconcave maximum (pseudo)-likelihood estimator and then state the conditions needed for our technical results.

2.1 Algorithm based on iterated reweighted L₁-penalty

The computation of the nonconcave maximum likelihood problems can be solved by a sequence of L₁-penalized likelihood problems via local linear approximation (Zou and Li 2008, Fan et al. 2009). For example, given the current estimate Ω_k = (ω_ij,k), by the local linear approximation to the penalty function,

$$q_1\left(\Omega \right)\approx \mathrm{tr}\left(\mathrm{S}\Omega \right)-\log \mid \Omega \mid +\sum _{i\ne j}[p_{{\lambda }_{n1}}\left(\mid {\omega }_{ij,k}\mid \right)+p_{{\lambda }_{n1}}^{\prime }\left(\mid {\omega }_{ij,k}\mid \right)(\mid {\omega }_{ij}\mid -\mid {\omega }_{ij,k}\mid )].$$ (2.1)

Hence, Ω_k+1 should be taken to maximize the right-hand side of (2.1):

$${\Omega }_{k+1}={\mathrm{argmax}}_{\Omega }\left[\mathrm{tr}\left(\mathrm{S}\Omega \right)-\log \mid \Omega \mid +\sum _{i\ne j}{p}_{{\lambda }_{n1}}^{\prime }\left(\mid {\omega }_{ij,k}\mid \right)\mid {\omega }_{ij}\mid \right],$$ (2.2)

after ignoring the two constant terms. Problem (2.2) is the weighted penalized L₁-likelihood. In particular, if we take the most primitive initial value Ω₀ = 0, then

$${\Omega }_1={\mathrm{argmax}}_{\Omega }\left[\mathrm{tr}\left(\mathrm{S}\Omega \right)-\log \mid \Omega \mid +{\lambda }_{n1}\sum _{i\ne j}\mid {\omega }_{ij}\mid \right],$$

is already a good estimator. Iterations of (2.2) reduces the biases of the estimator, as larger estimated coefficients in the previous iterations receive less penalty. In fact, in a different setup, Zou and Li (2008) showed that one iteration of such a procedure is sufficient as long as the initial values are good enough.

Fan et al. (2009) has implemented the above algorithm for optimizing (1.1). They have also demonstrated in Section 2.2 in their paper how to utilize the graphical lasso algorithm of Friedman, Hastie and Tibshirani (2008), which is essentially a group coordinate descent procedure, to solve problem (2.2) quickly, even when p_n > n. Such a group coordinate decent algorithm was also used by Meier et al. (2008) to solve the group LASSO problem. Thus iteratively, (2.2), and hence (1.1), can be solved quickly with the graphical lasso algorithm. See also Zhang (2007) for a general solution to the folded-concave penalized least-squares problem. The following is a brief summary of the numerical results in Fan et al. (2009).

2.2 Some numerical results

We give a brief summary of a breast cancer data analysis with p_n > n considered in Fan et al. (2009). For full details, please refer to Section 3.2 of Fan et al. (2009). Other simulation results are also in Section 4 in their paper.

Breast cancer data

Normalized gene expression data from 130 patients with stage I-III breast cancers are analyzed, with 33 of them belong to class 1 and 97 belong to class 2. The aim is to assess prediction accuracy in predicting which class a patient will belong to, using a set of pre-selected genes (p_n = 110, chosen by t-tests) as gene expression profile data. The data is randomly divided into training (n = 109) and testing sets. The mean vector for the genes expression levels is obtained from the training data, as well as the associated inverse covariance matrix estimated using LASSO, adaptive LASSO and SCAD penalties as three different regularization methods. A linear discriminant score is then calculated for each regularization method and applied to the testing set to predict if a patient belongs to class 1 or 2. This is repeated 100 times.

On average, the estimated precision matrix $\widehat{\Omega }$ using LASSO has many more nonzeros than that using SCAD (3923 versus 674). This is not surprising when we look at equation (2.3) in our paper, where the L₁ penalty imposes an upper bound on the tuning parameter λ_n1 for consistency, which links to reducing the bias in the estimation. This makes the λ_n1 in practice too small to set many of the elements in $\widehat{\Omega }$ to zero. While we do not know which elements in the true Ω are zero, the large number of nonzero elements in the L₁ penalized estimator seems spurious, and the resulting gene network is not easy to interpret.

On the other hand, SCAD-penalized estimator has a much smaller number of nonzero elements, since the tuning parameter λ_n1 is not bounded above under consistency of the resulting estimator. This makes the resulting gene network easier to interpret, with some clusters of genes identified.

Also, classification results on the testing set using the SCAD penalty for precision matrix estimation is better than that using the L₁ penalty, in the sense that the specificity (#True Negative/#class 2) is higher (0.794 to 0.768) while the sensitivity (#True Positive/#class 1) is similar to that using L₁-penalized precision matrix estimator.

2.3 Technical conditions

We now introduce some notations and present regularity conditions for the rate of convergence and sparsistency.

Let $S_1=\{(i,j):{\omega }_{ij}^0\ne 0\}$, where ${\Omega }_0=\left({\omega }_{ij}^0\right)$ is the true precision matrix. Denote by s_n1 = |S₁| − p_n, which is the number of nonzero elements in the off-diagonal entries of Ω₀. Define

$$a_{n1}=\underset{(i,j)\in S_1}{\max }{p}_{{\lambda }_{n1}}^{\prime }\left(\mid {\omega }_{ij}^0\mid \right),b_{n1}=\underset{(i,j)\in S_1}{\max }{p}_{{\lambda }_{n1}}^{″}\left(\mid {\omega }_{ij}^0\mid \right).$$

The term a_n1 is related to the asymptotic bias of the penalized likelihood estimate due to penalization. Note that for L₁-penalty, a_n1 = λ_n1 and b_n1 = 0, whereas for SCAD, a_n1 = b_n1 = 0 for sufficiently large n under the last assumption of condition (B) below.

We assume the following regularity conditions:

1. There are constants τ₁ and τ₂ such that

   $$0<{\tau }_1<{\lambda }_{\min }\left({\Sigma }_0\right)\le {\lambda }_{\max }\left({\Sigma }_0\right)<{\tau }_2<\infty \text{for all}n.$$

2. a_n1 = O({(1 + p_n/(s_n1 + 1)) log p_n/n}^1/2), b_n1 = o(1), and ${\min }_{(i,j)\in S_1}\mid {\omega }_{ij}^0\mid ∕{\lambda }_{n1}\to \infty$ as n → ∞.

3. The penalty p_λ(·) is singular at the origin, with lim_t↓0 p_λ(t)/(λt) = k > 0.

4. There are constants C and D such that, when ${\theta }_1,{\theta }_2<C{\lambda }_{n1},\mid p_{{\lambda }_{n1}}^{″}\left({\theta }_1\right)-p_{{\lambda }_{n1}}^{″}\left({\theta }_2\right)\mid \le D\mid {\theta }_1-{\theta }_2\mid$.

Condition (A) bounds uniformly the eigenvalues of Σ₀, which facilitates the proof of consistency. It also includes a wide class of covariance matrices as noted in Bickel and Levina (2008b). The rates a_n1 and b_n1 in condition (B) are also needed for proving consistency. If they are too large, the bias due to penalty can dominate the variance from the likelihood, resulting in poor estimates.

The last requirement in condition (B) states the rate at which the nonzero parameters should be distinguished from zero asymptotically. It is not explicitly needed in the proofs, but for asymptotically unbiased penalty functions, this is a necessary condition so that a_n1 and b_n1 are converging to zero fast enough as needed in the first part of condition (B). In particular, for the SCAD and hard-thresholding penalty functions, this condition implies that a_n1 = b_n1 = 0 exactly for sufficiently large n, thus allowing a flexible choice of λ_n1. For the SCAD penalty (1.2), the condition can be relaxed as ${\min }_{(i,j)\in S_1}\mid {\omega }_{ij}^0\mid ∕{\lambda }_{n1}<a$.

The singularity in condition (C) gives sparsity in the estimates [Fan and Li (2001)]. Finally, condition (D) is a smoothing condition for the penalty function, and is needed in proving asymptotic normality. The SCAD penalty, for instance, satisfies this condition by choosing the constant D, independent of n, to be large enough.

2.4 Properties of sparse precision matrix estimation

Minimizing (1.1) involves nonconvex minimization, and we need to prove that there exists a local minimizer $\widehat{\Omega }$ for the minimization problem with a certain rate of convergence, which is given under the Frobenius norm. The proof is given in Section 5. It is similar to the one given in Rothman et al. (2008), but now the penalty function is nonconvex.

Theorem 1 (Rate of convergence). Under regularity conditions (A)-(D), if $(p_n+s_{n1})\log p_n∕n=O\left({\lambda }_{n1}^2\right)$ and (p_n + s_n1)(log p_n)^k/n = O(1) for some k > 1, then there exists a local minimizer $\widehat{\Omega }$ such that ${\parallel \widehat{\Omega }-{\Omega }_0\parallel }_F^2=O_P\{(p_n+s_{n1})\log p_n∕n\}$. For the L₁-penalty, we only need $\log p_n∕n=O\left({\lambda }_{n1}^2\right)$.

The proofs of this theorem and others are relegated to Section 5 so that readers can get more quickly what the results are. As in Fan and Li (2001), the asymptotic bias due to the penalty for each nonzero parameter is a_n1. Since we penalized only on the off-diagonal elements, the total bias induced by the penalty is asymptotically of order s_n1a_n1. The square of this total bias over all nonzero elements is of order O_P{(p_n + s_n1) log p_n/n} under condition (B).

Theorem 1 states explicitly how the number of nonzero elements and dimensionality affect the rate of convergence. Since there are (p_n + s_n1) nonzero elements and each of them can be estimated at best with rate n^−1/2, the total square errors are at least of rate (p_n + s_n1)/n. The price that we pay for high-dimensionality is merely a logarithmic factor log p_n. The results holds as long as (p_n+s_n1)/n is at a rate O((log p_n)^−k) with some k > 1, which decays to zero slowly. This means that in practice p_n can be comparable to n without violating the results. The condition here is not minimum possible; we expect it holds for p ≫ n. Here, we refer the local minimizer as an interior point within a given close set such that it minimizes the target function. Following a similar argument to Huang et al. (2008), the local minimizer in Theorem 1 can be taken as the global minimizer with additional conditions on the tail of the penalty function.

Theorem 1 is also applicable to the L₁-penalty function, where the local minimizer becomes the global minimizer. The asymptotic bias of the L₁-penalized estimate is given in the term s_n1a_n1 = s_n1λ_n1 as shown in the technical proof. In order to control the bias, we impose condition (B), which entails an upper bound on λ_n1 = O({(1+p_n/(s_n1+1)) log p_n/n}^1/2). The bias problem due to the L₁-penalty for finite parameter has already been unveiled by Fan and Li (2001) and Zou (2006).

Next we show the sparsistency of the penalized estimator from (1.1). We use S^c to denote the complement of a set S.

Theorem 2 (Sparsistency). Under the conditions given in Theorem 1, for any local minimizer of (1.1) satisfying ${\parallel \widehat{\Omega }-{\Omega }_0\parallel }_F^2=O_P\{(p_n+s_{n1})\log p_n∕n\}$ and ${\parallel \widehat{\Omega }-{\Omega }_0\parallel }^2=O_P\left({\eta }_n\right)$ for a sequence of ${\eta }_n\to 0$, if $\log p_n∕n+{\eta }_n=O\left({\lambda }_{n1}^2\right)$, then with probability tending to 1, ${\widehat{\omega }}_{ij}=0$ for all $(i,j)\in S_1^c$.

First, since ${\parallel M\parallel }^2\le {\parallel M\parallel }_F^2$ for any matrix M, we can always take η_n = (p_n + s_n1) log p_n/n in Theorem 2, but this will result in more stringent requirement on the number of zero elements when L₁-penalty is used, as we now explain. The sparsistency requires a lower bound on the rate of the regularization parameter λ_n1. On the other hand, condition (B) imposes an upper bound on λ_n1 when L₁-penalty is used in order to control the biases. Explicitly, we need, for L₁-penalized likelihood,

$$\log p_n∕n+{\eta }_n=O\left({\lambda }_{n1}^2\right)=(1+p_n∕(s_{n1}+1))\log p_n∕n$$ (2.3)

for both consistency and sparsistency to be satisfied. We present two scenarios here for the two bounds to be compatible, making use of the inequalities ${\parallel M\parallel }_F^2∕p_n\le {\parallel M\parallel }^2\le {\parallel M\parallel }_F^2$ for a matrix M of size p_n.

1. We always have $\parallel \widehat{\Omega }-{\Omega }_0\parallel \le {\parallel \widehat{\Omega }-{\Omega }_0\parallel }_F$. In the worst case scenario where they have the same order, ${\parallel \widehat{\Omega }-{\Omega }_0\parallel }^2=O_P((p_n+s_{n1})\log p_n∕n)$, so that η_n = (p_n+s_n1) log p_n/n. It is then easy to see from (2.3) that the two bounds are compatible only when s_n1 = O(1).

2. We also have ${\parallel \widehat{\Omega }-{\Omega }_0\parallel }_F^2∕p_n\le {\parallel \widehat{\Omega }-{\Omega }_0\parallel }^2$. In the optimistic scenario where they have the same order,

   $${\parallel \widehat{\Omega }-{\Omega }_0\parallel }^2=O_P((1+s_{n1}∕p_n)\log p_n∕n).$$

   Hence, η_n = (1 + s_n1/p_n) log p_n/n, and compatibility of the bounds requires s_n1 = O(p_n).

Hence, even in the optimistic scenario, consistency and sparsistency are guaranteed only when s_n1 = O(p_n) if the L₁-penalty is used, i.e., the precision matrix has to be sparse enough.

However, if the penalty function used is unbiased, like the SCAD or the hard-thresholding penalty, we do not impose an extra upper bound for λ_n1 since its first derivative $p_{{\lambda }_{n1}}^{\prime }\left(\mid \theta \mid \right)$ goes to zero fast enough as |θ| increases (exactly equals zero for the SCAD and hard-thresholding penalty functions, when n is sufficiently large; see condition (B) and the explanation thereof). Thus, λ_n1 is allowed to decay to zero slowly, allowing even the largest order $s_{n1}=O\left(p_n^2\right)$.

We remark that asymptotic normality for the estimators of the elements in S₁ have been established in a previous version of this paper. We omit it here for brevity.

2.5 Properties of sparse inverse correlation matrix estimation

The inverse correlation matrix Ψ₀ retains the same sparsity structure of Ω₀. Consistency and sparsistency results can be achieved with p_n as large as log p_n = o(n), as long as (s_n1 + 1)(log p_n)^k/n = O(1) for some k > 1 as n → ∞. We minimize, w.r.t. Ψ = (ψ_ij),

$$\mathrm{tr}\left(\Psi {\widehat{\Gamma }}_{\mathrm{S}}\right)-\log \mid \Psi \mid +\sum _{i\ne j}{p}_{{\nu }_{n1}}\left(\mid {\psi }_{ij}\mid \right),$$ (2.4)

where ${\widehat{\Gamma }}_{\mathbf{S}}={\widehat{\mathbf{W}}}^{-1}\mathbf{S}{\widehat{\mathbf{W}}}^{-1}$ is the sample correlation matrix, with ${\widehat{\mathbf{W}}}^2={\mathbf{D}}_{\mathbf{S}}$ being the diagonal matrix with diagonal elements of S, and υ_n1 is a regularization parameter. After obtaining $\widehat{\Psi }$, Ω₀ can also be estimated by $\tilde{\Omega }={\widehat{\mathbf{W}}}^{-1}\widehat{\Psi }{\widehat{\mathbf{W}}}^{-1}$.

To present the rates of convergence for $\widehat{\Psi }$ and $\tilde{\Omega }$, we define

$$c_{n1}=\underset{(i,j)\in S_1}{\max }{p}_{{\nu }_{n1}}^{\prime }\left(\mid {\psi }_{ij}^0\mid \right),d_{n1}=\underset{(i,j)\in S_1}{\max }{p}_{{\nu }_{n1}}^{″}\left(\mid {\psi }_{ij}^0\mid \right),$$

where ${\Psi }_0=\left({\psi }_{ij}^0\right)$ and modify condition (D) to (D’) with λ_n1 there replaced by υ_n1, and impose (B’) c_n1 = O({log p_n/n}^1/2), d_n1 = o(1). Also, ${\min }_{(i,j)\in S_1}\mid {\psi }_{ij}^0\mid ∕{\nu }_{n1}\to \infty$ as n → ∞.

Theorem 3 Under regularity conditions (A),(B’),(C) and (D’), if (s_n1+1)(log p_n)^k/n = O(1) for some k > 1 and $(s_{n1}+1)\log p_n∕n=o\left({\nu }_{n1}^2\right)$, then there exists a local minimizer $\widehat{\Psi }$ for (2.4) such that ${\parallel \widehat{\Psi }-{\Psi }_0\parallel }_F^2=O_P(s_{n1}\log p_n∕n)$ and ${\parallel \tilde{\Omega }-{\Omega }_0\parallel }^2=O_P((s_{n1}+1)\log p_n∕n)$ under the operator norm. For the L₁-penalty, we only need $\log p_n∕n=O\left({\nu }_{n1}^2\right)$

Note that we can allow p_n ≫ n without violating the result as long as log p_n/n = o(1). Note also that an order of {p_n log p_n/n}^1/2 is removed by estimating the inverse correlation rather than the precision matrix, which is somewhat surprising since the inverse correlation matrix, unlike the correlation matrix, does not have known diagonal elements that contribute no errors to the estimation. This can be explained and proved as follows. If s_n1 = O(p_n), the result is obvious. When s_n1 = o(p_n), most of the off-diagonal elements are zero. Indeed, there are at most O(s_n1) columns of the inverse correlation matrix which contain at least one nonzero element. The rest of the columns that have all zero off-diagonal elements must have diagonal entries 1. These columns represent variables that are actually uncorrelated from the rest. Now, it is easy to see from (2.4) that these diagonal elements, which are one, are all estimated exactly as one with no estimation error. Hence, an order of (p_n log p_n/n)^1/2 is not present even in the case of estimating the inverse correlation matrix.

For the L₁-penalty, our result reduces to that given in Rothman et al. (2008). We offer the sparsistency result as follows.

Theorem 4 (Sparsistency) Under the conditions given in Theorem 3, for any local minimizer of (2.4) satisfying ${\parallel \widehat{\Psi }-{\Psi }_0\parallel }_F^2=O_P(s_{n1}\log p_n∕n)$ and ${\parallel \widehat{\Psi }-{\Psi }_0\parallel }^2=O_P\left(n_n\right)$ for some η_n → 0, if $\log p_n∕n+n_n=O\left({\nu }_{n1}^2\right)$, then with probability tending to 1, ${\widehat{\psi }}_{ij}=0$ for all $(i,j)\in S_1^c$.

The proof follows exactly the same as that for Theorem 2 in Section 2.4, and is thus omitted.

For the L₁-penalty, control of bias and sparsistency require υ_n1 to satisfy bounds like (2.3):

$$\log p_n∕n+{\eta }_n=O\left({\nu }_{n1}^2\right)=\log p_n∕n.$$ (2.5)

This leads to two scenarios:

1. The worst case scenario has

   $${\parallel \widehat{\Psi }-{\Psi }_0\parallel }^2={\parallel \widehat{\Psi }-{\Psi }_0\parallel }_F^2=O_P(s_{n1}\log p_n∕n),$$

   meaning η_n = s_n1 log p_n/n. Then compatibility of the bounds in (2.5) requires s_n1 = O(1).
2. The optimistic scenario has

   $${\parallel \widehat{\Psi }-{\Psi }_0\parallel }^2={\parallel \widehat{\Psi }-{\Psi }_0\parallel }_F^2∕p_n=O_P(s_{n1}∕p_n\cdot \log p_n∕n),$$

   meaning η_n = s_n1/p_n · log p_n/n. Then compatibility of the bounds in (2.5) requires s_n1 = O(p_n).

On the other hand, for penalties like the SCAD or the hard-thresholding penalty, we do not need an upper bound for s_n1. Hence, we only need (s_n + 1)(log p_n)^k/n = O(1) as n → ∞ for some k > 1. It is clear that SCAD results in better sampling properties than the L₁-penalized estimator in precision or inverse correlation matrix estimation.

3 Estimation of sparse covariance matrix

In this section, we analyze the sparse covariance matrix estimation using the penalized likelihood (1.3). Then it is modified to estimating the correlation matrix, which improves the rate of convergence. We assume that the y_i’s are i.i.d. N(0, Σ₀) throughout this section.

3.1 Properties of sparse covariance matrix estimation

Let $S_2=\{(i,j):{\sigma }_{ij}^0\ne 0\}$, where ${\Sigma }_0=\left({\sigma }_{ij}^0\right)$. Denote s_n2 = |S₂| − p_n, so that s_n2 is the number of nonzero elements in Σ₀ on the off-diagonal entries. Put

$$a_{n2}=\underset{(i,j)\in S_2}{\max }{p}_{{\lambda }_{n2}}^{\prime }\left(\mid {\sigma }_{ij}^0\mid \right),b_{n2}=\underset{(i,j)\in S_2}{\max }{p}_{{\lambda }_{n2}}^{″}\left(\mid {\sigma }_{ij}^0\mid \right).$$

Technical conditions in Section 2 need some revision. In particular, condition (D) now becomes condition (D2) with λ_n1 there replaced by λ_n2. Condition (B) should now be (B2) a_n2 = O({(1 + p_n/(s_n2 + 1)) log p_n/n}^1/2), b_n2 = o(1), and ${\min }_{(i,j)\in S_2}\mid {\sigma }_{ij}^0\mid ∕{\lambda }_{n2}\to \infty$ as n → ∞.

Theorem 5 (Rate of convergence). Under regularity conditions (A), (B2), (C) and (D2), if (p_n + s_n2)(log p_n)^k/n = O(1) for some k > 1 and (p_n + s_n2) log $p_n∕n=O\left({\lambda }_{n2}^2\right)$, then there exists a local minimizer $\widehat{\Sigma }$ such that ${\parallel \widehat{\Sigma }-{\Sigma }_0\parallel }_F^2=O_p((p_n+s_{n2})\log p_n∕n)$. For the L₁-penalty, we only need $\log p_n∕n=O\left({\lambda }_{n2}^2\right)$.

Like the case for precision matrix estimation, the asymptotic bias due to the L₁-penalty is of order s_n2a_n2 = s_n2λ_n2. To control this term, for the L₁-penalty, we require λ_n2 = O({(1 + p_n/(s_n2 + 1)) log p_n/n}^1/2).

Theorem 6 (Sparsistency). Under the conditions given in Theorem 5, for any local minimizer $\widehat{\Sigma }$ of (1.3) satisfying ${\parallel \widehat{\Sigma }-{\Sigma }_0\parallel }_F^2=O_P((p_n+s_{n2})\log p_n∕n)$ and ${\parallel \widehat{\Sigma }-{\Sigma }_0\parallel }^2=O_P\left(n_n\right)$ for some η_n → 0, if $\log p_n∕n+n_n=O\left({\lambda }_{n2}^2\right)$, then with probability tending to 1, ${\widehat{\sigma }}_{ij}=0$ for all $(i,j)\in S_2^c$.

For the L₁-penalized likelihood, controlling of bias for consistency together with sparsistency requires

$$\log p_n∕n+{\eta }_n=O\left({\lambda }_{n2}^2\right)=(1+p_n∕(s_{n2}+1))\log p_n∕n.$$ (3.1)

This is the same condition as (2.3), and hence in the worst case scenario where

$${\parallel \widehat{\Sigma }-{\Sigma }_0\parallel }^2={\parallel \widehat{\Sigma }-{\Sigma }_0\parallel }_F^2=O_P((p_n+s_{n2})\log p_n∕n),$$

we need s_n2 = O(1). In the optimistic scenario where

$${\parallel \widehat{\Sigma }-{\Sigma }_0\parallel }^2={\parallel \widehat{\Sigma }-{\Sigma }_0\parallel }_F^2∕p_n,$$

we need s_n2 = O(p_n). In both cases, the matrix Σ₀ has to be very sparse, but the former is much sparser.

On the other hand, if unbiased penalty functions like the SCAD or hard-thresholding penalty are used, we do not need an upper bound on λ_n2 since the bias a_n2 = 0 for sufficiently large n. This gives more flexibility on the order of s_n2.

Similar to Section 2, asymptotic normality for the estimators of the elements in S₂ can be proved under certain assumptions.

3.2 Properties of sparse correlation matrix estimation

The correlation matrix Γ₀ retains the same sparsity structure of Σ₀ with known diagonal elements. This special structure allows us to estimate Γ₀ more accurately. To take advantage of the known diagonal elements, the sparse correlation matrix Γ₀ is estimated by minimizing w.r.t. Γ = (γ_ij),

$$\mathrm{tr}\left({\Gamma }^{-1}{\widehat{\Gamma }}_{\mathrm{S}}\right)+\log \mid \Gamma \mid +\sum _{i\ne j}{p}_{{\nu }_{n2}}\left(\mid {\gamma }_{ij}\mid \right),$$ (3.2)

where υ_n2 is a regularization parameter. After obtaining $\widehat{\Gamma }$ Σ₀ can be estimated by $\tilde{\Sigma }=\widehat{\mathrm{W}}\widehat{\Gamma }\widehat{\mathrm{W}}$.

To present the rates of convergence for $\widehat{\Gamma }$ and $\tilde{\Sigma }$, we define

$$c_{n2}=\underset{(i,j)\in S_2}{\max }{p}_{{\nu }_{n2}}^{\prime }\left(\mid {\gamma }_{ij}^0\mid \right),d_{n2}=\underset{(i,j)\in S_2}{\max }{p}_{{\nu }_{n2}}^{″}\left(\mid {\gamma }_{ij}^0\mid \right),$$

where ${\Gamma }_0=\left({\gamma }_{ij}^0\right)$. We modify condition (D) to (D2′) with λ_n2 there replaced by υ_n2, and (B) to (B2′) as follows: (B2′) c_n2 = O({log p_n/n}^1/2), d_n2 = o(1), and ${\min }_{(i,j)\in S_2}\mid {\gamma }_{ij}^0\mid ∕\nu n_2\to \infty$ as n → ∞.

Theorem 7 Under regularity conditions (A),(B2′),(C) and (D2′), if (p_n+s_n2)(log p_n)^k/n = O(1) for some k > 1 and $(s_{n2}+1)\log p_n∕n=o\left({\nu }_{n2}^2\right)$, then there exists a local minimizer $\widehat{\Gamma }$ for (3.2) such that

$${\parallel \widehat{\Gamma }-{\Gamma }_0\parallel }_F^2=O_P(s_{n2}\log p_n∕n).$$

In addition, for the operator norm, we have

$${\parallel \tilde{\Sigma }-{\Sigma }_0\parallel }^2=O_P\{(s_{n2}+1)\log p_n∕n\}.$$

For the L₁-penalty, we only need $\log p_n∕n=O\left({\nu }_{n2}^2\right)$.

The proof is sketched in Section 5. This theorem shows that the correlation matrix, like the inverse correlation matrix, can be estimated more accurately, since diagonal elements are known to be one.

Theorem 8 (Sparsistency). Under the conditions given in Theorem 7, for any local minimizer $\widehat{\Gamma }$ of (3.2) satisfying ${\parallel \widehat{\Gamma }-{\Gamma }_0\parallel }_F^2=O_P(s_{n2}\log p_n∕n)$ and ${\parallel \widehat{\Gamma }-{\Gamma }_0\parallel }^2=O_P\left(n_n\right)$ for some η_n → 0 , if $\log p_n∕n+n_n=O\left({\nu }_{n2}^2\right)$, then with probability tending to 1, ${\widehat{\gamma }}_{ij}=0$ for all $(i,j)\in S_2^c$.

The proof follows exactly the same as that of Theorem 6 in Section 5, and is omitted. For the L₁-penalized likelihood, controlling of bias and sparsistency requires

$$\log p_n∕n+{\eta }_n=O\left({\nu }_{n2}^2\right)=\log p_n∕n.$$ (3.3)

This is the same condition as (2.5), hence in the worst scenario where

$${\parallel \widehat{\Gamma }-{\Gamma }_0\parallel }^2={\parallel \widehat{\Gamma }-{\Gamma }_0\parallel }_F^2=O_P(s_{n2}\log p_n∕n),$$

we need s_n2 = O(1). In the optimistic scenario where

$${\parallel \widehat{\Gamma }-{\Gamma }_0\parallel }^2={\parallel \widehat{\Gamma }-{\Gamma }_0\parallel }_F^2∕p_n=O_P(s_{n2}∕p_n\cdot \log p_n∕n),$$

we need s_n2 = O(p_n).

The use of unbiased penalty functions like the SCAD or the hard-thresholding penalty, similar to results in the previous sections, does not impose an upper bound on the regularization parameter since bias c_n2 = 0 for sufficiently large n. This gives more flexibility to the order of s_n2 allowed.

4 Extension to sparse Cholesky decomposition

Pourahmadi (1999) proposed the modified Cholesky decomposition (MCD) which facilitates the sparse estimation of Ω through penalization. The idea is to represent zero-mean data y = (y₁, ⋯ , y_pn)^T using the autoregressive model:

$$y_i=\sum _{j=1}^{i-1}{\varphi }_{ij}{y}_j+{∊}_i,\text{and}\mathrm{T}\Sigma {\mathrm{T}}^T=\mathrm{D},$$ (4.1)

where T is the unique unit lower triangular matrix with ones on its diagonal and (i, j)^th element being −ϕ_ij for j < i, and D is diagonal with i^th element being ${\sigma }_i^2=\mathrm{var}\left({∊}_i\right)$. The optimization problem is unconstrained (since the ϕ_ij’s are free variables), and the estimate for Ω is always positive-definite.

Huang et al. (2006) and Levina et al. (2008) both used the MCD for estimating Ω₀. The former maximized the log-likelihood (ML) over T and D simultaneously, while the latter suggested also a least square version (LS), with D being first set to the identity matrix and then minimizing over T to obtain $\widehat{\mathbf{T}}$. The latter corresponds to the original Cholesky decomposition. The sparse Cholesky factor can be estimated through minimizing

$$\left(ML\right):q_3(\mathbf{T},\mathbf{D})=\mathrm{tr}\left({\mathbf{T}}^T{\mathbf{D}}^{-1}\mathbf{TS}\right)+\log \mid \mathbf{D}\mid +2\sum _{i<j}{p}_{{\lambda }_{n3}}\left(\mid t_{ij}\mid \right).$$ (4.2)

This is indeed the same as (1.1) with the substitution of Ω = T^TD⁻¹T and penalization parameter λ_n3. Noticing that (4.1) can be written as $\mathbf{Ty}=\epsilon$, the least square version is to minimize $\mathrm{tr}\left(\epsilon {\epsilon }^T\right)=\mathrm{tr}\left({\mathbf{T}}^T{\mathbf{Tyy}}^T\right)$ in the matrix notation. Aggregating the n observations and adding penalty functions, the least-square criterion is to minimize

$$\left(LS\right):q_4\left(\mathbf{T}\right)=\mathrm{tr}\left({\mathbf{T}}^T\mathbf{TS}\right)+2\sum _{i<j}{p}_{{\lambda }_{n4}}\left(\mid t_{ij}\mid \right).$$ (4.3)

In view of the results in Sections 2.5 and 3.2, we can also write the sample covariance matrix in (4.2) as $\mathbf{S}=\widehat{\mathbf{W}}{\widehat{\Gamma }}_{\mathrm{S}}\widehat{\mathbf{W}}$ and then replace ${\mathbf{D}}^{-1∕2}\mathbf{T}\widehat{\mathbf{W}}$ by T, resulting in the normalized (NL) version as follows:

$$\left(NL\right):q_5\left(\mathbf{T}\right)=\mathrm{tr}\left({\mathbf{T}}^T\mathbf{T}{\widehat{\Gamma }}_{\mathrm{S}}\right)-2\log \mid \mathbf{T}\mid +2\sum _{i<j}{p}_{{\lambda }_{n5}}\left(\mid t_{ij}\mid \right).$$ (4.4)

We will also assume the y_i’s are i.i.d. N(0, Σ₀) as in the last section.

4.1 Properties of sparse Cholesky factor estimation

Since all the T’s introduced in the three models above have the same sparsity structure, let S and s_n3 be the nonzero set and number of nonzeros associated with each T above. Define

$$a_{n3}=\underset{(i,j)\in S}{\max }{p}_{{\lambda }_{n3}}^{\prime }\left(\mid t_{ij}^0\mid \right),b_{n3}=\underset{(i,j)\in S}{\max }{p}_{{\lambda }_{n3}}^{″}\left(\mid t_{ij}^0\mid \right).$$

For (ML), condition (D) is adapted to (D3) with λ_n1 there replaced by λ_n3. Condition (B) is modified as (B3) a_n3 = O({(1 + p_n/(s_n3 + 1)) log p_n/n}^1/2), b_n3 = o(1) and ${\min }_{(i,j)\in S}\mid {\varphi }_{ij}^0\mid ∕{\lambda }_{n3}\to \infty$ as n → ∞.

After obtaining $\widehat{\mathbf{T}}$ and $\widehat{\mathbf{D}}$ from minimizing (ML), we set $\widehat{\Omega }={\widehat{\mathbf{T}}}^T{\widehat{\mathbf{D}}}^{-1}\widehat{\mathbf{T}}$.

Theorem 9 Under regularity conditions (A),(B3),(C),(D3), if (p_n + s_n3)(log p_n)^k/n = O(1) for some k > 1 and $(p_n+s_{n3})\log p_n∕n=O\left({\lambda }_{n3}^2\right)$, then there exists a local minimizer $\widehat{\mathbf{T}}$ and $\widehat{\mathbf{D}}$ for (ML) such that ${\parallel \widehat{\mathbf{T}}-{\mathbf{T}}_0\parallel }_F^2=O_P(s_{n3}\log p_n∕n)$, ${\parallel \widehat{\mathbf{D}}-{\mathbf{D}}_0\parallel }_F^2=O_P(p_n\log p_n∕n)$ and ${\parallel \widehat{\Omega }-{\Omega }_0\parallel }_F^2=O_P\{(p_n+s_{n3})\log p_n∕n\}$. For the L₁-penalty, we only need $\log p_n∕n=O\left({\lambda }_{n3}^2\right)$.

The proof is similar to those of Theorems 5 and 7 and is omitted. The Cholesky factor T has ones on its main diagonal without the need for estimation. Hence, the rate of convergence is faster than $\widehat{\Omega }$.

Theorem 10 (Sparsistency). Under the conditions in Theorem 9, for any local minimizer $\widehat{\mathbf{T}}$, $\widehat{\mathbf{D}}$ of (4.2) satisfying ${\parallel \widehat{\mathbf{T}}-{\mathbf{T}}_0\parallel }_F^2=O_P(s_{n3}\log p_n∕n)$ and ${\parallel \widehat{\mathbf{D}}-{\mathbf{D}}_0\parallel }_F^2=O_P(p_n\log p_n∕n)$, if $\log p_n∕n+{\eta }_n+{\zeta }_n=O\left({\lambda }_{n3}^2\right)$, then sparsistency holds for $\widehat{\mathbf{T}}$, provided that ${\parallel \widehat{\mathbf{T}}-{\mathbf{T}}_0\parallel }^2=O_P\left({\eta }_n\right)$ and ${\parallel \widehat{\mathbf{D}}-{\mathbf{D}}_0\parallel }^2=O_P\left({\zeta }_n\right)$, for some η_n, ζ_n → 0.

The proof is in Section 5. For the L₁-penalized likelihood, control of bias and sparsistency impose the following:

$$\log p_n∕n+{\eta }_n+{\zeta }_n=O\left({\lambda }_{n3}^2\right)=(1+p_n∕(s_{n3}+1))\log p_n∕n.$$ (4.5)

The worst scenario corresponds to η_n = s_n3 log p_n/n and ζ_n = p_n log p_n/n, so that we need s_n3 = O(1). The optimistic scenario corresponds to η_n = s_n3/p_n · log p_n/n and ζ_n = log p_n/n, so that we need s_n3 = O(p_n).

On the other hand, such a restriction is not needed for unbiased penalties like the SCAD or hard-thresholding penalty, giving more flexibility on the order of s_n3.

4.2 Properties of sparse normalized Cholesky factor estimation

We now turn to analyzing the normalized penalized likelihood (4.4). With T = (t_ij) in (NL) which is lower triangular, define

$$a_{n5}=\underset{(i,j)\in S}{\max }{p}_{{\lambda }_{n5}}^{\prime }\left(\mid t_{ij}^0\mid \right),b_{n5}=\underset{(i,j)\in S}{\max }{p}_{{\lambda }_{n5}}^{″}\left(\mid t_{ij}^0\mid \right).$$

Condition (D) is now changed to (D5) with λ_n1 there replaced by λ_n5. Condition (B) is now substituted by (B5) $a_{n5}^2=O(\log p_n∕n)$, b_n5 = o(1), ${\min }_{(i,j)\in S}\mid t_{ij}^0\mid ∕{\lambda }_{n5}\to \infty$ as n → ∞.

Theorem 11 (Rate of convergence) Under regularity conditions (A),(B5),(C) and (D5), if s_n3(log p_n)^k/n = O(1) for some k > 1 and $(s_{n3}+1)\log p_n∕n=o\left({\lambda }_{n5}^2\right)$, then there exists a local minimizer $\widehat{\mathbf{T}}$ for (NL) such that ${\parallel \widehat{\mathbf{T}}-{\mathbf{T}}_0\parallel }_F^2=O_P(s_{n3}\log p_n∕n)$ and rate of convergence in the Frobenius norm

$${\parallel \widehat{\Omega }-{\Omega }_0\parallel }_F^2=O_P\{(p_n+s_{n3})\log p_n∕n\},$$

and in the operator norm, it is improved to

$${\parallel \widehat{\Omega }-{\Omega }_0\parallel }^2=O_P\left\{(s_{n3}+1)\log p_n∕n)\right\}.$$

For the L₁-penalty, we only need $\log p_n∕n=O\left({\lambda }_{n5}^2\right)$.

The proof is similar to that of Theorems 5 and 7 and is omitted. In this theorem, like Lemma 3, we can have p_n so that p_n/n goes to a constant less than 1. It is evident that normalizing with $\widehat{\mathbf{W}}$ results in an improvement in the rate of convergence in operator norm.

Theorem 12 (Sparsistency). Under the conditions given in Theorem 11, for any local minimizer $\widehat{\mathbf{T}}$ of (4.4) satisfying ${\parallel \widehat{\mathbf{T}}-{\mathbf{T}}_0\parallel }_F^2=O_P(s_{n3}\log p_n∕n)$ if $\log p_n∕n+{\eta }_n=O\left({\lambda }_{n5}^2\right)$, then sparsistency holds for $\widehat{\mathbf{T}}$, provided that ${\parallel \widehat{\mathbf{T}}-{\mathbf{T}}_0\parallel }^2=O\left({\eta }_n\right)$ for some η_n → 0.

Proof is omitted since it goes exactly the same as that of Theorem 10. The above results apply also to the L₁-penalized estimator. For simultaneous persistency and optimal rate of convergence using the L₁-penalty, the biases inherent in it induce the restriction s_n3 = O(1) in the worst scenario where ${\eta }_n^2=s_{n3}\log p_n∕n$, and s_n3 = O(p_n) in the optimistic scenario where ${\eta }_n^2=s_{n3}∕p_n\cdot \log p_n∕n$. This restriction does not apply to the SCAD and other asymptotically unbiased penalty functions.

5 Proofs

We first prove three lemmas. The first one concerns with inequalities involving the operator and the Frobenius norms. The other two concern with order estimation for elements in a matrix of the form A(S − Σ₀)B, which are useful in proving results concerning sparsistency.

Lemma 1 Let A and B be real matrices such that the product AB is defined. Then, defining ${\parallel \mathbf{A}\parallel }_{\min }^2={\lambda }_{\min }\left({\mathbf{A}}^T\mathbf{A}\right)$, we have

$${\parallel \mathrm{A}\parallel }_{\min }{\parallel \mathrm{B}\parallel }_F\le {\parallel \mathrm{AB}\parallel }_F\le \parallel \mathrm{A}\parallel {\parallel \mathrm{B}\parallel }_F.$$ (5.1)

In particular, if A = (a_ij), then |a_ij| ≤ ∥A∥ for each i, j.

Proof of Lemma 1. Write B = (b₁, ⋯ , b_q), where b_i is the i-th column vector in B. Then

$$\begin{array}{cc}\hfill {\parallel \mathrm{AB}\parallel }_F^2=\mathrm{tr}\left({\mathrm{B}}^T{\mathrm{A}}^T\mathrm{AB}\right)=\sum _{i=1}^q{\mathrm{b}}_i^T{\mathrm{A}}^T{\mathrm{Ab}}_i\le & {\lambda }_{\max }\left({\mathrm{A}}^T\mathrm{A}\right)\sum _{i=1}^q{\parallel {\mathrm{b}}_i\parallel }^2\hfill \\ \hfill =& {\parallel \mathrm{A}\parallel }^2{\parallel \mathrm{B}\parallel }_F^2.\hfill \end{array}$$

Similarly,

$$\begin{array}{cc}\hfill {\parallel \mathrm{AB}\parallel }_F^2=\sum _{i=1}^q{\mathrm{b}}_i^T{\mathrm{A}}^T{\mathrm{Ab}}_i\ge & {\lambda }_{\min }\left({\mathrm{A}}^T\mathrm{A}\right)\sum _{i=1}^q{\parallel {\mathrm{b}}_i\parallel }^2\hfill \\ \hfill =& {\parallel \mathrm{A}\parallel }_{\min }^2{\parallel \mathrm{B}\parallel }_F^2,\hfill \end{array}$$

which completes the proof of (5.1). To prove |a_ij| ≤ ∥A∥, note that $a_{ij}={\mathrm{e}}_i^T\mathbf{A}{\mathrm{e}}_j$, where e_i is the unit column vector with one at the i-th position, and zero elsewhere. Hence, using (5.1),

$$\mid a_{ij}\mid =\mid {\mathrm{e}}_i^T{\mathrm{Ae}}_j\mid \le {\parallel {\mathrm{Ae}}_j\parallel }_F\le \parallel \mathrm{A}\parallel \cdot {\parallel {\mathrm{e}}_j\parallel }_F=\parallel \mathrm{A}\parallel ,$$

and this completes the proof of the lemma. □

Lemma 2 Let S be a sample covariance matrix of a random sample {y_i}_1≤i≤n, with E(y_i) = 0 and var(y_i) = Σ₀. Let y_i = (y_i1, ⋯ , yip_n) with y_ij ~ F_j, where F_j is the c.d.f. of y_ij , and let G_j be the c.d.f. of $y_{ij}^2$, with

$$\underset{1\le i\le p_n}{\max }{\int }_0^{\infty }\exp \left(\lambda t\right)d{G}_j\left(t\right)<\infty ,0<\mid \lambda \mid <{\lambda }_0$$ (5.2)

for some λ₀ > 0. Assume log p_n/n = o(1), and that Σ₀ has eigenvalues uniformly bounded above as n → ∞. Then for constant matrices A and B with ∥A∥, ∥B∥ = O(1), we have max_i,j |(A(S − Σ₀)B)_ij | = O_P({log p_n/n}^1/2).

Remark: The conditions on the y_ij’s above are the same as those used in Bickel and Levina (2008b) for relaxing the normality assumption.

Proof of Lemma 2. Let x_i = Ay_i and ${\mathrm{w}}_i={\mathbf{B}}^T{\mathrm{y}}_i$. Define ${\mathrm{u}}_i={({\mathrm{x}}_i^T,{\mathrm{w}}_i^T)}^T$ , with covariance matrix

$${\Sigma }_{\mathrm{u}}=\mathrm{var}\left({\mathrm{u}}_i\right)=\left(\begin{array}{cc}\hfill \mathrm{A}{\Sigma }_0{\mathrm{A}}^T\hfill & \hfill \mathrm{A}{\Sigma }_0\mathrm{B}\hfill \\ \hfill {\mathrm{B}}^T{\Sigma }_0{\mathrm{A}}^T\hfill & \hfill {\mathrm{B}}^T{\Sigma }_0\mathrm{B}\hfill \end{array}\right).$$

Since ∥(A^T B)^T∥ ≤ (∥A∥² + ∥B∥²)^1/2 = O(1) and ∥Σ₀∥ = O(1) uniformly, we have ∥Σ_u∥ = O(1) uniformly, Then, with ${\mathrm{S}}_{\mathrm{u}}=n^{-1}{\sum }_{i=1}^n{\mathrm{u}}_i{\mathrm{u}}_i^T$, which is the sample covariance matrix for the random sample {u_i}_1≤i≤n, by Lemma A.3 of Bickel and Levina (2008b) which holds under the assumption for the y_ij’s and log p_n/n = o(1), we have

$$\underset{i,j}{\max }\mid {({\mathrm{S}}_{\mathrm{u}}-{\Sigma }_{\mathrm{u}})}_{ij}\mid =O_P\left({\{\log p_n∕n\}}^{1∕2}\right).$$

In particular, it means that

$$\underset{i,j}{\max }\mid {\left(\mathrm{A}(\mathrm{S}-{\Sigma }_0)\mathrm{B}\right)}_{ij}\mid ={\left(n^{-1}\sum _{r=1}^n{\mathrm{x}}_r{\mathrm{w}}_r^T-\mathrm{A}{\Sigma }_0\mathrm{B}\right)}_{ij}=O_P\left({\{\log p_n∕n\}}^{1∕2}\right),$$

which completes the proof of the lemma. □

Lemma 3 Let S be a sample covariance matrix of a random sample y_i1≤i≤n with y_i ~ N(0, Σ₀). Assume p_n/n → y ∈ [0, 1), Σ₀ has eigenvalues uniformly bounded as n → ∞, and A = A₀ + Δ₁, B = B₀ + Δ₂ are such that the constant matrices ∥A₀∥, ∥B₀∥ = O(1), with ∥Δ₁∥, ∥Δ₂∥ = o_P(1). Then we still have max_i,j|(A(S−Σ₀)B)_ij| = O_P({log p_n/n}^1/2).

Proof of Lemma 3. Consider

$$\mathrm{A}(\mathrm{S}-{\Sigma }_0)\mathrm{B}=K_1+K_2+K_3+K_4,$$ (5.3)

where K₁ = A₀(S − Σ₀)B₀, K₂ = Δ₁(S − Σ₀)B₀, K₃ = A₀(S − Σ₀)Δ₂ and K₄ = Δ₁(S − Σ₀)Δ₂. Now max_i,j |(K₁)_ij | = O_P({log p_n/n}^1/2) by Lemma 2. Consider K₂. Suppose the maximum element of the matrix is at the (i, j)-th position. Consider ((S − Σ₀)B₀)_ij, the (i, j)-th element of (S − Σ₀)B₀. Since each element in S − Σ₀ has a rate O_P(n^−1/2), the i-th row of S − Σ₀ has a norm of O_P({p_n/n}^1/2). Also, the j-th column of B₀ has ∥B₀e_j∥ ≤ ∥B₀∥ = O(1). Hence, ((S − Σ₀)B₀)_ij = O_P({p_n/n}^1/2).

Hence, we can find c_n = o({n/p_n}^1/2) such that each element in $c_n{\mathbf{B}}_0^T(\mathrm{S}-{\Sigma }_0)$ has an order larger than that in Δ₁, since ∥Δ₁∥ = o_P(1) implies that each element in Δ₁ is also o_P(1) by Lemma 1.

Then suitable choice of c_n leads to

$$\underset{i,j}{\max }\mid {\left({\Delta }_1(\mathrm{S}-{\Sigma }_0){\mathrm{B}}_0\right)}_{ij}\mid \le c_n\underset{k}{\max }\mid {\left({\mathrm{B}}_0^T{(\mathrm{S}-{\Sigma }_0)}^2{\mathrm{B}}_0\right)}_{kk}\mid .$$ (5.4)

At the same time, Theorem 5.10 in Bai and Silverstein (2006) implies that, for y_i ~ N(0, Σ₀) and p_n/n → y ∈ (0, 1), with probability one,

$$\begin{array}{cc}\hfill -2\sqrt{y}-y\le & \underset{n\to \infty }{\lim \inf }{\lambda }_{\min }({\Sigma }_0^{-1∕2}\mathrm{S}{\Sigma }_0^{-1∕2}-\mathrm{I})\hfill \\ \hfill \le & \underset{n\to \infty }{\lim \sup }{\lambda }_{\max }({\Sigma }_0^{-1∕2}\mathrm{S}{\Sigma }_0^{-1∕2}-\mathrm{I})\le 2\sqrt{y}+y.\hfill \end{array}$$

Hence, if we have p_n/n = o(1), we must have $\parallel {\Sigma }_0^{-1∕2}\mathrm{S}{\Sigma }_0^{-1∕2}-\mathbf{I}\parallel =o_P\left(1\right)$, or it will contradict the above. It means that ∥S−Σ₀∥ = o_P(1) since Σ₀ has eigenvalues uniformly bounded. Or, if p_n/n → y ∈ (0, 1), then we have ∥S − Σ₀∥ = O_P(1) by the above.

Since S − Σ₀ is symmetric, we can find a rotation matrix Q (i.e. Q^TQ = QQ^T = I) so that

$$\mathrm{S}-{\Sigma }_0=\mathrm{Q}\Lambda {\mathrm{Q}}^T,$$

where Λ is a diagonal matrix with real entries. Then we are free to control c_n again so as to satisfy further that c_n∥Λ∥² = o_P(∥Λ∥), since ∥Λ∥ = ∥S − Σ₀∥ = O_P(1) at most. Hence,

$$\begin{array}{cc}\hfill c_n\underset{k}{\max }\mid {\left({\mathrm{B}}_0^T{(\mathrm{S}-{\Sigma }_0)}^2{\mathrm{B}}_0\right)}_{kk}\mid =& \underset{k}{\max }\mid {\left({\mathrm{B}}_0^T\mathrm{Q}{c}_n{\Lambda }^2{\mathrm{Q}}^T{\mathrm{B}}_0\right)}_{kk}\mid \hfill \\ \hfill \le & \underset{k}{\max }\mid {\left({\mathrm{B}}_0^T\mathrm{Q}\Lambda {\mathrm{Q}}^T{\mathrm{B}}_0\right)}_{kk}\mid \hfill \\ \hfill =& \underset{k}{\max }\mid {\left({\mathrm{B}}_0^T(\mathrm{S}-{\Sigma }_0){\mathrm{B}}_0\right)}_{kk}\mid =O_P\left({\{\log p_n∕n\}}^{1∕2}\right),\hfill \end{array}$$

where the last line used the previous proof for constant matrix B₀. Hence, combining this with (5.4), we have max_i,j |(K₂)_ij| = O_P({log p_n/n}^1/2). Similar arguments go for K₃ and K₄. □

Proof of Theorem 1. The main idea of the proof is inspired by Fan and Li (2001) and Rothman et al. (2008). Let U be a symmetric matrix of size p_n, D_U be its diagonal matrix and R_U = U − D_U be its off-diagonal matrix. Set Δ_U = α_nR_U + β_nD_U. We would like to show that, for α_n = (s_n1 log p_n/n)^1/2 and β_n = (p_n log p_n/n)^1/2, and for a set $\mathcal{A}$ defined as $\mathcal{A}=\{U:{\parallel {\Delta }_U\parallel }_F^2=C_1^2{\alpha }_n^2+C_2^2{\beta }_n^2\}$,

$$P\left(\underset{U\in \mathcal{A}}{\inf }{q}_1({\Omega }_0+{\Delta }_U)>q_1\left({\Omega }_0\right)\right)\to 1,$$

for sufficiently large constants C₁ and C₂. This implies that there is a local minimizer in $\{{\Omega }_0+{\Delta }_U:{\parallel {\Delta }_U\parallel }_F^2\le C_1^2{\alpha }_n^2+C_2^2{\beta }_n^2\}$ such that ${\parallel \widehat{\Omega }-{\Omega }_0\parallel }_F=O_P({\alpha }_n+{\beta }_n)$ for sufficiently large n, since Ω₀ + Δ_U is positive definite. This is shown by noting that

$${\lambda }_{\min }({\Omega }_0+{\Delta }_U)\ge {\lambda }_{\min }\left({\Omega }_0\right)+{\lambda }_{\min }\left({\Delta }_U\right)\ge {\lambda }_{\min }\left({\Omega }_0\right)-{\parallel {\Delta }_U\parallel }_F>0,$$

since Ω₀ has eigenvalues uniformly bounded away from 0 and ∞ by condition (A), and ∥Δ_U∥_F = O(α_n + β_n) = o(1).

Consider, for Σ = Σ₀ + Δ_U, the difference

$$q_1\left(\Omega \right)-q_1\left({\Omega }_0\right)=I_1+I_2+I_3,$$

where

$$I_1=\mathrm{tr}\left(\mathrm{S}\Omega \right)-\log \mid \Omega \mid -(\mathrm{tr}\left(\mathrm{S}{\Omega }_0\right)-\log \mid {\Omega }_0\mid ),$$

$$I_2=\sum _{(i,j)\in S_1^c}(p_{{\lambda }_{n1}}\left(\mid {\omega }_{ij}\mid \right)-p_{{\lambda }_{n1}}\left(\mid {\omega }_{ij}^0\mid \right)),$$

$$I_3=\sum _{(i,j)\in S_1,i\ne j}(p_{{\lambda }_{n1}}\left(\mid {\omega }_{ij}\mid \right)-p_{{\lambda }_{n1}}\left(\mid {\omega }_{ij}^0\mid \right)).$$

It is sufficient to show that the difference is positive asymptotically with probability tending to 1. Using Taylor’s expansion with the integral remainder, we have I₁ = K₁+K₂, where

$$\begin{array}{c}{K}_1=\mathrm{tr}\left((\mathrm{S}-{\Sigma }_0){\Delta }_U\right),\hfill \\ K_2=\mathrm{vec}{\left({\Delta }_U\right)}^T\left\{{\int }_0^{1}g(v,{\Omega }_v)(1-v)dv\right\}\mathrm{vec}\left({\Delta }_U\right),\hfill \end{array}$$ (5.5)

with the definitions Ω_v = Ω₀ + vΔ_U, and $g(v,{\Omega }_v)={\Omega }_v^{-1}\otimes {\Omega }_v^{-1}$. Now,

$$\begin{array}{cc}\hfill K_2\ge & {\int }_0^1(1-v)\underset{0\le v\le 1}{\min }{\lambda }_{\min }({\Omega }_v^{-1}\otimes {\Omega }_v^{-1})dv\cdot {\parallel \mathrm{vec}\left({\Delta }_U\right)\parallel }^2\hfill \\ \hfill =& {\parallel \mathrm{vec}\left({\Delta }_U\right)\parallel }^2∕2\cdot \underset{0\le v\le 1}{\min }{\lambda }_{\max }^{-2}\left({\Omega }_v\right)\hfill \\ \hfill \ge & {\parallel \mathrm{vec}\left({\Delta }_U\right)\parallel }^2∕2\cdot {(\parallel {\Omega }_0\parallel +\parallel {\Delta }_U\parallel )}^{-2}\hfill \\ \hfill \ge & (C_1^2{\alpha }_n^2+C_2^2{\beta }_n^2)∕2\cdot ({\tau }_1^{-1}+o\left(1\right))-2,\hfill \end{array}$$

where we used ∥Δ_U∥ ≤ C₁α_n + C₂β_n = O((log p_n)^(1−k)/2) = o(1) by our assumption.

Consider K₁. It is clear that |K₁| ≤ L₁ + L₂, where

$$L_1=\mid \sum _{(i,j)\in S_1}{(\mathrm{S}-{\Sigma }_0)}_{ij}{\left({\Delta }_U\right)}_{ij}\mid ,$$

$$L_2=\mid \sum _{(i,j)\in S_1^c}{(\mathrm{S}-{\Sigma }_0)}_{ij}{\left({\Delta }_U\right)}_{ij}\mid .$$

Using Lemmas 1 and 2, we have

$$\begin{array}{cc}\hfill L_1\le & {(s_{n1}+p_n)}^{1∕2}\underset{i,j}{\max }\mid {(\mathrm{S}-{\Sigma }_0)}_{ij}\mid \cdot {\parallel {\Delta }_U\parallel }_F\hfill \\ \hfill \le & O_P({\alpha }_n+{\beta }_n)\cdot {\parallel {\Delta }_U\parallel }_F\hfill \\ \hfill =& O_P(C_1{\alpha }_n^2+C_2{\beta }_n^2),\hfill \end{array}$$

This is dominated by K₂ when C₁ and C₂ are sufficiently large.

Now, consider I₂ − L₂ for penalties other than L₁. Since ${\parallel {\Delta }_U\parallel }_F^2=C_1^2{\alpha }_n^2+C_2^2{\beta }_n^2$ on $\mathcal{A}$, we have that |ω_ij| = O(C₁α_n + C₂β_n) = o(1) for all $(i,j)\in S_1^c$. Also, note that the condition on λ_n1 ensures that, for $(i,j)\in S_1^c$, |ω_ij| = O(α_n + β_n) = o(λ_n1). Hence, by condition (C), for all $(i,j)\in S_1^c$, we can find a constant k₁ > 0 such that

$$p_{{\lambda }_{n1}}\left(\mid {\omega }_{ij}\mid \right)\ge {\lambda }_{n1}{k}_1\mid {\omega }_{ij}\mid .$$

This implies that

$$I_2=\sum _{(i,j)\in S_1^c}{p}_{{\lambda }_{n1}}\left(\mid {\omega }_{ij}\mid \right)\ge {\lambda }_{n1}{k}_1\sum _{(i,j)\in S_1^c}\mid {\omega }_{ij}\mid .$$

Hence,

$$\begin{array}{cc}\hfill I_2-L_2\ge & \sum _{(i,j)\in S_1^c}\{{\lambda }_{n1}{k}_1\mid {\omega }_{ij}\mid -\mid {(\mathrm{S}-{\Sigma }_0)}_{ij}\mid \cdot \mid {\omega }_{ij}\mid \}\hfill \\ \hfill \ge & \sum _{(i,j)\in S_1^c}[{\lambda }_{n1}{k}_1-O_P\left({\{\log p_n∕n\}}^{1∕2}\right)]\cdot \mid {\omega }_{ij}\mid \hfill \\ \hfill =& {\lambda }_{n1}\sum _{(i,j)\in S_1^c}[k_1-O_P\left({\lambda }_{n1}^{-1}{\{\log p_n∕n\}}^{1∕2}\right)]\cdot \mid {\omega }_{ij}\mid .\hfill \end{array}$$

With the assumption that $(p_n+s_{n1})\log p_n∕n=O\left({\lambda }_{n1}^2\right)$, we see from the above that I₂−L₂ ≥ 0 since $O_P=\left({\lambda }_{n1}^{-1}{\{\log p_n∕n\}}^{1∕2}\right)=o_P\left(1\right)$, using $\log p_n∕n=o((p_n+s_{n1})\log p_n∕n)=o\left({\lambda }_{n1}^2\right)$.

For the L₁-penalty, since we have max_i≠j |S − Σ₀| = O_P((log p_n/n)^1/2) by Lemma 2, we can find a positive W = O_P(1) such that

$$\underset{i\ne j}{\max }\mid \mathrm{S}-{\Sigma }_0\mid =W{(\log p_n∕n)}^{1∕2}.$$

Then we can set λ_n1 = 2W(log p_n/n)^1/2 or one with order greater than (log p_n/n)^1/2, and the above arguments are still valid, so that I₂ − L₂ > 0.

Now, with L₁ dominated by K₂ and I₂ − L₂ ≥ 0, the proof completes if we can show that I₃ is also dominated by K₂, since we have proved that K₂ > 0. Using Taylor’s expansion, we can arrive at

$$\mid I_3\mid \le \min {(C_1,C_2)}^{-1}\cdot O\left(1\right)\cdot (C_1^2{\alpha }_n^2+C_2^2{\beta }_n^2)+o\left(1\right)\cdot (C_1^2{\alpha }_n^2+C_2^2{\beta }_n^2),$$

where o(1) and O(1) are the terms independent of C₁ and C₂. By condition (B), we have

$$\mid I_3\mid =C\cdot O({\alpha }_n^2+{\beta }_n^2)+C^2\cdot o({\alpha }_n^2+{\beta }_n^2),$$

which is dominated by K₂ with large enough constants C₁ and C₂. This completes the proof of the theorem. □

Proof of Theorem 2. For Ω a minimizer of (1.1), the derivative for q₁(Ω) w.r.t. ω_ij for $(i,j)\in S_2^c$ is

$$\frac{\partial q_1\left(\Omega \right)}{\partial {\omega }_{ij}}=2(s_{ij}-{\sigma }_{ij}+p_{{\lambda }_{n1}}^{\prime }\left(\mid {\omega }_{ij}\mid \right)\mathrm{sgn}\left({\omega }_{ij}\right)),$$

where sgn(a) denotes the sign of a. If we can show that the sign of ∂q₁(Ω)/∂ω_ij depends on sgn(ω_ij) only with probability tending to 1, the optimum will be at 0, so that ${\widehat{\omega }}_{ij}=0$ for all $(i,j)\in S_2^c$ with probability tending to 1. We need to estimate the order of s_ij − σ_ij independent of i and j.

Decompose s_ij − σ_ij = I₁ + I₂, where

$$I_1=s_{ij}-{\sigma }_{ij}^0,I_2={\sigma }_{ij}^0-{\sigma }_{ij}.$$

By Lemma 2 or Lemma A.3 of Bickel and Levina (2008b), it follows that max_i,j |I₁| = O_P({log p_n/n}^1/2). It remains to estimate the order of I₂.

By Lemma 1, $\mid {\sigma }_{ij}-{\sigma }_{ij}^0\mid \le \parallel \Sigma -{\Sigma }_0\parallel$, which has order

$$\begin{array}{cc}\hfill \parallel \Sigma -{\Sigma }_0\parallel =& \parallel \Sigma (\Omega -{\Omega }_0){\Sigma }_0\parallel \hfill \\ \hfill \le & \parallel \Sigma \parallel \cdot \parallel \Omega -{\Omega }_0\parallel \cdot \parallel {\Sigma }_0\parallel \hfill \\ \hfill =& O\left(\parallel \Omega -{\Omega }_0\parallel \right),\hfill \end{array}$$

where we used condition (A) to get ∥Σ₀∥ = O(1), and using η_n → 0 so that ${\lambda }_{\min }(\Omega -{\Omega }_0)=o\left(1\right)\mathrm{for}\parallel \Omega -{\Omega }_0\parallel =O\left({\eta }_n^{1∕2}\right)$,

$$\begin{array}{cc}\hfill \parallel \Sigma \parallel ={\lambda }_{\min }^{-1}\left(\Omega \right)\le & {({\lambda }_{\min }\left({\Omega }_0\right)+{\lambda }_{\min }(\Omega -{\Omega }_0))}^{-1}\hfill \\ \hfill =& {(O\left(1\right)+o\left(1\right))}^{-1}=O\left(1\right).\hfill \end{array}$$

Hence, $\parallel \Omega -{\Omega }_0\parallel =O\left({\eta }_n^{1∕2}\right)$ implies $\mid I_2\mid =O\left({\eta }_n^{1∕2}\right)$.

Combining the last two results yields that

$$\begin{array}{cc}\hfill \underset{i,j}{\max }\mid s_{ij}-{\sigma }_{ij}\mid =& O_P(\mid s_{ij}-{\sigma }_{ij}^0\mid +{\eta }_n^{1∕2})\hfill \\ \hfill =& O_P({\{\log p_n∕n\}}^{1∕2}+{\eta }_n^{1∕2}).\hfill \end{array}$$

By conditions (C) and (D), we have

$$p_{{\lambda }_{n1}}^{\prime }\left(\mid {\omega }_{ij}\mid \right)=C_3{\lambda }_{n1}$$

for ω_ij in a small neighborhood of 0 (excluding 0 itself) and some positive constant C₃. Hence, if ω_ij lies in a small neighborhood of 0, we need to have $\log p_n∕n+n_n=O\left({\lambda }_{n1}^2\right)$ in order to have the sign of ∂q₁(Ω)/∂ω_ij depends on sgn(ω_ij) only with probability tending to 1. The proof of the theorem is completed. □

Proof of Theorem 3. Because of the similarity between equations (2.4) and (1.1), the Frobenius norm result has nearly identical proof as Theorem 1, except that we now set Δ_U = α_nU. For the operator norm result, we refer readers to the proof of Theorem 2 of Rothman et al. (2008). □

Proof of Theorem 5. The proof is similar to that of Theorem 1. We only sketch briefly the proof, pointing out the important differences.

Let α_n = (s_n2 log p_n/n)^1/2 and β_n = (p_n log p_n/n)^1/2, and define $\mathcal{A}=\{U:{\parallel {\Delta }_U\parallel }_F^2=C_1^2{\alpha }_n^2+C_2^2{\beta }_n^2\}$. Want to show

$$P\left(\underset{U\in \mathcal{A}}{\inf }{q}_2({\Sigma }_0+{\Delta }_U)>q_2\left({\Sigma }_0\right)\right)\to 1,$$

for sufficiently large constants C₁and C₂.

For Σ = Σ₀ + Δ_U, the difference

$$q_2\left(\Sigma \right)-q_2\left({\Sigma }_0\right)=I_1+I_2+I_3,$$

where

$$I_1=\mathrm{tr}\left(S\Omega \right)+\log \mid \Sigma \mid -(\mathrm{tr}\left(S{\Omega }_0\right)+\log \mid {\Sigma }_0\mid ),$$

$$I_2=\sum _{(i,j)\in S_2^c}(p_{{\lambda }_{n2}}\left(\mid {\sigma }_{ij}\mid \right)-p_{{\lambda }_{n2}}\left(\mid {\sigma }_{ij}^0\mid \right)),$$

$$I_3=\sum _{(i,j)\in S_2,i\ne j}(p_{{\lambda }_{n2}}\left(\mid {\sigma }_{ij}\mid \right)-p_{{\lambda }_{n2}}\left(\mid {\sigma }_{ij}^0\mid \right)),$$

with I₁ = K₁ + K₂, where

$$\begin{array}{c}{K}_1=-\mathrm{tr}\left((\mathrm{S}-{\Sigma }_0){\Omega }_0{\Delta }_U{\Omega }_0\right)=-\mathrm{tr}\left(({\mathrm{S}}_{{\Omega }_0}-{\Omega }_0){\Delta }_U\right),\hfill \\ K_2=\mathrm{vec}{\left({\Delta }_U\right)}^T\left\{{\int }_0^{1}g(v,{\Sigma }_v)(1-v)dv,\right\}\mathrm{vec}\left({\Delta }_U\right),\hfill \end{array}$$ (5.6)

and Σ_v = Σ₀ + vΔ_U, S_Ω0 is the sample covariance matrix of a random sample {x_i}_1≤i≤n having x_i ~ N(0, Ω₀). Also,

$$g(v,{\Sigma }_v)={\Sigma }_v^{-1}\otimes {\Sigma }_v^{-1}\mathrm{S}{\Sigma }_v^{-1}+{\Sigma }_v^{-1}\mathrm{S}{\Sigma }_v^{-1}\otimes {\Sigma }_v^{-1}-{\Sigma }_v^{-1}\otimes {\Sigma }_v^{-1}.$$ (5.7)

The treatment of K₂is different from that in Theorem 1. By condition (A), and (p_n + s_n2)(log p_n)^k/n = O(1) for some k > 1, we have

$$\parallel v{\Delta }_U{\Omega }_0\parallel \le \parallel {\Delta }_U\parallel \parallel {\Omega }_0\parallel \le {\tau }_1^{-1}(C_1{\alpha }_n+C_2{\beta }_n)=O\left({\left(\log p_n\right)}^{1-k}\right)=o\left(1\right).$$

Thus, we can use the Neumann series expansion to arrive at

$${\Sigma }_v^{-1}={\Omega }_0{(I+v{\Delta }_U{\Omega }_0)}^{-1}={\Omega }_0(I-v{\Delta }_U{\Omega }_0+o\left(1\right)),$$

where the little o (or o_P, O or O_P in any matrix expansions in the remainder of this proof) represents a function of the L₂ norm of the residual matrix in the expansion. That is, ${\sum }_{\nu }^{-1}={\Omega }_0+O_p({\alpha }_n+{\beta }_n)$, and $\parallel {\sum }_{\nu }^{-1}\parallel ={\tau }_1^{-1}+O_p({\alpha }_n+{\beta }_n)$. With S_I difined as the sample covariance matrix formed from a random sample {x_i}_1≤i≤n having x_i ~ N(0, I),

$$\parallel \mathrm{S}-{\Sigma }_0\parallel =O_P\left(\parallel {\mathrm{S}}_I-I\parallel \right)=o_P\left(1\right)$$

(see arguments in Lemma 3). These entail

$$\begin{array}{cc}\hfill \mathrm{S}{\Sigma }_v^{-1}=& (\mathrm{S}-{\Sigma }_0){\Sigma }_v^{-1}+{\Sigma }_0{\Sigma }_v^{-1}\hfill \\ \hfill =& o_P\left(1\right)+I+O_P({\alpha }_n+{\beta }_n)\hfill \\ \hfill =& I+o_P\left(1\right).\hfill \end{array}$$

Combining these results, we have

$$g(v,{\Sigma }_v)={\Omega }_0\otimes {\Omega }_0+O_P({\alpha }_n+{\beta }_n).$$

Consequently,

$$\begin{array}{cc}\hfill K_2=& \mathrm{vec}{\left({\Delta }_U\right)}^T\left\{{\int }_0^1{\Omega }_0\otimes {\Omega }_0(1+o_P\left(1\right))(1-v)dv\right\}\mathrm{vec}\left({\Delta }_U\right)\hfill \\ \hfill \ge & {\lambda }_{\min }({\Omega }_0\otimes {\Omega }_0){\parallel \mathrm{vec}\left({\Delta }_U\right)\parallel }^2∕2\cdot (1+o_P\left(1\right))\hfill \\ \hfill =& {\tau }_1^{-2}(C_1^2{\alpha }_n^2+C_2^2{\beta }_n^2)∕2\cdot (1+o_P\left(1\right)).\hfill \end{array}$$

All other terms are dealt with similarly as in the proof of Theorem 1, and hence we omit them. □

Proof of Theorem 6. The proof is similar to that of Theorem 2. We only show the main differences.

It is easy to show

$$\frac{\partial q_2\left(\Sigma \right)}{\partial {\sigma }_{ij}}=2(-{\left(\Omega \mathrm{S}\Omega \right)}_{ij}+{\omega }_{ij}+p_{{\lambda }_n}^{\prime }\left(\mid {\sigma }_{ij}\mid \right)\mathrm{sgn}\left({\sigma }_{ij}\right)).$$

Our aim is to estimate the order of |(−ΩSΩ + Ω)_ij|, finding an upper bound which is independent of both i and j.

Write

$$-\Omega \mathrm{S}\Omega +\Omega =I_1+I_2,$$

where I₁ = −Ω(S − Σ₀)Ω and I₂ = Ω(Σ − Σ₀)Ω. Since

$$\begin{array}{cc}\hfill \parallel \Omega \parallel =& {\lambda }_{\min }^{-1}\left(\Sigma \right)\le {({\lambda }_{\min }\left({\Sigma }_0\right)+{\lambda }_{\min }(\Sigma -{\Sigma }_0))}^{-1}\hfill \\ \hfill =& {\tau }_1^{-1}+o\left(1\right),\hfill \end{array}$$

we have

$$\Omega ={\Omega }_0+(\Omega -{\Omega }_0)={\Omega }_0-\Omega (\Sigma -{\Sigma }_0){\Omega }_0={\Omega }_0+\Delta ,$$

where $\parallel \Delta \parallel \le \parallel \Omega \parallel \cdot \parallel \Sigma -{\Sigma }_0\parallel \cdot \parallel {\Omega }_0\parallel =O\left({\eta }_n^{1∕2}\right)=o\left(1\right)$ by Lemma 1, with ∥Σ−Σ₀∥² = O(η_n). Hence, we can apply Lemma 3 and conclude that max_i,j |(I₁)_ij| = O_P({log p_n/n}^1/2).

For I₂, we have

$$\underset{i,j}{\max }\mid {\left(I_2\right)}_{ij}\mid \le \parallel \Omega \parallel \cdot \parallel \Sigma -{\Sigma }_0\parallel \cdot \parallel \Omega \parallel =O\left(\parallel \Sigma -{\Sigma }_0\parallel \right)=O\left({\eta }_n^{1∕2}\right).$$

Hence, we have

$$\underset{i,j}{\max }\mid {(-\Omega \mathrm{S}\Omega +\Omega )}_{ij}\mid =O({\{\log p_n∕n\}}^{1∕2}+{\eta }_n^{1∕2}).$$

The rest goes similar to the proof of Theorem 2, and is omitted. □

Proof of Theorem 7. The proof is nearly identical to that of Theorem 5, except that we now set Δ_U = α_nU. The fact that ${\left({\widehat{\Gamma }}_{\mathrm{S}}\right)}_{ii}=1={\gamma }_{ii}^0$ has no estimation error eliminates an order (p_n log p_n/n)^1/2 that contributes from estimating $\mathrm{tr}\left(({\widehat{\Gamma }}_{\text{S}}-{\Gamma }_0){\Psi }_0{\Delta }_U{\Psi }_0\right)$ for (3.2). This is why we can estimate a sparse correlation matrix more accurately.

For the operator norm result, we refer readers to the proof of Theorem 2 of Rothman et al. (2008). □

Proof of Theorem 10. For (T,D) a minimizer of (4.2), the derivative for q₃(T,D) w.r.t. t_ij for $(i,j)\in S_3^c$ is

$$\frac{\partial q_3(\mathrm{T},\mathrm{D})}{\partial t_{ij}}=2({\left({\mathrm{ST}}^T{\mathrm{D}}^{-1}\right)}_{ji}+p_{{\lambda }_{n3}}^{\prime }\left(\mid t_{ij}\mid \right)\mathrm{sgn}\left(t_{ij}\right)).$$

Now ST^TD⁻¹ = I₁ + I₂ + I₃ + I₄, where

$$I_1=(\mathrm{S}-{\Sigma }_0){\mathrm{T}}^T{\mathrm{D}}^{-1}{I}_2={\Sigma }_0{(\mathrm{T}-{\mathrm{T}}_0)}^T{\mathrm{D}}^{-1},$$

$$I_3={\Sigma }_0{\mathrm{T}}_0^T({\mathrm{D}}^{-1}-{\mathrm{D}}_0^{-1}),I_4={\Sigma }_0{\mathrm{T}}_0^T{\mathrm{D}}_0^{-1}.$$

By the MCD (4.1), $I_4=T_0^{-1}$. Since i > j for $(i,j)\in S_3^c$, we must have ${\left(T_0^{-1}\right)}_{ji}=0$. Hence, we can ignore I₄.

Since ∥T−T₀∥² = O(η_n) and ∥D−D₀∥² = O(ζ_n) with η_n, ζ_n = o(1), and by condition (A) we can easily show $\parallel {\mathrm{D}}^{-1}-{\mathrm{D}}_0^{-1}\parallel =O\left(\parallel \mathrm{D}-{\mathrm{D}}_0\parallel \right)=O\left({\zeta }_n^{1∕2}\right)$. Then we can apply Lemma 3 to show that max_ij |(I₁)_ij| = (log p_n/n)^1/2.

For I₂, we have ${\text{max}}_{ij}\mid {\left(I_2\right)}_{ij}\mid \le \parallel {\Sigma }_0\parallel \cdot \parallel \mathrm{T}-{\mathrm{T}}_0\parallel \cdot \parallel {\mathrm{D}}^{-1}\parallel =O\left({\eta }_n^{1∕2}\right)$. And finally. ${\text{max}}_{ij}\mid {\left(I_3\right)}_{ij}\mid \le \parallel {\Sigma }_0\parallel \cdot \parallel {\mathrm{T}}_0\parallel \cdot \parallel {\mathrm{D}}^{-1}-{\mathrm{D}}_0^{-1}\parallel =O\left({\zeta }_n^{1∕2}\right)$.

With all these, we have ${\text{max}}_{\left(ij\right)\in S_3^c}{\mid {\left({\mathrm{ST}}^T{\mathrm{D}}^{-1}\right)}_{ij}\mid }^2=\log p_n∕n+{\eta }_n+{\zeta }_n$. The rest of the proof goes like that of Theorem 2 or 6. □