Asymptotically Normal and Efficient Estimation of Covariate-Adjusted Gaussian Graphical Model

Abstract

A tuning-free procedure is proposed to estimate the covariate-adjusted Gaussian graphical model. For each finite subgraph, this estimator is asymptotically normal and efficient. As a consequence, a confidence interval can be obtained for each edge. The procedure enjoys easy implementation and efficient computation through parallel estimation on subgraphs or edges. We further apply the asymptotic normality result to perform support recovery through edge-wise adaptive thresholding. This support recovery procedure is called ANTAC, standing for Asymptotically Normal estimation with Thresholding after Adjusting Covariates. ANTAC outperforms other methodologies in the literature in a range of simulation studies. We apply ANTAC to identify gene-gene interactions using an eQTL dataset. Our result achieves better interpretability and accuracy in comparison with CAMPE.

1. Introduction

Graphical models have been successfully applied to a broad range of studies that investigate the relationships among variables in a complex system. With the advancement of high-throughput technologies, an unprecedented amount of features can be collected for a given system. Therefore, the inference with graphical models has become more challenging. To better understand the complex system, novel methods under high dimensional setting are extremely needed. Among graphical models, Gaussian graphical models have recently received considerable attention for their applications in the analysis of gene expression data. It provides an approach to discover and analyze gene relationships, which offers insights into gene regulatory mechanism. However gene expression data alone are not enough to fully capture the complexity of gene regulation. Genome-wide expression quantitative trait loci (eQTL) studies, which simultaneously measure genetic variation and gene expression levels, reveal that genetic variants account for a large proportion of the variability of gene expression across different individuals (Rockman & Kruglyak 2006). Some genetic variants may confound the genetic network analysis, thus ignoring the influence of them may lead to false discoveries. Adjusting the effect of genetic variants is of importance for the accurate inference of genetic network at the expression level. A few papers in the literature have considered to accommodate covariates in graphical models. See, for example, Li, Chun & Zhao (2012), Yin & Li (2013) and Cai, Li, Liu & Xie (2013) introduced Gaussian graphical model with adjusted covariates, and Cheng, Levina, Wang & Zhu (2012) introduced additional covariates to Ising models.

This problem has been naturally formulated as joint estimation of the multiple regression coefficients and the precision matrix in Gaussian settings. Since it is widely believed that genes operate in biological pathways, the graph for gene expression data is expected to be sparse. Many regularization-based approaches have been proposed in the literature. Some use a joint regularization penalty for both the multiple regression coefficients and the precision matrix and solve iteratively (Obozinski, Wainwright & Jordan 2011; Yin & Li 2011; Peng, Zhu, Bergamaschi, Han, Noh, Pollack & Wang 2010). Others apply a two-stage strategy: estimating the regression coefficients in the first stage and then estimating the precision matrix based on the residuals from the first stage. For all these methods, the thresholding level for support recovery depends on the unknown matrix l₁ norm of the precision matrix or an irrepresentable condition on the Hessian tensor operator, thus those theoretically justified procedures can not be implemented practically. In practice, the thresholding level is often selected through cross-validation. When the dimension p of the precision matrix is relatively high, cross-validation is computationally intensive, with a jeopardy that the selected thresholding level is very different from the optimal one. As we show in the simulation studies presented in Section 5, the thresholding levels selected by the cross-validation tend to be too small, leading to an undesired denser graph estimation in practice. In addition, for current methods in the literature, the thresholding level for support recovery is set to be the same for all entries of the precision matrix, which makes the procedure non-adaptive.

In this paper, we propose a tuning free methodology for the joint estimation of the regression coefficients and the precision matrix. The estimator for each entry of the precision matrix or each partial correlation is asymptotically normal and efficient. Thus a P-value can be obtained for each edge to reflect the statistical significance of each entry. In the gene expression analysis, the P-value can be interpreted as the significance of the regulatory relationships among genes. This method is easy to implement and is attractive in two aspects. First, it has the scalability to handle large datasets. Estimation on each entry is independent and thus can be parallelly computed. As long as the capacity of instrumentation is adequate, those steps can be distributed to accommodate the analysis of high dimensional data. Second, it has the modulability to estimate any subgraph with special interests. For example, biologists may be interested in the interaction of genes play essential roles in certain biological processes. This method allows them to specifically target the estimation on those genes. An R package implementing our method has been developed and is available on the CRAN website.

We apply the asymptotic normality and efficiency result to do support recovery by edge-wise adaptive thresholding. This rate-optimal support recovery procedure is called ANTAC, standing for Asymptotically Normal estimation with Thresholding after Adjusting Covariates. This work is closely connected to a growing literature on optimal estimation of large covariance and precision matrices. Many regularization methods have been proposed and studied. For example, Bickel and Levina (Bickel & Levina 2008a; Bickel & Levina 2008b) proposed banding and thresholding estimators for estimating bandable and sparse covariance matrices respectively and obtained rate of convergence for the two estimators. See also El Karoui (2008) and Lam & Fan (2009). Cai, Zhang & Zhou (2010) established the optimal rates of convergence for estimating bandable covariance matrices. Cai & Zhou (2012) and Cai, Liu & Zhou (2012) obtained the minimax rate of convergence for estimating sparse covariance and precision matrices under a range of losses including the spectral norm loss. Most closely related to this paper is the work in Ren, Sun, Zhang & Zhou (2013) where fundamental limits were given for asymptotically normal and efficient estimation of sparse precision matrices. Due to the complication of the covariates, the analysis in this paper is more involved.

We organize the rest of the paper as follows. Section 2 describes the covariate-adjusted Gaussian graphical model and introduces our novel two-step procedure. Corresponding theoretical studies on asymptotic normal distribution and adaptive support recovery are presented in Sections 3-4. Simulation studies are carried out in Section 5. Section 6 presents the analysis of eQTL data. Proofs for theoretical results are collected in Section 7. We collect a key lemma and auxiliary results for proving the main results in Section 8 and Appendix 9.

2. Covariate-Adjusted Gaussian Graphical Model and Methodology

In this section we first formally introduce the covariate-adjusted Gaussian graphical model, and then propose a two-step procedure for estimation of the model.

2.1 Covariate-adjusted Gaussian Graphical Model

Let (X⁽ⁱ⁾,Y⁽ⁱ⁾), i = 1, …, n, be i.i.d. with

$$Y^{(i)}={\mathrm{\Gamma }}_{p\times q}{X}^{(i)}+Z^{(i)},$$ (1)

where Γ is a p × q unknown coefficient matrix, and Z⁽ⁱ⁾ is a p—dimensional random vector following a multivariate Gaussian distribution N (0, Ω⁻¹) and is independent of X⁽ⁱ⁾. For the genome-wide expression quantitative trait studies, Y⁽ⁱ⁾ is the observed expression levels for p genes of the i–th subject and X⁽ⁱ⁾ is the corresponding values of q genetic markers. We will assume that Ω and Γ_p×q are sparse. The precision matrix Ω is assumed to be sparse partly due to the belief that genes operate in biological pathways, and the sparseness structure of Γ reflects the sensitivity of confounding of genetic variants in the genetic network analysis.

We are particularly interested in the graph structure of random vector Z⁽ⁱ⁾, which represents the genetic networks after removing the effect of genetic markers. Let G = (V, E) be an undirected graph representing the conditional independence relations between the components of a random vector Z⁽¹⁾ = (Z₁₁, …, Z_1p)^T. The vertex set V = {V₁, …, V_p} represents the components of Z. The edge set E consists of pairs (i, j) indicating the conditional dependence between Z_1i and Z_1j given all other components. In the genetic network analysis, the following question is fundamental: Is there an edge between V_i and V_j? It is well known that recovering the structure of an undirected Gaussian graph G = (V, E) is equivalent to recovering the support of the population precision matrix Ω = (ω_ij) of the data in the Gaussian graphical model. There is an edge between V_i and V_j, i.e., (i, j) ∈ E, if and only if ω_ij ≠ 0. See, for example, Lauritzen (1996). Consequently, the support recovery of the precision matrix Ω yields the recovery of the structure of the graph G.

Motivated by biological applications, we consider the high-dimensional case in this paper, allowing the dimension to exceed or even be far greater than the sample size, min {p, q} ≥ n. The main goal of this work is not only to provide a fully data driven and easily implementable procedure to estimate the network for the covariate-adjusted Gaussian graphical model, but also to provide a confidence interval for estimation of each entry of the precision matrix Ω.

2.2 A Two-step Procedure

In this section, we propose a two-step procedure to estimate Ω. In the first step of the two-step procedure, we apply a scaled lasso method to obtain an estimator Γ̂ = (γ̂₁, …, γ̂_p)^T of Γ. This procedure is tuning free. This is different from other procedures in the literature for the sparse linear regression, such as standard lasso and Dantzig selector which select tuning parameters by cross-validation and can be computationally very intensive for high dimensional data. In the second step, we approximate each Z⁽ⁱ⁾ by Ẑ⁽ⁱ⁾ = Y⁽ⁱ⁾ – Γ̂_p×qX⁽ⁱ⁾, then apply the tuning-free methodology proposed in Ren et al. (2013) for the standard Gaussian graphical model to estimate each entry ω_ij of Ω, pretending that each Ẑ⁽ⁱ⁾ was Z⁽ⁱ⁾. As a by-product, we have an estimator Γ̂ of Γ, which is shown to be rate optimal under different matrix norms, however our main goal is not to estimate Γ, but to make inference on Ω.

Step 1 Denote the n by q dimensional explanatory matrix by X = (X⁽¹⁾, …, X⁽ⁿ⁾)^T , where the ith row of matrix is from the i–th sample X⁽ⁱ⁾. Similarly denote the n by p dimensional response matrix by Y = (Y⁽¹⁾, …, Y⁽ⁿ⁾)^T and the noise matrix by Z = (Z⁽¹⁾, …, Z⁽ⁿ⁾)^T . Let Y_j and Z_j be the j–th column of Y and Z respectively For each j = 1, …, p, we apply a scaled lasso penalization to the univariate linear regression of Y_j against X as follows,

$$\mathbf{\text{Step}}\mathbf{1}:\{{\widehat{\gamma }}_j,{\widehat{\sigma }}_{jj}^{1/2}\}=arg\underset{b\in {ℝ}^q,\theta \in {ℝ}^{+}}{min}\{\frac{{\Vert {\mathbf{Y}}_j-\mathbf{X}b\Vert }^2}{2n\theta }+\frac{\theta }{2}+{\lambda }_1\sum _{k=1}^q\frac{\Vert {\mathbf{X}}_k\Vert }{\sqrt{n}}|b_k|\},$$ (2)

where the weighted penalties are chosen to be adaptive to each variance Var (X_1k) such that an explicit value can be given for the parameter λ₁, for example, one of the theoretically justified choices is ${\lambda }_1=\sqrt{2(1+\frac{logp}{logq})/n}$. The scaled lasso (2) is jointly convex in b and θ. The global optimum can be obtained through alternatively updating between b and θ. The computational cost is nearly the same as that of the standard lasso. For more details about its algorithm, please refer to Sun & Zhang (2012).

Define the estimate of “noise” Z_j as the residue of the scaled lasso regression by

$${\widehat{\mathbf{Z}}}_j={\mathbf{Y}}_j-\mathbf{X}{\widehat{\gamma }}_j,$$ (3)

which will be used in the second step to make inference for Ω.

Step 2 In the second step, we propose a tuning-free regression approach to estimate Ω based on Ẑ defined in Equation (3), which is different from other methods proposed in the literature, including Cai et al. (2013) or Yin & Li (2013). An advantage of our approach is the ability to provide an asymptotically normal and efficient estimation of each entry of the precision matrix Ω.

We first introduce some convenient notations for a subvector or a submatrix. For any index subset A of {1, 2,…,p} and a vector W of length p, we use W_A to denote a vector of length |A| with elements indexed by A. Similarly for a matrix U and two index subsets A and B of {1, 2,…,p}, we can define a submatrix U_A,B of size |A| × |B| with rows and columns of U indexed by A and B, respectively Let W = (W₁, …, W_p)^T, representing each Z⁽ⁱ⁾, follow a Gaussian distribution N (0, Ω⁻¹). It is well known that

$$W_A|W_{A^c}=N(-{\mathrm{\Omega }}_{A,A}^{-1}{\mathrm{\Omega }}_{A,A^c}{W}_{A^c},{\mathrm{\Omega }}_{A,A}^{-1}).$$ (4)

For A = {i, j}, equivalently we may write

$$(W_i,W_j)=W_{{\{i,j\}}^c}^T\mathit{\beta }+({\eta }_i,{\eta }_j),$$ (5)

where the coefficients and error distributions are

$$\mathit{\beta }=-{\mathrm{\Omega }}_{A^c,A}{\mathrm{\Omega }}_{A,A}^{-1},{({\eta }_i,{\eta }_j)}^T~N(0,{\mathrm{\Omega }}_{A,A}^{-1}).$$ (6)

Based on the regression interpretation (5), we have the following data version of the multivariate regression model

$${\mathbf{Z}}_A={\mathbf{Z}}_{A^c}\mathit{\beta }+{\mathit{\epsilon }}_A,$$ (7)

where β is a (p − 2) by 2 dimensional coefficient matrix. If we know Z_A and β, an asymptotically normal and efficient estimator of Ω_A,A is ${({\mathit{\epsilon }}_A^T{\mathit{\epsilon }}_A/n)}^{-1}$.

But of course β is unknown and we only have access to the estimated observations Ẑ from Equation (3). We replace Z_A and Z_A^c by Ẑ_A and Ẑ_A^c respectively in the regression (7) to estimate β as follows. For each m ∈ A = {i, j}, we apply a scaled lasso penalization to the univariate linear regression of Ẑ_m against Ẑ_A^c,

$$\mathbf{\text{Step}}\mathbf{2}:\{{\widehat{\mathit{\beta }}}_m,{\widehat{\psi }}_{mm}^{1/2}\}=arg\underset{b\in {ℝ}^{p-2},\sigma \in {ℝ}^{+}}{min}\{\frac{{\Vert {\widehat{\mathbf{Z}}}_m-{\widehat{\mathbf{Z}}}_{A^c}b\Vert }^2}{2n\sigma }+\frac{\sigma }{2}+{\lambda }_2\sum _{k\in A^c}\frac{\Vert {\widehat{\mathbf{Z}}}_k\Vert }{\sqrt{n}}|b_k|\},$$ (8)

where the vector b is indexed by A^c, and one of the theoretically justified choices of λ₂ is ${\lambda }_2=\sqrt{\frac{2logp}{n}}$. Denote the residuals of the scaled lasso regression by

$${\widehat{\mathit{\epsilon }}}_A={\widehat{\mathbf{Z}}}_A-{\widehat{\mathbf{Z}}}_{A^c}\widehat{\mathit{\beta }},$$ (9)

and then define

$${\widehat{\mathrm{\Omega }}}_{A,A}={({\widehat{\mathit{\epsilon }}}_A^T{\widehat{\mathit{\epsilon }}}_A/n)}^{-1}.$$ (10)

This extends the methodology proposed in Ren et al. (2013) for Gaussian graphical model to corrupted observations. The approximation error Ẑ – Z affects inference for Ω. Later we show if Γ is sufficient sparse, Γ_p×qX can be well estimated so that the approximation error is negligible. When both Ω and Γ are sufficiently sparse, Ω̂_A,A can be shown to be asymptotically normal and efficient. An immediate application of the asymptotic normality result is to perform adaptive graphical model selection by explicit entry-wise thresholding, which yields a rate-optimal adaptive estimation of the precision matrix Ω under various matrix l_w norms. See Theorems 2, 3 and 4 in Section 3 and 4 for more details.

3. Asymptotic Normality Distribution of the Estimator

In this section we first give theoretical properties of the estimator Ẑ as well as Γ̂, then present the asymptotic normality and efficiency result for estimation of Ω.

We assume the coefficient matrix Γ is sparse, and entries of X with mean zero are bounded since the gene marker is usually bounded.

1. The coefficient matrix Γ satisfies the following sparsity condition,

   $$\underset{i}{max}{\sum }_{j\ne i}min\{1,\frac{|{\gamma }_{ij}|}{{\lambda }_1}\}=s_1,$$ (11)

   where in this paper λ₁ is at an order of $\sqrt{\frac{logq}{n}}$. Note that s₁ ≤ max_i Σ_j≠iI {γ_ij ≠ 0}, the maximum of the exact row sparseness among all rows of Γ.

2. There exist positive constants M₁ and M₂ such that 1/M₁ ≤ λ_min (Cov(X⁽¹⁾)) and 1/M₂ ≤ λ_min (Ω) ≤ λ_max (Ω) ≤ M₂.

3. There is a constant B > 0 such that

   $$|X_{ij}|\le B\text{for all}i\text{and}j.$$ (12)

It is worthwhile to note that the boundedness assumption (12) does not imply the X⁽¹⁾ is jointly sub-gaussian, i.e., X⁽¹⁾ is allowed to be not jointly sub-gaussian as long as above conditions are satisfied. In the high dimensional regression literature, it is common to assume the joint sub-gaussian condition on the design matrix as follows,

• 3′. We shall assume that the distribution of X⁽¹⁾ is jointly sub-gaussian with parameter (M₁)^1/2 > 0 in the sense that

  $$\mathbb{P}\{|v^T{X}^{(1)}|>t\}\le e^{-t^2/2M_1}\text{for all}t>0\text{and}{\Vert v\Vert }_2=1.$$ (13)

We analyze the Step 1 of the procedure in Equation (2) under Conditions 1-3 as well as Conditions 1-2 and 3′. The optimal rates of convergence are obtained under the matrix l_∞ norm and Frobenius norm for estimation of Γ, which yield a rate of convergence for estimation of each Z_j under the l₂ norm.

Theorem 1

Let ${\lambda }_1=(1+{\epsilon }_1)\sqrt{\frac{2{\delta }_{1}logq}{n}}$ for any δ₁ ≥ 1 and ε₁ > 0 in Equation (2). Assume that

$$s_1=o(min\{\frac{n}{{log}^{3}nlogq},\sqrt{\frac{n}{logq}}\left\}\right).$$

Under Conditions 1-3 we have

$$\mathbb{P}\{\frac{1}{n}{\Vert {\widehat{\mathbf{Z}}}_j-{\mathbf{Z}}_j\Vert }^2>C_1{s}_1\frac{logq}{n}\}\le o(q^{-{\delta }_1+1})\mathit{\text{for each}}j,$$ (14)

$$\mathbb{P}\{{\Vert \widehat{\mathbf{\Gamma }}-\mathbf{\Gamma }\Vert }_{l_{\infty }}>C_2{s}_1\sqrt{\frac{logq}{n}}\}\le o(p\cdot q^{-{\delta }_1+1}),$$ (15)

$$\mathbb{P}\{\frac{1}{p}{\Vert \widehat{\mathbf{\Gamma }}-\mathbf{\Gamma }\Vert }_F^2>C_3{s}_1\frac{logq}{n}\}\le o(p\cdot q^{-{\delta }_1+1}).$$ (16)

Moreover, if we replace Condition 3 by the weaker version Condition 3′, all results above still hold under a weaker assumption on s₁,

$$s_1=o\left(\frac{n}{logq}\right).$$ (17)

The proof of Theorem 1 is provided in the Section 7.1.

Remark 1

Under the assumption that the l_r norm of each row of Γ is bounded by $k_{n,q}^{1/r}$, an immediate application of Theorem 1 yields corresponding results for l_r ball sparseness. For example,

$$\mathbb{P}\{\frac{1}{p}{\Vert \widehat{\mathbf{\Gamma }}-\mathbf{\Gamma }\Vert }_F^2>C_4{k}_{n,q}{\left(\frac{logq}{n}\right)}^{1-r/2}\}\le o(p\cdot q^{-{\delta }_1+1}),$$

provided that $k_{n,q}=o{\left(\frac{n}{logq}\right)}^{1-r/2}$ and Conditions 1-2, 3′ hold.

Remark 2

Cai et al. (2013) assumes that the matrix l₁ norm of (Cov (X⁽¹⁾))⁻¹, the inverse of the covariance matrix of X⁽¹⁾, is bounded, and their tuning parameter depends on the unknown l₁ norm. In Theorem 1 we don't need the assumption on the l₁ norm of (Cov (X⁽¹⁾))⁻¹ and the tuning parameter λ₁ is given explicitly.

To analyze the Step 2 of the procedure in Equation (8), we need the following assumptions for Ω.

• 4. The precision matrix Ω = (ω_ij)_p×q has the following sparsity condition

  $$\underset{i}{max}{\sum }_{j\ne i}min\{1,\frac{|{\omega }_{ij}|}{{\lambda }_2}\}=s_2,$$ (18)

  where λ₂ is at an order of $\sqrt{\frac{logp}{n}}$.

• 5. There exists a positive constant M₂ such that ‖Ω‖_l∞ ≤ M₂.

It is convenient to introduce a notation for the covariance matrix of (η_i, η_j)^T in Equation (5). Let

$${\mathrm{\Psi }}_{A,A}={\mathrm{\Omega }}_{A,A}^{-1}=\left(\begin{array}{cc}{\psi }_{ii}& {\psi }_{ij}\\ {\psi }_{ji}& {\psi }_{jj}\end{array}\right).$$

We will estimate Ψ_A,A first and show that an efficient estimator of Ψ_A,A yields an efficient estimation of entries of Ω_A,A by inverting the estimator of Ψ_A,A. Denote a sample version of Ψ_A,A by

$${\mathrm{\Psi }}_{A,A}^{\mathit{\text{ora}}}={({\psi }_{kl}^{\mathit{\text{ora}}})}_{k\in A,l\in A}={\mathit{\epsilon }}_A^T{\mathit{\epsilon }}_A/n,$$ (19)

which is an oracle MLE of Ψ_A,A, assuming that we know β, and

$${\mathrm{\Omega }}_{A,A}^{\mathit{\text{ora}}}={({\omega }_{kl}^{\mathit{\text{ora}}})}_{k\in A,l\in A}={({\mathrm{\Psi }}_{A,A}^{\mathit{\text{ora}}})}^{-1}.$$ (20)

Let

$${\widehat{\mathrm{\Psi }}}_{A,A}={\widehat{\mathit{\epsilon }}}_A^T{\widehat{\mathit{\epsilon }}}_A/n,$$ (21)

where ε̂_A is defined in Equation (9). Note that Ω̂_A,A defined in Equation (10) is simply the inverse of the estimator Ψ̂_A,A. The following result shows that Ω̂_A,A is asymptotically normal and efficient when both Γ and Ω are sufficient sparse.

Theorem 2

Let λ₁ be defined as in Theorem 1 with ${\delta }_1\ge 1+\frac{logp}{logq}$ and ${\lambda }_2=(1+{\epsilon }_2)\sqrt{\frac{2{\delta }_{2}logp}{n}}$ for any δ₂ ≥ 1 and ε₂ > 0 in Equation (8). Assume that

$$s_1=o\left(\sqrt{\frac{n}{logq}}\right)\mathit{\text{and}}{s}_2=o\left(\sqrt{\frac{n}{logp}}\right).$$ (22)

Under Conditions 1-2 and 4-5, and Condition 3 or 3′, we have

$$\mathbb{P}\{{\Vert {\widehat{\mathrm{\Psi }}}_{A,A}-{\widehat{\mathrm{\Psi }}}_{A,A}^{\mathit{\text{ora}}}\Vert }_{\infty }>C_5(s_2\frac{logp}{n}+s_1\frac{logq}{n}\left)\right\}\le o(p^{-{\delta }_2+1}+p{q}^{-{\delta }_1+1}),$$ (23)

$$\mathbb{P}\{{\Vert {\widehat{\mathrm{\Omega }}}_{A,A}-{\mathrm{\Omega }}_{A,A}^{\mathit{\text{ora}}}\Vert }_{\infty }>C_6(s_2\frac{logp}{n}+s_1\frac{logq}{n}\left)\right\}\le o(p^{-{\delta }_2+1}+p{q}^{-{\delta }_1+1}),$$ (24)

for some positive constants C₅ and C₆. Furthermore, ω̂_ij is asymptotically efficient

$$\sqrt{n{F}_{ij}}({\widehat{\omega }}_{ij}-{\omega }_{ij})\stackrel{D}{\leftrightarrow }N(0,1),$$ (25)

when $s_2=o\left(\frac{\sqrt{n}}{logp}\right)$ and $s_1=o\left(\frac{\sqrt{n}}{logq}\right)$, where

$$F_{ij}^{-1}={\omega }_{ii}{\omega }_{jj}+{\omega }_{ij}^2.$$

Remark 3

The asymptotic normality result can be obtained for estimation of the partial correlation. Let r_ij = −ω_ij/(ω_iiω_jj)^1/2 be the partial correlation between Z_i and Z_j. Define r̂_ij = −ω̂_ij/(ω̂_iiω̂_jj)^1/2. Under the same assumptions in Theorem 2, the estimator r̂_ij is asymptotically efficient, i.e., $\sqrt{n{(1-r_{ij}^2)}^{-2}}({\widehat{r}}_{ij}-r_{ij})\stackrel{D}{\leftrightarrow }N(0,1)$, when $s_2=o(\sqrt{n}/logp)$ and $s_1=o(\sqrt{n}/logq)$.

Remark 4

In Equations (4) and (7), we can replace A = {i, j} by a bounded size subset B ⊂ [1 : p] with cardinality more than 2. Similar to the analysis of Theorem 2, we can show the estimator for any smooth functional of ${\mathrm{\Omega }}_{B,B}^{-1}$ is asymptotic normality as shown in Ren et al. (2013) for Gaussian graphical model.

Remark 5

A stronger result can be obtained for the choice of λ₁ and λ₂. Theorems 1 and 2 sill hold, when ${\lambda }_1=(1+{\epsilon }_1)\sqrt{\frac{2{\delta }_{1}log(q/s_{max,1})}{n}}$ and ${\lambda }_2=(1+{\epsilon }_2)\sqrt{\frac{2{\delta }_{2}log(p/s_{max,2})}{n}}$, where $s_{max,1}=o\left(\frac{\sqrt{n}}{logq}\right)$ and $s_{max,2}=o\left(\frac{\sqrt{n}}{logp}\right)$. Another alternative choice of λ₁ and λ₂ will be introduced in Section 5.

4. Adaptive Support Recovery and Estimation of Ω Under Matrix Norms

In this section, the asymptotic normality result obtained in Theorem 2 is applied to perform adaptive support recovery and to obtain rate-optimal estimation of the precision matrix under various matrix l_ω norms. The two-step procedure for support recovery is first removing the effect of the co-variate X, then applying ANT (Asymptotically Normal estimation with Thresholding) procedure. We thus call it ANTAC, which stands for ANT after Adjusting Covariates.

4.1 ANTAC for Support Recovery of Ω

The support recovery on covariate-adjusted Gaussian graphical model has been studied by several papers, for example, Yin & Li (2013) and Cai et al. (2013). Denote the support of Ω by Supp(Ω). In these literature, the theoretical properties on the support recovery were obtained but they all assumed that ${min}_{(i,j)\in \mathit{\text{Supp}}(\mathrm{\Omega })}|{\omega }_{ij}|\ge C{M}_{n,p}^2\sqrt{\frac{logp}{n}}$, where M_n,p is either the matrix l_∞ norm or related to the irrepresentable condition on Ω, which is unknown. The ANTAC procedure, based on the asymptotic normality estimation in Equation (25), performs entry-wise thresholding adaptively to recover the graph with explicit thresholding levels.

Recall that in Theorem 2 we have

$$\sqrt{n{F}_{ij}}({\widehat{\omega }}_{ij}-{\omega }_{ij})\stackrel{D}{\leftrightarrow }N(0,1),$$

where $F_{ij}={({\omega }_{ii}{\omega }_{jj}+{\omega }_{ij}^2)}^{-1}$ is the Fisher information of estimating ω_ij. Suppose we know this Fisher information, we can apply a thresholding level $\sqrt{\frac{2\xi ({\omega }_{ii}{\omega }_{jj}+{\omega }_{ij}^2)logp}{n}}$ with any ξ ≥ 2 for ω̂_ij to correctly distinguish zero and nonzero entries, noting the total number of edges is p (p − 1) /2. However, when the variance ${\omega }_{ii}{\omega }_{jj}+{\omega }_{ij}^2$ is unknown, all we need is to plug in a consistent estimator. The ANTAC procedure is defined as follows

$${\widehat{\mathrm{\Omega }}}_{\mathit{\text{thr}}}={\left({\widehat{\omega }}_{ij}^{\mathit{\text{thr}}}\right)}_{p\times p},\text{where}{\widehat{\omega }}_{ii}^{\mathit{\text{thr}}}={\widehat{\omega }}_{ii},\text{and}{\widehat{\omega }}_{ij}^{\mathit{\text{thr}}}={\widehat{\omega }}_{ij}1\{|{\widehat{\omega }}_{ij}|\ge {\tau }_{ij}\}$$ (26)

$$\text{with}{\tau }_{ij}=\sqrt{\frac{2{\xi }_0({\widehat{\omega }}_{ii}{\widehat{\omega }}_{jj}+{\widehat{\omega }}_{ij}^2)logp}{n}}\text{for}i\ne j,$$ (27)

where ω̂_kl is the consistent estimator of ω_kl defined in (10) and ξ₀ is a tuning parameter which can be taken as fixed at any ξ₀ > 2.

The following sufficient condition for support recovery is assumed in Theorem 3 below. Define the sign of Ω by (Ω) = {sgn(ω_ij), 1 ≤ i, j ≤ p}. Assume that

$$|{\omega }_{ij}|\ge 2\sqrt{\frac{2{\xi }_0({\omega }_{ii}{\omega }_{jj}+{\omega }_{ij}^2)logp}{n}},\forall {\omega }_{ij}\in \mathit{\text{Supp}}(\mathrm{\Omega }).$$ (28)

The following result shows that not only the support of Ω but also the signs of the nonzero entries can be recovered exactly by Ω̂_thr.

Theorem 3

Assume that Conditions 1-2 and 4-5, and Condition 3 or 3′ hold. Let λ₁ be defined as in Theorem 1 with ${\delta }_1\ge 1+\frac{logp}{logq}$ and ${\lambda }_2=(1+{\epsilon }_2)\sqrt{\frac{2{\delta }_{2}logp}{n}}$ with any δ₂ ≥ 3 and ε₂ > 0 in Equation (8). Also let ξ₀ > 2 in the thresholding level (27). Under the assumptions (22) and (28), we have that the ANTAC defined in (26) recovers the support (Ω) consistently, i.e.,

$$\underset{n\to \infty }{lim}\mathbb{P}(\mathcal{S}({\widehat{\mathrm{\Omega }}}_{\mathit{\text{thr}}})=\mathcal{S}(\mathrm{\Omega }))=1.$$ (29)

Remark 6

If the assumption (28) does not hold, the procedure recovers part of the true graph with high partial correlation.

The proof of Theorem 3 depends on the oracle inequality (24) in Theorem 2, a moderate deviation result of the oracle ω̂_ij and a union bound. The detail of the proof is in spirit the same as that of Theorem 6 in Ren et al. (2013), and thus will be omitted due to the limit of space.

4.2 ANTAC for Estimation under the Matrix l_w Norm

In this section, we consider the rate of convergence of a thresholding estimator of Ω under the matrix l_w norm, including the spectral norm. The convergence under the spectral norm leads to the consistency of eigenvalues and eigenvectors estimation. Define Ω̆_thr, a modification of Ω̂_thr defined in (26), as follows

$${\stackrel{⌣}{\mathrm{\Omega }}}_{\mathit{\text{thr}}}={({\widehat{\omega }}_{ij}^{\mathit{\text{thr}}}1\{|{\widehat{\omega }}_{ij}|\le logp\})}_{p\times p}.$$ (30)

From the idea of the proof of Theorem 3 (see also the proof of Theorem 6 in Ren et al. (2013)), we see that with high probability ‖Ω̂−Ω‖_∞ is dominated by ${\Vert {\mathrm{\Omega }}^{\mathit{\text{ora}}}-\mathrm{\Omega }\Vert }_{\infty }=O_p\left(\sqrt{\frac{logp}{n}}\right)$ under the sparsity assumptions (22). The key of the proof in Theorem 4 is to derive the upper bound under matrix l₁ norm based on the entry-wise supnorm ‖Ω̆_thr− Ω‖_∞. Then the theorem follows immediately from the inequality |M‖_lw ≤ ‖M‖_l1 for any symmetric matrix M and 1 ≤ w ≤ ∞, which can be proved by applying the Riesz-Thorin interpolation theorem. The proof follows similarly from that of Theorem 3 in Cai & Zhou (2012). We omit the proof due to the limit of space.

Theorem 4

Assume that Conditions 1-2 and 4-5, and Condition 3 or 3′ hold. Under the assumptions (22) and n = max {O (p^ξ₁), O (q^ξ₂)} with some ξ₁, ξ₂ > 0, the Ω̆_thr defined in (30) with sufficiently large δ₁ and δ₂ satisfies, for all 1 ≤ w ≤ ∞,

$$\mathbb{E}{\Vert {\stackrel{⌣}{\mathrm{\Omega }}}_{\mathit{\text{thr}}}-\mathrm{\Omega }\Vert }_{l_w}^2\le C{s}_2^2\frac{logp}{n}.$$ (31)

Remark 7

The rate of convergence result in Theorem 4 also can be easily extended to the parameter space in which each row of Ω is in a l_r ball of radius $k_{n,q}^{1/r}$. See, e.g., Theorem 3 in Cai & Zhou (2012). Under the same assumptions of Theorem 4 except replacing $s_2=o(\sqrt{n/logp})$ by $k_{n,p}^2=o({(n/logp)}^{1-r})$, we have

$$\mathbb{E}{\Vert {\stackrel{⌣}{\mathrm{\Omega }}}_{\mathit{\text{thr}}}-\mathrm{\Omega }\Vert }_{l_w}^2\le C{k}_{n,p}^2{\left(\frac{logp}{n}\right)}^{1-r}.$$ (32)

Remark 8

For the Gaussian graphical model without covariate variables, Cai et al. (2012) showed the rates obtained in Equations (31) and (32) are optimal when p ≥ cn^γ for some γ > 1 and k_n,p = o (n^1/2 (logp)^−3/2) for the corresponding parameter spaces of Ω. This implies that our estimator is rate optimal.

5. Simulation Studies

5.1 Asymptotic Normal Estimation

In this section, we compare the sample distribution of the proposed estimator for each edge ω_ij with the normal distribution in Equation (25). Three models are considered with corresponding {p, q, n} listed in Table 1. Based on 200 replicates, the distributions of the estimators match the asymptotic distributions very well.

Table 1.

Model parameters and simulation results: mean and standard deviation (in parentheses) of the proposed estimator for the randomly selected entry with value v_ω based on 200 replicates.

(p, q, n)	π = P(ωij ≠ 0)	vω = 0	vω = 0.3	vω = 0.6	vω = 1
(200, 100, 400)	0.025	-0.015 (0.168)	0.289 (0.184)	0.574 (0.165)	0.986 (0.182)
(400, 100, 400)	0.010	-0.003 (0.24)	0.268 (0.23)	0.606 (0.23)	0.954 (0.244)
(1000, 100, 400)	0.005	0.011 (0.21)	0.292 (0.26)	0.507 (0.232)	0.862 (0.236)

Three sparse models are generated in a similar way to those in Cai et al. (2013). The p × q coefficient matrix Γ is generated as following for all three models,

$${\mathrm{\Gamma }}_{ij}\stackrel{i.i.d.}{~}N(0,1)\cdot \text{Bernoulli}(0.025),$$

where the Bernoulli random variable is independent with the standard normal variable, taking one with probability 0.025 and zero otherwise. We then generate the p × p precision matrix Ω with identical diagonal entries ω_ii = 4 for the model of p = 200 or 400 and ω_ii = 5 for the model of p = 1000, respectively. The off-diagonal entries of Ω are generated i.i.d. as follows for each model,

$${\omega }_{ij}=\{\begin{array}{cc}0.3& \text{with probability}\frac{\pi }{3}\\ 0.6& \text{with probability}\frac{\pi }{3}\\ 1& \text{with probability}\frac{\pi }{3}\\ 0& \text{otherwise}\end{array},\text{for}i\ne j$$

where the probability of being nonzero π = P(ω_ij ≠ 0) for three models is shown in Table 1. Once both Γ and Ω are chosen for each model, the n × p outcome matrix Y is simulated from Y = XΓ^T + Z where rows of Z are i.i.d. N(0, Ω⁻¹) and rows of X are i.i.d. N(0, I_q×q). We generate 200 replicates of X and Y for each model.

We randomly select four entries of Ω with values v_ω of 0, 0.3, 0.6 and 1 in each model and draw histograms of our estimators for those four entries based on the 200 replicates. The penalty parameter λ₁, which controls the weight of penalty in the regression of the first step (2), is set to be $B_1/\sqrt{n-1+B_1^2}$, where $B_1=q_t(1-\frac{1}{2}{(s_{max,1}/q)}^{1+\frac{logp}{logq}},n-1)$, and q_t(·, n) is the quantile function of t distribution with degrees of freedom n. This parameter λ₁ is a finite sample version of the asymptotic level $\sqrt{2(1+\frac{logp}{logq})log(q/s_{max,1})/n}$ we proposed in Theorem 2 and Remark 5. Here we pick s_{max, 1} = √n/ log q. The penalty parameter λ₂, which controls the weight of penalty in the second step (8), is set to be $B_2/\sqrt{n-1+B_2^2}$ where B₂ = q_t(1 − s_max,2/ (2p), n − 1), which is asymptotically equivalent to $\sqrt{2log(p/s_{max,2})/n}$. The s_max,2 is set to be √n/ log p.

In Figure 1, we show the histograms of the estimators with the theoretical normal density super-imposed for those randomly selected four entries with values v_ω of 0, 0.3, 0.6 and 1 in each of the three models. The distributions of our estimators match well with the theoretical normal distributions.

Figure 1.

The histograms of the estimators for randomly selected entries with values v_ω = 0, 0.3,0.6 and 1 in three models listed in Table 1. The theoretical normal density curves are shown as solid curves. The variance for each curve is $({\omega }_{ii}{\omega }_{jj}+{\omega }_{ij}^2)/n$, the inverse of the Fisher information.

5.2 Support recovery

In this section, we evaluate the performance of the proposed ANTAC method and competing methods in support recovery with different simulation settings. ANTAC always performs among the best under all model settings. Under the Heterogeneous Model setting, the ANTAC achieves superior precision and recall rates and performs significantly better than others. Besides, ANTAC is computationally more efficient compared to a state-of-art method CAPME due to its tuning free property.

Homogeneous Model

We consider three models with corresponding {p, q, n} listed in Table 2, which are similar to the models listed in Table 2 and used in (Cai et al. 2013). Since every model has identical values along the diagonal, we call them “Homogeneous Model”. In terms of the support recovery, ANTAC performs among the best in all three models, although the performance from all procedures is not satisfactory due to the intrinsic difficulty of support recovery problem for models considered.

Table 2.

Model parameters used in the simulation of support recovery.

	(p, q, n)	P(Γij ≠ 0)	π = P(ωij ≠ 0), i ≠ j
Model 1	(200, 200, 200)	0.025	0.025
Model 2	(200, 100, 300)	0.025	0.025
Model 3	(800, 200, 200)	0.025	0.010

We generate the p × q coefficient matrix Γ in the same way as Section 5.1,

$${\mathrm{\Gamma }}_{ij}\stackrel{i.i.d.}{~}N(0,1)\cdot \text{Bernoulli}(0.025).$$

The off-diagonal entries of the p × p precision matrix Ω are generated as follows,

$${\omega }_{ij}\stackrel{i.i.d.}{~}N(0,1)\cdot \text{Bernoulli}(\pi ),$$

where the probability of being nonzero π = P(ω_ij ≠ 0) is shown in Table 2 for three models respectively. We generate 50 replicates of X and Y for each of the three models.

We compare our method with graphical Lasso (GLASSO) (Friedman, Hastie & Tibshirani 2008), a state-of-art method — CAPME (Cai et al. 2013) and a conditional GLASSO procedure (short as cGLASSO), where we apply the same scaled lasso procedure as the first stage of the proposed method and then estimate the precision matrix by GLASSO. This cGLASSO procedure is similar to that considered in Yin & Li (2013) except that in the first stage Yin & Li (2013) applies ordinary lasso rather than the scaled lasso, which requires another cross-validation for this step. For GLASSO, the precision matrix is estimated directly from the sample covariance matrix without taking into account the effects from X. The tuning parameter for the l₁ penalty is selected using five-fold cross validation by maximizing the log-likelihood function. For CAPME, the tuning parameters λ₁ and λ₂, which control the penalty in the two stages of regression, are chosen using five-fold cross validation by maximizing the log-likelihood function. The optimum is achieved via a grid search on {(λ₁, λ₂)}. For Models 1 and 2, 10 × 10 grid is used and for Model 3, 5 × 5 grid is used because of the computational burden. Specifically, we use the CAPME package implemented by the authors of (Cai et al. 2013). For Model 3, each run with 5 × 5 grid search and five-fold cross validation takes 160 CPU hours using one core from PowerEdge M600 nodes 2.33 GHz and 16 – 48 GB RAM, whereas ANTAC takes 46 CPU hours. For ANTAC, the parameter λ₁ is set to be $B_1/\sqrt{n-1+B_1^2}$, where $B_1=q_t(1-\frac{1}{2}{(s_{max,1}/q)}^{1+\frac{logp}{logq}},n-1)$, q_t(·, n) is the quantile function of t distribution with degrees of freedom n and s_{max, 1} = √n/ log q. The parameter λ₂, is set to be $B_2/\sqrt{n-1+B_2^2}$ where B₂ = q_t(1 − (s_max,2/p)³ /2, n − 1). For cGLASSO, the first step is the same as ANTAC. In the second step, the precision matrix is estimated by applying GLASSO to the estimated Z, where the tuning parameter is selected using five-fold cross validation by maximizing the log-likelihood function.

We evaluate the performance of the estimators for support recovery problem in terms of the misspecification rate, specificity, sensitivity, precision and Matthews correlation coefficient, which are defined as,

$$\text{MISR}(\widehat{\mathrm{\Omega }},\mathrm{\Omega })=\frac{\text{FN}+\text{FP}}{p(p-1)},\text{SPE}=\frac{\text{TN}}{\text{TN}+\text{FP}},\text{SEN}=\frac{\text{TP}}{\text{TP}+\text{FN}},\text{PRE}=\frac{\text{TP}}{\text{TP}+\text{FP}},\text{MCC}=\frac{\text{TP}\times \text{TN}-\text{FP}\times \text{FN}}{{[(\text{TP}+\text{FP})(\text{TP}+\text{FN})(\text{TN}+\text{FP})(\text{TN}+\text{FN})]}^{1/2}}.$$

Here, TP, TN, FP, FN are the numbers of true positives, true negatives, false positives and false negatives respectively. True positives are defined as the correctly identified nonzero entries of the off-diagonal entries of Ω. For GLASSO and CAPME, nonzero entries of Ω̂ are selected as edges with no extra thresholding applied. For ANTAC, edges are selected by the theoretical bound with ξ₀ = 2. The results are summarized in Table 3. It can be seen that ANTAC achieves superior specificity and precision. Besides, ANTAC has the best overall performance in terms of the Matthews correlation coefficient.

Table 3.

Simulation results of the support recovery for homogeneous models based on 50 replications. Specifically, the performance is measured by misspecification rate, specificity, sensitivity (recall rate), precision and the Matthews correlation coefficient with all the values multiplied by 100. Numbers in parentheses are the simulation standard deviations.

	(p, q, n)	Method	MISR	SPE	SEN	PRE	MCC
Model 1	(200, 200, 200)	GLASSO	35(1)	65(1)	37(2)	2(0)	1(1)
		cGLASSO	25(6)	76(6)	64(7)	6(1)	8(1)
		CAPME	2(0)	100(0)	4(1)	96(1)	21(1)
		ANTAC	2(0)	100(0)	4(0)	88(8)	18(2)
Model 2	(200, 100, 300)	GLASSO	43(0)	57(0)	51(2)	3(0)	3(1)
		cGLASSO	5(0)	97(0)	47(1)	25(1)	32(1)
		CAPME	4(0)	97(0)	56(1)	29(1)	39(1)
		ANTAC	2(0)	100(0)	22(1)	97(2)	46(1)
Model 3	(800, 200, 200)	GLASSO	19(1)	81(1)	19(1)	1(0)	0(0)
		cGLASSO	1(0)	100(0)	0(0)	100(0)	2(0)
		CAPME	1(0)	100(0)	0(0)	0(0)	0(0)
		ANTAC	1(0)	100(0)	7(0)	71(2)	22(1)

We further construct ROC curves to check how this result would vary by changing the tuning parameters. For GLASSO or cGLASSO, the ROC curve is obtained by varying the tuning parameter. For CAPME, λ₁ is fixed as the value selected by the cross validation and the ROC curve is obtained by varying λ₂. For proposed ANTAC method, the ROC curve is obtained by varying the thresholding level ξ₀. When p is small, CAPME, cGLASSO and ANTAC have comparable performance. As p grows, both ANTAC and cGLASSO outperform CAPME.

The purpose of simulating “Homogeneous Model” is to compare the performance of ANTAC and other procedures under models with similar settings used in Cai et al. (2013). Overall the performance from all procedures is not satisfactory due to the difficulty of support recovery problem. All nonzero entries are sampled from a standard normal. Hence, most signals are very weak and hard to be recovered by any method, although ANTAC performs among the best in all three models.

Heterogeneous Model

We consider some models where the diagonal entries of the precision matrix have different values. These models are different from “Homogeneous Model” and we call them “Heterogeneous Model”. The performance of ANTAC and other procedures are explored under “Magnified Block” model and “Heterogeneous Product” model, respectively. The ANTAC performs significantly better than GLASSO, cGLASSO and CAMPE in both settings.

In “Magnified Block” model, we apply the following randomized procedure to choose Ω and Γ. We first simulate a 50 × 50 matrix Ω_B with diagonal entries being 1 and each non-diagonal entry i.i.d. being nonzero with P(ω_ij ≠ 0) = 0.02. If ω_ij ≠ 0, we sample ω_ij from {0.4, 0.5}. Then we generate two matrices by multiplying Ω_B by 5 and 10, respectively. Then we align three matrices along the diagonal, resulting in a block diagonal matrix Ω with sequentially magnified signals. A visualization of the simulated precision matrix is shown in Figure 3. The 150 × 100 matrix Γ is simulated with each entry being nonzero i.i.d. follows N(0,1) with P(Γ_ij ≠ 0) = 0.05. Once the matrices Ω and Γ are chosen, 50 replicates of X and Y are generated.

Figure 3.

Heatmap of support recovery using different methods for a “Magnified Block” model.

In “Heterogeneous Product” model, the matrices Ω and Γ are chosen in the following randomized way. We first simulate a 200 × 200 matrix Ω with diagonal entries being 1 and each non-diagonal entry i.i.d. being nonzero with P(ω_ij ≠ 0) = 0.005. If ω_ij ≠ 0, we sample ω_ij from {0.4, 0.5}. Then we replace the 100 × 100 submatrix at bottom-right corner by multiplying the 100 × 100 submatrix at up-left corner by 2, which results in a precision matrix with possibly many different product values ω_iiσ_jj over all i,j pairs, where σ_jj is the jth diagonal entry of the covariance matrix Σ = (σ_kl)_{p × p} = Ω⁻¹. Thus we call it “Heterogeneous Product” model. A visualization of the simulated precision matrix is shown in Figure 4. The 200 × 100 matrix Γ is simulated with each entry being nonzero i.i.d. follows N(0,1) with P(Γ_ij ≠ 0) = 0.05. Once Ω and Γ are chosen, 50 replicates of X and Y are generated.

Figure 4.

Heatmap of support recovery using different methods for a “Heterogeneous Product” model.

We compare our method with GLASSO, CAPME and cGLASSO procedures in “Heterogeneous Model”. We first compare the performance of support recovery when a single procedure from each method is applied. The tuning parameters for each procedure are set in the same way as in “Homogeneous Model” except that for CAPME, the optimal tuning parameter is achieved via a 10 × 10 grid search on {(λ₁, λ₂)} by five-fold cross validation. We summarize the support recovery results in Table 4. A visualization of the support recovery result for a replicate of “Magnified Block” model and a replicate of “Heterogeneous Product” model are shown in Figure 3 and 4 respectively. In both models, ANTAC significantly outperforms others and achieves high precision and recall rate. Specifically, ANTAC has precision of 0.99 for two models respectively while no other procedure achieves precision rate higher than 0.21 in either model. Besides, ANTAC returns true sparse graph structure while others report much denser results.

Table 4.

Simulation results of the support recovery for heterogeneous models based on 50 replications. The performance is measured by overall error rate, specificity, sensitivity (recall rate), precision and the Matthews correlation coefficient with all the values multiplied by 100. Numbers in parentheses are the simulation standard deviations.

	(p, q, n)	Method	MISR	SPE	SEN	PRE	MCC
Magnified Block	(150, 100, 300)	GLASSO	54(0)	46(0)	80(4)	1(0)	4(1)
		cGLASSO	14(0)	86(0)	99(1)	4(0)	19(0)
		CAPME	1(0)	99(0)	99(0)	24(2)	58(1)
		ANTAC	0(0)	100(0)	98(1)	99(1)	99(1)
Heterogeneous Product	(200, 100, 300)	GLASSO	42(0)	58(0)	85(3)	1(0)	6(0)
		cGLASSO	12(0)	88(0)	100(0)	4(0)	18(0)
		CAPME	4(0)	96(0)	97(0)	15(0)	33(0)
		ANTAC	0(0)	100(0)	80(3)	99(1)	89(2)

Moreover, we construct the precision-recall curve to compare a sequence of procedures from different methods. In terms of precision-recall curve, CAPME has closer performance as the proposed method in “Magnified Block” model whereas cGLASSO has closer performance as the proposed method in “Heterogeneous Product” model, which indicates the proposed method performs comparable to the better of CAPME and cGLASSO. Another implication indicates from the precision-recall curve is that while the tuning free ANTAC method is always close to the best point along the curve created by using different values of threshold ξ₀, CAPME and cGLASSO via cross validation cannot select the optimal parameters on their corresponding precision-recall curves, even though one of them has potentially good performance when using appropriate tuning parameters. Here is an explanation why the ANTAC procedure is better than CAPME and cGLASSO in “Magnified Block” model and “Heterogeneous Product” model settings. Recall that in the second stage, CAPME applies the same penalty level λ for each entry of the difference ΩΣ̂ − I, where Σ̂ denotes the sample covariance matrix, but the i,j entry has variance ω_iiσ_jj after scaling. Thus CAPME may not recover the support well in the “Heterogeneous Product” model settings, where the variances of different entries may be very different. As for the cGLASSO, we notice that essentially the same level of penalty is put on each entry ω_ij while the variance of each entry in the ith row Ω_i. depends on ω_ii. Hence we cannot expect cGLASSO performs very well in the “Magnified Block” model settings, where the diagonals ω_ii vary a lot. In contrast, the ANTAC method adaptively puts the right penalty level (asymptotic variance) for each estimate of ω_ij, therefore it works well in either setting.

Overall, the simulation results on heterogeneous models reveal the appealing practical properties of the ANTAC procedure. Our procedure enjoys tuning free property and has superior performance. In contrast, it achieves better precision and recall rate than the results from CAPME and cGLASSO using cross validation. Although in terms of precision-recall curve, the better of CAPME and cGLASSO is comparable with our procedure, generally the optimal sensitivity and specificity could not be obtained through cross-validation.

6. Application to an EQTL Study

We apply the ANTAC procedure to a yeast dataset from Smith & Kruglyak (2008) (GEO accession number GSE9376), which consists of 5,493 gene expression probes and 2,956 genotyped markers measured in 109 segregants derived from a cross between BY and RM treated with glucose. We find the proposed method achieves both better interpretability and accuracy in this example.

There are many mechanisms leading to the dependency of genes at the expression level. Among those, the dependency between transcription factors (TFs) and their regulated genes has been intensively investigated. Thus the gene-TF binding information could be utilized as an external biological evidence to validate and interpret the estimation results. Specifically, we used the high-confidence TF binding site (TFBS) profiles from m:Explorer, a database recently compiled using perturbation microarray data, TF-DNA binding profiles and nucleosome positioning measurements (Reimand, Aun, Vilo, Vaquerizas, Sedman & Luscombe 2012).

We first focus our analysis on a medium size dataset that consists of 121 genes on the yeast cell cycle signaling pathway (from the Kyoto Encyclopedia of Genes and Genomes database (Kanehisa & Goto 2000)). There are 119 markers marginally associated with at least 3 of those 121 genes with a Bonferroni corrected P-value less than 0.01. The parameters λ₁ and λ₂ for the ANTAC method are set as described in Theorem 3. 55 edges are identified using a cutoff of 0.01 on the FDR controlled P-values and 200 edges with a cutoff of 0.05. The number of edges goes to 375 using a cutoff of 0.1. For the purpose of visualization and interpretation, we further focus on 55 edges resulted from the cutoff of 0.01.

We then check how many of these edges involve at least one TF and how many TF-gene pairs are documented in the m:Explorer database. In 55 detected edges, 12 edges involve at least one TF and 2 edges are documented. In addition, we obtain the estimation of precision matrix from CAPME, where the tuning parameters λ₁ = 0.356 and λ₂ = 0.5 are chosen by five-fold cross validation. To compare with the results from ANTAC, we select top 55 edges from CAPME solution based on the magnitude of partial correlation. Within these 55 edges, 13 edges involve at least one TF and 2 edges are documented. As shown in Figure 6, 22 edges are detected by both methods. Our method identifies a promising cell cycle related subnetwork featured by CDC14, PDS1, ESP1 and DUN1, connecting through GIN4, CLB3 and MPS1. In the budding yeast, CDC14 is a phosphatase functions essentially in late mitosis. It enables cells to exit mitosis through dephosphorylation and activation of the enemies of CDKs (Wurzenberger & Gerlich 2011). Throughout G1, S/G2 and early mitosis, CDC14 is inactive. The inactivation is partially achieved by PDS1 via its inhibition on an activator ESP1 (Stegmeier, Visintin, Amon et al. 2002). Moreover, DUN1 is required for the nucleolar localization of CDC14 in DNA damage-arrested yeast cells (Liang & Wang 2007).

Figure 6.

Visualization of the network constructed from yeast cell cycle expression data by CAPME and the proposed ANTAC method. For ANTAC, 55 edges are identified using a cutoff of 0.01 on the FDR controlled p-values. For CAPME, top 55 edges are selected based on the magnitude of partial correlation. 22 common edges detected by both methods are shown in dashed lines. Edges only detected by the proposed method are shown in solid lines. CAPME-specific edges are shown in dotted lines.

We then extend the analysis to a larger dataset constructed from GSE9376. For 285 TFs documented in m:Explorer database, expression levels of 20 TFs are measured in GSE9376 with variances greater than 0.25. For these 20 TFs, 875 TF-gene interactions with 377 genes with variances greater than 0.25 are documented in m:Explorer. Applying the screening strategy as the previous example, we select 644 genetic markers marginally associated with at least 5 of the 377 genes with a Bonferroni corrected P-value less than 0.01. We apply the proposed ANTAC method and CAPME to this new dataset. For ANTAC, the parameters λ₁ and λ₂ are set as described in Theorem 3. For CAPME, the tuning parameters λ₁ = 0.089 and λ₂ = 0.281 are chosen by five-fold cross validation. We use TF-gene interactions documented in m:Explorer as an external biological evidence to validate the results. The results are summarized in Table 5. For ANTAC, 540 edges are identified using a cutoff of 0.05 on the FDR controlled P-values. Within these edges, 67 edges are TF-gene interactions and 44 out of 67 are documented in m:Explorer. In comparison, 8499 nonzero edges are detected by CAPME, where 915 edges are TF-gene interactions and 503 out of 915 are documented. This result is hard to interpret biologically. We further ask if identifying the same number of TF-gene interactions, which method achieves higher accuracy according to the concordance with m:Explorer. Based on the magnitude of partial correlation, we select top 771 edges from the CAPME solution, which capture 67 TF-gene interactions. Within these interactions, 38 are documented in m:Explorer. Thus in this example, the proposed ANTAC method achieves both better interpretability and accuracy.

Table 5.

Results for a dataset consists of 644 markers and 377 genes, which was constructed from GSE9376.

Method	Solution Criteria	Total # of gene-gene interactions	# of TF-gene interactions	# of TF-gene interactions documented	Documented Proportion
ANTAC	FDR controlled P-values ≤0.05	540	67	44	65%
CAPME	Magnitude of partial correlation	771	67	38	57%
CAPME	Nonzero entries	8499	915	503	55%

7. Proof of Main Theorems

In this section, we will prove the main results Theorems 1 and 2.

7.1 Proof of Theorem 1

This proof is based on the key lemma, Lemma 5, which is deterministic in nature. We apply Lemma 5 with R = Db^true + E replaced by Y_j= Xγ_j + Z_j, λ replaced by λ₁ and sparsity s replaced by s₁. The following lemma is the key to the proof.

Lemma 1

There exist some constants $C_k^{\prime },1\le k\le 3$ such that for each 1 ≤ j ≤ p,

$${\Vert {\gamma }_j-{\widehat{\gamma }}_j\Vert }_1\le C_1^{\prime }{\lambda }_1{s}_1,$$ (33)

$$\Vert {\gamma }_j-{\widehat{\gamma }}_j\Vert \le C_2^{\prime }{\lambda }_1\sqrt{s_1},$$ (34)

$${\Vert \mathbf{X}({\gamma }_j-{\widehat{\gamma }}_j)\Vert }^2/n\le C_3^{\prime }{\lambda }_1^2{s}_1,$$ (35)

with probability 1 − o(q^−δ₁+1).

With the help of the lemma above, it's trivial to finish our proof. In fact, ‖Ẑ_j − Z_j‖² = ‖X(γ_j − γ̂_j)‖² and hence Equation (14) immediately follows from result (35). Equations (15) and (16) are obtained by the union bound and Equations (33) and (34) because of the following relationship,

$${\Vert \widehat{\mathbf{\Gamma }}-\mathbf{\Gamma }\Vert }_{l_{\infty }}=\underset{j}{max}{\Vert {\gamma }_j-{\widehat{\gamma }}_j\Vert }_1,$$

$$\frac{1}{p}{\Vert \widehat{\mathbf{\Gamma }}-\mathbf{\Gamma }\Vert }_F^2=\frac{1}{p}\sum _{j=1}^p{\Vert {\gamma }_j-{\widehat{\gamma }}_j\Vert }^2.$$

It is then enough to prove Lemma 1 to complete the proof of Theorem 1.

Proof of Lemma 1

The Lemma is an immediate consequence of Lemma 5 applied to Y_j= Xγ_j + Z_j with tuning parameter λ₁ and sparsity s₁. To show the union of Equations (33)-(35) holds with high probability 1 − o(q^−δ₁+1), we only need to check the following conditions of the Lemma 5 hold with probability at least 1 − o(q^−δ₁+1),

$$I_1=\{\nu \le {\theta }^{\mathit{\text{ora}}}{\lambda }_1\frac{\xi -1}{\xi +1}(1-\tau )\text{for some}\xi >1\},$$

$$I_2=\{\frac{\Vert {\mathbf{X}}_k\Vert }{\sqrt{n}}\in [1/A_1,A_1]\text{for all}k\},$$

$$I_3=\{{\theta }^{\mathit{\text{ora}}}\in [1/A_2,A_2],\text{where}{\theta }^{\mathit{\text{ora}}}=\frac{\Vert {\mathbf{Z}}_j\Vert }{\sqrt{n}}.\},$$

$$I_4=\left\{\begin{array}{c}\frac{{\mathbf{w}}^T\mathbf{w}}{n}\text{satisfies lower}-\text{RE with}({\alpha }_1,\zeta (n,q))\mathrm{s}.\mathrm{t}.\\ s_1\zeta (n,q)8{(1+\xi )}^2{A}_1^2\le min\{\frac{{\alpha }_1}{2},1\}\end{array}\right\},$$

where we set ξ = 3/ε₁ + 1 for the current setting, $A_1=C_{A_1}max\{B,\sqrt{M_1}\}$ under Condition 3 and $A_1=C_{A_1}\sqrt{M_1}$ under Condition 3′ for some universal constant C_A1 > 0, $A_2=\sqrt{2M_2}$, ${\alpha }_1=\frac{1}{2M_1}$ and ζ (n, q) = o(1/s₁). Let us still define $\mathbf{w}=\mathbf{X}\cdot \mathit{\text{diag}}\left(\frac{\sqrt{n}}{\Vert {\mathbf{X}}_k\Vert }\right)$ as the standardized X in Lemma 5 of Section 8.

We will show that

$$\mathbb{P}\{I_1^c\}\le O(q^{-{\delta }_1+1}/\sqrt{logq}),$$

$$\mathbb{P}\{I_i^c\}\le o(q^{-{\delta }_1})\text{for}i=2,3\text{and}4,$$

which implies

$$\mathbb{P}\{E_j\}\ge 1-o(q^{-{\delta }_1+1}),$$

where E_j is the union of I₁ to I₄. We will first consider $\mathbb{P}\{I_2^c\}$ and $\mathbb{P}\{I_3^c\}$, then $\mathbb{P}\{I_4^c\}$, and leave $\mathbb{P}\{I_1^c\}$ to the last, which relies on the bounds for $\mathbb{P}\{I_i^c\}$, 2 ≤ i ≤ 4.

(1). To study $\mathbb{P}\{I_2^c\}$ and $\mathbb{P}\{I_3^c\}$, we need the following Bernstein-type inequality (See e.g. (Vershynin 2010), Section 5.2.4 for the inequality and definitions of norms ‖·‖_φ2 and ‖·‖_φ1) to control the tail bound for sum of i.i.d. sub-exponential variables,

$$\mathbb{P}\left\{\right|\sum _{i=1}^n{U}_i|\ge tn\}\le 2exp(-cnmin\{\frac{t^2}{K^2},\frac{t}{K}\left\}\right),$$ (36)

for t > 0, where U_i are i.i.d. centered sub-exponential variables with parameter ‖U_i‖_φ1 ≤ K and c is some universal constant. Notice that θ^ora = ‖Z_j‖ /√n with ${\mathbf{Z}}_j^{(1)}~N(0,{\sigma }_{jj})$, and ${\mathbf{X}}_k^{(1)}$ is sub-gaussian with parameter ${\Vert {\mathbf{X}}_k^{(1)}\Vert }_{{\phi }_2}\in [c_{\phi 2}{M}_1^{-1/2},C_{\phi 2}B]$ by Conditions 2 and 3 with some universal constants c_φ2 and C_φ2 (all formulas involving φ₁ or φ₂ parameter of ${\mathbf{X}}_k^{(1)}$ replace “B” by $“\sqrt{M_1}”$ under Condition 3′ hereafter). The fact that sub-exponential is sub-gaussian squared implies that there exists some universal constant C₁ > 0 such that

$$n({({\theta }^{\mathit{\text{ora}}})}^2-{\sigma }_{jj})\text{is sum of}\mathrm{i}.\mathrm{i}.\mathrm{d}.sub-\text{exponential with}{\phi }_1\text{parameter}{C}_1{\sigma }_{jj},{\Vert {\mathbf{X}}_k\Vert }^2-n\mathbb{E}{x}_k^2\text{is sum of}\mathrm{i}.\mathrm{i}.\mathrm{d}.sub-\text{exponential with}{\phi }_1\text{parameter}{\Vert {\left({\mathbf{X}}_k^{(1)}\right)}^2\Vert }_{{\phi }_1}.$$

Note that σ_jj ∈ [1/M₂,M₂] by Condition 2 and ${\Vert {\left({\mathbf{X}}_k^{(1)}\right)}^2\Vert }_{{\phi }_1}\in [c_{\phi 1}^{\prime }{M}_1^{-1},C_{\phi 1}^{\prime }{B}^2]$ with some universal constants $c_{\phi 1}^{\prime }$ and $C_{\phi 1}^{\prime }$. Let $A_1=C_{A_1}max\{B,\sqrt{M_1}\}$ and $A_2=\sqrt{2M_2}$ for some large constant C_A1. Equation (36) with a sufficiently small constant c₀ implies

$$\mathbb{P}\{I_3^c\}=\mathbb{P}\{{({\theta }^{\mathit{\text{ora}}})}^2\notin [1/A_2^2,A_2^2]\}\le \mathbb{P}\{n|{({\theta }^{\mathit{\text{ora}}})}^2-{\sigma }_{jj}|\ge \frac{{\sigma }_{jj}}{2}n\}\le 2exp(-C\prime n)\le o(q^{-{\delta }_1}),$$ (37)

and

$$\mathbb{P}\{I_2^c\}=\mathbb{P}\{\frac{{\Vert {\mathbf{X}}_k\Vert }^2}{n}\notin [1/A_1^2,A_1^2]\text{for some}k\}\le q\mathbb{P}\{|{\Vert {\mathbf{X}}_k\Vert }^2-n\mathbb{E}{x}_k^2|>c_0{\Vert {\left({\mathbf{X}}_k^{(1)}\right)}^2\Vert }_{{\phi }_1}n\}\le q2exp(-C\prime n)\le o(q^{-{\delta }_1}).$$ (38)

(2). To study the lower-RE condition of $\frac{{\mathbf{W}}^T\mathbf{W}}{n}$, we essentially need to study $\frac{{\mathbf{X}}^T\mathbf{X}}{n}$. On the event I₂, it's easy to see that if $\frac{{\mathbf{X}}^T\mathbf{X}}{n}$ satisfies lower-RE with (α_x, ζ_x (n, q)), then $\frac{{\mathbf{W}}^T\mathbf{W}}{n}$ satisfies lower-RE with $({\alpha }_1,\zeta )=({\alpha }_x{A}_1^{-2},{\zeta }_x{A}_1^2)$, since $\mathbf{W}=\mathbf{X}\mathit{\text{diag}}\left(\frac{\sqrt{n}}{\Vert X_k\Vert }\right)$ and $\frac{\Vert {\mathbf{X}}_k\Vert }{\sqrt{n}}\in [1/A_1,A_1]$ on event I₂. Moreover, to study the lower-RE condition, the following lemma implies that we only need to consider the behavior of $\frac{{\mathbf{X}}^T\mathbf{X}}{n}$ on sparse vectors.

Lemma 2

For any symmetric matrix Δ_q×q, suppose |v^T Δv| ≤ δ for any unit 2s sparse vector v ∈ ℝ^q, i.e. ‖v‖ = 1 and $v\in {\mathbb{B}}_0(2s)=\{a:\sum _{i=1}^{q}1\{a_i\ne 0\}\le 2s\}$, then we have

$$|v^T\mathrm{\Delta }v|\le 27\delta ({\Vert v\Vert }^2+\frac{1}{s}{\Vert v\Vert }_1^2)\mathit{\text{for any}}v\in {\mathbb{R}}^q.$$

See Supplementary Lemma 12 in (Loh & Wainwright 2012) for the proof. With a slight abuse of notation, we define Σ_x = Cov (X⁽¹⁾). By applying Lemma 2 on $\frac{{\mathbf{X}}^T\mathbf{X}}{n}-{\sum }_x$ and Condition 2, $v^T{\sum }_{x}v\ge \frac{1}{M_1}{\Vert v\Vert }^2$, we know that $\frac{{\mathbf{X}}^T\mathbf{X}}{n}$ satisfies lower-RE (α_x, ζ_x (n, q)) with

$${\alpha }_x\ge \frac{1}{M_1}(1-\frac{27}{L})\text{and}{\zeta }_x(n,q)\le \frac{27}{s_1{M}_{1}L},$$ (39)

Provided

$$\left|v^T\right(\frac{{\mathbf{X}}^T\mathbf{X}}{n}-{\sum }_x\left)v\right|\le \frac{1}{L{M}_1}\text{for all}v\in {\mathbb{B}}_0(2s_1)\text{with}\Vert v\Vert =1,$$ (40)

which implies that the population covariance matrix Σ_x and its sample version $\frac{{\mathbf{X}}^T\mathbf{X}}{n}$ behave similarly on all 2s₁ sparse vectors.

Now we show Equation (40) holds under Conditions 3 and 3′ respectively for a sufficiently large constant L such that the inequality in the event I₄ holds. Under Condition 3′, X⁽¹⁾ is jointly sub-gaussian with parameter (M₁)^1/2 . A routine one-step chaining (or δ-net) argument implies that there exists some constant c_sg > 0 such that

$$\mathbb{P}\left\{\underset{v\in {\mathbb{B}}_0(2s_1)}{sup}\right|v^T(\frac{{\mathbf{X}}^T\mathbf{X}}{n}-{\sum }_x)v|>t{\Vert v\Vert }^2\}\le 2exp(-c_{sg}nmin\{\frac{t^2}{M_1^2},\frac{t}{M_1}\}+2s_{1}logq).$$ (41)

See e.g. Supplementary Lemma 15 in (Loh & Wainwright 2012) for the proof. Hence by picking small $t=\frac{1}{L{M}_1}$ with any fixed but arbitrary large L, the sparsity condition (17) $s_1=o\left(\frac{n}{logq}\right)$ and Equation (41) imply that Equation (40) holds with probability 1 − o (p^−δ₁).

Under Condition 3, if $s_1=o\left(\sqrt{\frac{n}{logq}}\right)$, Hoeffding's inequality and a union bound imply that

$$\mathbb{P}\{{\Vert \frac{{\mathbf{X}}^T\mathbf{X}}{n}-{\sum }_x\Vert }_{\infty }>2B\sqrt{\frac{(1+{\delta }_1)logq}{n}}\}\le 2q^{-2{\delta }_1},$$

where the norm ‖·‖_∞ denotes the entry-wise supnorm and ‖X⁽¹⁾‖_∞ ≤ B (see, e.g. (Massart 2007) Proposition 2.7). Thus with probability 1 − o (q^−δ₁), we have for any v ∈ ₀ (2s₁) with‖v‖ = 1,

$$\left|v^T\right(\frac{{\mathbf{X}}^T\mathbf{X}}{n}-{\sum }_x\left)v\right|\le {\Vert \frac{{\mathbf{X}}^T\mathbf{X}}{n}-{\sum }_x\Vert }_{\infty }{\Vert v\Vert }_1^2\le 2B\sqrt{\frac{(1+{\delta }_1)logq}{n}}\frac{1}{2s_1}=o(1),$$

where the last inequality follows from ${\Vert v\Vert }_1\le \sqrt{2s_1}\Vert v\Vert$ for any v ∈ ₀ (2s₁). Therefore we have Equation (40) holds with probability 1 − o (q^−δ₁) for any arbitrary large L. If $s_1=o\left(\frac{n}{{log}^{3}nlogq}\right)$, an involved argument using Dudley's entropy integral and Talagrand's concentration theorem for empirical processes implies (see, (Rudelson & Zhou 2013) Theorem 23 and its proof) Equation (40) holds with probability 1 − o (q^−δ₁) for any fixed but arbitrary large L.

Therefore we showed that under Condition 3 or 3′, Equation (40) holds with probability 1 − o (q^−δ₁) for any arbitrary large L. Consequently, Equation (39) with $({\alpha }_1,\zeta )=({\alpha }_x{A}_1^{-2},{\zeta }_x{A}_1^2)$ on event I₂ implies that we can pick ${\alpha }_1=\frac{1}{2M_1}$ and sufficiently small ζ such that event I₄ holds with probability 1 − o (q^−δ₁).

(3). Finally we study the probability of event I₁. The following tail probability of t distribution is helpful in the analysis.

Proposition 1

Let T_n follows a t distribution with n degrees of freedom. Then there exists ε_n → 0 as n → ∞ such that ∀t > 0

$$\mathbb{P}\{T_n^2>n(e^{2t^2/(n-1)}-1\left)\right\}\le (1+{\epsilon }_n)e^{-t^2}/({\pi }^{1/2}t).$$

Please refer to (Sun & Zhang 2012) Lemma 1 for the proof. Recall that $I_1=\{\frac{\nu }{{\theta }^{\mathit{\text{ora}}}}\le {\lambda }_1\frac{\xi -1}{\xi +1}(1-\tau )\}$, where τ defined in Equation (67) satisfies that $\tau =O(s_1{\lambda }_1^2)=o(1)$ on ${\cap }_{i=2}^4{I}_i$. From the definition of ν in Equation (66) we have

$$\frac{\nu }{{\theta }^{\mathit{\text{ora}}}}=\underset{k}{max}|h_k|,\text{with}{h}_k=\frac{{\mathbf{W}}_k^T{\mathbf{Z}}_j}{n{\theta }^{\mathit{\text{ora}}}},$$

where each column of W has norm ‖W_k‖ =√n. Given X, equivalently W, it's not hard to check that we have $\frac{\sqrt{n-1}{h}_k}{\sqrt{1-h_k^2}}~t_{(n-1)}$ by the normality of Z_j, where t_(n−1) is the t distribution with (n − 1) degrees of freedom. From Proposition 1 we have

$$\mathbb{P}\{|h_k|>\sqrt{\frac{2t^2}{n}}\}=\mathbb{P}\{\frac{(n-1)h_k^2}{1-h_k^2}>\frac{2(n-1)t^2/n}{1-2t^2/n}\}\le \mathbb{P}\{\frac{(n-1)h_k^2}{1-h_k^2}>\frac{2(n-1)t^2/(n-2)}{1-t^2/(n-2)}\}\le \mathbb{P}\{\frac{(n-1)h_k^2}{1-h_k^2}>(n-1)(e^{2t^2/(n-2)}-1\left)\right\}\le (1+{\epsilon }_{n-1})e^{-t^2}/({\pi }^{1/2}t),$$

where the first inequality holds when t² > 2, and the second inequality follows from the fact $e^x-1\le x/(1-\frac{x}{2})$ for 0 < x < 2. Now let t² = δ₁ log q > 2, and ${\lambda }_1=(1+{\epsilon }_1)\sqrt{\frac{2{\delta }_{1}logq}{n}}$ with ξ = 3/ε₁ + 1, then we have ${\lambda }_1\frac{\xi -1}{\xi +1}(1-\tau )>\sqrt{\frac{2{\delta }_{1}logq}{n}}$ and

$$\mathbb{P}\{{\cap }_{i=1}^4{I}_i\}\ge \mathbb{P}\{\frac{\nu }{{\theta }^{\mathit{\text{ora}}}}\le \sqrt{\frac{2{\delta }_{1}logq}{n}}\}-\mathbb{P}\{{({\cap }_{i=2}^4{I}_i)}^c\}\ge 1-q\cdot \mathbb{P}\{|h_k|>\sqrt{\frac{2{\delta }_{1}logq}{n}}\}-\mathbb{P}\{{({\cap }_{i=2}^4{I}_i)}^c\}\ge 1-(\frac{1}{\sqrt{\pi {\delta }_1}}+o(1))\frac{q^{-{\delta }_1+1}}{\sqrt{logq}},$$

which immediately implies $\mathbb{P}\{I_1^c\}\le O(q^{-{\delta }_1+1}/\sqrt{logq})$.

7.2 Proof of Theorem 2

The whole proof is based on the results in Theorem 1. In particular with probability 1 − o (p·q^−δ₁+1) the following events hold,

$$\frac{1}{n}{\Vert {\widehat{\mathbf{Z}}}_j-{\mathbf{Z}}_j\Vert }^2\le C_1{s}_1\frac{logq}{n}\text{for all}j,$$ (42)

$${\Vert {\widehat{\gamma }}_j-{\gamma }_j\Vert }_1\le C_2{s}_1\sqrt{\frac{logq}{n}}\text{for all}j.$$ (43)

From now on the analysis is conditioned on the two events above. This proof is also based on the key lemma, Lemma 5. We apply Lemma 5 with R = Db^true + E replaced by Ẑ_m= Ẑ_A^cβ_m + E_m for each m ∈ A = {i, j}, λ replaced by λ₂ and sparsity s replaced by C_βs₂, where E_m is defined by the regression model Equation (7) as

$${\mathbf{E}}_m\equiv {\epsilon }_m+{\mathrm{\Delta }}_{Z_m}={\epsilon }_m+({\widehat{\mathbf{Z}}}_m-{\mathbf{Z}}_m)+({\mathbf{Z}}_{A^c}-{\widehat{\mathbf{Z}}}_{A^c}){\beta }_m,$$ (44)

and C_β = 2M₂ because the definition of β_m in Equation (6) implies it is a weighted sum of two columns of Ω with weight bounded by M₂.

To obtain our desired result $|{\widehat{\psi }}_{kl}-{\psi }_{kl}^{\mathit{\text{ora}}}|=|{\epsilon }_k^T{\epsilon }_l/n-{\widehat{\epsilon }}_k^T{\widehat{\epsilon }}_l/n|$ for each pair k,l ∈ A= {i,j}, it's sufficient for us to bound $|{\widehat{\epsilon }}_k^T{\widehat{\epsilon }}_l/n-{\mathbf{E}}_k^T{\mathbf{E}}_l/n|$ and $|{\epsilon }_k^T{\epsilon }_1/n-{\mathbf{E}}_k^T{\mathbf{E}}_l/n|$ separately and then to apply the triangle inequality The following two lemmas are useful to establish those two bounds.

Lemma 3

There exists some constant C_in > 0 such that $|{\epsilon }_k^T{\epsilon }_l/n-{\mathbf{E}}_k^T{\mathbf{E}}_l/n|\le C_{\mathit{\text{in}}}{\lambda }_1^2{s}_1$ with probability 1 − o (q^−δ₁).

Lemma 4

There exist some constants $C_k^{\prime }$, 1 ≤ k ≤ 3 such that for each m ∈ A = {i, j},

$${\Vert {\beta }_m-{\widehat{\beta }}_m\Vert }_1\le C_1^{\prime }{\lambda }_2{s}_2,$$

$${\Vert {\widehat{\mathbf{Z}}}_{A^c}({\beta }_m-{\widehat{\beta }}_m)\Vert }^2/n\le C_2^{\prime }{\lambda }_2^2{s}_2,$$

$${\Vert {\widehat{\mathbf{Z}}}_{A^c}^T{\mathbf{E}}_m/n\Vert }_{\infty }\le C_3^{\prime }{\lambda }_2,$$

with probability 1 − o (q^−δ₂+1).

Before moving on, we point out a fact we will use several times in the proof,

$${\Vert {\beta }_m\Vert }_1\le 2M_2^2,$$ (45)

which follows from Equation (6) and Condition 5. Hence we have

$$\Vert {\mathrm{\Delta }}_{Z_m}\Vert /\sqrt{n}\le \frac{\Vert {\widehat{\mathbf{Z}}}_m-{\mathbf{Z}}_m\Vert }{\sqrt{n}}+\frac{{max}_k\Vert {\widehat{\mathbf{Z}}}_k-{\mathbf{Z}}_k\Vert }{\sqrt{n}}{\Vert {\beta }_m\Vert }_1\le \sqrt{C_3{s}_1{\lambda }_1^2},$$ (46)

for some constant C₃ > 0.

To bound the term $|{\widehat{\epsilon }}_k^T{\widehat{\epsilon }}_l/n-{\mathbf{E}}_k^T{\mathbf{E}}_l/n|$, we note that for any k,l ∈ A= {i,j},

$$|{\widehat{\epsilon }}_k^T{\widehat{\epsilon }}_l/n-{\mathbf{E}}_k^T{\mathbf{E}}_l/n|=\left|{({\mathbf{E}}_k+{\widehat{\mathbf{Z}}}_{A^c}({\beta }_k-{\widehat{\beta }}_k\left)\right)}^T\right({\mathbf{E}}_l+{\widehat{\mathbf{Z}}}_{A^c}({\beta }_l-{\widehat{\beta }}_1))/n-{\mathbf{E}}_k^T{\mathbf{E}}_l/n|\le {\Vert {\widehat{\mathbf{Z}}}_{A^c}^T{\mathbf{E}}_k/n\Vert }_{\infty }{\Vert {\beta }_k-{\widehat{\beta }}_k\Vert }_1+{\Vert {\widehat{\mathbf{Z}}}_{A^c}^T{\mathbf{E}}_l/n\Vert }_{\infty }{\Vert {\beta }_l-{\widehat{\beta }}_l\Vert }_1+\Vert {\widehat{\mathbf{Z}}}_{A^c}({\beta }_k-{\widehat{\beta }}_k)\Vert \cdot \Vert {\widehat{Z}}_{A^c}({\beta }_l-{\widehat{\beta }}_l)\Vert /n\le (2C_1^{\prime }{C}_3^{\prime }+C_2^{\prime }){\lambda }_2^2{s}_2,$$ (47)

where we applied Lemma 4 in the last inequality.

Lemma 3, together with Equation (47), immediately implies the desired result (23),

$$|{\widehat{\psi }}_{kl}-{\psi }_{kl}^{\mathit{\text{ora}}}|\le C_4^{\prime }(s_2\frac{logp}{n}+s_1\frac{logq}{n}),$$

for some constant $C_4^{\prime }$ with probability 1 − o (p · q ^−δ₁+1 + p^−δ₂+1). Since the spectrum of Ψ_A,A is bounded below by $M_2^{-1}$ and above by M₂ and the functional ${({\mathrm{\Omega }}_{A,A})}_{kl}={({\mathrm{\Psi }}_{A,A}^{-1})}_{kl}$ is Lipschitz in a neighborhood of Ψ_A,A for k,l ∈ A, we obtain that Equation (24) is an immediate consequence of Equation (23). Note that ${\omega }_{ij}^{\mathit{\text{ora}}}$ is the MLE of ω_ij in the model ${({\eta }_i,{\eta }_j)}^T~N(0,{\mathrm{\Omega }}_{A,A}^{-1})$ with three parameters given n samples. Whenever $s_2=o\left(\frac{\sqrt{n}}{logp}\right)$ and $s_1=o\left(\frac{\sqrt{n}}{logq}\right)$, we have $s_2\frac{logp}{n}+s_1\frac{logq}{n}=o(\frac{1}{\sqrt{n}})$. Therefore we have ${\omega }_{ij}^{\mathit{\text{ora}}}-{\widehat{\omega }}_{ij}=o_p({\omega }_{ij}^{\mathit{\text{ora}}}-{\omega }_{ij})$, which immediately implies Equation (25) in Theorem 2,

$$\sqrt{n{F}_{ij}}({\widehat{\omega }}_{ij}-{\omega }_{ij})\stackrel{D}{~}\sqrt{n{F}_{ij}}({\omega }_{ij}^{\mathit{\text{ora}}}-{\omega }_{ij})\stackrel{D}{\to }N(0,1),$$

where F_ij is the Fisher information of ω_ij.

It is then enough to prove Lemma 3 and Lemma 4 to complete the proof of Theorem 2.

Proof of Lemma 3

We show that $|{\epsilon }_k^T{\epsilon }_l/n-{\mathbf{E}}_k^T{\mathbf{E}}_l/n|\le C_{\mathit{\text{in}}}{\lambda }_1^2{s}_1$ with probability 1 − o (q^−δ₁) in this section. By Equation (46), we have

$$|{\epsilon }_k^T{\epsilon }_l/n-{\mathbf{E}}_k^T{\mathbf{E}}_l/n|\le \left|\frac{{\epsilon }_k^T{\mathrm{\Delta }}_{Z_l}+{\epsilon }_l^T{\mathrm{\Delta }}_{Z_k}+{\mathrm{\Delta }}_{Z_l}^T{\mathrm{\Delta }}_{Z_k}}{n}\right|\le C_3{s}_1{\lambda }_1^2+\frac{\left|{\epsilon }_k^T{\mathrm{\Delta }}_{Z_l}\right|+\left|{\epsilon }_l^T{\mathrm{\Delta }}_{Z_k}\right|}{n}.$$ (48)

To bound the term $\left|{\epsilon }_k^T\mathrm{\Delta }{Z}_l\right|$, we note that by the definition of Δ_Zt in Equation (44) there exists some constant C₄ such that,

$$\frac{|{\epsilon }_k^T{\mathrm{\Delta }}_{z_l}|}{n}=\left|{\epsilon }_k^T\right[({\widehat{\mathbf{Z}}}_l-{\mathbf{Z}}_l)+({\mathbf{Z}}_{A^c}-{\widehat{\mathbf{Z}}}_{A^c}){\beta }_m\left]\right|/n=|{\epsilon }_k^T\mathbf{X}[({\gamma }_l-{\widehat{\gamma }}_l)+({\widehat{\gamma }}_{A^c}-{\gamma }_{A^c}){\beta }_m]|/n\le {\Vert {\epsilon }_k^T\mathbf{X}/n\Vert }_{\infty }{\Vert ({\gamma }_l-{\widehat{\gamma }}_l)+({\widehat{\gamma }}_{A^c}-{\gamma }_{A^c}){\beta }_m\Vert }_1\le {\Vert {\epsilon }_k^T\mathbf{X}/n\Vert }_{\infty }\underset{i}{max}{\Vert {\gamma }_i-{\widehat{\gamma }}_i\Vert }_1(1+{\Vert {\beta }_m\Vert }_1)\le {\Vert {\epsilon }_k^T\mathbf{X}/n\Vert }_{\infty }{C}_4{s}_1{\lambda }_1,$$ (49)

where the last inequality follows from Equations (43) and (45). Since ${\epsilon }_k\stackrel{i.i.d}{\sim }N(0,{\psi }_{kk})$ and X are independent, it can be seen that each coordinate of ${\epsilon }_k^T\mathbf{X}$ is a sum of n i.i.d. sub-exponential variables with bounded parameter under either Condition 3 or 3′. A union bound with q coordinates and another application of Bernstein inequality in Equation (36) with $t=C_5\sqrt{\frac{logq}{n}}$ imply that ${\Vert {\epsilon }_k^T\mathbf{X}/n\Vert }_{\infty }\le C_5\sqrt{\frac{logq}{n}}$ with probability 1 − o (q^−δ₁) for some large constant C₅ > 0. This fact, together with Equation (49), implies that $|{\epsilon }_k^T{\mathrm{\Delta }}_{Z_l}/n|=O(s_1{\lambda }_1^2)$ with probability 1 − o (q^−δ₁). Similar result holds for $|{\epsilon }_l^T{\mathrm{\Delta }}_{Z_k}/n|$. Together with Equation (48), this result completes our claim on $|{\epsilon }_k^T{\epsilon }_l/n-{\mathbf{E}}_k^T{\mathbf{E}}_l/n|$ and finishes the proof of Lemma 3.

Proof of Lemma 4

The Lemma 4 is an immediate consequence of Lemma 5 applied to Ẑ_m= Ẑ_A^cβ_m+ E_m for each m ∈ A = {i,j} with parameter λ₂ and sparsity C_βs₂. We check the following conditions I₁ − I₄ in the Lemma 5 hold with probability 1 − o (p^−δ₂+1) to finish our proof. This part of the proof is similar to the proof of Lemma 1. We thus directly apply those facts already shown in the proof of Lemma 1 whenever possible. Let

$$I_1=\{\nu \le {\theta }^{\mathit{\text{ora}}}{\lambda }_2\frac{\xi -1}{\xi +1}(1-{\tau }_2)\text{for some}\xi >1\},$$

$$I_2=\{\frac{\Vert {\widehat{\mathbf{Z}}}_k\Vert }{\sqrt{n}}\in [1/A_1^{\prime },A_1^{\prime }]\text{for all}k\in A^c\},$$

$$I_3=\{{\theta }^{\mathit{\text{ora}}}\in [1/A_2^{\prime },A_2^{\prime }],\text{where}{\theta }^{\mathit{\text{ora}}}=\frac{\Vert {\mathbf{E}}_m\Vert }{\sqrt{n}}.\},$$

$$I_4=\left\{\begin{array}{c}\frac{{\mathbf{w}}^T\mathbf{w}}{n}\text{satisfies lower}-\text{RE with}({\alpha }_1,\zeta (n,p))\mathrm{s}.\mathrm{t}.\\ s_2\zeta (n,p)8{(\mathrm{l}+\xi )}^2{A}_1^{\prime 2}\le min\{\frac{{\alpha }_1}{2},1\}\end{array}\right\},$$

where we can set ξ = 3/ε₂ +1 for the current setting, $A_1^{\prime }=A_2^{\prime }=\sqrt{3M_2},{\alpha }_1=\frac{1}{2M_2}$ and ζ (n, p) = o(1/s₂). Let us still define $\mathbf{W}={\widehat{\mathbf{Z}}}_{A^c}\cdot \mathit{\text{diag}}\left(\frac{\sqrt{n}}{\Vert {\widehat{\mathbf{Z}}}_k\Vert }\right)$ as the standardized Ẑ_A^c in the Lemma 5 of Section 8. The strategy is to show that $\mathbb{P}\left\{I_1^c\right\}\le O(p^{-{\delta }_2+1}/\sqrt{logp})$ and $\mathbb{P}\left\{I_i^c\right\}\le o(p^{-{\delta }_2})$ for i = 2, 3 and 4, which completes our proof.

(1). To study $\mathbb{P}\left\{I_2^c\right\}$ and $\mathbb{P}\left\{I_3^c\right\}$, we note that $\frac{\Vert {\widehat{\mathbf{Z}}}_k\Vert }{\sqrt{n}}\le \frac{\Vert {\mathbf{Z}}_k\Vert }{\sqrt{n}}+\frac{\Vert {\widehat{\mathbf{Z}}}_k-{\mathbf{Z}}_k\Vert }{\sqrt{n}}$, where Z_k ∼ N (0, σ_kkI) and $\frac{\Vert {\widehat{\mathbf{Z}}}_k-{\mathbf{Z}}_k\Vert }{\sqrt{n}}=O\left(\sqrt{s_1{\lambda }_1^2}\right)=o(1)$ according to Equation (42). Similarly $\frac{\Vert {\mathbf{E}}_m\Vert }{\sqrt{n}}\le \frac{\Vert {\mathit{\epsilon }}_m\Vert }{\sqrt{n}}+\frac{\Vert {\mathrm{\Delta }}_{Z_m}\Vert }{\sqrt{n}}$ where ε_m ∼ N (0, ψ_mmI) and $\frac{\Vert {\mathrm{\Delta }}_{Zm}\Vert }{\sqrt{n}}=o(1)$ from Equation (46). Noting that ψ_mm, ${\sigma }_{kk}\in [M_2^{-1},M_2]$, we use the same argument as that for $\mathbb{P}\left\{I_3^c\right\}$ in the proof of Lemma 1 to obtain

$$\mathbb{P}\{I_2^c\}\le \mathbb{P}\{\frac{\Vert {\mathbf{Z}}_k\Vert }{\sqrt{n}}\notin [1/\sqrt{2M_2},\sqrt{2M_2}\left]\text{for some}k\right\}\le o(p^{-{\delta }_2}),$$

$$\mathbb{P}\{I_3^c\}\le \mathbb{P}\{\frac{\Vert {\mathit{\epsilon }}_m\Vert }{\sqrt{n}}\notin [1/\sqrt{2M_2},\sqrt{2M_2}\left]\right\}\le o(p^{-{\delta }_2}).$$

To study the lower-RE condition of $\frac{{\mathbf{W}}^T\mathbf{W}}{n}$, as what we did in the proof of Lemma 1 and Lemma 2, we essentially need to study $\frac{{\widehat{\mathbf{Z}}}_{A^c}^T{\widehat{\mathbf{Z}}}_{A^c}}{n}$ and to show the following fact

$$\left|v^T\right(\frac{{\widehat{\mathbf{Z}}}_{A^c}^T{\widehat{\mathbf{Z}}}_{A^c}}{n}-{\sum }_{A^c,A^c}\left)v\right|\le \frac{1}{L{M}_2}\text{for all}v\in {\mathbb{B}}_0(2C_{\beta }{s}_2)\text{with}\Vert v\Vert =1,$$

where λ_min (Σ_A^c,A^c) ≥ 1/M₂. Following the same line of the proof in Lemma 1 for the lower-RE condition of $\frac{{\mathbf{X}}^T\mathbf{X}}{n}$ with normality assumption on Z and sparsity assumption $s_2=o\left(\sqrt{\frac{n}{logp}}\right)$, we can obtain that with probability 1 − o (p^−δ₂),

$$\underset{v\in {\mathbb{B}}_0(2C_{\beta }{s}_2)}{sup}\left|v^T\right(\frac{{\mathbf{Z}}_{A^c}^T{\mathbf{Z}}_{A^c}}{n}-{\sum }_{A^c,A^c}\left)v\right|\le \frac{{\Vert v\Vert }^2}{2L{M}_2}.$$ (50)

Therefore all we need to show in the current setting is that with probability 1 − o (p^−δ₂),

$$\left|v^T\right(\frac{{\widehat{\mathbf{Z}}}_{A^c}^T{\widehat{\mathbf{Z}}}_{A^c}}{n}-\frac{{\mathbf{Z}}_{A^c}^T{\mathbf{Z}}_{A^c}}{n}\left)v\right|\le \frac{1}{2L{M}_2}\text{for all}v\in {\mathbb{B}}_0(2C_{\beta }{s}_2)\text{with}\Vert v\Vert =1.$$ (51)

To show Equation (51), we notice

$$\left|v^T\right(\frac{{\widehat{\mathbf{Z}}}_{A^c}^T{\widehat{\mathbf{Z}}}_{A^c}}{n}-\frac{{\mathbf{Z}}_{A^c}^T{\mathbf{Z}}_{A^c}}{n}\left)v\right|\le 2\left|v^T\right(\frac{{\mathbf{Z}}_{A^c}^T({\widehat{\mathbf{Z}}}_{A^c}-{\mathbf{Z}}_{A^c})}{n}\left)v\right|+\frac{{\Vert ({\widehat{\mathbf{Z}}}_{A^c}-{\mathbf{Z}}_{A^c})v\Vert }^2}{n}\equiv D_1+D_2.$$

To control D₂, we find

$$\sqrt{D_2}\le \underset{k}{max}\frac{\Vert {\widehat{\mathbf{Z}}}_k-{\mathbf{Z}}_k\Vert }{\sqrt{n}}{\Vert v\Vert }_1\le \sqrt{s_1}{\lambda }_1\sqrt{2C_{\beta }{s}_2}=o(1),$$ (52)

by Equation (42), ${\Vert v\Vert }_1\le \sqrt{2C_{\beta }{s}_2}\Vert v\Vert$ and sparsity assumptions (22).

To control D₁, we find $D_1\le \sqrt{D_2}\cdot \frac{\Vert {\mathbf{Z}}_{A^c}v\Vert }{\sqrt{n}}$, which is o(1) with probability 1 − o (p^−δ₂) by Equation (52) and the following result $\frac{\Vert {\mathbf{Z}}_{A^c}v\Vert }{\sqrt{n}}=O(1)$. Equation (50) implies that with probability 1 − o (p^−δ₂),

$$\frac{\Vert {\mathbf{Z}}_{A^c}v\Vert }{\sqrt{n}}\le \frac{\Vert v\Vert }{\sqrt{2L{M}_2}}+\Vert {\sum }_{A^c,A^c}^{1/2}v\Vert =O(1)\text{for all}\Vert v\Vert =1.$$

(3). Finally we study the probability of event I₁. In the current setting,

$$\nu ={\Vert {\mathbf{W}}^T{\mathbf{E}}_m/n\Vert }_{\infty }={\Vert {\mathbf{W}}^T{\mathit{\epsilon }}_m/n\Vert }_{\infty }+O({\Vert {\mathbf{W}}^T{\mathrm{\Delta }}_{Z_m}/n\Vert }_{\infty }),$$

$${\theta }^{\mathit{\text{ora}}}=\frac{\Vert {\mathbf{E}}_m\Vert }{\sqrt{n}}=\frac{\Vert {\mathit{\epsilon }}_m\Vert }{\sqrt{n}}+O\left(\frac{\Vert {\mathrm{\Delta }}_{Z_m}\Vert }{\sqrt{n}}\right).$$

Following the same line of the proof for event I₁ in Lemma 1, we obtain that with probability 1 − o (p^−δ₂+1),

$${\Vert {\mathbf{W}}^T{\mathit{\epsilon }}_m/n\Vert }_{\infty }<\frac{\Vert {\mathit{\epsilon }}_m\Vert }{\sqrt{n}}{\lambda }_2\frac{\xi -1}{\xi +1}(1-{\tau }_2).$$

Thus to prove event $I_1=\{\nu \le {\theta }^{\mathit{\text{ora}}}{\lambda }_2\frac{\xi -1}{\xi +1}(1-{\tau }_2)\}$ holds with desired probability, we only need to show with probability 1 − o (p^−δ₂+1),

$${\Vert {\mathbf{W}}^T{\mathrm{\Delta }}_{Z_m}/n\Vert }_{\infty }=o({\lambda }_2)\text{and}\frac{\Vert {\mathrm{\Delta }}_{Z_m}\Vert }{\sqrt{n}}=o(1).$$ (53)

Equation (46) immediately implies $\frac{\Vert {\mathrm{\Delta }}_{Zm}\Vert }{\sqrt{n}}=O\left(\sqrt{s_1{\lambda }_1^2}\right)=o(1)$ by the sparsity assumptions (22). To study ‖W^TΔ_Zm/n‖_∞, we obtain that there exists some constant C₆ > 0 such that,

$${\Vert {\mathbf{W}}^T{\mathrm{\Delta }}_{Z_m}/n\Vert }_{\infty }={\Vert \mathit{\text{diag}}\left(\frac{\sqrt{n}}{\Vert {\widehat{\mathbf{Z}}}_k\Vert }\right){\widehat{\mathbf{Z}}}_{A^c}^T{\mathrm{\Delta }}_{Z_m}/n\Vert }_{\infty }\le A_1^{\prime }{\Vert {\mathbf{Z}}_{A^c}^T+({\widehat{\mathbf{Z}}}_{A^c}^T-{\mathbf{Z}}_{A^c}^T){\mathrm{\Delta }}_{Z_m}/n\Vert }_{\infty }\le A_1^{\prime }({\Vert {\mathbf{Z}}_{A^c}^T{\mathrm{\Delta }}_{Z_m}/n\Vert }_{\infty }+\underset{i}{max}\Vert ({\widehat{\mathbf{Z}}}_i-{\mathbf{Z}}_i)/\sqrt{n}\Vert \Vert {\mathrm{\Delta }}_{Z_m}/\sqrt{n}\Vert )\le A_1^{\prime }{\Vert {\mathbf{Z}}_{A^c}^T{\mathrm{\Delta }}_{Z_m}/n\Vert }_{\infty }+C_6{s}_1{\lambda }_1^2,$$ (54)

where we used $\frac{\Vert {\widehat{\mathbf{Z}}}_k\Vert }{\sqrt{n}}\in [1/A_1^{\prime },A_1^{\prime }]$ for all k on I₂ in the first inequality and Equations (42) and (46) in the last inequality By the sparsity assumptions (22), we have $s_1{\lambda }_1^2=o({\lambda }_2)$. Thus it's sufficient show ${\Vert Z_{A^c}^T{\mathrm{\Delta }}_{Z_m}/n\Vert }_{\infty }=o({\lambda }_2)$ with probability 1 − o (p^−δ₂+1). In fact,

$${\Vert {\mathbf{Z}}_{A^c}^T{\mathrm{\Delta }}_{Z_m}/n\Vert }_{\infty }\le {\Vert {\mathbf{Z}}_{A^c}^T\mathbf{X}[({\gamma }_m-{\widehat{\gamma }}_m)+({\widehat{\gamma }}_{A^c}-{\gamma }_{A^c}){\beta }_m]/n\Vert }_{\infty }\le {\Vert {\mathbf{Z}}_{A^c}^T\mathbf{X}/n\Vert }_{\infty }{\Vert [({\gamma }_m-{\widehat{\gamma }}_m)+({\widehat{\gamma }}_{A^c}-{\gamma }_{A^c}){\beta }_m]\Vert }_1\le {\Vert {\mathbf{Z}}_{A^c}^T\mathbf{X}/n\Vert }_{\infty }\underset{i}{max}{\Vert {\widehat{\gamma }}_1-{\gamma }_i\Vert }_1(1+{\Vert {\beta }_m\Vert }_1)\le {\Vert {\mathbf{Z}}_{A^c}^T\mathbf{X}/n\Vert }_{\infty }C{s}_1{\lambda }_1=o({\Vert {\mathbf{Z}}_{A^c}^T\mathbf{X}/n\Vert }_{\infty }),$$ (55)

where the last inequality follows from Equations (43) and (45).

Since each ${\mathbf{Z}}_k^T\stackrel{i.i.d}{~}N(0,{\sigma }_{kk})$ and is independent of X, it can be seen that each entry of ${\mathbf{Z}}_{A^c}^T\mathbf{X}$ is a sum of n i.i.d. sub-exponential variables with finite parameter under either Condition 3 or 3′. A union bound with pq entries and an application of Bernstein inequality in Equation (36) with $t=C_7\sqrt{\frac{log(pq)}{n}}$ imply that ${\Vert {\mathbf{Z}}_{A^c}^T\mathbf{X}/n\Vert }_{\infty }\le C_7\sqrt{\frac{log(pq)}{n}}$ with probability 1 − o ((pq)^−δ₁) for some large constant C₇ > 0. This result, together with Equation (55), implies that ${\Vert {\mathbf{Z}}_{A^c}^T{\mathrm{\Delta }}_{Z_m}/n\Vert }_{\infty }=o({\lambda }_2)$ with probability 1 − o (p^−δ₂+1). Now we finish the proof of ‖W^TΔ_Zm/n‖_∞ = o(λ₂) by Equation (54) and hence the proof of the Equation (53) with probability 1 − o (p^−δ₂+1). This completes our proof of Lemma 4.

8. A Key Lemma

The lemma of scaled lasso introduced in this section is deterministic in nature. It could be applied in different settings in which the assumptions of design matrix, response variables and noise distribution may vary, as long as the conditions of the lemma are satisfied. Therefore it's a very useful building block in the analysis of many different problems. In particular, the main theorems of both steps are based on this key lemma. The proof of this lemma is similar as that in (Sun & Zhang 2012) and (Ren et al. 2013), but we use the restrict eigenvalue condition for the gram matrix instead of CIF condition to easily adapt to different settings of design matrix for our probabilistic analysis.

Consider the following general scaled l₁ penalized regression problem. Denote the n by p₀ dimensional design matrix by D = (D₁, …, D_p0), the n dimensional response variable R = (R₁, …, R_n)^T and the noise variable E = (E₁, …, E_n)^T. The scaled lasso estimator with tuning parameter λ of the regression

$$\mathbf{R}=\mathbf{D}{b}^{\mathit{\text{true}}}+\mathbf{E},$$ (56)

is defined as

$$\{\widehat{b},\widehat{\theta }\}=arg\underset{b\in {\mathbb{R}}^{p_0},\theta \in \mathbb{R}+}{min}\{\frac{{\Vert \mathbf{R}-\mathbf{D}b\Vert }^2}{2n\theta }+\frac{\theta }{2}+\lambda \sum _{k=1}^{p_0}\frac{\Vert {\mathbf{D}}_k\Vert }{\sqrt{n}}|b_k|\},$$ (57)

where the sparsity s of the true coefficient b^true is defined as follows,

$$s={\sum }_{j=1}^{p_0}min\{1,|b_j^{\mathit{\text{true}}}|/\lambda \},$$ (58)

which is a generalization of exact sparseness (the number of nonzero entries).

We first normalize each column of the design matrix D to make the analysis cleaner by setting

$$d_k=\frac{\Vert {\mathbf{D}}_k\Vert }{\sqrt{n}}{b}_k,\text{and}\mathbf{W}=\mathbf{D}\cdot \mathit{\text{diag}}\left(\frac{\sqrt{n}}{\Vert {\mathbf{D}}_k\Vert }\right),$$ (59)

and then rewrite the model (56) and the penalized procedure (57) as follows,

$$\mathbf{R}=\mathbf{W}{d}^{\mathit{\text{true}}}+{\mathbf{E}}_j,$$ (60)

and

$$L_{\lambda }(d,\theta )=\frac{{\Vert \mathbf{R}-\mathbf{W}d\Vert }^2}{2n\theta }+\frac{\theta }{2}+\lambda {\Vert d\Vert }_1,$$ (61)

$$\{\widehat{d},\widehat{\theta }\}=arg\underset{d\in {\mathbb{R}}^{p_0},\theta \in \mathbb{R}+}{min}{L}_{\lambda }(d,\theta ),$$ (62)

where the true coefficients and the estimator of the standardized scaled lasso regression (61) are $d_k^{\mathit{\text{true}}}=\frac{\Vert {\mathbf{D}}_k\Vert }{\sqrt{n}}{b}_k^{\mathit{\text{true}}}$ and ${\widehat{d}}_k={\widehat{b}}_k\frac{\Vert {\mathbf{D}}_k\Vert }{\sqrt{n}}$ respectively.

For this standardized scaled lasso regression, we introduce some important notation, including the lower-RE condition on the gram matrix $\frac{{\mathbf{W}}^T\mathbf{W}}{n}$. The oracle estimator θ^ora of the noise level can be defined as

$${\theta }^{\mathit{\text{ora}}}=\frac{\Vert \mathbf{R}-\mathbf{W}{d}^{\mathit{\text{true}}}\Vert }{\sqrt{n}}=\frac{\Vert \mathbf{E}\Vert }{\sqrt{n}}.$$ (63)

Let |K| be the cardinality of an index set K. Define T as the index set of those large coefficients of d^true,

$$T=\{k:|d_k^{\mathit{\text{true}}}|\ge \lambda \}.$$ (64)

We say the gram matrix $\frac{{\mathbf{W}}^T\mathbf{W}}{n}$ satisfies a lower-RE condition with curvature α₁ > 0 and tolerance ζ (n, p₀) > 0 if

$${\mu }^T\frac{{\mathbf{W}}^T\mathbf{W}}{n}\mu \ge {\alpha }_1{\Vert \mu \Vert }^2-\zeta (n,p_0){\Vert \mu \Vert }_1^2\text{for all}\mu \in \mathbb{R}.$$ (65)

Moreover, we define

$$\nu ={\Vert {\mathbf{W}}^T(\mathbf{R}-\mathbf{W}{d}^{\mathit{\text{true}}})/n\Vert }_{\infty }={\Vert {\mathbf{W}}^T\mathbf{E}/n\Vert }_{\infty },$$ (66)

$$\tau =s{\lambda }^2\cdot A_2\frac{2\xi }{(1+\xi )}{C}_1(\frac{3{\theta }^{\mathit{\text{ora}}}}{2},4\xi A_1,{\alpha }_1),$$ (67)

with constants ξ > 1, A₁ and A₂ introduced in Lemma 5. C₁( $\frac{3{\theta }^{\mathit{\text{ora}}}}{2}$, 4ξ A₁, α₁) is a constant depending on $\frac{3{\theta }^{\mathit{\text{ora}}}}{2}$, ξA₁ and α₁ with its definition in Equation (76). It is bounded above if $\frac{3{\theta }^{\mathit{\text{ora}}}}{2}$ and 4ξA₁ are bounded above and α₁ is bounded below by some universal constants, respectively. With the notation we can state the key lemma as follows.

Lemma 5

Consider the scaled l₁ penalized regression procedure (57). Whenever there exist constants ξ > 1, A₁ and A₂ such that the following conditions are satisfied

1. $\nu \le {\theta }^{\mathit{\text{ora}}}\lambda \frac{\xi -1}{\xi +1}(1-\tau )$ for some ξ > 1 and τ ≤ 1/2 defined in (67);

2. $\frac{\Vert {\mathbf{D}}_k\Vert }{\sqrt{n}}\in [1/A_1,A_1]$ for all k;

3. θ^ora ∈ [1/A₂,A₂];

4. $\frac{{\mathbf{W}}^T\mathbf{W}}{n}$ satisfies the lower-RE condition with α₁ and ζ (n, p₀) such that

   $$s\zeta (n,p_0)8{(1+\xi )}^2{A}_1^2\le min\{\frac{{\alpha }_1}{2},1\},$$ (68)
   we have the following deterministic bounds

   $$|\widehat{\theta }-{\theta }^{\mathit{\text{ora}}}|\le C_1{\lambda }^{2}s,$$ (69)

   $${\Vert \widehat{b}-b^{\mathit{\text{true}}}\Vert }^2\le C_2{\lambda }^{2}s,$$ (70)

   $${\Vert \widehat{b}-b^{\mathit{\text{true}}}\Vert }_1\le C_3\lambda s,$$ (71)

   $${\Vert \mathbf{D}(\widehat{b}-b^{\mathit{\text{true}}})\Vert }^2/n\le C_4{\lambda }^{2}s,$$ (72)

   $${\Vert {\mathbf{D}}^T\mathbf{E}/n\Vert }_{\infty }\le C_5\lambda ,$$ (73)

   where constants C_i(i = 1, …, 5) only depend on A₁, A₂, α₁ and ξ.

Figure 2.

The ROC curves for different methods. For GLASSO or cGLASSO, the ROC curve is obtained by varying its tuning parameter. For CAPME, λ₁ is fixed as the value selected by the cross validation and the ROC curve is obtained by varying λ₂. For ANTAC, the ROC curve is obtained by varying the cut-off on P-values.

Figure 5.

The precision-recall curves for “Magnified Block” model and “Heterogeneous Product” model using different methods. For GLASSO or cGLASSO, the curve is obtained by varying its tuning parameter. For CAPME, λ₁ is fixed as the value selected by the cross validation and the curve is obtained by varying λ₂. For ANTAC, the precision-recall curve is obtained by varying threshold level ξ₀. The points on the curves correspond to the results obtained by cross-validation for GLASS, cGLASS and CAPME and by using theoretical threshold level ξ₀ = 2 for tuning free ANTAC.

9. Appendix

9.1 Proof of Lemma 5

The function L_λ (d, θ) in Equation (61) is jointly convex in (d, θ). For fixed θ > 0, denote the minimizer of L_λ (d, θ) over all d ∈ ℝ^po by d̂ (θλ), a function of θλ, i.e.,

$$\widehat{d}(\theta \lambda )=arg\underset{d\in {\mathbb{R}}^{p_0}}{min}{L}_{\lambda }(d,\theta )=arg\underset{d\in {\mathbb{R}}^{p_0}}{min}\{\frac{{\Vert \mathbf{R}-\mathbf{W}d\Vert }^2}{2n}+\lambda \theta {\Vert d\Vert }_1\},$$ (74)

then if we knew θ̂ in the solution of Equation (62), the solution for the equation is {d̂ (θ̂λ), θ̂}. We recognize that d̂ (θ̂λ) is just the standard lasso with the penalty θ̂λ, however we don't know the estimator θ̂. The strategy of our analysis is that we first show that θ̂ is very close to its oracle estimator θ^ora, then the standard lasso analysis would imply the desired result Equations (70)-(73) under the assumption that θ̂/θ^ora = 1 + O(λ²s). For the standard lasso analysis, some kind of regularity condition is assumed on the design matrix W^TW/n in the regression literature. In this paper we use the lower-RE condition, which is one of the most general conditions.

Let μ = λθ. From the Karush-Kuhn-Tucker condition, d̂ (μ) is the solution to the Equation (74) if and only if

$${\mathbf{W}}_k^T(\mathbf{R}-\mathbf{W}\widehat{d}(\mu ))/n=\mu \cdot \mathit{\text{sgn}}({\widehat{d}}_k(\mu )),\text{if}{\widehat{d}}_k(\mu )\ne 0,{\mathbf{W}}_k^T(\mathbf{R}-\mathbf{W}\widehat{d}(\mu ))/n\in [-\mu ,\mu ],\text{if}{\widehat{d}}_k(\mu )=0.$$ (75)

Let C₂ (a₁, a₂) and C₁ (a₁₁, a₁₂, a₂) be constants depending on a₁, a₂ and a₁₁, a₁₂, a₂, respectively. The constant C₂ is bounded above if a₁ is bounded above and a₂ is bounded below by constants, respectively. The constant C₁ is bounded above whenever a₁₁ and a₁₂ are bounded above and a₂ is bounded below by constants. The explicit formulas of C₁ and C₂ are given as follows,

$$C_2(a_1,a_2)=\frac{a_1}{a_2}+{[{\left(\frac{a_1}{a_2}\right)}^2+\frac{2(a_1+1)}{a_2}]}^{1/2},C_1(a_{11},a_{12},a_2)=a_{12}(1+C_2(a_{11}\times a_{12},a_2)).$$ (76)

The following propositions are helpful to establish our result. The proof is given in Sections 9.2 and 9.3.

Proposition 2

The sparsity s is defined in Equation (58). For any ξ > 1, assuming $\nu \le \mu \frac{\xi -1}{\xi +1}$ and conditions 2, 4 in Lemma 5 hold, we have

$${\Vert \widehat{d}(\mu )-d^{\mathit{\text{true}}}\Vert }_1\le C_1(\frac{\mu }{\lambda },4\xi A_1,{\alpha }_1)\lambda s,$$ (77)

$$\Vert \widehat{d}(\mu )-d^{\mathit{\text{true}}}\Vert \le C_2(4\xi A_1\frac{\mu }{\lambda },{\alpha }_1)\lambda \sqrt{s},$$ (78)

$$\frac{1}{n}{\Vert \mathbf{W}(d^{\mathit{\text{true}}}-\widehat{d}(\mu ))\Vert }^2\le (\nu +\mu ){\Vert \widehat{d}(\mu )-d^{\mathit{\text{true}}}\Vert }_1.$$ (79)

Proposition 3

Let {d̂, θ̂} be the solution of the scaled lasso (62). For any ξ > 1, assuming conditions 1 − 4 in Lemma 5 hold, then we have

$$|\frac{\widehat{\theta }}{{\theta }^{\mathit{\text{ora}}}}-1|\le \tau .$$ (80)

Now we finish our proof with these two propositions. According to Conditions 1 − 4 in Lemma 5, Proposition 3 implies $\nu \le \mu \frac{\xi -1}{\xi +1}$ with μ = λθ̂ and Proposition 2 further implies that there exist some constants c₁, c₂ and c₃ such that

$${\Vert \widehat{d}(\lambda \widehat{\theta })-d^{\mathit{\text{true}}}\Vert }_1\le c_1\lambda s,\Vert \widehat{d}(\lambda \widehat{\theta })-d^{\mathit{\text{true}}}\Vert \le c_2\lambda \sqrt{s},\frac{{\Vert \mathbf{W}(d^{\mathit{\text{true}}}-\widehat{d}(\lambda \widehat{\theta }))\Vert }^2}{n}\le c_3{\lambda }^{2}s.$$

Note that $\frac{\mu }{\lambda }=\widehat{\theta }\in [{\theta }^{\mathit{\text{ora}}}(1-\tau ),{\theta }^{\mathit{\text{ora}}}(1+\tau )]\subset [\frac{1}{2A_2},\frac{3A_2}{2}]$. Thus the constants c₁, c₂ and c₃ only depends on A₁, A₂, α₁ and ξ. Now we transfer the results above on standardized scaled lasso (62) back to the general scaled lasso (57) through the bounded scaling constants $\left\{\frac{\sqrt{n}}{\Vert {\mathbf{D}}_k\Vert }\right\}$ and immediately have the desired results (70)-(72). Result (69) is an immediate consequence of Proposition 3 and Result (73) is an immediate consequence of assumptions 1 − 3.

9.2 Proof of Proposition 2

Notice that

$$\frac{1}{n}{\Vert \mathbf{W}(d^{\mathit{\text{true}}}-\widehat{d}(\mu ))\Vert }^2=\frac{{(d^{\mathit{\text{true}}}-\widehat{d}(\mu ))}^T({\mathbf{W}}^T(\mathbf{R}-\mathbf{W}\widehat{d}(\mu ))-{\mathrm{W}}^T(\mathbf{R}-\mathbf{W}{d}^{\mathit{\text{true}}}))}{n}\le \mu ({\Vert d^{\mathit{\text{true}}}\Vert }_1-{\Vert \widehat{d}(\mu )\Vert }_1)+\nu {\Vert d^{\mathit{\text{true}}}-\widehat{d}(\mu )\Vert }_1$$ (81)

$$\le (\mu +\nu ){\Vert {(d^{\mathit{\text{true}}}-\widehat{d}(\mu ))}_T\Vert }_1+2\mu {\Vert {(d^{\mathit{\text{true}}})}_{T^c}\Vert }_1-(\mu -\nu ){\Vert {(d^{\mathit{\text{true}}}-\widehat{d}(\mu ))}_{T^c}\Vert }_1,$$ (82)

where the first inequality follows from the KKT conditions (75).

Now define Δ = d̂(μ) − d^true. Equation (81) also implies the desired inequality (79)

$${\mathrm{\Delta }}^T\frac{{\mathbf{W}}^T\mathbf{W}}{n}\mathrm{\Delta }\le (\mu +\nu ){\Vert \mathrm{\Delta }\Vert }_1$$ (83)

We will first show that

$${\Vert {\mathrm{\Delta }}_{T^c}\Vert }_1\le max\{2(1+\xi ){\Vert {(d^{\mathit{\text{true}}})}_{T^c}\Vert }_1,2\xi {\Vert {\mathrm{\Delta }}_T\Vert }_1\},$$ (84)

then we are able to apply the lower-RE condition (65) to derive the desired results.

To show Equation (84), we note that our assumption $\nu \le \mu \frac{\xi -1}{\xi +1}$ with Equation (82) implies

$$\frac{1}{n}{\Vert \mathbf{W}\mathrm{\Delta }\Vert }^2\le 2\mu (\frac{\xi {\Vert {\mathrm{\Delta }}_T\Vert }_1}{\xi +1}+{\Vert {(d^{\mathit{\text{true}}})}_{T^c}\Vert }_1-\frac{{\Vert {\mathrm{\Delta }}_{T^c}\Vert }_1}{\xi +1}).$$ (85)

Suppose that

$${\Vert {\mathrm{\Delta }}_{T^c}\Vert }_1\ge 2(1+\xi ){\Vert {(d^{\mathit{\text{true}}})}_{T^c}\Vert }_1,$$ (86)

then the inequality (85) becomes

$$0\le \frac{1}{n}{\Vert \mathbf{W}\mathrm{\Delta }\Vert }^2\le \frac{\mu }{\xi +1}(2\xi {\Vert {\mathrm{\Delta }}_T\Vert }_1-{\Vert {\mathrm{\Delta }}_{T^c}\Vert }_1),$$

which implies

$${\Vert {\mathrm{\Delta }}_{T^c}\Vert }_1\le 2\xi {\Vert {\mathrm{\Delta }}_T\Vert }_1.$$ (87)

Therefore the complement of inequality (86) and Equation (87) together finish our proof of Equation (84).

Before proceeding, we point out two facts which will be used below several times. Note the sparseness s is defined in terms of the true coefficients b^true in Equation (58) before standardization but the index set T is defined in term of d^true in Equation (64) after standardization. Condition 2 implies that this standardization step doesn't change the sparseness up to a factor A₁. Hence it's not hard to see that |T| ≤ A₁s and ‖(d^true)_T^c‖₁ ≤ A₁λs.

Now we are able to apply the lower-RE condition of $\frac{{\mathbf{W}}^T\mathbf{W}}{n}$ to Equation (83) and obtain that

$${\mathrm{\Delta }}^T\frac{{\mathbf{W}}^T\mathbf{W}}{n}\mathrm{\Delta }\ge {\alpha }_1{\Vert \mathrm{\Delta }\Vert }^2-\zeta (n,p_0){\Vert \mathrm{\Delta }\Vert }_1^2\ge {\alpha }_1{\Vert \mathrm{\Delta }\Vert }^2-\zeta (n,p_0)8{(1+\xi )}^2({\Vert {(d^{\mathit{\text{true}}})}_{T^c}\Vert }_1^2+{\Vert {\mathrm{\Delta }}_T\Vert }_1^2)\ge ({\alpha }_1-|T|\zeta (n,p_0)8{(1+\xi )}^2){\Vert \mathrm{\Delta }\Vert }^2-\zeta (n,p_0)8{(1+\xi )}^2{A}_1^2{\lambda }^2{s}^2\ge \frac{{\alpha }_1}{2}{\Vert \mathrm{\Delta }\Vert }^2-{\lambda }^{2}s,$$

where in the second, third and last inequalities we applied the acts (84), ${\Vert {\mathrm{\Delta }}_T\Vert }_1^2\le |T|{\Vert {\mathrm{\Delta }}_T\Vert }^2$, ‖T‖ ≤ A₁s, ‖(d^true)_T^c‖₁ ≤ A₁λs and $s\zeta (n,p_0)8{(1+\xi )}^2{A}_1^2\le min\{\frac{{\alpha }_1}{2},1\}$. Moreover, by applying those facts used in last equation again, we have

$$(\mu +\nu ){\Vert \mathrm{\Delta }\Vert }_1\le 4\xi \mu ({\Vert {(d^{\mathit{\text{true}}})}_{T^c}\Vert }_1+{\Vert {\mathrm{\Delta }}_T\Vert }_1)\le 4\xi \mu (A_1\lambda s+\sqrt{|T|}\Vert \mathrm{\Delta }\Vert )\le 4\xi A_1\frac{\mu }{\lambda }({\lambda }^{2}s+\sqrt{s}\lambda \Vert \mathrm{\Delta }\Vert ).$$

The above two inequalities together with Equation (83) imply that

$$4\xi A_1\frac{\mu }{\lambda }({\lambda }^{2}s+\sqrt{s}\lambda \Vert \mathrm{\Delta }\Vert )\ge \frac{{\alpha }_1}{2}{\Vert \mathrm{\Delta }\Vert }^2-{\lambda }^{2}s.$$

Define $S_u=4\xi A_1\frac{\mu }{\lambda }$. Some algebra about this quadratic inequality implies the bound of Δ under l₂ norm

$$\Vert \mathrm{\Delta }\Vert \le (\frac{S_u}{{\alpha }_1}+{[{\left(\frac{S_u}{{\alpha }_1}\right)}^2+\frac{2(S_u+1)}{{\alpha }_1}]}^{1/2})\lambda \sqrt{s}\equiv C_2(4\xi A_1\frac{\mu }{\lambda },{\alpha }_1)\lambda \sqrt{s}.$$

Combining this fact with Equation (84), we finally obtain the bound under l₁ norm (77)

$${\Vert \mathrm{\Delta }\Vert }_1\le {\Vert {\mathrm{\Delta }}_T\Vert }_1+{\Vert {\mathrm{\Delta }}_{T^c}\Vert }_1\le 2(1+\xi )({\Vert {(d^{\mathit{\text{true}}})}_{T^c}\Vert }_1+{\Vert {\mathrm{\Delta }}_T\Vert }_1)\le 2(1+\xi )(A_{1}s\lambda +\sqrt{|T|}\Vert \mathrm{\Delta }\Vert )\le 4\xi A_1(1+C_2(4\xi A_1\frac{\mu }{\lambda },{\alpha }_1\left)\right)s\lambda \equiv C_1(\frac{\mu }{\lambda },4\xi A_1,{\alpha }_1)s\lambda .$$

9.3 Proof of Proposition 3

For τ defined in Equation (67), we need to show that θ̂ ≥ θ^ora (1 − τ) and θ̂ ≤ θ^ora (1 + τ) on the event $\{\nu \le {\theta }^{\mathit{\text{ora}}}\lambda \frac{\xi -1}{\xi +1}(1-\tau )\}$. Let d̂ (θλ) be the solution of (74) as a function of θ, then

$$\frac{\partial }{\partial \theta }{L}_{\lambda }(\widehat{d}(\theta \lambda ),\theta )=\frac{1}{2}-\frac{{\Vert \mathbf{R}-\mathbf{W}\widehat{d}(\theta \lambda )\Vert }^2}{2n{\theta }^2},$$ (88)

since ${\{\frac{\partial }{\partial d}{L}_{\lambda }(d,\theta ){|}_{d=\widehat{d}(\theta \lambda )}\}}_k=0$ for all d̂_k (θλ) ≠ 0, and ${\{\frac{\partial }{\partial \theta }\widehat{d}(\theta \lambda )\}}_k=0$ for all d̂_k (θλ) = 0 which follows from the fact that {k : d̂_k (θλ) = 0 } is unchanged in a neighborhood of θ for almost all θ. Equation (88) plays a key in the proof.

(1). To show that θ̂ ≥ θ^ora (1 − τ) it's enough to show

$$\frac{\partial }{\partial \theta }{L}_{\lambda }(\widehat{d}(\theta \lambda ),\theta ){|}_{\theta =t_1}<0,$$

where t₁ = θ^ora (1 − τ), due to the strict convexity of the objective function L_λ (d, θ) in θ. Equation (88) implies that

$$2t_1^2\frac{\partial }{\partial \theta }{L}_{\lambda }(\widehat{d}(\theta \lambda ),\theta ){|}_{\theta =t_1}=t_1^2-\frac{{\Vert \mathbf{R}-\mathbf{W}\widehat{d}(t_1\lambda )\Vert }^2}{n}\le t_1^2-\frac{{\Vert \mathbf{R}-\mathbf{W}{d}^{\mathit{\text{true}}}+\mathbf{W}(\widehat{d}(t_1\lambda )-d^{\mathit{\text{true}}})\Vert }^2}{n}\le t_1^2-{({\theta }^{\mathit{\text{ora}}})}^2+2{(d^{\mathit{\text{true}}}-\widehat{d}(t_1\lambda ))}^T\frac{{\mathbf{W}}^T(\mathbf{R}-\mathbf{W}{d}^{\mathit{\text{true}}})}{n}\le 2t_1(t_1-{\theta }^{\mathit{\text{ora}}})+2\nu {\Vert d^{\mathit{\text{true}}}-\widehat{d}(t_1\lambda )\Vert }_1.$$ (89)

On the event $\{\nu \le t_1\lambda \frac{\xi -1}{\xi +1}\}=\{\nu /{\theta }^{\mathit{\text{ora}}}<\lambda \frac{\xi -1}{\xi +1}(1-\tau )\}$ we have

$$2t_1^2\frac{\partial }{\partial \theta }{L}_{\lambda }(\widehat{d}(\theta \lambda ),\theta ){|}_{\theta =t_1}\le 2t_1(t_1-{\theta }^{\mathit{\text{ora}}})+2t_1\lambda \frac{\xi -1}{\xi +1}{\Vert d^{\mathit{\text{true}}}-\widehat{d}(t_1\lambda )\Vert }_1\le 2t_1[-\tau {\theta }^{\mathit{\text{ora}}}+\lambda \frac{\xi -1}{\xi +1}{\Vert d^{\mathit{\text{true}}}-\widehat{d}(t_1\lambda )\Vert }_1]<0.$$

The last inequality follows from the definition of τ and the l₁ error bound in Equation (77) of Proposition 2. Note that for λ²s sufficiently small, we have small τ < 1/2. In fact, although ‖d^true − d̂ (t₁λ)‖₁ also depends on τ, our choice of τ is well-defined and is larger than $\frac{\lambda }{{\theta }^{\mathit{\text{ora}}}}\frac{3(\xi -1)}{2(\xi +1)}{\Vert d^{\mathit{\text{true}}}-\widehat{d}(t_1\lambda )\Vert }_1$.

(2). Let t₂ = θ^ora (1 + τ). To show the other side θ̂ ≥ θ^ora (1 + τ) it is enough to show

$$\frac{\partial }{\partial \theta }{L}_{\lambda }(\widehat{d}(\theta \lambda ),\theta ){|}_{\theta =t_2}>0.$$

Equation (88) implies that on the event $\{\nu \le t_2\lambda \frac{\xi -1}{\xi +1}\}=\{\nu /{\theta }^{\mathit{\text{ora}}}<\lambda \frac{\xi -1}{\xi +1}(1+\tau )\}$ we have

$$2t_2^2\frac{\partial }{\partial \theta }{L}_{\lambda }(\widehat{d}(\theta \lambda ),\theta ){|}_{\theta =t_2}=t_2^2-\frac{{\Vert \mathbf{R}-\mathbf{W}\widehat{d}(t_2\lambda )\Vert }^2}{n}=t_2^2-{({\theta }^{\mathit{\text{ora}}})}^2+{({\theta }^{\mathit{\text{ora}}})}^2-\frac{{\Vert \mathbf{R}-\mathbf{W}\widehat{d}(t_2\lambda )\Vert }^2}{n}=t_2^2-{({\theta }^{\mathit{\text{ora}}})}^2+\frac{{\Vert \mathbf{R}-\mathbf{W}{d}^{\mathit{\text{true}}}\Vert }^2-{\Vert \mathbf{R}-\mathbf{W}\widehat{d}(t_2\lambda )\Vert }^2}{n}=t_2^2-{({\theta }^{\mathit{\text{ora}}})}^2+\frac{{(\widehat{d}(t_2\lambda )-d^{\mathit{\text{true}}})}^T{\mathbf{W}}^T(\mathbf{R}-\mathbf{W}{d}^{\mathit{\text{true}}}+\mathbf{R}-\mathbf{W}\widehat{d}(t_2\lambda ))}{n}\ge t_2^2-{({\theta }^{\mathit{\text{ora}}})}^2-{\Vert \widehat{d}(t_2\lambda )-d^{\mathit{\text{true}}}\Vert }_1(\nu +t_2\lambda )\ge (t_2+{\theta }^{\mathit{\text{ora}}}){\theta }^{\mathit{\text{ora}}}\tau -\frac{2\xi }{\xi +1}{t}_2\lambda {\Vert \widehat{d}(t_2\lambda )-d^{\mathit{\text{true}}}\Vert }_1\ge 2{\theta }^{\mathit{\text{ora}}}(\tau {\theta }^{\mathit{\text{ora}}}-\frac{3\xi }{2(\xi +1)}\lambda {\Vert \widehat{d}(t_2\lambda )-d^{\mathit{\text{true}}}\Vert }_1)>0,$$

where the second last inequality is due to the fact τ < 1/2 and the last inequality follows from the definition of τ and the l₁ error bound in Equation (77) of Proposition 2. Still, our choice of τ is well-defined and is larger than $\frac{\lambda }{{\theta }^{\mathit{\text{ora}}}}\frac{4\xi }{2(\xi +1)}{\Vert d^{\mathit{\text{true}}}-\widehat{d}(t_2\lambda )\Vert }_1$.