Application of fused graphical lasso to statistical inference for multiple sparse precision matrices

Abstract

In this paper, the fused graphical lasso (FGL) method is used to estimate multiple precision matrices from multiple populations simultaneously. The lasso penalty in the FGL model is a restraint on sparsity of precision matrices, and a moderate penalty on the two precision matrices from distinct groups restrains the similar structure across multiple groups. In high-dimensional settings, an oracle inequality is provided for FGL estimators, which is necessary to establish the central limit law. We not only focus on point estimation of a precision matrix, but also work on hypothesis testing for a linear combination of the entries of multiple precision matrices. We apply a de-biasing technology, which is used to obtain a new consistent estimator with known distribution for implementing the statistical inference, and extend the statistical inference problem to multiple populations. The corresponding de-biasing FGL estimator and its asymptotic theory are provided. A simulation study and an application of the diffuse large B-cell lymphoma data show that the proposed test works well in high-dimensional situation.

Introduction

Undirected graphical models are popular tools for representing the network structure of data and have been widely applied in many domains, such as machine learning, gene pattern recognition, and financial data analysis. Letting x = (x₁, …, x^p)^T be a p-variate normal random vector with mean vector μ and covariance Σ₀ (Σ₀ is positive definite), the precision matrix (or concentration matrix) is denoted the inverse of the covariance matrix, i.e., ${\Theta }_0≔{\Sigma }_0^{-1}$. The graphical models capture conditional dependence relationships between random variables via non-zero entries in a precision matrix. If Θ_0ij ≠ 0, xⁱ and x^j, i, j = 1, …, p are dependent on each other, given all other variables. Meanwhile, the zero entries in the precision matrix correspond to pairs of variables that are conditionally independent given other variables. Therefore, the graph model is closely related to the precision matrix. The research of estimating and testing of a precision matrix have been a rapidly growing research direction in the past few years.

Letting x₁, …, x_n be a sequence of independent and identically distributed (i.i.d.) observations from the population x, X_p×n ≔ (x₁, …, x_n). A natural estimator of the precision matrix is the inverse of the sample covariance matrix $\widehat{\Sigma }$, where $\widehat{\Sigma }=\frac{1}{n}{\mathbf{\text{X}}}^T\mathbf{\text{X}}$. On the one hand, in high-dimensional settings, Johnstone [1] proposed that the eigenvalues of the sample covariance matrix do not converge to the corresponding eigenvalue of the population covariance matrix for Σ = I. Consequently, this estimator becomes invalid when the dimension p is comparable to the sample size n. On the other hand, the sample covariance matrix is singular in a p > n − 1 setting. This will produce non-negligible errors in using ${\widehat{\Sigma }}_n^{-1}$ to estimate Θ₀. In addition, a sparse (i.e., many entries are either zero or nearly so) assumption for a high-dimensional precision matrix is essential, since the zero entries imply the conditional independence structures, which are what we are most concerned with in the graphical model. In general, ${\widehat{\Sigma }}_n^{-1}$ does not have a sparsity construction. How to estimate the sparse precision matrix in high-dimensional settings is an intractable problem.

In recent years, various proposals have been put forward for estimating a precision matrix in high-dimensional situations, among which the graphical model with sparsity-promoting penalties is valid for obtaining a sparse estimator. By applying the l₁ (lasso penalty) to the entries of the concentration matrix, Yuan and Lin [2] proposed a max-det algorithm to obtain the estimator of Θ₀. The convergence result of the estimator is derived under a p fixed assumption. Using a coordinate descent procedure, Friedman et al. [3] provided an algorithm for solving a graphical Lasso estimator that is remarkably fast, even if p > n. Rothman et al. [4] investigated a sparse permutation invariant covariance estimator, and established a convergence rate of the estimator in the Frobenius norm as both data dimension p and sample size n are allowed to grow, and showed that the rate explicitly depends on how sparse the true concentration matrix is. For additional theoretical details on penalized likelihood methods for graphical models, see Fan et al. [5], Ravikumar et al. [6], Xue and Zou [7], and Yuan et al. [8].

The above-mentioned methods focus on estimating a single graphical model. However, joint estimators perform better in recovering the truth graphs for multiple graphical models, when graphs sharing the similar structure. Guo et al. [9] studied joint estimation of precision matrices that have a hierarchical structure assumption. Zhang et al. [10] proposed a new joint group lasso penalty to restore the joint graphical model. Their method was applied for multiple gene networks data with several subpopulations and data types. A fused graphical lasso was proposed by Danaher et al. [11] with a penalty imposing a similar structure of a precision matrix across groups. Supposing that ${\mathbf{\text{X}}}_{p\times n_k}^{\left[k\right]}≔({\mathbf{\text{x}}}_1^{\left[k\right]},\dots ,{\mathbf{\text{x}}}_{n_k}^{\left[k\right]})$ are sample matrices, and ${\mathbf{\text{x}}}_i^{\left[k\right]}\in R^p(i=1,\dots ,n_k)$ are sampled i.i.d. from a distribution with mean μ^[k] and covariance ${\Sigma }_0^{\left[k\right]}$, for k = 1, …, K, we assume μ^[k] = 0 without loss of generality. To simplify notation, we omit the subscript of ${\mathbf{\text{X}}}_{p\times n_k}^{\left[k\right]}$, and denote the sample matrices as X^[k]. The population precision matrix is defined as the inverse of the population covariance matrix, i.e., ${\Theta }_0^{\left[k\right]}={\left({\Sigma }_0^{\left[k\right]}\right)}^{-1}$. The estimators of precision matrices $\left\{{\Theta }_0^{\left[k\right]}\right\}$ are investigated by minimizing the negative penalized log likelihood

$$\begin{array}{c}\hfill \left\{{\widehat{\Theta }}^{\left[k\right]}\right\}={\text{arg}\text{min}}_{\{{\Theta }^{\left[k\right]}\in {\mathcal{S}}^{++}\}}\sum _k\{tr\left({\widehat{\Sigma }}^{\left[k\right]}{\Theta }^{\left[k\right]}\right)-\text{log}\text{det}\left({\Theta }^{\left[k\right]}\right)\}+\mathbf{\text{P}}\left(\left\{{\Theta }^{\left[k\right]}\right\}\right),\end{array}$$ (1)

where P({Θ^[k]}) denotes the penalty function, the $\left\{{\widehat{\Theta }}^{\left[k\right]}\right\}$ are the minimizers of (1), and we optimize over the symmetric positive-definite matrices set ${\mathcal{S}}^{++}$. The fused graphical lasso (FGL) is the solution to optimization problem (1) with the fused lasso penalty

$$\begin{array}{c}\hfill \mathbf{\text{P}}\left(\left\{{\Theta }^{\left[k\right]}\right\}\right)=\lambda \sum _{k=1}^K\left|\right|{\left({\Theta }^{\left[k\right]}\right)}^{-}{\left|\right|}_1+\rho \sum _{k<k^{\prime }}\left|\right|{({\Theta }^{\left[k\right]}-{\Theta }^{\left[k^{\prime }\right]})}^{-}{\left|\right|}_1,\end{array}$$ (2)

where λ and ρ are non-negative regularization parameters, (Θ^[k])⁻ represents the matrix obtained by setting the diagonal elements of (Θ^[k]) to zero, and || ⋅ ||₁ denotes the l₁ norm of a vector or matrix. It is reasonable to restrict non-diagonal elements of Θ^[k], since we are most concerned with the conditional independence cross-different variables. Note that the first term in (2) is the classical lasso penalty, which shrinks the coefficients toward 0 as λ increases. It guarantees discovery of the sparse estimators $\left\{{\widehat{\Theta }}^{\left[k\right]}\right\}$ of the model. The penalty on (Θ^[k] − Θ^[k′])⁻ indicates that the elements of ${\widehat{\Theta }}^{\left[1\right]},\dots ,{\widehat{\Theta }}^{\left[K\right]}$ have a similar network structure across classes.

An approach for the estimation of the joint graphical models largely relies on penalized estimation. The penalty biases the estimates toward the assumed structure, which makes hypothesis tests for precision matrices more challenging. Work on statistical inference for low-dimensional parameters in graphical models has recently been carried out (Janková and van de Geer [12]; Janková and van de Geer [13]; Ren et al. [14]; Yu et al. [15]) based on the l₁-penalized estimator. Janková and van de Geer [12] provided a de-biasing technique to obtain a new consistent estimator with known distribution. However, these approaches were developed only in the setting in which the parameters of one graph are inferred. In contrast, studies of inference techniques using estimators obtained from cross-group penalization are much fewer. The work on statistical inference for multiple graphical models is an interesting area open for future research. Inspired by Janková and van de Geer [12], we not only give FGL estimators of multiple precision matrices from co-movement data, but also test the linear combination of the entries of these precision matrices. The core of the proposed method is based on the de-biasing technique, and we implement statistical inference of the precision matrices under high-dimensional settings according to the proposed central limit theorem.

The rest of this paper is organized as follows. In Main results section, we give the oracle inequality for multiple estimators with a FGL penalty and its weighted version. Testing the hypothesis for the linear combination of corresponding entries of multiple precision matrices is also considered in this section. Based on de-biasing technology, the CLT of the proposed statistics for multiple populations is also derived in this section. In Numerical study part, we report the results of simulations. In Real Data Application, we apply the proposed method to identification of gene-to-gene interaction of the diffuse large B-cell lymphoma data. All technical details are relegated to the Proof of Theorem part.

Main results

We assume following notation throughout the paper. For a matrix $A={\left(a_{ij}\right)}_{i,j=1}^p$, we denote (A)_ij as (i, j)-entry of A, or denote its (i, j)-entry as A_ij to simplify the notation. We write $\left|A\right|$ for the determinant of A, and the trace of matrix A is denoted tr(A). Letting A⁺ = diag(A) for a diagonal matrix with the same diagonal as A, A⁻ = A − A⁺. ${\left|\right|A\left|\right|}_F^2={\sum }_{i,j}{a}_{ij}^2$ denotes the Frobenius norm (also known as the matrix 2-norm). We use the notation ||A||_∞ = max_i,j|a_ij| for the supremum norm of a matrix A, and |||A|||₁ ≔ max_j ∑_i|a_ij| for the l₁-operator norm.

We write $f\left(n\right)=\mathcal{O}\left(g\right(n\left)\right)$ if f(n) ≤ cg(n) for some constant c < ∞, and f(n) = Ω(g(n)) if f(n) ≥ c′g(n) for some constant c′ > 0. The notation f(n) ≍ g(n) means that $f\left(n\right)=\mathcal{O}\left(g\right(n\left)\right)$ and f(n) = Ω(g(n)). In the common high-dimensional setting, the dimension p is allowed to grow to infinity. The dimension is comparable, substantially larger or smaller than the sample size. We set sample sizes n₁ ≍ … ≍ n_K ≍ n throughout the paper, and n* = n₁ + … + n_K going to infinity. Furthermore, for notational simplicity, we assume that n₁ = … = n_K = n.

Oracle inequality

To obtain the oracle inequality of multiple estimators of FGL models, we introduce some notation related to the sparsity assumptions on the entries of the true precision matrix. Letting

$$\begin{array}{c}\hfill S_k≔\{(i,j):{\Theta }_{0ij}^{\left[k\right]}\ne 0,i\ne j\},\end{array}$$

where ${\Theta }_{0ij}^{\left[k\right]}$ is the (i, j)-entry of ${\Theta }_0^{\left[k\right]}$ and s_k = |S_k| is the cardinality of S_k, we adopt the boundedness of the eigenvalues of the true precision matrix and certain tail conditions proposed by Janková and Van De Geer [12].

Condition 1 (Bounded eigenvalues) There exist universal constants L for k such that

$$\begin{array}{c}\hfill 0<L<{\Lambda }_{\text{min}}\left({\Theta }_0^{\left[k\right]}\right)\le {\Lambda }_{\text{max}}\left({\Theta }_0^{\left[k\right]}\right)<1/L<\infty ,\end{array}$$

where Λ_min and Λ_max denote the minimum and maximum eigenvalues of a matrix, respectively.

Condition 2 (Sub-Gaussianity vector condition) The observations ${\mathbf{\text{x}}}_i^{\left[k\right]}$ , i = 1, …, n_k, are uniformly sub-Gaussian vectors in the respective groups.

We propose the oracle inequality for FGL lasso under the K = 2 situation.

Theorem 1 Supposing that Conditions 1 and 2 hold, for k = 1, 2, tuning parameter λ satisfying 2(ρ + λ₀) ≤ λ ≤ c/8L, and $\frac{8{\lambda }^2(s_1+s_2)}{c}+\frac{4p{\lambda }_0^2}{c}\le {\lambda }_0/2L$. On the set $\left\{{\text{max}}_k\right||{\widehat{\Sigma }}^{\left[k\right]}-{\Sigma }_0^{\left[k\right]}|{|}_{\infty }\le {\lambda }_0\}$, k = 1, 2, it holds that

$$\begin{array}{c}\hfill c\sum _{k=1}^2\left|\right|{\widehat{\Theta }}^{\left[k\right]}-{\Theta }_0^{\left[k\right]}{\left|\right|}_F^2+\lambda \sum _{k=1}^2\left|\right|{({\widehat{\Theta }}^{\left[k\right]}-{\Theta }_0^{\left[k\right]})}^{-}{\left|\right|}_1\le \frac{8{\lambda }^2(s_1+s_2)}{c}+\frac{4p{\lambda }_0^2}{c},\end{array}$$

and

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \sum _{k=1}^2\left|\right||{\widehat{\Theta }}^{\left[k\right]}-{\Theta }_0^{\left[k\right]}{\left|\right||}_1\le \frac{4\lambda (8s_1+8s_2+p)}{c},\end{array}\end{array}$$

where c = 1/(8L²).

Remark 1 From the inequality, we must select λ so that λp → 0 as n → ∞ to ensure consistency, which is not satisfied by a sub-Gaussianity random vector. Thus, the condition λp → 0 excludes the p ≫ n situation.

The FGL does not take into account that the variables have, in general, different scaling. Thus, we consider the weighted FGL. The minimizer of the optimization problem (1) with weighted FGL penalty

$$\begin{array}{c}\hfill \mathbf{\text{P}}\left(\left\{{\Theta }^{\left[k\right]}\right\}\right)=\lambda \sum _k\sum _{i\ne j}{\widehat{W}}_{ii}^{\left[k\right]}{\widehat{W}}_{jj}^{\left[k\right]}|{\Theta }_{ij}^{\left[k\right]}|+\rho \sum _{k<k^{\prime }}\sum _{i\ne j}|{\widehat{W}}_{ii}^{\left[k\right]}{\widehat{W}}_{jj}^{\left[k\right]}{\Theta }_{ij}^{\left[k\right]}-{\widehat{W}}_{ii}^{\left[k^{\prime }\right]}{\widehat{W}}_{jj}^{\left[k^{\prime }\right]}{\Theta }_{ij}^{\left[k^{\prime }\right]}|\end{array}$$ (3)

is denoted $\left\{{\widehat{\Theta }}_w^{\left[k\right]}\right\}$, where ${\widehat{W}}^{\left[k\right]}=\left[diag\right({\widehat{\Sigma }}^{\left[k\right]}){]}^{\frac{1}{2}}$. Further, the population correlation matrix is denoted $R_0^{\left[k\right]}$ and the sample correlation matrix is denoted

$$\begin{array}{c}\hfill {\widehat{R}}^{\left[k\right]}={\left({\widehat{W}}^{\left[k\right]}\right)}^{-1}{\widehat{\Sigma }}^{\left[k\right]}{\left({\widehat{W}}^{\left[k\right]}\right)}^{-1}.\end{array}$$

If we substitute ${\widehat{R}}^{\left[k\right]}$ for ${\widehat{\Sigma }}^{\left[k\right]}$, the minimizer of

$$\begin{array}{c}\hfill {\text{arg}\text{min}}_{\{{\Theta }^{\left[k\right]}\in {\mathcal{S}}^{++}\}}\sum _k\{tr\left({\widehat{R}}^{\left[k\right]}{\Theta }^{\left[k\right]}\right)-\text{log}\text{det}\left({\Theta }^{\left[k\right]}\right)\}+\mathbf{\text{P}}\left(\left\{{\Theta }^{\left[k\right]}\right\}\right)\end{array}$$ (4)

with a FGL penalty (2) is denoted $\left\{{\widehat{\Theta }}_R^{\left[k\right]}\right\}$, which is a matter of estimating the parameter by the normalized data. Then,

$$\begin{array}{c}\hfill {\widehat{\Theta }}_R^{\left[k\right]}={\widehat{W}}^{\left[k\right]}{\widehat{\Theta }}_w^{\left[k\right]}{\widehat{W}}^{\left[k\right]},\end{array}$$

which means, essentially, that ${\widehat{\Theta }}_R^{\left[k\right]}$ are the estimators of ${\Theta }_{R0}^{\left[k\right]}≔(R_0^{\left[k\right]}{)}^{-1}$.

Theorem 2 Under the conditions of Theorem 1, on the set $\left\{{\text{max}}_k\right||{\widehat{R}}^{\left[k\right]}-R_0^{\left[k\right]}|{|}_{\infty }\le {\lambda }_0\}$ , k = 1, 2, it holds that

$$\begin{array}{c}\hfill \begin{array}{c}\hfill c\sum _{k=1}^2\left|\right|{\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]}{\left|\right|}_F^2+\lambda \sum _{k=1}^2\left|\right|{({\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]})}^{-}{\left|\right|}_1\le \frac{8{\lambda }^2(s_1+s_2)}{c},\end{array}\end{array}$$ (5)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \sum _{k=1}^2\left|\right||{\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]}{\left|\right||}_1\le \frac{32\lambda (s_1+s_2)}{c},\end{array}\end{array}$$ (6)

and

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \sum _{k=1}^2\left|\right||{\widehat{\Theta }}_w^{\left[k\right]}-{\Theta }_0^{\left[k\right]}{\left|\right||}_1\le \frac{32\lambda (s_1+s_2)}{c}.\end{array}\end{array}$$ (7)

It is natural to extend this conclusion to the K > 2 FGL model. For k = 1, …, K and the K > 2 situation, we obtain the following theorem.

Theorem 3 (Multiple FGL model) Supposing that Conditions 1 and 2 hold, for K > 2, $2(\frac{K(K-1)}{2}\rho +{\lambda }_0)\le \lambda \le c/8L$, and $\frac{8{\lambda }^2{\sum }_{k=1}^K{s}_k}{c}+\frac{2Kp{\lambda }_0^2}{c}\le {\lambda }_0/2L$, on the set $\left\{{\text{max}}_k\right||{\widehat{\Sigma }}^{\left[k\right]}-{\Sigma }_0^{\left[k\right]}|{|}_{\infty }\le {\lambda }_0\}$, k = 1, …, K, it holds that

$$\begin{array}{c}\hfill \begin{array}{c}\hfill c\sum _{k=1}^K\left|\right|{\widehat{\Theta }}^{\left[k\right]}-{\Theta }_0^{\left[k\right]}{\left|\right|}_F^2+\lambda \sum _{k=1}^K\left|\right|{({\widehat{\Theta }}^{\left[k\right]}-{\Theta }_0^{\left[k\right]})}^{-}{\left|\right|}_1\le \frac{8{\lambda }^2{\sum }_{k=1}^K{s}_k}{c}+\frac{2Kp{\lambda }_0^2}{c}\end{array}\end{array}$$ (8)

and

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \sum _{k=1}^K\left|\right||{\widehat{\Theta }}^{\left[k\right]}-{\Theta }_0^{\left[k\right]}{\left|\right||}_1\le \frac{2K\lambda (8{\sum }_{k=1}^K{s}_k+\frac{Kp}{2})}{c}.\end{array}\end{array}$$ (9)

Theorem 4 (Multiple FGL model for weighted version) Under the conditions of Theorem 3, on the set $\left\{{\text{max}}_k\right||{\widehat{R}}^{\left[k\right]}-R_0^{\left[k\right]}|{|}_{\infty }\le {\lambda }_0\}$ , k = 1, 2, it holds that

$$\begin{array}{c}\hfill \begin{array}{c}\hfill c\sum _{k=1}^K\left|\right|{\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]}{\left|\right|}_F^2+\lambda \sum _{k=1}^K\left|\right|{({\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]})}^{-}{\left|\right|}_1\le \frac{8{\lambda }^2{\sum }_{k=1}^K{s}_k}{c},\end{array}\end{array}$$ (10)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \sum _{k=1}^K\left|\right||{\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]}{\left|\right||}_1\le \frac{16K\lambda {\sum }_{k=1}^K{s}_k}{c},\end{array}\end{array}$$ (11)

and

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \sum _{k=1}^K\left|\right||{\widehat{\Theta }}_w^{\left[k\right]}-{\Theta }_0^{\left[k\right]}{\left|\right||}_1\frac{16K\lambda {\sum }_{k=1}^K{s}_k}{c}.\end{array}\end{array}$$ (12)

Asymptotic property

We not only focus on the point estimation of multiple precision matrices, but also on hypothesis testing for the linear combination of the entries of the precision matrices over two groups. One may want to test whether the elements of the precision matrix over two groups are equal:

$$\begin{array}{c}\hfill H_0:{\Theta }_{0ij}^{\left[1\right]}={\Theta }_{0ij}^{\left[2\right]}vs.H_1:{\Theta }_{0ij}^{\left[1\right]}\ne {\Theta }_{0ij}^{\left[2\right]}.\end{array}$$ (13)

To test Hypothesis (13), we aim to obtain confidence intervals for estimators based on the de-biasing technique, which is imposed for eliminating the bias associated with the penalty. The de-biasing estimator is defined as ${\widehat{\Theta }}_d^{\left[k\right]}=2{\widehat{\Theta }}^{\left[k\right]}-{\widehat{\Theta }}^{\left[k\right]}{\widehat{\Sigma }}^{\left[k\right]}{\widehat{\Theta }}^{\left[k\right]}$. The difference between the de-biasing estimator and the true value can be decomposed into two parts as follows:

$$\begin{array}{c}\hfill {\widehat{\Theta }}_d^{\left[k\right]}-{\Theta }_0^{\left[k\right]}={\Xi }^{\left[k\right]}+{\Upsilon }^{\left[k\right]},\end{array}$$

where

$$\begin{array}{cc}& {\Xi }^{\left[k\right]}=-{\Theta }_0^{\left[k\right]}({\widehat{\Sigma }}^{\left[k\right]}-{\Sigma }_0^{\left[k\right]}){\Theta }_0^{\left[k\right]},\\ & {\Upsilon }^{\left[k\right]}=-({\widehat{\Theta }}^{\left[k\right]}-{\Theta }_0^{\left[k\right]})({\widehat{\Sigma }}^{\left[k\right]}-{\Sigma }_0^{\left[k\right]}){\Theta }_0^{\left[k\right]}-({\widehat{\Theta }}^{\left[k\right]}-{\Theta }_0^{\left[k\right]})({\widehat{\Sigma }}^{\left[k\right]}{\widehat{\Theta }}^{\left[k\right]}-{\mathbf{\text{I}}}_p).\end{array}$$

Under the compatibility conditions, Janková and van de Geer [16] proposed that the (i, j)-entry of ${\widehat{\Theta }}_d^{\left[k\right]}-{\Theta }_0^{\left[k\right]}$ has an asymptotic normality property, and $\sqrt{n}\left|\right|{\Upsilon }^{\left[k\right]}{\left|\right|}_{\infty }$ converges to zero in probability. Thus, for testing Hypothesis (13), we construct the testing statistic

$$\begin{array}{c}\hfill T_{ij}≔({\widehat{\Theta }}_d^{\left[1\right]}-{\widehat{\Theta }}_d^{\left[2\right]}{)}_{ij}=[2{\widehat{\Theta }}^{\left[1\right]}-{\widehat{\Theta }}^{\left[1\right]}{\widehat{\Sigma }}^{\left[1\right]}{\widehat{\Theta }}^{\left[1\right]}-(2{\widehat{\Theta }}^{\left[2\right]}-{\widehat{\Theta }}^{\left[2\right]}{\widehat{\Sigma }}^{\left[2\right]}{\widehat{\Theta }}^{\left[2\right]}){]}_{ij}\end{array}$$ (14)

using de-biasing estimators.

For K = 2, we let

$$\begin{array}{c}\hfill s=\text{max}\{s_1,s_2\},d=\text{max}\{d_1,d_2\},\end{array}$$

where

$$\begin{array}{c}\hfill d_k=\underset{j=1,\dots ,p}{\text{max}}\left|D_j^{\left[k\right]}\right|,D_j^{\left[k\right]}=\{(i,j):{\Theta }_{0ij}^{\left[k\right]}\ne 0,i\ne j\}.\end{array}$$

Next, we establish the central limit theorem for T_ij.

Theorem 5 Assuming Conditions 1, 2, and $\lambda \asymp \rho \asymp \sqrt{\text{log}p/n}$ and $(p+s)\sqrt{d}=o(\sqrt{n}/\text{log}p)$, it holds that

$$\begin{array}{c}\hfill {\widehat{\Theta }}_d^{\left[1\right]}-{\widehat{\Theta }}_d^{\left[2\right]}-({\Theta }_0^{\left[1\right]}-{\Theta }_0^{\left[2\right]})={\Xi }^{\left[1\right]}-{\Xi }^{\left[2\right]}+rem,\end{array}$$ (15)

where

$$\begin{array}{c}\hfill {\left|\right|rem\left|\right|}_{\infty }=\left|\right|{\Upsilon }^{\left[1\right]}-{\Upsilon }^{\left[2\right]}{\left|\right|}_{\infty }=o_p(1/\sqrt{n}),\end{array}$$ (16)

and o_p denotes the convergence in probability. Moreover,

$$\begin{array}{c}\hfill \sqrt{n}[T_{ij}-{\mathbf{\Theta }}_{0ij}]{\to }_{D}N(0,{\sigma }_{ij}^2),\end{array}$$ (17)

where ${\mathbf{\Theta }}_{0ij}={({\Theta }_0^{\left[1\right]}-{\Theta }_0^{\left[2\right]})}_{ij}$.

To complete the testing procedure, we use the consistent estimator ${\widehat{\sigma }}_{ij}^2={\left({\widehat{\Theta }}^{\left[1\right]}\right)}_{ii}{\left({\widehat{\Theta }}^{\left[1\right]}\right)}_{jj}+{\left({\widehat{\Theta }}^{\left[1\right]}\right)}_{ij}^2+{\left({\widehat{\Theta }}^{\left[2\right]}\right)}_{ii}{\left({\widehat{\Theta }}^{\left[2\right]}\right)}_{jj}+{\left({\widehat{\Theta }}^{\left[2\right]}\right)}_{ij}^2$ for Theorem 5. Theorem 5 provide a practical and efficient way of obtaining the p value and critical value for the test statistic. Under a null hypothesis, we observe that ${\Theta }_{0ij}^{\left[1\right]}-{\Theta }_{0ij}^{\left[2\right]}=0$. For an α level of significance, we reject H₀ if $|\sqrt{n}{T}_{ij}/{\widehat{\sigma }}_{ij}^2|>{\xi }_{\alpha /2}$, where ξ_α is the 1 − α upper quantile of the standard normal distribution.

Theorem 5 requires a stronger sparsity condition than the corresponding oracle-type inequality in Theorem 1. According to the convergence rate of $(p+s)\sqrt{d}$, Theorem 5 applies to the p ≪ n situation. For p ≫ n, we provide the following theorem.

Theorem 6 Assuming Conditions 1, 2, and $\lambda \asymp \rho \asymp \sqrt{\text{log}p/n}$ and $s\sqrt{d}=o(\sqrt{n}/\text{log}p)$ , for the p ≪ n regime, the Eq (22) holds with ${\widehat{\Theta }}_w^{\left[k\right]}$, where

$$\begin{array}{c}\hfill {\left|\right|rem\left|\right|}_{\infty }=o_p(1/\sqrt{n}).\end{array}$$ (18)

In addition,

$$\begin{array}{c}\hfill \sqrt{n}[T_{wij}-{\mathbf{\Theta }}_{0ij}]{\to }_{D}N(0,{\sigma }_{ij}^2),\end{array}$$ (19)

where $T_{wij}={(2{\widehat{\Theta }}_w^{\left[1\right]}-{\widehat{\Theta }}_w^{\left[1\right]}{\widehat{\Sigma }}^{\left[1\right]}{\widehat{\Theta }}_w^{\left[1\right]})}_{ij}-{(2{\widehat{\Theta }}_w^{\left[2\right]}-{\widehat{\Theta }}_w^{\left[2\right]}{\widehat{\Sigma }}^{\left[2\right]}{\widehat{\Theta }}_w^{\left[2\right]})}_{ij}$.

We do not need to impose the so-called irrepresentability condition on Σ to derive the theoretical properties of our estimators, in contrast to Brownlees et al. [17].

In addition, for the multi-sample precision matrix hypothesis problem, one may want to test a linear hypothesis testing problem:

$$\begin{array}{c}\hfill H_0:a_1{\Theta }_{0ij}^{\left[1\right]}+\dots +a_K{\Theta }_{0ij}^{\left[K\right]}=0vs.H_1:\text{not}{H}_0,\end{array}$$ (20)

where a₁, …, a_K are known constants. Similar to the two-sample case, we proposed the test statistic

$$\begin{array}{c}\hfill a_1{\widehat{\Theta }}_{dij}^{\left[1\right]}+\dots +a_K{\widehat{\Theta }}_{dij}^{\left[K\right]}.\end{array}$$ (21)

For the K > 2 multiple situation, we assume s = max{s₁, …, s_K} and d = max{d₁, …, d_K}. Consequently, we establish the asymptotic normality of the proposed statistic in the following corollary, i.e., Corollary 1.

Corollary 1 Under the assumptions of Theorem 5, it holds that

$$\begin{array}{c}\hfill f({\widehat{\Theta }}_d^{\left[1\right]},\dots ,{\widehat{\Theta }}_d^{\left[K\right]})-f({\Theta }_0^{\left[1\right]},\dots ,{\Theta }_0^{\left[K\right]})=f({\Xi }^{\left[1\right]},\dots ,{\Xi }^{\left[K\right]})+rem,\end{array}$$ (22)

$$\begin{array}{c}\hfill {\left|\right|rem\left|\right|}_{\infty }=\left|\right|f({\Upsilon }^{\left[1\right]},\dots ,{\Upsilon }^{\left[K\right]}){\left|\right|}_{\infty }=o_p(1/\sqrt{n}),\end{array}$$ (23)

where f(x₁, …, x_K) = a₁x₁ + … + a_Kx_K. In addition,

$$\begin{array}{c}\hfill \sqrt{n}[T_{ij}-{\mathbf{\Theta }}_{0ij}]{\to }_{D}N(0,{\sigma }_{ij}^2),\end{array}$$ (24)

where $T_{ij}=f({\widehat{\Theta }}_{dij}^{\left[1\right]},\dots ,{\widehat{\Theta }}_{dij}^{\left[K\right]})$ and ${\mathbf{\Theta }}_{0ij}=f({\Theta }_{0ij}^{\left[1\right]},\dots ,{\Theta }_{0ij}^{\left[K\right]})$.

The asymptotic variance σ_ij in Corollary 1 is unknown, so to construct confidence intervals we use a consistent estimator

$$\begin{array}{c}\hfill {\widehat{\sigma }}_{ij}^2=f_v\left(\right[{\left({\widehat{\Theta }}^{\left[1\right]}\right)}_{ii}{\left({\widehat{\Theta }}^{\left[1\right]}\right)}_{jj}+{\left({\widehat{\Theta }}^{\left[1\right]}\right)}_{ij}^2],\dots ,[{\left({\widehat{\Theta }}^{\left[K\right]}\right)}_{ii}{\left({\widehat{\Theta }}^{\left[K\right]}\right)}_{jj}+{\left({\widehat{\Theta }}^{\left[K\right]}\right)}_{ij}^2\left]\right),\end{array}$$

where $f_v(x_1,\dots ,x_K)=a_1^2{x}_1+\dots +a_K^2{x}_K$. In addition, a weighted version is proposed as follows.

Corollary 2 Under the assumptions of Theorem 6, the residual term in (23) converges in probability with rate $1/\sqrt{n}$, and CLT in (24) holds by replacing ${\widehat{\Theta }}^{\left[k\right]}$ by ${\widehat{\Theta }}_w^{\left[k\right]}$, which is obtained by solving the weighted FGL optimization problem.

Numerical study

Simulation experiments were carried out to evaluate the performance of the proposed de-biasing FGL test. We considered the sparse graphical model, and a random sample was generated from the multivariate normal distribution $N(0_p,{\left({\Theta }_0^{\left[k\right]}\right)}^{-1})$ with a population covariance matrix defined as the inverse of the population precision matrix.

To solve the graphical lasso problem with a certain penalty, we refer to the alternating direction method of multiplier (ADMM) algorithm, since it is guaranteed to converge to the global optimum. For more details, the reader is referred to Boyd et al. [18] and Danaher et al. [11]. When an objective method for selecting tuning parameters λ and ρ is required, the approximations of the Akaike information criterion (AIC), Bayesian information criterion, or cross-validation method can be used to select tuning parameters. The AIC method was chosen for the following simulation, and λ and ρ both range from 0.05 to 0.3 with a step of 0.0086, where the step is derived by (0.3 − 0.05)/(30 − 1).

In addition, all the reported simulation results are based on 500 simulations with a nominal significance level of 0.05, and we set the dimension to 100.

Fluctuations of test

We illustrated the theoretical asymptotic normality result on simulated data for testing the two-sample problem (13), and we set precision matrices equal under a null hypothesis, i.e., ${\Theta }_0^{\left[1\right]}={\Theta }_0^{\left[2\right]}$.

Letting G be a p × p symmetric graph matrix with diagonal entries 0 and $\tilde{\alpha }$ percent of off-diagonal elements 1, and U be p × p matrix with elements i.i.d. generated from the uniformly distribution on the interval (0, 1), i.e., U(0, 1), we denote the elements of the symmetric matrix $\tilde{\mathbf{\Theta }}$ as ${\tilde{\theta }}_{ij}$. For i > j,

$$\begin{array}{c}\hfill {\tilde{\theta }}_{ij}=\frac{g_{ij}{u}_{ij}+g_{ji}{u}_{ji}}{2}-{\mathbf{1}}_{\{\frac{g_{ij}{u}_{ij}+g_{ji}{u}_{ji}}{2}<0.5\}},\end{array}$$ (25)

where g_ij and u_ij are the (i, j)-entry of G and U, respectively, and 1_{·} is the indicator function. For i < j, we set ${\tilde{\theta }}_{ij}={\tilde{\theta }}_{ji}$. The diagonal entries of matrix $\tilde{\mathbf{\Theta }}$ are zeros. Then, the precision matrix is generated as

$$\begin{array}{c}\hfill {\Theta }_0^{\left[k\right]}=\tilde{\mathbf{\Theta }}+\left(|{\Lambda }_{\text{min}}\left(\tilde{\mathbf{\Theta }}\right)|+0.1\right){\mathbf{\text{I}}}_p.\end{array}$$ (26)

This shows that the matrix generated is symmetric and positive definite. To make the non-zero entries go away from 0 and to generate a sparse matrix, we subtract 1 from the non-zero elements. In addition, the precision matrix generation procedure shows that $\tilde{\alpha }$ is a parameter controlling the sparsity. When $\tilde{\alpha }=1$, a dense matrix is generated. As is well known, the sparsity of a matrix not only requires a small quantity of non-zero elements, but also a large absolute value of non-zero elements. The parameter $\tilde{\alpha }$ controls sparsity in terms of the number of sparse elements.

We examined the fluctuation of $\sqrt{n}{T}_{ij}/{\widehat{\sigma }}_{ij}$ under (p, n) = (100, 200) and (p, n) = (100, 400) settings for the extremely sparse and dense precision matrix cases, respectively. For the extremely sparse precision matrix case, we set the parameter $\tilde{\alpha }=0.01$, and for dense case we use $\tilde{\alpha }=1$.

We simulated the fluctuation for the extremely sparse case as shown in Fig 1 and the dense case in Fig 2. The index (i, j) in the simulation was intermittently chosen. In fact, the CLT provides the method for testing any element of the linear combination of the precision matrix. Theoretically, we can test for any index (i, j)-entry of Θ₀ whether the true value is zero or not.

Fig 1. The fluctuation for two-sample case with sparse precision matrix.

Histogram of $\sqrt{n}{T}_{ij}/{\widehat{\sigma }}_{ij}$ for $\tilde{\alpha }=0.01$. Here, T_(i,j) = T_ij and ${\widehat{\sigma }}_{(i,j)}={\widehat{\sigma }}_{ij}$. The setting is (p, n) = (100, 200) with (i, j) ∈ {(1, 1), (1, 30), (1, 60), (1, 90)} for four graphs in the first line. The sample size and dimension were set as (p, n) = (100, 400) for four graphs in the second line.

Fig 2. The fluctuation for two-sample case with dense precision matrix.

Histogram of $\sqrt{n}{T}_{ij}/{\widehat{\sigma }}_{ij}$ for $\tilde{\alpha }=1$. Here, T_(i,j) = T_ij and ${\widehat{\sigma }}_{(i,j)}={\widehat{\sigma }}_{ij}$. The setting is (p, n) = (100, 200) with (i, j) ∈ {(1, 1), (1, 30), (1, 60), (1, 90)} for four graphs in the first line. The sample size and dimension were set to (p, n) = (100, 400) for four graphs in the second line.

Average coverage probabilities

We demonstrate the performance of the test method for the K = 2 situation on testing the hypothesis as follows.

• Equal Null. Testing hypothesis (13);

• Linear Null. Testing the linear null hypothesis $H_0:a_1{\Theta }_{0ij}^{\left[1\right]}+a_2{\Theta }_{0ij}^{\left[2\right]}=0$, i.e., $H_0:{\Theta }_{0ij}^{\left[2\right]}=-\frac{a_1}{a_2}{\Theta }_{0ij}^{\left[1\right]}$. Without loss generation, we chose $-\frac{a_1}{a_2}=0.5$ and ${\Theta }_{0ij}^{\left[1\right]}$ generated from (26).

From the global perspective, we used the average coverage, which is also considered in Janková and van de Geer [12]. Letting

$$\begin{array}{c}\hfill I_{ij}≔[T_{ij}-1.96\frac{{\sigma }_{ij}}{\sqrt{n}},T_{ij}+1.96\frac{{\sigma }_{ij}}{\sqrt{n}}]\end{array}$$ (27)

be the 95% asymptotic confidence interval for Θ_0ij, we substitute the estimator ${\widehat{\sigma }}_{ij}$ for σ_ij to obtain the empirical version. The frequency of the true value being covered by the confidence interval (27) is defined as ${\widehat{\vartheta }}_{ij}$. Then, the average coverage over a set A is denoted

$$\begin{array}{c}\hfill Avgco{v}_A=\frac{1}{\left|A\right|}\sum _{(i,j)\in A}{\widehat{\vartheta }}_{ij}.\end{array}$$ (28)

S denotes the set of non-zero entries of ${\Theta }_{0ij}^{\left[1\right]}$. It is easy to check that S = S₁ = S₂ for the reason that ${\Theta }_{0ij}^{\left[1\right]}$ and ${\Theta }_{0ij}^{\left[2\right]}$ have same structure of sparsity for the Equal Null and Linear Null cases. Thus, for the different null hypotheses, we simulated the average coverage over S and its complementary set S^c. The parameter of sparsity is $\tilde{\alpha }=0.1,0.5,$ and 0.9.

Partial results in Table 1 meet our expectation. However, we do not deny that the simulations are affected by randomness. In addition, the proposed method is based on the combination of estimation and hypothesis testing, which accumulates error. The simulation results provide guidance for practice.

Table 1. Estimated average coverage probabilities for K = 2 situation.

2*$\tilde{\alpha }$	2*n	Equal Null		Linear Null
		S	S c	S	S c
2*0.1	200	0.9886	0.9875	0.9101	0.9824
	400	0.9885	0.9867	0.8607	0.9762
2*0.5	200	0.9880	0.9878	0.9384	0.9745
	400	0.9870	0.9868	0.8820	0.9647
2*0.9	200	0.9901	0.9899	0.9509	0.9751
	400	0.9889	0.9890	0.9091	0.9639

Multiple FGL case

For the multiple FGL case, we examined the fluctuation of the statistic T_ij for the K = 3 situation on testing the hypothesis as follows.

• Three-sample Linear Null. Testing hypothesis $H_0:{\Theta }_{0ij}^{\left[3\right]}=-\frac{a_1}{a_3}{\Theta }_{0ij}^{\left[1\right]}-\frac{a_2}{a_3}{\Theta }_{0ij}^{\left[2\right]}$, where $-\frac{a_1}{a_3}=0.6$ and $-\frac{a_2}{a_3}=0.9$ are both generated from U(0, 1). ${\Theta }_{0ij}^{\left[1\right]}$ and ${\Theta }_{0ij}^{\left[2\right]}$ are both generated from (26) with parameters 0.01 and 0.1, respectively.

We set $-\frac{a_1}{a_3}$ and $-\frac{a_2}{a_3}$ to positive numbers, since the setting of hypothesis testing should guarantee that ${\left\{{\Theta }_{0ij}^{\left[k\right]}\right\}}_{k=1}^3$ are symmetric positive-definite matrices. Besides, for Three-sample Linear Null, S denotes the set of non-zero entries of ${\Theta }_{0ij}^{\left[1\right]}+\frac{a_2}{a_3}{\Theta }_{0ij}^{\left[2\right]}$. The dimension and sample size are (p, n) = (100, 200) and (p, n) = (100, 400), respectively. Histograms of the proposed statistic T_ij at the

$$\begin{array}{c}\hfill (i,j)\in \left\{\right(1,1),(1,10),(1,20),(1,30\left)\right\}\end{array}$$

locations of the precision matrix are presented in Fig 3.

Fig 3. The fluctuation for multiple-sample case with dense precision matrix.

Histogram of $\sqrt{n}{T}_{ij}/{\widehat{\sigma }}_{ij}$ for $\tilde{\alpha }=1$. Here, T_(i,j) = T_ij and ${\widehat{\sigma }}_{(i,j)}={\widehat{\sigma }}_{ij}$. The setting is (p, n) = (100, 200) with (i, j) ∈ {(1, 1), (1, 10), (1, 20), (1, 30)} for four graphs in the first line. The sample size and dimension were set to (p, n) = (100, 400) for four graphs in the second line.

Real data application

The lymphoma is a malignant tumor with increasing incidence and mortality year by year. In this part, we apply the proposed method to two sets of diffuse large B-cell lymphoma (DLBCL) data, denoted DLBCL-A [19] and DLBCL-B [20], which is available at http://portals.broadinstitute.org/cgibin/cancer/datasets.cgi. Some brief information on these datasets can be found in Table 2. The DLBCL-A and DLBCL-B datasets have 3 subgroups, and the label and sample size of each subgroup are shown in the 5th column in Table 2. Both DLBCL-A dataset and DLBCL-B dataset have a high dimension with 662 genes but only a few observations with the sample size 141 for the DLBCL-A dataset and 180 for the DLBCL-B dataset.

Table 2. Brief introduction to the gene profile expression datasets.

Dataset	n	p	Subgroups	Subgroup label (sample size)
DLBCL-A	141	662	3	I (49), II (50), III (42)
DLBCL-B	180	662	3	I (42), II (51), III (87)

Typically, we test for differences in mean vectors across different disease subgroups, however, the role of gene-to-gene interactions in the data across different subtypes remains unclear. In this section, we use our test approach to identify whether the gene-to-gene interactions that controls lymphoma most behave the same across different disease subtypes. For distinct gene subtypes of the same disease gene data, we focus on testing the equality of two precision matrices. The hypothesis testing problem is

$$\begin{array}{c}\hfill H_0:{\Theta }_{0ij}^{typei}={\Theta }_{0ij}^{typej}vs.H_1:{\Theta }_{0ij}^{typei}\ne {\Theta }_{0ij}^{typej}\end{array}$$

where type i and type i are chosen from I, II, III set in Table 2 and type i ≠ type j. We tune parameters with weighted FGL penalty in (3) by AIC criterion. After the tuning procedure, we estimate precision matrices, and then return a p × p matrix, whose (i, j)-th elements are p-value of statistic T_ij. The results are demonstrated in Figs 4 and 5.

Fig 4. The p-values of proposed test for DLBCL-A dataset.

P-values of T_ij by comparing subtype I and subtype II (left), subtype II and subtype III (middle), and subtype I and subtype III (right) with DLBCL-A dataset.

Fig 5. The p-values of proposed test for DLBCL-B dataset.

P-values of T_ij by comparing subtype I and subtype II (left), subtype II and subtype III (middle), and subtype I and subtype III (right) with DLBCL-B dataset.

As can be seen in the figure, the interactions between genes of DLBCL-A dataset are not the same among three different subtypes, while for DLBCL-B dataset, the interactions between genes of three different subtypes are mostly similar.

Proof of theorem

Proof of Theorem 1

To prove Theorem 1, we need a lemma of Janková and Van de Geer [16], which is present as follow.

Lemma 7 Let f(Δ) ≔ tr(ΔΣ₀) − [log det(Δ + Θ₀) − log det(Θ₀)]. Assume that 1/L ≤ λ_min(Θ₀) ≤ λ_max(Θ₀) ≤ L for some constant L ≥ 1. Then for all Δ such that ||Δ||_F ≤ 1/(2L), f(Δ) is well defined and

$$\begin{array}{c}\hfill f(\Delta )\ge \frac{1}{2{(L+1/\left(2L\right))}^2}{\left|\right|\Delta \left|\right|}_F^2.\end{array}$$

To simplify the notation, we substitute ${\widehat{\Sigma }}_k$, Σ_0k, ${\widehat{\Theta }}_k$, Θ_0k for ${\widehat{\Sigma }}^{\left[k\right]}$, ${\Sigma }_0^{\left[k\right]}$, ${\widehat{\Theta }}^{\left[k\right]}$, ${\Theta }_0^{\left[k\right]}$ respectively.

Proof 1 Note that ${\widehat{\Theta }}_k$ is the minimum value of the fused graphical Lasso for k = 1, 2. Let ${\tilde{\Theta }}_k={\alpha }_k{\widehat{\Theta }}_k+(1-{\alpha }_k){\Theta }_{0k}$, and ${\alpha }_k=\frac{M}{M+\left|\right|{\widehat{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F}$. According to the definitions of ${\tilde{\Theta }}_k$, and the convexity of loss function

$$\begin{array}{ccc}& & F_n({\Theta }_1,{\Theta }_2)\hfill \\ & =& tr\left({\Theta }_1{\widehat{\Sigma }}_1\right)-\text{log}\text{det}\left({\Theta }_1\right)+tr\left({\Theta }_2{\widehat{\Sigma }}_2\right)-\text{log}\text{det}\left({\Theta }_2\right)\hfill \\ & & +\lambda \left|\right|{\Theta }_1^{-}{\left|\right|}_1+\lambda \left|\right|{\Theta }_2^{-}{\left|\right|}_1+\rho \left|\right|{\Theta }_1^{-}-{\Theta }_2^{-}{\left|\right|}_1,\hfill \end{array}$$

we obtain

$$\begin{array}{c}\hfill F_n({\tilde{\Theta }}_1,{\tilde{\Theta }}_2)\le F_n({\Theta }_{01},{\Theta }_{02}).\end{array}$$

That is

$$\begin{array}{ccc}& & \sum _{k=1}^2\{tr({\tilde{\Theta }}_k-{\Theta }_{0k}){\widehat{\Sigma }}_k-(\text{log}\text{det}\left({\tilde{\Theta }}_k\right)-\text{log}\text{det}\left({\Theta }_{0k}\right))+\lambda \left|\right|{\tilde{\Theta }}_k^{-}{\left|\right|}_1\}+\rho \left|\right|{\tilde{\Theta }}_1^{-}-{\tilde{\Theta }}_2^{-}{\left|\right|}_1\hfill \\ & \le & \lambda \left|\right|{\Theta }_{01}^{-}{\left|\right|}_1+\lambda \left|\right|{\Theta }_{02}^{-}{\left|\right|}_1+\rho \left|\right|{\Theta }_{01}^{-}-{\Theta }_{02}^{-}{\left|\right|}_1.\hfill \end{array}$$ (29)

Let ${\Delta }_k={\tilde{\Theta }}_k-{\Theta }_{0k}$ , and

$$\begin{array}{c}\hfill f\left({\Delta }_k\right)≔tr\left({\Delta }_k{\Sigma }_{0k}\right)-[\text{log}\text{det}({\Delta }_k+{\Theta }_{0k})-\text{log}\text{det}\left({\Theta }_{0k}\right)],\end{array}$$

subtracting $tr\left({\Delta }_1({\widehat{\Sigma }}_1-{\Sigma }_{01})\right)+tr\left({\Delta }_2({\widehat{\Sigma }}_2-{\Sigma }_{02})\right)$ on the both sides of the inequality (29), we get

$$\begin{array}{c}\hfill \begin{array}{cc}& f\left({\Delta }_1\right)+f\left({\Delta }_2\right)+\lambda \left|\right|{\tilde{\Theta }}_1^{-}{\left|\right|}_1+\lambda \left|\right|{\tilde{\Theta }}_2^{-}{\left|\right|}_1+\rho \left|\right|{\tilde{\Theta }}_1^{-}-{\tilde{\Theta }}_2^{-}{\left|\right|}_1\hfill \\ \hfill \le & -tr\left({\Delta }_1({\widehat{\Sigma }}_1-{\Sigma }_{01})\right)-tr\left({\Delta }_2({\widehat{\Sigma }}_2-{\Sigma }_{02})\right)+\lambda \left|\right|{\Theta }_{01}^{-}{\left|\right|}_1+\lambda \left|\right|{\Theta }_{02}^{-}{\left|\right|}_1+\rho \left|\right|{\Theta }_{01}^{-}-{\Theta }_{02}^{-}{\left|\right|}_1.\hfill \end{array}\end{array}$$ (30)

For $tr\left({\Delta }_k({\widehat{\Sigma }}_k-{\Sigma }_{0k})\right)$ term, we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \left|tr\right({\Delta }_k({\widehat{\Sigma }}_k-{\Sigma }_{0k})\left)\right|& =\left|G\right({\Delta }_k\circ ({\widehat{\Sigma }}_k-{\Sigma }_{0k})\left)\right|\hfill \\ & \le |G({\Delta }_k^{-}\circ ({\widehat{\Sigma }}_k^{-}-{\Sigma }_{0k}^{-}))|+|G({\Delta }_k^{+}\circ ({\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}))|,\hfill \end{array}\end{array}$$

where function G(M) takes the summation of all the elements of the matrix M, and ∘ is Hadamard product. According to Cauchy-Schwarz inequality, on the sets $\{{\text{max}}_k\left\{\right||{\widehat{\Sigma }}_k-{\Sigma }_{0k}{\left|\right|}_{\infty }\}\le {\lambda }_0\}$,

$$\begin{array}{c}\hfill \begin{array}{cc}& |G({\Delta }_k^{-}\circ ({\widehat{\Sigma }}_k^{-}-{\Sigma }_{0k}^{-}))|+|G({\Delta }_k^{+}\circ ({\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}))|\hfill \\ & \le \left|\right|{\widehat{\Sigma }}_k^{-}-{\Sigma }_{0k}^{-}{\left|\right|}_{\infty }\left|\right|{\Delta }_k^{-}{\left|\right|}_1+\left|\right|{\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}{\left|\right|}_F\left|\right|{\Delta }_k^{+}{\left|\right|}_F\hfill \\ & \le {\lambda }_0\left|\right|{\Delta }_k^{-}{\left|\right|}_1+\left|\right|{\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}{\left|\right|}_F\left|\right|{\Delta }_k^{+}{\left|\right|}_F.\hfill \end{array}\end{array}$$

Hence,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill -tr\left({\Delta }_k({\widehat{\Sigma }}_k-{\Sigma }_{0k})\right)& \le \left|tr\right({\Delta }_k({\widehat{\Sigma }}_k-{\Sigma }_{0k})\left)\right|\hfill \\ & \le {\lambda }_0\left|\right|{\Delta }_k^{-}{\left|\right|}_1+\left|\right|{\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}{\left|\right|}_F\left|\right|{\Delta }_k^{+}{\left|\right|}_F.\hfill \end{array}\end{array}$$ (31)

Next, for L_k ≥ 1 satisfying condition

$$\begin{array}{c}\hfill 1/L_k\le {\lambda }_{min}\left({\Theta }_{0k}\right)\le {\lambda }_{max}\left({\Theta }_{0k}\right)\le L_k,\end{array}$$

we choose L > 1 satisfying 1/L ≤ 1/L_k and L_k ≤ L, k = 1, 2. Based on the definitions of Δ_k and ${\tilde{\Theta }}_k$, we get

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left|\right|{\Delta }_k{\left|\right|}_F={\alpha }_k\left|\right|{\widehat{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F=\frac{\left|\right|{\widehat{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F}{M+\left|\right|{\widehat{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F}M,\end{array}\end{array}$$ (32)

for arbitrary M in (0, 1/2L]. Thus, ||Δ_k||_F is bounded by M, i.e., ||Δ_k||_F ≤ M. For f(Δ_k) term, based on Lemma 7, we have

$$\begin{array}{c}\hfill f\left({\Delta }_k\right)\ge c\left|\right|{\tilde{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F^2,\end{array}$$ (33)

where $c=\frac{1}{2{(L+1/\left(2L\right))}^2}$. In particular, we choose c = 1/(8L²), and the inequality (33) still holds.

Using bounds (31) and (33), the inequality (30) turns to be

$$\begin{array}{c}\hfill \begin{array}{cc}& c\left|\right|{\tilde{\Theta }}_1-{\Theta }_{01}{\left|\right|}_F^2+c\left|\right|{\tilde{\Theta }}_2-{\Theta }_{02}{\left|\right|}_F^2+\lambda \left|\right|{\tilde{\Theta }}_1^{-}{\left|\right|}_1+\lambda \left|\right|{\tilde{\Theta }}_2^{-}{\left|\right|}_1+\rho \left|\right|{\tilde{\Theta }}_1^{-}-{\tilde{\Theta }}_2^{-}{\left|\right|}_1\hfill \\ \hfill \le & {\lambda }_0\left|\right|{\Delta }_1^{-}{\left|\right|}_1+{\lambda }_0\left|\right|{\Delta }_2^{-}{\left|\right|}_1+\left|\right|{\widehat{\Sigma }}_1^{+}-{\Sigma }_{01}^{+}{\left|\right|}_F\left|\right|{\Delta }_1^{+}{\left|\right|}_F+\left|\right|{\widehat{\Sigma }}_2^{+}-{\Sigma }_{02}^{+}{\left|\right|}_F\left|\right|{\Delta }_2^{+}{\left|\right|}_F\hfill \\ & +\lambda \left|\right|{\Theta }_{01}^{-}{\left|\right|}_1+\lambda \left|\right|{\Theta }_{02}^{-}{\left|\right|}_1+\rho \left|\right|{\Theta }_{01}^{-}-{\Theta }_{02}^{-}{\left|\right|}_1.\hfill \end{array}\end{array}$$ (34)

We move some terms of the inequality (34) and combine them to get the following inequality

$$\begin{array}{c}\hfill \begin{array}{cc}& c\left|\right|{\tilde{\Theta }}_1-{\Theta }_{01}{\left|\right|}_F^2+c\left|\right|{\tilde{\Theta }}_2-{\Theta }_{02}{\left|\right|}_F^2\hfill \\ & +\lambda \{\left|\right|{\tilde{\Theta }}_1^{-}{\left|\right|}_1-\left|\right|{\Theta }_{01}^{-}{\left|\right|}_1+\left|\right|{\tilde{\Theta }}_2^{-}{\left|\right|}_1-\left|\right|{\Theta }_{02}^{-}{\left|\right|}_1\}\hfill \\ \hfill \le & {\lambda }_0\{\left|\right|{\tilde{\Theta }}_1^{-}-{\Theta }_{01}^{-}{\left|\right|}_1+\left|\right|{\tilde{\Theta }}_2^{-}-{\Theta }_{02}^{-}{\left|\right|}_1\}+\rho \{\left|\right|{\Theta }_{01}^{-}-{\Theta }_{02}^{-}{\left|\right|}_1-\left|\right|{\tilde{\Theta }}_1^{-}-{\tilde{\Theta }}_2^{-}{\left|\right|}_1\}\hfill \\ & +\left|\right|{\widehat{\Sigma }}_1^{+}-{\Sigma }_{01}^{+}{\left|\right|}_F\left|\right|{\tilde{\Theta }}_1^{+}-{\Theta }_{01}^{+}{\left|\right|}_F+\left|\right|{\widehat{\Sigma }}_2^{+}-{\Sigma }_{02}^{+}{\left|\right|}_F\left|\right|{\tilde{\Theta }}_2^{+}-{\Theta }_{02}^{+}{\left|\right|}_F.\hfill \end{array}\end{array}$$ (35)

Next we need to prove three inequations:

$$\begin{array}{c}\hfill \left|\right|{\tilde{\Theta }}_k^{-}{\left|\right|}_1-\left|\right|{\Theta }_{0k}^{-}{\left|\right|}_1\ge \left|\right|{\Delta }_{k{S}_k^c}^{-}{\left|\right|}_1-\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_1,\end{array}$$ (36)

$$\begin{array}{c}\hfill \left|\right|{\tilde{\Theta }}_k^{-}-{\Theta }_{0k}^{-}{\left|\right|}_1\le \left|\right|{\Delta }_{k{S}_k^c}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_1,\end{array}$$ (37)

$$\begin{array}{c}\hfill \left|\right|{\Theta }_{01}^{-}-{\Theta }_{02}^{-}{\left|\right|}_1-\left|\right|{\tilde{\Theta }}_1^{-}-{\tilde{\Theta }}_2^{-}{\left|\right|}_1\le \left|\right|{\tilde{\Theta }}_1^{-}-{\Theta }_{01}^{-}{\left|\right|}_1+\left|\right|{\tilde{\Theta }}_2^{-}-{\Theta }_{02}^{-}{\left|\right|}_1.\end{array}$$ (38)

Because

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \left|\right|{\tilde{\Theta }}_k^{-}{\left|\right|}_1& =\left|\right|{\Theta }_{0k}^{-}+{\Delta }_k^{-}{\left|\right|}_1\hfill \\ \hfill & =\left|\right|{\Theta }_{0k{S}_k}^{-}+{\Delta }_{k{S}_k}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{k{S}_k^c}^{-}{\left|\right|}_1,\hfill \end{array}\end{array}$$

and

$$\begin{array}{c}\hfill \left|\right|{\Theta }_{0k}^{-}{\left|\right|}_1=\left|\right|{\Theta }_{0k{S}_k}^{-}{\left|\right|}_1\end{array}$$

hold. Thus,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \left|\right|{\tilde{\Theta }}_k^{-}{\left|\right|}_1-\left|\right|{\Theta }_{0k}^{-}{\left|\right|}_1=& \left|\right|{\Theta }_{0k{S}_k}^{-}+{\Delta }_{k{S}_k}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{k{S}_k^c}^{-}{\left|\right|}_1-\left|\right|{\Theta }_{0k{S}_k}^{-}{\left|\right|}_1\hfill \\ \hfill \ge & \left|\right|{\Delta }_{k{S}_k^c}^{-}{\left|\right|}_1-\left|\right||{\Theta }_{0k{S}_k}^{-}+{\Delta }_{k{S}_k}^{-}{\left|\right|}_1-\left|\right|{\Theta }_{0k{S}_k}^{-}{\left|\right|}_1|\hfill \\ \hfill \ge & \left|\right|{\Delta }_{k{S}_k^c}^{-}{\left|\right|}_1-\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_1,\hfill \end{array}\end{array}$$

which proves inequality (36). By the triangle inequality, we naturally obtain

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \left|\right|{\tilde{\Theta }}_k^{-}-{\Theta }_{0k}^{-}{\left|\right|}_1=& \left|\right|{\Delta }_k^{-}{\left|\right|}_1\hfill \\ \hfill =& \left|\right|{\Delta }_{k{S}_k^c}^{-}+{\Delta }_{k{S}_k}^{-}{\left|\right|}_1\hfill \\ \hfill \le & \left|\right|{\Delta }_{k{S}_k^c}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_1.\hfill \end{array}\end{array}$$

Thus, the inequation (37) holds. For inequation (38), we have

$$\begin{array}{c}\hfill \begin{array}{cc}& \left|\right|{\Theta }_{01}^{-}-{\Theta }_{02}^{-}{\left|\right|}_1-\left|\right|{\tilde{\Theta }}_1^{-}-{\tilde{\Theta }}_2^{-}{\left|\right|}_1\hfill \\ \hfill =& \left|\right|{\Theta }_{01}^{-}-{\tilde{\Theta }}_1^{-}+{\tilde{\Theta }}_1^{-}-{\tilde{\Theta }}_2^{-}+{\tilde{\Theta }}_2^{-}-{\Theta }_{02}^{-}{\left|\right|}_1-\left|\right|{\tilde{\Theta }}_1^{-}-{\tilde{\Theta }}_2^{-}{\left|\right|}_1\hfill \\ \hfill \le & \left|\right|{\tilde{\Theta }}_1^{-}-{\Theta }_{01}^{-}{\left|\right|}_1+\left|\right|{\tilde{\Theta }}_2^{-}-{\Theta }_{02}^{-}{\left|\right|}_1.\hfill \end{array}\end{array}$$

Thus, the inequality (35) yields

$$\begin{array}{c}\hfill \begin{array}{cc}& c\left|\right|{\tilde{\Theta }}_1-{\Theta }_{01}{\left|\right|}_F^2+c\left|\right|{\tilde{\Theta }}_2-{\Theta }_{02}{\left|\right|}_F^2\hfill \\ & +\lambda \{\left|\right|{\Delta }_{1S_1^c}^{-}{\left|\right|}_1-\left|\right|{\Delta }_{1S_1}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{2S_2^c}^{-}{\left|\right|}_1-\left|\right|{\Delta }_{2S_2}^{-}{\left|\right|}_1\}\hfill \\ \hfill \le & (\rho +{\lambda }_0)\{\left|\right|{\Delta }_{1S_1^c}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{1S_1}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{2S_2^c}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{2S_2}^{-}{\left|\right|}_1\}\hfill \\ & +\left|\right|{\widehat{\Sigma }}_1^{+}-{\Sigma }_{01}^{+}{\left|\right|}_F\left|\right|{\tilde{\Theta }}_1^{+}-{\Theta }_{01}^{+}{\left|\right|}_F+\left|\right|{\widehat{\Sigma }}_2^{+}-{\Sigma }_{02}^{+}{\left|\right|}_F\left|\right|{\tilde{\Theta }}_2^{+}-{\Theta }_{02}^{+}{\left|\right|}_F.\hfill \end{array}\end{array}$$

By taking 2(ρ + λ₀) < λ, we conclude that

$$\begin{array}{c}\hfill \begin{array}{cc}& 2c\{\left|\right|{\tilde{\Theta }}_1-{\Theta }_{01}{\left|\right|}_F^2+\left|\right|{\tilde{\Theta }}_2-{\Theta }_{02}{\left|\right|}_F^2\}+\lambda \{\left|\right|{\Delta }_{1S_1^c}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{2S_2^c}^{-}{\left|\right|}_1\}\hfill \\ \hfill \le & 3\lambda \{\left|\right|{\Delta }_{1S_1}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{2S_2}^{-}{\left|\right|}_1\}\hfill \\ & +2\{\left|\right|{\widehat{\Sigma }}_1^{+}-{\Sigma }_{01}^{+}{\left|\right|}_F\left|\right|{\tilde{\Theta }}_1^{+}-{\Theta }_{01}^{+}{\left|\right|}_F+\left|\right|{\widehat{\Sigma }}_2^{+}-{\Sigma }_{02}^{+}{\left|\right|}_F\left|\right|{\tilde{\Theta }}_2^{+}-{\Theta }_{02}^{+}{\left|\right|}_F\}.\hfill \end{array}\end{array}$$

By the definition of Δ_k, we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \left|\right|{\Delta }_k^{-}{\left|\right|}_1=& \left|\right|{\Delta }_{k{S}_k}^{-}+{\Delta }_{k{S}_k^c}^{-}{\left|\right|}_1\hfill \\ \hfill \le & \left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{k{S}_k^c}^{-}{\left|\right|}_1.\hfill \end{array}\end{array}$$ (39)

So we deduce

$$\begin{array}{c}\hfill \begin{array}{cc}& 2c\{\left|\right|{\tilde{\Theta }}_1-{\Theta }_{01}{\left|\right|}_F^2+\left|\right|{\tilde{\Theta }}_2-{\Theta }_{02}{\left|\right|}_F^2\}+\lambda \{\left|\right|{\Delta }_1^{-}{\left|\right|}_1+\left|\right|{\Delta }_2^{-}{\left|\right|}_1\}\hfill \\ \hfill \le & 4\lambda \{\left|\right|{\Delta }_{1S_1}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{2S_2}^{-}{\left|\right|}_1\}\hfill \\ & +2\{\left|\right|{\widehat{\Sigma }}_1^{+}-{\Sigma }_{01}^{+}{\left|\right|}_F\left|\right|{\tilde{\Theta }}_1^{+}-{\Theta }_{01}^{+}{\left|\right|}_F+\left|\right|{\widehat{\Sigma }}_2^{+}-{\Sigma }_{02}^{+}{\left|\right|}_F\left|\right|{\tilde{\Theta }}_2^{+}-{\Theta }_{02}^{+}{\left|\right|}_F\}\hfill \end{array}\end{array}$$

holds. Since the inequality of arithmetic and geometric means, the inequality $\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_1\le \sqrt{s_k}\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_F$ holds. Thus

$$\begin{array}{c}\hfill \begin{array}{cc}& 2c\{\left|\right|{\tilde{\Theta }}_1-{\Theta }_{01}{\left|\right|}_F^2+\left|\right|{\tilde{\Theta }}_2-{\Theta }_{02}{\left|\right|}_F^2\}+\lambda \{\left|\right|{\Delta }_1^{-}{\left|\right|}_1+\left|\right|{\Delta }_2^{-}{\left|\right|}_1\}\hfill \\ \hfill \le & 4\lambda \{\sqrt{s_1}\left|\right|{\Delta }_{1S_1}^{-}{\left|\right|}_F+\sqrt{s_2}\left|\right|{\Delta }_{2S_2}^{-}{\left|\right|}_F\}\hfill \\ & +2\{\left|\right|{\widehat{\Sigma }}_1^{+}-{\Sigma }_{01}^{+}{\left|\right|}_F\left|\right|{\tilde{\Theta }}_1^{+}-{\Theta }_{01}^{+}{\left|\right|}_F+\left|\right|{\widehat{\Sigma }}_2^{+}-{\Sigma }_{02}^{+}{\left|\right|}_F\left|\right|{\tilde{\Theta }}_2^{+}-{\Theta }_{02}^{+}{\left|\right|}_F\}.\hfill \end{array}\end{array}$$ (40)

Using xy ≤ (x² + y²)/2, the inequality (40) infer that

$$\begin{array}{c}\hfill \begin{array}{cc}& 2c\{\left|\right|{\tilde{\Theta }}_1-{\Theta }_{01}{\left|\right|}_F^2+\left|\right|{\tilde{\Theta }}_2-{\Theta }_{02}{\left|\right|}_F^2\}+\lambda \{\left|\right|{\Delta }_1^{-}{\left|\right|}_1+\left|\right|{\Delta }_2^{-}{\left|\right|}_1\}\hfill \\ \hfill \le & \frac{1}{2}(c\left|\right|{\Delta }_{1S_1}^{-}{\left|\right|}_F^2+\frac{16{\lambda }^2{s}_1}{c}+c\left|\right|{\Delta }_{2S_2}^{-}{\left|\right|}_F^2+\frac{16{\lambda }^2{s}_2}{c})\hfill \\ & +\frac{1}{2}(c\left|\right|{\tilde{\Theta }}_1^{+}-{\Theta }_{01}^{+}{\left|\right|}_F^2+\frac{4\left|\right|{\widehat{\Sigma }}_1^{+}-{\Sigma }_{01}^{+}{\left|\right|}_F^2}{c}+c\left|\right|{\tilde{\Theta }}_2^{+}-{\Theta }_{02}^{+}{\left|\right|}_F^2+\frac{4\left|\right|{\widehat{\Sigma }}_2^{+}-{\Sigma }_{02}^{+}{\left|\right|}_F^2}{c}).\hfill \end{array}\end{array}$$

Because

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill c\left|\right|{\tilde{\Theta }}_k^{+}-{\Theta }_{0k}^{+}{\left|\right|}_F^2+c\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_F^2\le & \{c\left|\right|{\tilde{\Theta }}_k^{+}-{\Theta }_{0k}^{+}{\left|\right|}_F^2+c\left|\right|{\Delta }_k^{-}{\left|\right|}_F^2\}\hfill \\ & +\{c\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_F^2+c\left|\right|{\Delta }_{k{S}_k^c}^{-}{\left|\right|}_F^2+c\left|\right|{\Delta }_k^{+}{\left|\right|}_F^2\}\hfill \\ \hfill =& 2c\left|\right|{\Delta }_k{\left|\right|}_F^2,\hfill \end{array}\end{array}$$ (41)

we obtain

$$\begin{array}{c}\hfill \begin{array}{cc}& 2c\{\left|\right|{\tilde{\Theta }}_1-{\Theta }_{01}{\left|\right|}_F^2+\left|\right|{\tilde{\Theta }}_2-{\Theta }_{02}{\left|\right|}_F^2\}+\lambda \{\left|\right|{\Delta }_1^{-}{\left|\right|}_1+\left|\right|{\Delta }_2^{-}{\left|\right|}_1\}\hfill \\ \hfill \le & c\{\left|\right|{\Delta }_1{\left|\right|}_F^2+\left|\right|{\Delta }_2{\left|\right|}_F^2\}+\frac{8{\lambda }^2(s_1+s_2)}{c}+\frac{2\left|\right|{\widehat{\Sigma }}_1^{+}-{\Sigma }_{01}^{+}{\left|\right|}_F^2}{c}+\frac{2\left|\right|{\widehat{\Sigma }}_2^{+}-{\Sigma }_{02}^{+}{\left|\right|}_F^2}{c}.\hfill \end{array}\end{array}$$

Thus,

$$\begin{array}{c}\hfill \begin{array}{cc}& c\{\left|\right|{\Delta }_1{\left|\right|}_F^2+\left|\right|{\Delta }_2{\left|\right|}_F^2\}+\lambda \{\left|\right|{\Delta }_1^{-}{\left|\right|}_1+\left|\right|{\Delta }_2^{-}{\left|\right|}_1\}\hfill \\ \hfill \le & \frac{8{\lambda }^2(s_1+s_2)}{c}+\frac{2\left|\right|{\widehat{\Sigma }}_1^{+}-{\Sigma }_{01}^{+}{\left|\right|}_F^2}{c}+\frac{2\left|\right|{\widehat{\Sigma }}_2^{+}-{\Sigma }_{02}^{+}{\left|\right|}_F^2}{c}.\hfill \end{array}\end{array}$$ (42)

Based on the inequality $\left|\right|{\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}{\left|\right|}_F\le \sqrt{p}\left|\right|{\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}{\left|\right|}_{\infty }$, we have

$$\begin{array}{c}\hfill \begin{array}{c}\hfill c\{\left|\right|{\Delta }_1{\left|\right|}_F^2+\left|\right|{\Delta }_2{\left|\right|}_F^2\}+\lambda \{\left|\right|{\Delta }_1^{-}{\left|\right|}_1+\left|\right|{\Delta }_2^{-}{\left|\right|}_1\}\le \frac{8{\lambda }^2(s_1+s_2)}{c}+\frac{4p{\lambda }_0^2}{c}.\end{array}\end{array}$$ (43)

Next, we prove that substituting ${\widehat{\Theta }}_k$ for ${\tilde{\Theta }}_k$, the conclusion still holds. According to the condition,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \left|\right|{\Delta }_1{\left|\right|}_F^2+\left|\right|{\Delta }_2{\left|\right|}_F^2\le & \frac{{\lambda }_0}{2cL}\le \frac{\lambda }{4cL}\le \frac{1}{32L^2}.\hfill \end{array}\end{array}$$

Taking $M=1/\left(2\sqrt{2}L\right)<1/2L$, we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \left|\right|{\Delta }_1{\left|\right|}_F^2+\left|\right|{\Delta }_2{\left|\right|}_F^2\le & M^2/4.\hfill \end{array}\end{array}$$

Thus, ||Δ_k||_F is bounded by M/2. In addition,

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left|\right|{\widehat{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F=\frac{M\left|\right|{\Delta }_k{\left|\right|}_F}{M-\left|\right|{\Delta }_k{\left|\right|}_F},\end{array}\end{array}$$

which means $\left|\right|{\widehat{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F$ is monotone increasing function of ||Δ_k||_F on set (0, M). We obtain that $\left|\right|{\widehat{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F\le M$. Therefore, we can substitute ${\widehat{\Theta }}_k$ for ${\tilde{\Theta }}_k$, and that leads to the inequality (43) holds for ${\widehat{\Theta }}_k$.

According to inequality (43), we get

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \left|\right|{\widehat{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F^2\le & \frac{8{\lambda }^2(s_1+s_2)}{c^2}+\frac{4p{\lambda }_0^2}{c^2}\hfill \\ \hfill \le & \frac{{\lambda }^2(8s_1+8s_2+p)}{c^2},\hfill \end{array}\end{array}$$

and

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \left|\right|{\widehat{\Theta }}_k^{-}-{\Theta }_{0k}^{-}{\left|\right|}_1\le & \frac{8\lambda (s_1+s_2)}{c}+\frac{4p{\lambda }_0^2}{\lambda c}\hfill \\ \hfill \le & \frac{\lambda (8s_1+8s_2+p)}{c}.\hfill \end{array}\end{array}$$

Thus, we conclude the upper bound of ${\sum }_{k=1}^2\left|\right||{\widehat{\Theta }}_k-{\Theta }_{0k}{\left|\right||}_1$,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \sum _{k=1}^2\left|\right||{\widehat{\Theta }}_k-{\Theta }_{0k}{\left|\right||}_1\le & \sum _{k=1}^2(\left|\right|{\widehat{\Theta }}_k^{+}-{\Theta }_{0k}^{+}{\left|\right|}_{\infty }+\left|\right|{\widehat{\Theta }}_k^{-}-{\Theta }_{0k}^{-}{\left|\right|}_1)\hfill \\ \hfill \le & \sum _{k=1}^2(\left|\right|{\widehat{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F+\left|\right|{\widehat{\Theta }}_k^{-}-{\Theta }_{0k}^{-}{\left|\right|}_1)\hfill \\ \hfill \le & \frac{2\lambda \sqrt{8s_1+8s_2+p}}{c}+\frac{2\lambda (8s_1+8s_2+p)}{c}\hfill \\ \hfill \le & \frac{4\lambda (8s_1+8s_2+p)}{c}.\hfill \end{array}\end{array}$$

Proof of Theorem 2

Proof 2 The minimizer $({\widehat{\Theta }}_R^{\left[1\right]},{\widehat{\Theta }}_R^{\left[2\right]})$ satisfying inequality (42), that is

$$\begin{array}{c}\hfill \begin{array}{cc}& c\{\left|\right|{\widehat{\Theta }}_R^{\left[1\right]}-{\Theta }_{R0}^{\left[1\right]}{\left|\right|}_F^2+\left|\right|{\widehat{\Theta }}_R^{\left[2\right]}-{\Theta }_{R0}^{\left[2\right]}{\left|\right|}_F^2\}+\lambda \{\left|\right|{({\widehat{\Theta }}_R^{\left[1\right]}-{\Theta }_{R0}^{\left[1\right]})}^{-}{\left|\right|}_1+\left|\right|{({\widehat{\Theta }}_R^{\left[2\right]}-{\Theta }_{R0}^{\left[2\right]})}^{-}{\left|\right|}_1\}\hfill \\ \hfill \le & \frac{8{\lambda }^2(s_1+s_2)}{c}+\frac{2\left|\right|{({\widehat{R}}^{\left[1\right]}-R_0^{\left[1\right]})}^{+}{\left|\right|}_F^2}{c}+\frac{2\left|\right|{({\widehat{R}}^{\left[2\right]}-R_0^{\left[2\right]})}^{+}{\left|\right|}_F^2}{c}.\hfill \end{array}\end{array}$$

The diagonal elements of ${\widehat{R}}^{\left[k\right]}$ and $R_0^{\left[k\right]}$ are all 1. Thus

$$\begin{array}{c}\hfill \begin{array}{cc}& c\{\left|\right|{\widehat{\Theta }}_R^{\left[1\right]}-{\Theta }_{R0}^{\left[1\right]}{\left|\right|}_F^2+\left|\right|{\widehat{\Theta }}_R^{\left[2\right]}-{\Theta }_{R0}^{\left[2\right]}{\left|\right|}_F^2\}+\lambda \{\left|\right|{({\widehat{\Theta }}_R^{\left[1\right]}-{\Theta }_{R0}^{\left[1\right]})}^{-}{\left|\right|}_1+\left|\right|{({\widehat{\Theta }}_R^{\left[2\right]}-{\Theta }_{R0}^{\left[2\right]})}^{-}{\left|\right|}_1\}\hfill \\ \hfill \le & \frac{8{\lambda }^2(s_1+s_2)}{c}.\hfill \end{array}\end{array}$$

Moreover, for the conclusion of the l₁-operator norm, we get

$$\begin{array}{c}\hfill \begin{array}{cc}& \left|\right||{\widehat{\Theta }}_R^{\left[1\right]}-{\Theta }_{R0}^{\left[1\right]}{\left|\right||}_1+\left|\right||{\widehat{\Theta }}_R^{\left[2\right]}-{\Theta }_{R0}^{\left[2\right]}{\left|\right||}_1\hfill \\ \hfill \le & \sum _{k=1}^2(\left|\right|{({\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]})}^{+}{\left|\right|}_{\infty }+\left|\right|{({\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]})}^{-}{\left|\right|}_1)\hfill \\ \hfill \le & \sum _{k=1}^2(\left|\right|{\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]}{\left|\right|}_F+\left|\right|{({\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]})}^{-}{\left|\right|}_1)\hfill \\ \hfill \le & \frac{32\lambda (s_1+s_2)}{c}.\hfill \end{array}\end{array}$$

For the minimizer $({\widehat{\Theta }}_w^{\left[1\right]},{\widehat{\Theta }}_w^{\left[2\right]})$ , following inequality holds

$$\begin{array}{c}\hfill \begin{array}{cc}& \left|\right||{\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]}{\left|\right||}_1\hfill \\ \hfill =& \left|\right||{\widehat{W}}^{\left[k\right]}{\widehat{\Theta }}_w^{\left[k\right]}{\widehat{W}}^{\left[k\right]}-W_0^{\left[k\right]}{\Theta }_{w0}^{\left[k\right]}{W}_0^{\left[k\right]}{\left|\right||}_1\hfill \\ \hfill \le & \left|\right|{\widehat{W}}^{\left[k\right]}{\left|\right|}_{\infty }^2\left|\right||{\widehat{\Theta }}_w^{\left[k\right]}-{\Theta }_{w0}^{\left[k\right]}{\left|\right||}_1+\left|\right|{\widehat{W}}^{\left[k\right]}-W_0^{\left[k\right]}{\left|\right|}_{\infty }\left|\right||{\Theta }_{w0}^{\left[k\right]}{\left|\right||}_1\left|\right|{\widehat{W}}^{\left[k\right]}{\left|\right|}_{\infty }\hfill \\ & +\left|\right|W_0^{\left[k\right]}{\left|\right|}_{\infty }\left|\right||{\Theta }_{w0}^{\left[k\right]}{\left|\right||}_1\left|\right|{\widehat{W}}^{\left[k\right]}-W_0^{\left[k\right]}{\left|\right|}_{\infty }.\hfill \end{array}\end{array}$$ (44)

To draw the conclusion, we have the following facts:

• The Sub-Gaussian vector with covariance ${\Sigma }_0^{\left[k\right]}$ implies that $\sqrt{n/\text{log}p}\left|\right|({\widehat{\Sigma }}^{\left[k\right]}-{\Sigma }_0^{\left[k\right]}){\left|\right|}_{\infty }$ is bounded in probability.

• The eigenvalues of ${\Theta }_{w0}^{\left[k\right]}$ are bounded by a constant.

Thus, $\left|\right||{\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]}{\left|\right||}_1$ and $\left|\right||{\widehat{\Theta }}_w^{\left[k\right]}-{\Theta }_{w0}^{\left[k\right]}{\left|\right||}_1$ share the same boundary.

Proof of Theorem 3

Proof 3 Similarly, ${\widehat{\Theta }}_k$ are the minimum value of the fused graphical Lasso for k = 1, 2, ⋯, K. Let ${\tilde{\Theta }}_k={\alpha }_k{\widehat{\Theta }}_k+(1-{\alpha }_k){\Theta }_{0k}$, and ${\alpha }_k=\frac{M}{M+\left|\right|{\widehat{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F}$. Denotes

$$\begin{array}{c}\hfill F_n({\Theta }_1,\cdots ,{\Theta }_K)=\sum _{k=1}^K\{tr\left({\Theta }_k{\widehat{\Sigma }}_k\right)-\text{log}\text{det}\left({\Theta }_k\right)\}+\lambda \sum _{k=1}^K\left|\right|{\Theta }_k^{-}{\left|\right|}_1+\rho \sum _{k<k^{\prime }}\left|\right|{\Theta }_k^{-}-{\Theta }_{k^{\prime }}^{-}{\left|\right|}_1,\end{array}$$

we obtain

$$\begin{array}{c}\hfill F_n({\tilde{\Theta }}_1,{\tilde{\Theta }}_2,\cdots ,{\tilde{\Theta }}_K)\le F_n({\Theta }_{01},{\Theta }_{02},\cdots ,{\Theta }_{0K}).\end{array}$$

Thus,

$$\begin{array}{ccc}& & \sum _{k=1}^K\{tr({\tilde{\Theta }}_k-{\Theta }_{0k}){\widehat{\Sigma }}_k-(\text{log}\text{det}\left({\tilde{\Theta }}_k\right)-\text{log}\text{det}\left({\Theta }_{0k}\right))+\lambda \left|\right|{\tilde{\Theta }}_k^{-}{\left|\right|}_1\}\hfill \\ & & +\rho \sum _{k<k^{\prime }}\left|\right|{\tilde{\Theta }}_k^{-}-{\tilde{\Theta }}_{k^{\prime }}^{-}{\left|\right|}_1\hfill \\ & \le & \lambda \sum _{k=1}^K\left|\right|{\Theta }_{0k}^{-}{\left|\right|}_1+\rho \sum _{k<k^{\prime }}\left|\right|{\Theta }_{0k}^{-}-{\Theta }_{0k^{\prime }}^{-}{\left|\right|}_1.\hfill \end{array}$$

Using the notations that ${\Delta }_k={\tilde{\Theta }}_k-{\Theta }_{0k}$ and

$$f\left({\Delta }_k\right)≔tr\left({\Delta }_k{\Sigma }_{0k}\right)-[\text{log}\text{det}({\Delta }_k+{\Theta }_{0k})-\text{log}\text{det}\left({\Theta }_{0k}\right)],$$

we yield the following expression

$$\begin{array}{c}\hfill \begin{array}{cc}& \sum _{k=1}^{K}f\left({\Delta }_k\right)+\lambda \sum _{k=1}^K\left|\right|{\tilde{\Theta }}_k^{-}{\left|\right|}_1+\rho \sum _{k<k^{\prime }}\left|\right|{\tilde{\Theta }}_k^{-}-{\tilde{\Theta }}_{k^{\prime }}^{-}{\left|\right|}_1\hfill \\ \hfill \le & -\sum _{k=1}^K\left(tr\right({\Delta }_k({\widehat{\Sigma }}_k-{\Sigma }_{0k})\left)\right)-tr\left({\Delta }_2({\widehat{\Sigma }}_2-{\Sigma }_{02})\right)\hfill \\ & +\lambda \sum _{k=1}^K\left|\right|{\Theta }_{0k}^{-}{\left|\right|}_1+\rho \sum _{k<k^{\prime }}\left|\right|{\Theta }_{0k}^{-}-{\Theta }_{0k^{\prime }}^{-}{\left|\right|}_1.\hfill \end{array}\end{array}$$ (45)

For L_k ≥ 1, k = 1, 2, ⋯, K, the minimum and maximum eigenvalues of Θ_0k hold that

$$\begin{array}{c}\hfill 1/L_k\le {\lambda }_{min}\left({\Theta }_{0k}\right)\le {\lambda }_{max}\left({\Theta }_{0k}\right)\le L_k.\end{array}$$

For multiple case, we select a constant L satisfying 1/L ≤ 1/L_k and L_k ≤ L. By similar analysis, for M in (0, 1/2L], the inequality (32) and the inequality (33) still hold.

For K groups data, based on the inequalities (31) and (33). Then, the inequality (45) turns to be

$$\begin{array}{c}\hfill \begin{array}{cc}& c\sum _{k=1}^K\left|\right|{\tilde{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F^2+\lambda \sum _{k=1}^K\left|\right|{\tilde{\Theta }}_k^{-}{\left|\right|}_1+\rho \sum _{k<k^{\prime }}\left|\right|{\tilde{\Theta }}_k^{-}-{\tilde{\Theta }}_{k^{\prime }}^{-}{\left|\right|}_1\hfill \\ \hfill \le & \sum _{k=1}^K\{{\lambda }_0\left|\right|{\Delta }_k^{-}{\left|\right|}_1+\left|\right|{\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}{\left|\right|}_F\left|\right|{\Delta }_k^{+}{\left|\right|}_F\}+\lambda \sum _{k=1}^K\left|\right|{\Theta }_{0k}^{-}{\left|\right|}_1+\rho \sum _{k<k^{\prime }}\left|\right|{\Theta }_{0k}^{-}-{\Theta }_{0k^{\prime }}^{-}{\left|\right|}_1.\hfill \end{array}\end{array}$$

Thus,

$$\begin{array}{c}\hfill \begin{array}{cc}& c\sum _{k=1}^K\left|\right|{\tilde{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F^2+\lambda \sum _{k=1}^K\{\left|\right|{\tilde{\Theta }}_k^{-}{\left|\right|}_1-\left|\right|{\Theta }_{0k}^{-}{\left|\right|}_1\}\hfill \\ \hfill \le & {\lambda }_0\sum _{k=1}^K\left|\right|{\tilde{\Theta }}_k^{-}-{\Theta }_{0k}^{-}{\left|\right|}_1+\rho \sum _{k<k^{\prime }}\{\left|\right|{\Theta }_{0k}^{-}-{\Theta }_{0k^{\prime }}^{-}{\left|\right|}_1-\left|\right|{\tilde{\Theta }}_k^{-}-{\tilde{\Theta }}_{k^{\prime }}^{-}{\left|\right|}_1\}\hfill \\ & +\sum _{k=1}^K\{\left|\right|{\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}{\left|\right|}_F\left|\right|{\tilde{\Theta }}_k^{+}-{\Theta }_{0k}^{+}{\left|\right|}_F\}.\hfill \end{array}\end{array}$$ (46)

When k = 1, 2, ⋯, K, the inequations (36) and (37) still hold. Similarly, we have the following inequality

$$\begin{array}{c}\hfill \begin{array}{cc}& \left|\right|{\Theta }_{0k}^{-}-{\Theta }_{0k^{\prime }}^{-}{\left|\right|}_1-\left|\right|{\tilde{\Theta }}_k^{-}-{\tilde{\Theta }}_{k^{\prime }}^{-}{\left|\right|}_1\hfill \\ \hfill =& \left|\right|{\Theta }_{0k}^{-}-{\tilde{\Theta }}_k^{-}+{\tilde{\Theta }}_k^{-}-{\tilde{\Theta }}_{k^{\prime }}^{-}+{\tilde{\Theta }}_{k^{\prime }}^{-}-{\Theta }_{0k^{\prime }}^{-}{\left|\right|}_1-\left|\right|{\tilde{\Theta }}_k^{-}-{\tilde{\Theta }}_{k^{\prime }}^{-}{\left|\right|}_1\hfill \\ \hfill \le & \left|\right|{\tilde{\Theta }}_k^{-}-{\Theta }_{0k}^{-}{\left|\right|}_1+\left|\right|{\tilde{\Theta }}_{k^{\prime }}^{-}-{\Theta }_{0k^{\prime }}^{-}{\left|\right|}_1.\hfill \end{array}\end{array}$$ (47)

Thus, by the Eqs (36), (37) and (47) the inequality (46) yields

$$\begin{array}{c}\hfill \begin{array}{cc}& c\sum _{k=1}^K\left|\right|{\tilde{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F^2+\lambda \sum _{k=1}^K\{\left|\right|{\Delta }_{k{S}_k^c}^{-}{\left|\right|}_1-\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_1\}\hfill \\ \hfill \le & {\lambda }_0\sum _{k=1}^K\{\left|\right|{\Delta }_{k{S}_k^c}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_1\}\hfill \\ & +\rho \sum _{k<k^{\prime }}\{\left|\right|{\Delta }_{k{S}_k^c}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{k^{\prime }{S}_{k^{\prime }}^c}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{k^{\prime }{S}_{k^{\prime }}}^{-}{\left|\right|}_1\}\hfill \\ & +\sum _{k=1}^K\{\left|\right|{\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}{\left|\right|}_F\left|\right|{\tilde{\Theta }}_k^{+}-{\Theta }_{0k}^{+}{\left|\right|}_F\}\hfill \\ \hfill \le & (\frac{K(K-1)}{2}\rho +{\lambda }_0)\sum _{k=1}^K\{\left|\right|{\Delta }_{k{S}_k^c}^{-}{\left|\right|}_1+\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_1\}\hfill \\ & +\sum _{k=1}^K\{\left|\right|{\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}{\left|\right|}_F\left|\right|{\tilde{\Theta }}_k^{+}-{\Theta }_{0k}^{+}{\left|\right|}_F\}.\hfill \end{array}\end{array}$$

Since K is a fixed constant, and $2(\frac{K(K-1)}{2}\rho +{\lambda }_0)<\lambda$ , we can obtain

$$\begin{array}{c}\hfill \begin{array}{cc}& 2c\sum _{k=1}^K\left|\right|{\tilde{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F^2+\lambda \sum _{k=1}^K\left|\right|{\Delta }_{k{S}_k^c}^{-}{\left|\right|}_1\hfill \\ \hfill \le & 3\lambda \sum _{k=1}^K\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_1+2\sum _{k=1}^K\{\left|\right|{\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}{\left|\right|}_F\left|\right|{\tilde{\Theta }}_k^{+}-{\Theta }_{0k}^{+}{\left|\right|}_F\}.\hfill \end{array}\end{array}$$

On the basis of the inequality (39), we deduce

$$\begin{array}{c}\hfill \begin{array}{cc}& 2c\sum _{k=1}^K\left|\right|{\tilde{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F^2+\lambda \sum _{k=1}^K\left|\right|{\Delta }_k^{-}{\left|\right|}_1\hfill \\ \hfill \le & 4\lambda \sum _{k=1}^K\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_1+2\sum _{k=1}^K\{\left|\right|{\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}{\left|\right|}_F\left|\right|{\tilde{\Theta }}_k^{+}-{\Theta }_{0k}^{+}{\left|\right|}_F\}\hfill \end{array}\end{array}$$

holds. In addition, one can get the inequality $\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_1\le \sqrt{s_k}\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_F$. Thus

$$\begin{array}{c}\hfill \begin{array}{cc}& 2c\sum _{k=1}^K\left|\right|{\tilde{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F^2+\lambda \sum _{k=1}^K\left|\right|{\Delta }_k^{-}{\left|\right|}_1\hfill \\ \hfill \le & 4\lambda \sum _{k=1}^K\left(\sqrt{s_k}\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_F\right)+2\sum _{k=1}^K\{\left|\right|{\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}{\left|\right|}_F\left|\right|{\tilde{\Theta }}_k^{+}-{\Theta }_{0k}^{+}{\left|\right|}_F\}.\hfill \end{array}\end{array}$$ (48)

Based on xy ≤ (x² + y²)/2 and the inequality (41), the inequality (48) infer that

$$\begin{array}{c}\hfill \begin{array}{cc}& 2c\sum _{k=1}^K\left|\right|{\tilde{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F^2+\lambda \sum _{k=1}^K\left|\right|{\Delta }_k^{-}{\left|\right|}_1\hfill \\ \hfill \le & \frac{1}{2}\sum _{k=1}^K(c\left|\right|{\Delta }_{k{S}_k}^{-}{\left|\right|}_F^2+\frac{16{\lambda }^2{s}_k}{c})+\frac{1}{2}\sum _{k=1}^K(c\left|\right|{\tilde{\Theta }}_k^{+}-{\Theta }_{0k}^{+}{\left|\right|}_F^2+\frac{4\left|\right|{\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}{\left|\right|}_F^2}{c})\hfill \\ \hfill \le & c\sum _{k=1}^K\left|\right|{\Delta }_k{\left|\right|}_F^2+\frac{8{\lambda }^2{\sum }_{k=1}^K{s}_k}{c}+\frac{2{\sum }_{k=1}^K\left|\right|{\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}{\left|\right|}_F^2}{c}.\hfill \end{array}\end{array}$$

Thus,

$$\begin{array}{c}\hfill c\sum _{k=1}^K\left|\right|{\Delta }_k{\left|\right|}_F^2+\lambda \sum _{k=1}^K\left|\right|{\Delta }_k^{-}{\left|\right|}_1\le \frac{8{\lambda }^2{\sum }_{k=1}^K{s}_k}{c}+\frac{2{\sum }_{k=1}^K\left|\right|{\widehat{\Sigma }}_k^{+}-{\Sigma }_{0k}^{+}{\left|\right|}_F^2}{c}.\end{array}$$ (49)

Using the relation between the Frobenius norm and the supremum norm, we have

$$\begin{array}{c}\hfill \begin{array}{c}\hfill c\sum _{k=1}^K\left|\right|{\Delta }_k{\left|\right|}_F^2+\lambda \sum _{k=1}^K\left|\right|{\Delta }_k^{-}{\left|\right|}_1\le \frac{8{\lambda }^2{\sum }_{k=1}^K{s}_k}{c}+\frac{2Kp{\lambda }_0^2}{c}.\end{array}\end{array}$$ (50)

According to the inequality (50), we get

$$\begin{array}{c}\hfill \sum _{k=1}^K\left|\right|{\Delta }_k{\left|\right|}_F^2\le \frac{{\lambda }_0}{2cL}.\end{array}$$

According to λ₀ ≤ λ/2 and the condition λ ≤ c/8L, we get

$$\begin{array}{c}\hfill \sum _{k=1}^K\left|\right|{\Delta }_k{\left|\right|}_F^2\le \frac{1}{32L^2}.\end{array}$$

Taking $M=1/\left(2\sqrt{2}L\right)<1/2L$ , we have

$$\begin{array}{c}\hfill \sum _{k=1}^K\left|\right|{\Delta }_k{\left|\right|}_F^2\le M^2/4.\end{array}$$

Thus, ||Δ_k||_F is bounded by M/2. Further, we can derive $\left|\right|{\widehat{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F\le M$ which means that we can substitute ${\widehat{\Theta }}_k$ for ${\tilde{\Theta }}_k$, and that leads to the inequality (50) holds for ${\widehat{\Theta }}_k$, i.e.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill c\sum _{k=1}^K\left|\right|{\widehat{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F^2+\lambda \sum _{k=1}^K\left|\right|{({\widehat{\Theta }}_k-{\Theta }_{0k})}^{-}{\left|\right|}_1\le \frac{8{\lambda }^2{\sum }_{k=1}^K{s}_k}{c}+\frac{2Kp{\lambda }_0^2}{c},\end{array}\end{array}$$

That implies

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \sum _{k=1}^K\left|\right||{\widehat{\Theta }}_k-{\Theta }_{0k}{\left|\right||}_1\le & \sum _{k=1}^K(\left|\right|{\widehat{\Theta }}_k^{+}-{\Theta }_{0k}^{+}{\left|\right|}_{\infty }+\left|\right|{\widehat{\Theta }}_k^{-}-{\Theta }_{0k}^{-}{\left|\right|}_1)\hfill \\ \hfill \le & \sum _{k=1}^K(\left|\right|{\widehat{\Theta }}_k-{\Theta }_{0k}{\left|\right|}_F+\left|\right|{\widehat{\Theta }}_k^{-}-{\Theta }_{0k}^{-}{\left|\right|}_1)\hfill \\ \hfill \le & K[\frac{\lambda \sqrt{8{\sum }_{k=1}^K{s}_k+\frac{Kp}{2}}}{c}+\frac{\lambda (8{\sum }_{k=1}^K{s}_k+\frac{Kp}{2})}{c}]\hfill \\ \hfill \le & \frac{2K\lambda (8{\sum }_{k=1}^K{s}_k+\frac{Kp}{2})}{c},\hfill \end{array}\end{array}$$

which completes the proof.

Proof of Theorem 4

Proof 4 We get from (49)

$$\begin{array}{c}\hfill \begin{array}{cc}& c\sum _{k=1}^K\left|\right|{\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]}{\left|\right|}_F^2+\lambda \sum _{k=1}^K\left|\right|{({\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]})}^{-}{\left|\right|}_1\hfill \\ \hfill \le & \frac{8{\lambda }^2{\sum }_{k=1}^K{s}_k}{c}+\frac{2{\sum }_{k=1}^K\left|\right|{({\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]})}^{+}{\left|\right|}_F^2}{c},\hfill \end{array}\end{array}$$

and similarly derive

$$\begin{array}{c}\hfill c\sum _{k=1}^K\left|\right|{\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]}{\left|\right|}_F^2+\lambda \sum _{k=1}^K\left|\right|{({\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]})}^{-}{\left|\right|}_1\le \frac{8{\lambda }^2{\sum }_{k=1}^K{s}_k}{c}.\end{array}$$

Using

$$\begin{array}{c}\hfill \sum _{k=1}^K\left|\right||{\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]}{\left|\right||}_1\le \sum _{k=1}^K(\left|\right|{\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]}{\left|\right|}_F+\left|\right|{({\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]})}^{-}{\left|\right|}_1)\end{array}$$

we have

$$\begin{array}{c}\hfill \sum _{k=1}^K\left|\right||{\widehat{\Theta }}_R^{\left[k\right]}-{\Theta }_{R0}^{\left[k\right]}{\left|\right||}_1\le \frac{16K\lambda {\sum }_{k=1}^K{s}_k}{c}.\end{array}$$

At last, using the inequality (44), based on the analysis of the upper bound of $\left|\right|W_0^{\left[k\right]}{\left|\right|}_{\infty }$ and $\left|\right|{\widehat{W}}^{\left[k\right]}{\left|\right|}_{\infty }$, and the convergence rate of $\left|\right|({\widehat{\Sigma }}^{\left[k\right]}-{\Sigma }_0^{\left[k\right]}){\left|\right|}_{\infty }$, we draw the conclusion that

$$\begin{array}{c}\hfill \sum _{k=1}^K\left|\right||{\widehat{\Theta }}_w^{\left[k\right]}-{\Theta }_0^{\left[k\right]}{\left|\right||}_1\frac{16K\lambda {\sum }_{k=1}^K{s}_k}{c}.\end{array}$$

Proof of Theorem 5

Proof 5 First of all, we prove that the remainder converge in probability with a $1/\sqrt{n}$ convergence rate. On account of Theorem 1, we get

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\left|\right|rem\left|\right|}_{\infty }\le \sum _{k=1}^2\left|\right|({\widehat{\Theta }}^{\left[k\right]}-{\Theta }_0^{\left[k\right]})({\widehat{\Sigma }}^{\left[k\right]}-{\Sigma }_0^{\left[k\right]}){\Theta }_0^{\left[k\right]}{\left|\right|}_{\infty }+\sum _{k=1}^2\left|\right|({\widehat{\Theta }}^{\left[k\right]}-{\Theta }_0^{\left[k\right]})({\widehat{\Sigma }}^{\left[k\right]}{\widehat{\Theta }}^{\left[k\right]}-{\mathbf{\text{I}}}_p){\left|\right|}_{\infty }\end{array}\end{array}$$

Define

$$\begin{array}{c}\hfill l(\Theta )=\sum _{k=1}^2\{tr\left({\widehat{\Sigma }}^{\left[k\right]}{\Theta }^{\left[k\right]}\right)-\text{log}\text{det}\left({\Theta }^{\left[k\right]}\right)\}+\lambda \sum _{k=1}^2\left|\right|{\left({\Theta }^{\left[k\right]}\right)}^{-}{\left|\right|}_1+\rho \left|\right|{({\Theta }^{\left[1\right]}-{\Theta }^{\left[2\right]})}^{-}{\left|\right|}_1.\end{array}$$

By the Karush-Kuhn-Tucker (KKT) conditions, we yield

$$\begin{array}{c}\hfill {\widehat{\Sigma }}^{\left[1\right]}-{\left({\widehat{\Theta }}^{\left[1\right]}\right)}^{-1}+(\lambda +\rho ){\widehat{Z}}^{\left[1\right]}=0,\end{array}$$ (51)

and

$$\begin{array}{c}\hfill {\widehat{\Sigma }}^{\left[2\right]}-{\left({\widehat{\Theta }}^{\left[2\right]}\right)}^{-1}+(\lambda -\rho ){\widehat{Z}}^{\left[2\right]}=0,\end{array}$$ (52)

where ${\widehat{Z}}_{ij}^{\left[k\right]}=sign\left({\widehat{\Theta }}_{ij}^{\left[k\right]}\right)$ if ${\widehat{\Theta }}_{ij}^{\left[k\right]}\ne 0$ , and satisfying $\left|\right|{\widehat{Z}}^{\left[k\right]}{\left|\right|}_{\infty }\le 1$. Multiplying by ${\widehat{\Theta }}^{\left[1\right]}$ on the Eq (51), we get

$$\begin{array}{c}\hfill {\mathbf{\text{I}}}_p-{\widehat{\Sigma }}^{\left[1\right]}{\widehat{\Theta }}^{\left[1\right]}=(\lambda +\rho ){\widehat{Z}}^{\left[1\right]}{\widehat{\Theta }}^{\left[1\right]}.\end{array}$$

Similarly, we have

$$\begin{array}{c}\hfill {\mathbf{\text{I}}}_p-{\widehat{\Sigma }}^{\left[2\right]}{\widehat{\Theta }}^{\left[2\right]}=(\lambda -\rho ){\widehat{Z}}^{\left[2\right]}{\widehat{\Theta }}^{\left[2\right]}.\end{array}$$

Thus,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\left|\right|rem\left|\right|}_{\infty }\le & \sum _{k=1}^2\left|\right||({\widehat{\Theta }}^{\left[k\right]}-{\Theta }_0^{\left[k\right]}){\left|\right||}_1\left|\right|({\widehat{\Sigma }}^{\left[k\right]}-{\Sigma }_0^{\left[k\right]}){\left|\right|}_{\infty }\left|\right||{\Theta }_0^{\left[k\right]}{\left|\right||}_1\hfill \\ & +(\lambda +\rho )\sum _{k=1}^2\left|\right||({\widehat{\Theta }}^{\left[k\right]}-{\Theta }_0^{\left[k\right]}){\left|\right||}_1\left|\right|{\widehat{Z}}^{\left[k\right]}{\left|\right|}_{\infty }\left|\right||{\widehat{\Theta }}^{\left[k\right]}{\left|\right||}_1.\hfill \end{array}\end{array}$$

To draw the conclusion, we have

$$\begin{array}{c}\hfill \left|\right||({\widehat{\Theta }}^{\left[k\right]}-{\Theta }_0^{\left[k\right]}){\left|\right||}_1\le b(p+s)\lambda ,\end{array}$$ (53)

where b is a constant and is related to L. According to the Schwarz inequality and Weyl inequality, we get

$$\begin{array}{c}\hfill \left|\right||{\Theta }_0^{\left[k\right]}{\left|\right||}_1\le \sqrt{d+1}{\Lambda }_{\text{max}}\left({\Theta }_0^{\left[k\right]}\right).\end{array}$$ (54)

The bound of $\left|\right||{\widehat{\Theta }}^{\left[k\right]}{\left|\right||}_1$ is derived by

$$\begin{array}{c}\hfill \left|\right||{\widehat{\Theta }}^{\left[k\right]}{\left|\right||}_1\le \left|\right||{\widehat{\Theta }}^{\left[k\right]}-{\Theta }_0^{\left[k\right]}{\left|\right||}_1+\left|\right||{\Theta }_0^{\left[k\right]}{\left|\right||}_1.\end{array}$$ (55)

According to the rate of λ, we conclude that

$$\begin{array}{c}\hfill \left|\right||{\widehat{\Theta }}^{\left[k\right]}{\left|\right||}_1\le \sqrt{d+1}{\Lambda }_{\text{max}}\left({\Theta }_0^{\left[k\right]}\right).\end{array}$$ (56)

Besides, the Sub-Gaussian random vector with covariance ${\Sigma }_0^{\left[k\right]}$ implies that $\left|\right|{\widehat{\Sigma }}^{\left[k\right]}-{\Sigma }_0^{\left[k\right]}{\left|\right|}_{\infty }=O_p\left(\sqrt{\text{log}\left(p\right)/n}\right)$, where O_p denotes bounded in probability. We get

$$\begin{array}{ccc}\hfill {\left|\right|rem\left|\right|}_{\infty }& \le & \frac{4\lambda (8s_1+8s_2+p)}{c}\sqrt{\frac{\text{log}p}{n}}\sqrt{d+1}\text{max}\{{\Lambda }_{\text{max}}\left({\Theta }_0^{\left[1\right]}\right),{\Lambda }_{\text{max}}\left({\Theta }_0^{\left[2\right]}\right)\}\hfill \\ & & +(\lambda +\rho )\frac{4\lambda (8s_1+8s_2+p)}{c}\sqrt{d+1}\text{max}\{{\Lambda }_{\text{max}}\left({\Theta }_0^{\left[1\right]}\right),{\Lambda }_{\text{max}}\left({\Theta }_0^{\left[2\right]}\right)\}.\hfill \end{array}$$

For λ ≍ ρ, ||rem||_∞ is bounded by $\tilde{b}(p+s)\sqrt{d+1}{\lambda }^2$ in probability, where $\tilde{b}$ is a constant related to L. Based on the condition $(p+s)\sqrt{d}=o(\sqrt{n}/\text{log}p)$, ${\left|\right|rem\left|\right|}_{\infty }=o_p(1/\sqrt{n})$. According to the bounded fourth moments of ${\left({\widehat{\Theta }}^{\left[k\right]}\right)}_{ii}{\left({\widehat{\Theta }}^{\left[k\right]}\right)}_{jj}+{\left({\widehat{\Theta }}^{\left[k\right]}\right)}_{ij}^2$ and Lindeberg central limit theorem, we complete the proof of the Theorem 5.

Proof of Theorem 6

Proof 6 The conclusions of Theorem 6 can be obtained from the arguments (53)–(56). For weighted version, ||rem||_∞ can be bounded by $\tilde{b}s\sqrt{d+1}{\lambda }^2$, which completes the proof.

Data Availability

All data come from website http://portals.broadinstitute.org/cgibin/cancer/datasets.cgi. However, we do not have the right to share this data.

Funding Statement

the author Q.Y. Zhang is supported by Program for youth innovation Research in Capital University of Economics and Business(QNTD202207).