Minimax Rate-optimal Estimation of High-dimensional Covariance Matrices with Incomplete Data

Abstract

Missing data occur frequently in a wide range of applications. In this paper, we consider estimation of high-dimensional covariance matrices in the presence of missing observations under a general missing completely at random model in the sense that the missingness is not dependent on the values of the data. Based on incomplete data, estimators for bandable and sparse covariance matrices are proposed and their theoretical and numerical properties are investigated.

Minimax rates of convergence are established under the spectral norm loss and the proposed estimators are shown to be rate-optimal under mild regularity conditions. Simulation studies demonstrate that the estimators perform well numerically. The methods are also illustrated through an application to data from four ovarian cancer studies. The key technical tools developed in this paper are of independent interest and potentially useful for a range of related problems in high-dimensional statistical inference with missing data.

1 Introduction

The problem of missing data arises frequently in a wide range of fields, including biomedical studies, social science, engineering, economics, and computer science. Statistical inference in the presence of missing observations has been well studied in classical statistics. See, e.g., Ibrahim and Molenberghs [18] for a review of missing data methods in longitudinal studies and Schafer [26] for literature on handling multivariate data with missing observations. See Little and Rubin [20] and the references therein for a comprehensive treatment of missing data problems.

Missing data also occurs in contemporary high-dimensional inference problems, whose dimension p can be comparable to or even much larger than the sample size n. For example, in large-scale genome-wide association studies (GWAS), it is common for many subjects to have missing values on some genetic markers due to various reasons, including insufficient resolution, image corruption, and experimental error during the laboratory process. Also, different studies may have different volumes of genomic data available by design. For instance, the four genomic ovarian cancer studies discussed in Section 4 have throughput measurements of mRNA gene expression levels, but only one of these also has microRNA measurements (Cancer Genome Atlas Research Network [11], Bonome et al. [4], Tothill et al. [27] and Dressman et al. [15]). Discarding samples with any missingness is highly inefficient and could induce bias due to non-random missingness. It is of significant interest to integrate multiple high-throughput studies of the same disease, not only to boost statistical power but also to improve the biological interpretability. However, considerable challenges arise when integrating such studies due to missing data.

Although there have been significant recent efforts to develop methodologies and theories for high dimensional data analysis, there is a paucity of methods with theoretical guarantees for statistical inference with missing data in the high-dimensional setting. Under the assumption that the components are missing uniformly and completely at random (MUCR), Loh and Wainwright [21] proposed a non-convex optimization approach to high-dimensional linear regression, Lounici [23] introduced a method for estimating a low-rank covariance matrix and Lounici [22] considered sparse principal component analysis. In these papers, theoretical properties of the procedures were analyzed. These methods and theoretical results critically depend on the MUCR assumption.

Covariance structures play a fundamental role in high-dimensional statistics. It is of direct interest in a wide range of applications including genomic data analysis, particularly for hypothesis generation. Knowledge of the covariance structure is critical to many statistical methods, including discriminant analysis, principal component analysis, clustering analysis, and regression analysis. In the high-dimensional setting with complete data, inference on the covariance structure has been actively studied in recent years. See Cai, Ren and Zhou [7] for a survey of recent results on minimax and adaptive estimation of high-dimensional covariance and precision matrices under various structural assumptions. Estimation of high-dimensional covariance matrices in the presence of missing data also has wide applications in biomedical studies, particularly in integrative genomic analysis which holds great potential in providing a global view of genome function (see Hawkins et al. [17]).

In this paper, we consider estimation of high-dimensional covariance matrices in the presence of missing observations under a general missing completely at random (MCR) model in the sense that the missingness is not dependent on the values of the data. Let X₁, . . . , X_n be n independent copies of a p dimensional random vector X with mean μ and covariance matrix Σ. Instead of observing the complete sample {X₁, . . . , X_n, one observes the sample with missing values, where the observed coordinates of X_k are indicated by a vector S_k ∈ {0, 1}^p, k = 1, . . ., n. That is,

$$X_{jk}\text{is observed if}{S}_{jk}=1\text{and}{X}_{jk}\text{is missing if}{S}_{jk}=0.$$ (1)

Here X_jk and S_jk are respectively the jth coordinate of the vectors X_k and S_k. We denote the incomplete sample with missing values by ${\mathbf{X}}^{\ast }=\{{\mathbf{X}}_1^{\ast },\dots ,{\mathbf{X}}_n^{\ast }\}$. The major goal of the present paper is to estimate Σ, the covariance matrix of X, with theoretical guarantees based on the incomplete data X* in the high-dimensional setting where p can be much larger than n.

This paper focuses on estimation of high-dimensional bandable covariance matrices and sparse covariance matrices in the presence of missing data. These two classes of covariance matrices arise frequently in many applications, including genomics, econometrics, signal processing, temporal and spatial data analyses, and chemometrics. Estimation of these high-dimensional structured covariance matrices have been well studied in the setting of complete data in a number of recent papers, e.g., Bickel and Levina [2, 3], Karoui [16], Rothman et al. [24], Cai and Zhou [10], Cai and Liu [5], Cai et al. [6, 9] and Cai and Yuan [8]. Given an incomplete sample X* with missing values, we introduced a “generalized” sample covariance matrix, which can be viewed as an analog of the usual sample covariance matrix in the case of complete data. For estimation of bandable covariance matrices, where the entries of the matrix decay as they move away from the diagonal, a blockwise tridiagonal estimator is introduced and is shown to be rate-optimal. We then consider estimation of sparse covariance matrices. An adaptive thresholding estimator based on the generalized sample covariance matrix is proposed. The estimator is shown to achieve the optimal rate of convergence over a large class of approximately sparse covariance matrices under mild conditions.

The technical analysis for the case of missing data is much more challenging than that for the complete data, although some of the basic ideas are similar. To facilitate the theoretical analysis of the proposed estimators, we establish two key technical results, first, a large deviation result for a sub-matrix of the generalized sample covariance matrix and second, a large deviation bound for the self-normalized entries of the generalized sample covariance matrix. These technical tools are not only important for the present paper, but also useful for other related problems in high-dimensional statistical inference with missing data.

A simulation study is carried out to examine the numerical performance of the proposed estimation procedures. The results show that the proposed estimators perform well numerically. Even in the MUCR setting, our proposed procedures for estimating bandable, sparse covariance matrices, which do not rely on the information of the missingness mechanism, outperform the ones specifically designed for MUCR. The advantages are more significant under the setting of missing completely at random but not uniformly. We also illustrate our procedure with an application to data from four ovarian cancer studies that have different volumes of genomic data by design. The proposed estimators enable us to estimate the covariance matrix by integrating the data from all four studies and lead to a more accurate estimator. Such high-dimensional covariance matrix estimation with missing data is also useful for other types of data integration. See further discussions in Section 4.4.

The rest of the paper is organized as follows. Section 2 considers estimation of bandable covariance matrices with incomplete data. The minimax rate of convergence is established for the spectral norm loss under regularity conditions. Section 3 focuses on estimation of high-dimensional sparse covariance matrices and introduces an adaptive thresholding estimator in the presence of missing observations. Asymptotic properties of the estimator under the spectral norm loss is also studied. Numerical performance of the proposed methods is investigated in Section 4 through both simulation studies and an analysis of an ovarian cancer dataset. Section 5 discusses a few related problems. Finally the proofs of the main results are given in Section 6 and the Supplement.

2 Estimation of Bandable Covariance Matrices

In this section, we consider estimation of bandable covariance matrices with incomplete data. Bandable covariance matrices, whose entries decay as they move away from the diagonal, arise frequently in temporal and spatial data analysis. See, e.g., Bickel and Levina [2] and Cai et al. [7] and the references therein. The procedure relies on a “generalized” sample covariance matrix. We begin with basic notation and definitions that will be used throughout the rest of the paper.

2.1 Notation and Definitions

Matrices and vectors are denoted by boldface letters. For a vector $\beta \in {\mathbb{R}}^p$, we denote the Euclidean q-norm by ${\Vert \beta \Vert }_q$, i.e., ${\Vert \beta \Vert }_q=\sqrt[q]{{\sum }_{i=1}^p{\mid {\beta }_i\mid }^q}$. Let $\mathbf{A}={\mathbf{UDV}}^{\mathsf{T}}={\sum }_i{\lambda }_i\left(\mathbf{A}\right){\mathbf{u}}_i{\mathbf{v}}_i^{\mathsf{T}}$ be the singular value decomposition of a matrix $\mathbf{A}\in {\mathbb{R}}^{p_1\times p_2}$, where D = diag{λ₁(A),...} with λ₁ (A) ≥ · · · ≥ 0 being the singular values. For 1 ≤ q ≤ ∞, the Schatten-q norm ∥A∥_q is defined by ${\Vert \mathbf{A}\Vert }_q={\left\{{\sum }_i{\lambda }_i^q\left(\mathbf{A}\right)\right\}}^{1∕q}$. In particular, ${\Vert \mathbf{A}\Vert }_2=\sqrt{{\sum }_i{\lambda }_i^2\left(\mathbf{A}\right)}$ is the Frobenius norm of A and will be denoted as ∥A∥_F; ∥A∥_∞ = λ₁(A) is the spectral norm of A and will be simply denoted as ∥A∥. For 1 ≤ q ≤ ∞ and $A\in {\mathbb{R}}^{p_1\times p_2}$, we denote the operator ${\ell }_q$ norm of A by ${\Vert \mathbf{A}\Vert }_{{\ell }_q}$ which is defined as ${\Vert \mathbf{A}\Vert }_{{\ell }_q}={\max }_{x\in {\mathbb{R}}^{p_2}}{\Vert \mathbf{Ax}\Vert }_q∕{\Vert \mathbf{x}\Vert }_q$. The following are well known facts about the various norms of a matrix A = (aij),

$${\Vert \mathbf{A}\Vert }_{{\ell }_1}=\underset{j}{\max }\sum _{i=1}^{p_1}\mid a_{ij}\mid ,{\Vert \mathbf{A}\Vert }_{{\ell }_2}=\Vert \mathbf{A}\Vert ={\lambda }_1\left(\mathbf{A}\right),{\Vert \mathbf{A}\Vert }_{{\ell }_{\infty }}=\underset{i}{\max }\sum _{j=1}^{p_2}\mid a_{ij}\mid ,$$ (2)

and, if A is symmetric, ${\Vert \mathbf{A}\Vert }_{{\ell }_1}={\Vert \mathbf{A}\Vert }_{{\ell }_{\infty }}\ge {\Vert \mathbf{A}\Vert }_{{\ell }_2}$. When R₁, R₂ are two subsets of {1,..., p₁}, {1,..., p₂} respectively, we note A_{R1 × R2} = (a_ij)_{i∈R1,j∈R2} as the sub-matrix of A with indices R₁ and R₂. In addition, we simply write A_{R1 × R1} as A_R1.

We denote by X₁,..., X_n a complete random sample (without missing observations) from a p-dimensional distribution with mean μ and covariance matrix Σ. The sample mean and sample covariance matrix are defined as

$$\stackrel{‒}{\mathbf{X}}=\frac{1}{n}\sum _{k=1}^n{\mathbf{X}}_k,\widehat{\Sigma }=\frac{1}{n}\sum _{k=1}^n({\mathbf{X}}_k-\stackrel{‒}{\mathbf{X}}){({\mathbf{X}}_k-\stackrel{‒}{\mathbf{X}})}^{\mathsf{T}}.$$ (3)

Now we introduce the notation related to the incomplete data with missing observations. Generally, we use the superscript “*” to denote objects related to missing values. Let S₁, . . ., S_n be the indicator vectors for the observed values (see (1)) and let ${\mathbf{X}}^{\ast }=\{{\mathbf{X}}_1^{\ast },\dots ,{\mathbf{X}}_n^{\ast }\}$ be the observed incomplete data where the observed entries are indexed by the vectors S₁, . . ., S_n ∈ {0, 1}^p. In addition, we define

$$n_{ij}^{\ast }=\sum _{k=1}^n{S}_{ik}{S}_{jk},1\le i,j\le p.$$ (4)

Here $n_{ij}^{\ast }$ is the number of vectors ${\mathbf{X}}_k^{\ast }$ in which the i^th and j^th entries are both observed. For convenience, we also denote

$$n_i^{\ast }=n_{ii}^{\ast },n_{\min }^{\ast }=\underset{i,j}{\min }{n}_{ij}^{\ast },$$ (5)

Given a sample ${\mathbf{X}}^{\ast }=\{{\mathbf{X}}_1^{\ast },\dots ,{\mathbf{X}}_n^{\ast }\}$ with missing values, the sample mean and sample covariance matrix can no longer be calculated in the usual way. Instead, we propose the “generalized sample mean” ${\stackrel{‒}{\mathbf{X}}}^{\ast }$ defined by

$${\stackrel{‒}{\mathbf{X}}}^{\ast }={\left({\stackrel{‒}{X}}_i^{\ast }\right)}_{1\le i\le p}\text{with}{\stackrel{‒}{X}}_i^{\ast }=\frac{1}{n_i^{\ast }}\sum _{k=1}^n{X}_{ik}{S}_{ik},1\le i\le p,$$ (6)

where X_ik is the ith entry of X_k, and the “generalized sample covariance matrix” ${\widehat{\Sigma }}^{\ast }$ defined by

$${\widehat{\Sigma }}^{\ast }={\left({\widehat{\sigma }}_{ij}^{\ast }\right)}_{1\le i,j\le p}\text{with}{\widehat{\sigma }}_{ij}^{\ast }=\frac{1}{n_{ij}^{\ast }}\sum _{k=1}^n(X_{ik}-{\stackrel{‒}{X}}_i^{\ast })(X_{jk}-{\stackrel{‒}{X}}_j^{\ast })S_{ik}{S}_{jk}.$$ (7)

As will be seen later, the generalized sample mean ${\stackrel{‒}{\mathbf{X}}}^{\ast }$ and the generalized sample covariance matrix ${\widehat{\Sigma }}^{\ast }$ play similar roles as those of the conventional sample mean and sample covariance matrix in inference problems, but the technical analysis can be much more involved. Some distinctions between the generalized sample covariance matrix ${\widehat{\Sigma }}^{\ast }$ and the usual sample covariance matrix $\widehat{\Sigma }$ are that ${\widehat{\Sigma }}^{\ast }$ is in general not non-negative definite, and each entry ${\widehat{\sigma }}_{ij}^{\ast }$ is the average of a varying number $\left(n_{ij}^{\ast }\right)$ of samples, which create additional difficulties in the technical analysis.

Regarding the mechanism of missingness, the assumption we use for the theoretical analysis is missing completely at random. This is a more general setting than the one considered previously by Loh and Wainwright [21] and Lounici [22].

Assumption 2.1 (Missing Completely at Random (MCR)) S = {S₁, . . . , S_n} is not dependent on the values of X. Here S can be either deterministic or random, but independent of X.

We adopt Assumption 1 in Chen et al. [13] and assume that the random vector X is sub-Gaussian satisfying the following assumption.

Assumption 2.2 (Sub-Gaussian Assumption) X = {X₁, . . . , X_n}. Here the columns X_k are i.i.d. and can be expressed as

$${\mathbf{X}}_k=\Gamma {\mathbf{Z}}_k+\mu ,k=1,\dots ,n,$$ (8)

where μ is a fixed p-dimensional mean vector, $\Gamma \in {\mathbb{R}}^{p\times q}$ is a fixed matrix with q ≥ p so that ΓΓ^⊤ = Σ, Z_k = (Z_ik, . . . , Z_mk)^⊤ is an m-dimensional random vector with the components mean 0, variance 1, and i.i.d. sub-Gaussian, with the exception of i.i.d. Rademacher. More specifically, each Z_ik satisfies that $\mathrm{E}{Z}_{ik}=0,\mathrm{var}\left(Z_{ik}\right)=1,0<\mathrm{var}\left(Z_{ik}^2\right)<\infty$, and there exists τ > 0 such that Ee^tZ_ik ≤ exp(τt²/2) for all t > 0.

Note that the exclusion of the Rademacher distribution in Assumption 2.2 is only required for estimation of sparse covariance matrices. See Remark 3.3 for further discussions.

2.2 Rate-optimal Blockwise Tridiagonal Estimator

We follow Bickel [2] and Cai et al. [9] and consider estimating the covariance matrix Σ over the parameter space ${\mathcal{U}}_{\alpha }={\mathcal{U}}_{\alpha }(M_0,M)$ where

$${\mathcal{U}}_{\alpha }(M_0,M)=\left\{\Sigma :\underset{j}{\max }\sum _i\{\mid {\sigma }_{ij}\mid :\mid i-j\mid >k\}\le M{k}^{-\alpha }\text{for all}k,\Vert \Sigma \Vert \le M_0\right\}.$$ (9)

Suppose we have n i.i.d. samples with missing values ${\mathbf{X}}_1^{\ast },\dots ,{\mathbf{X}}_n^{\ast }$ with covariance matrix $\Sigma \in {\mathcal{U}}_{\alpha }(M_0,M)$. We propose a blockwise tridiagonal estimator ${\widehat{\Sigma }}^{\mathrm{bt}}$ to estimate Σ. We begin by dividing the generalized sample covariance matrix ${\widehat{\Sigma }}^{\ast }$ given by (7) into blocks of size k × k for some k. More specifically, pick an integer k and let N = ⌈p/k⌉. Set I_j = {(j – 1)k + 1, . . ., jk} for 1 ≤ j ≤ N – 1, and I_N = {(N – 1)k + 1, . . ., p} For 1 ≤ j, j′ ≤ N and A = (a_i1,i2)_p×p, define

$${\mathbf{A}}_{I_j\times I_{j^{\prime }}}={\left(a_{i_1,i_2}\right)}_{i_1\in I_j,i_2\in I_{j^{\prime }}}$$

and define the blockwise tridiagonal estimator ${\widehat{\Sigma }}^{\mathrm{bt}}$ by

$${\widehat{\Sigma }}_{I_j\times I_{j^{\prime }}}=\{\begin{array}{cc}{\widehat{\Sigma }}_{I_j\times I_{j^{\prime }}}^{\ast },\hfill & \text{if}\mid j-j^{\prime }\mid \le 1;\hfill \\ 0,\hfill & \text{otherwise}.\hfill \end{array}\phantom{\}}$$ (10)

That is, ${\widehat{\Sigma }}_{I_j\times I_{j^{\prime }}}$, is estimated by its sample counterpart of and only j and j′ differ by at most 1. The weight matrix of the blockwise tridiagonal estmator ${\widehat{\Sigma }}^{\mathrm{bt}}$ is illustrated in Figure 1.

Figure 1.

Weight matrix for the blockwise tridiagonal estimator.

Theorem 2.1 Suppose Assumptions 2.1 and 2.2 hold. Then, conditioning on S, the blockwise tridiagonal ${\widehat{\Sigma }}^{\mathrm{bt}}$ with $k={\left(n_{\min }^{\ast }\right)}^{1∕(2\alpha +1)}$ satisfies

$$\underset{\Sigma \in {\mathcal{U}}_{\alpha }(M,M_0)}{\sup }\mathrm{E}{\Vert {\widehat{\Sigma }}^{\mathrm{bt}}-\Sigma \Vert }^2\le C{\left(n_{\min }^{\ast }\right)}^{-2\alpha ∕(2\alpha +1)}+C\frac{\ln p}{n_{\min }^{\ast }},$$ (11)

where C is a constant depending only on M, M₀, and τ from Assumption 2.2.

The optimal choice of block size k depends on the unknown “smoothness parameter” α. In practice, k can be chosen by cross-validation. See Section 4.1 for further discussions. Moreover, the convergence rate in (11) is optimal as we also have the following lower bound result.

Proposition 2.1 For any n₀ ≥ 1 such that p ≤ exp(γn₀) for some constant γ < 0, conditioning on S we have

$$\underset{\widehat{\Sigma }}{\inf }\underset{\begin{array}{c}\hfill \Sigma \in {\mathcal{U}}_{\alpha }(M,M_0)\hfill \\ \hfill \mathbf{S}:n_{\min }^{\ast }\ge n_0\hfill \end{array}}{\sup }\mathrm{E}\left({\Vert \widehat{\Sigma }-\Sigma \Vert }^2\right)\ge C{\left(n_0\right)}^{-2\alpha ∕(2\alpha +1)}+C\frac{\ln p}{n_0}.$$

Remark 2.1 (Tapering and banding estimators) It should be noted that the same rate of convergence can also be attained by tapering and banding estimators with suitable choices of tapering and banding parameters. Specifically, let ${\widehat{\Sigma }}^{\mathrm{tp}}$ and ${\widehat{\Sigma }}^{\mathrm{bd}}$ be respectively the tapering and banded estimators proposed in Cai et al. [9] and Bickel and Levina [2] with

$${\widehat{\Sigma }}^{\mathrm{tp}}={\widehat{\Sigma }}_k^{\mathrm{tp}}={\left(w_{ij}^{\mathrm{tp}}{\widehat{\sigma }}_{ij}^{\ast }\right)}_{1\le i,j\le p}\text{and}{\widehat{\Sigma }}^{\mathrm{bd}}={\widehat{\Sigma }}_k^{\mathrm{bd}}={\left(W_{ij}^{\mathrm{bd}}{\widehat{\sigma }}_{ij}^{\ast }\right)}_{1\le i,j\le p},$$ (12)

where $w_{ij}^{\mathrm{tp}}$ and $w_{ij}^{\mathrm{bd}}$ are the weights defined as

$$w_{ij}^{\mathrm{tp}}=\{\begin{array}{cc}1,\hfill & \text{when}\mid i-j\mid \le k∕2,\hfill \\ 2-\frac{\mid i-j\mid }{k_h},\hfill & \text{when}k∕2<\mid i-j\mid <k\hfill \\ 0,\hfill & \text{otherwise}\hfill \end{array}\phantom{\}}\text{and}{w}_{ij}^{\mathrm{bd}}=\{\begin{array}{cc}1,\hfill & \text{when}\mid i-j\mid \le k,\hfill \\ 0,\hfill & \text{otherwise}\hfill \end{array}\phantom{\}}.$$ (13)

Then the estimators ${\widehat{\Sigma }}^{\mathrm{tp}}$ and ${\widehat{\Sigma }}^{\mathrm{bd}}$ with $k={\left(n_{\min }^{\ast }\right)}^{1∕(2\alpha +1)}$ attains the rate given in (11).

The proof of Theorem 2.1 shares some basic ideas with that for the complete data case (See, e.g. Theorem 2 in Cai et al. [9]). However, it relies on a new key technical tool which is a large deviation result for a sub-matrix of the generalized sample covariance matrix under the spectral norm. This random matrix result for the case of missing data, stated in the following lemma, can be potentially useful for other, related high-dimensional missing data problems. The proof of Lemma 2.1, given in Section 6, is more involved than the complete data case, as in the generalized sample covariance matrix each entry, ${\widehat{\sigma }}_{ij}^{\ast }$, is the average of a varying number of samples.

Lemma 2.1 Suppose Assumptions 2.1 and 2.2 hold. Let ${\widehat{\Sigma }}^{\ast }$ be the generalized sample covariance matrix defined in (7) and let A and B be two subsets of {1, . . ., p}. Then, conditioning on S, the submatrix ${\widehat{\Sigma }}_{A\times B}^{\ast }$ satisfies

$$\mathrm{Pr}\left(\Vert {\widehat{\Sigma }}_{A\times B}^{\ast }-{\Sigma }_{A\times B}\Vert \le x\right)\ge 1-C\cdot {\left(49\right)}^{\mid A\cup B\mid }\exp \left\{-c{n}_{\min }^{\ast }\min \left(\frac{x^2}{{\tau }^4\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert },\frac{x}{{\tau }^2{\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}}\right)\right\}$$ (14)

for all x > 0. Here C > 0 and c > 0 are two absolute constants.

3 Estimation of Sparse Covariance Matrices

In this section, we consider estimation of high-dimensional sparse covariance matrices in the presence of missing data. We introduce an adaptive thresholding estimator based on incomplete data and investigate its asymptotic properties.

3.1 Adaptive Thresholding Procedure

Sparse covariance matrices arise naturally in a range of applications including genomics. Estimation of sparse covariance matrices has been considered in several recent papers in the setting of complete data (see, e.g., Bickel and Levina [3], El Karoui [16], Rothman et al. [24], Cai and Zhou [10] and Cai and Liu [5]). Estimation of a sparse covariance matrix is intrinsically a heteroscedastic problem in the sense that the variances of the entries of the sample covariance matrix can vary over a wide range. To treat the heteroscedasticity of the sample covariances, Cai and Liu [5] introduced an adaptive thresholding procedure which adapts to the variability of the individual entries of the sample covariance matrix and outperforms the universal thresholding method. The estimator is shown to be simultaneously rate optimal over collections of sparse covariance matrices.

In the present setting of missing data, the usual sample covariance matrix is not available. Instead we apply the idea of adaptive thresholding to the generalized sample covariance matrix ${\widehat{\Sigma }}^{\ast }$. The procedure can be described as follows. Note that ${\widehat{\Sigma }}^{\ast }$ defined in (7) is a nearly unbiased estimate of Σ, we may write it element-wise as

$${\widehat{\sigma }}_{ij}^{\ast }\approx {\sigma }_{ij}+\sqrt{\frac{{\theta }_{ij}}{n_{ij}^{\ast }}}{z}_{ij},1\le i,j\le p,$$

where z_i is approximately normal with mean 0 and variance 1, and θ_ij describes the uncertainty of estimator ${\sigma }_{ij}^{\ast }$ to σ_ij such that

$${\theta }_{ij}=\mathrm{var}\{(X_i-{\mu }_i)(X_j-{\mu }_j)-{\sigma }_{ij}\}.$$

We can estimate θ_ij by

$${\widehat{\theta }}_{ij}^{\ast }=\frac{1}{n_{ij}^{\ast }}\sum _{k=1}^n{\{(X_{ik}-{\stackrel{‒}{X}}_i^{\ast })(X_{jk}-{\stackrel{‒}{X}}_j^{\ast })-{\widehat{\sigma }}_{ij}^{\ast }\}}^2{S}_{ik}{S}_{jk}.$$ (15)

Lemma 3.1 given at the end of this section shows that ${\widehat{\theta }}_{ij}^{\ast }$ is a good estimate of θ_ij.

Since the covariance matrix Σ is assumed to be sparse, it is natural to estimate Σ by individually thresholding ${\widehat{\theta }}_{ij}^{\ast }$ according to its own variability as measured by ${\widehat{\theta }}_{ij}^{\ast }$. Define the thresholding level λ_ij by

$${\lambda }_{ij}=\delta \sqrt{\frac{{\widehat{\theta }}_{ij}^{\ast }\ln p}{n_{ij}^{\ast }}},1\le i,j\le p,$$

where δ is a thresholding constant which can be taken as 2.

Let T_λ be a thresholding function satisfying the following conditions,

• (1).|T_λ(z)| ≤ c_T|y| for all z, y such that |z – y| ≤ λ;
• (2).T_λ(z) = 0 for |z| ≤ λ;
• (3).|T_λ(z) – z| ≤ λ, for all $z\in \mathbb{R}$.

These conditions are met by many well-used thresholding functions, including the soft thresholding rule T_λ(z) = sgn(z)(z – λ)₊, where sgn(z) is the sign function such that sgn(z) = 1 if z > 0, sgn(z) = 0 if z = 0, and sgn(z) = −1 if z < 0, and the adaptive lasso rule T_λ(z) = z(1 – |λ/z|^η)₊ with η ≤ 1 (see Rothman et al. [24]). The hard thresholding function does not satisfy Condition (1), but our analysis also applies to hard thresholding under similar conditions.

The covariance matrix Σ is estimated by ${\widehat{\Sigma }}^{\mathrm{at}}={\left({\widehat{\sigma }}_{ij}^{\mathrm{at}}\right)}_{1\le i,j\le p}$ where ${\widehat{\sigma }}_{ij}^{\mathrm{at}}$ is the thresholding estimator defined by

$${\widehat{\sigma }}_{ij}^{\mathrm{at}}=T_{{\lambda }_{ij}}\left({\widehat{\sigma }}_{ij}^{\ast }\right).$$ (16)

Note that here each entry ${\widehat{\sigma }}_{ij}^{\ast }$ is thresholded according to its own variability.

3.2 Asymptotic Properties

We now investigate the properties of the thresholding estimator ${\widehat{\Sigma }}^{\mathrm{at}}$ over the following parameter space for sparse covariance matrices,

$$\mathcal{H}\left(c_{n,p}\right)=\left\{\Sigma =\left({\sigma }_{ij}\right):\underset{1\le i\le p}{\max }\sum _{j=1}^p\min \left\{{\left({\sigma }_{ii}{\sigma }_{jj}\right)}^{1∕2},\frac{\mid {\sigma }_{ij}\mid }{\sqrt{\left(\ln p\right)∕n}}\right\}\le c_{n,p}\right\}.$$ (17)

The parameter space $\mathcal{H}\left(c_{n,p}\right)$ contains a large collection of sparse covariance matrices and does not impose any constraint on the variances σ_ii, i = 1, . . ., p. The collection $\mathcal{H}\left(c_{n,p}\right)$ contains some other commonly used classes of sparse covariance matrices in the literature, including an ℓ_q ball assumption ${\max }_i{\sum }_{j=1}^p{\mid {\sigma }_{ij}\mid }^q\le s_{n,p}$ in Bickel and Levina [3], and weak ℓ_q ball assumption ${\max }_{1\le j\le p}\left\{{\mid {\sigma }_{j\left[k\right]}\mid }^q\right\}\le s_{n,p}∕k$ for each integer k in Cai and Zhou [10] where |σ_j[k]| is the kth largest entry in magnitude of the jth row (σ_ij)_1≤i≤p. See Cai et al. [7] for more discussions.

We have the following result on the performance of ${\widehat{\Sigma }}^{\mathrm{at}}$ over the parameter space $\mathcal{H}\left(c_{n,p}\right)$.

Theorem 3.1 Suppose that δ ≥ 2, ln $p=o\left({\left(n_{\min }^{\ast }\right)}^{1∕3}\right)$ and Assumptions 2.1 and 2.2 hold. Then, conditioning on S, there exists a constant C not depending on p, $n_{\min }^{\ast }$ or n such that for any $\Sigma \in \mathcal{H}\left(c_{n,p}\right)$,

$$\mathrm{Pr}\left(\Vert {\widehat{\Sigma }}^{\mathrm{at}}-\Sigma \Vert \le C{c}_{n,p}\sqrt{\frac{\ln p}{n_{\min }^{\ast }}}\right)\ge 1-O\left\{{\left(\ln p\right)}^{-1∕2}{p}^{-\delta +2}\right\}.$$ (18)

Moreover, if we further assume that $p\ge {\left(n_{\min }^{\ast }\right)}^{\xi }$ and δ ≥ 4 + 1/ξ, we in addition have

$$\mathrm{E}\left({\Vert {\widehat{\Sigma }}^{\mathrm{at}}-\Sigma \Vert }^2\right)\le C{c}_{n,p}^2\frac{\ln p}{n_{\min }^{\ast }}.$$ (19)

Moreover, the lower bound result below shows that the rate in (19) is optimal.

Proposition 3.1 For any n₀ ≥ 1 and c_n,p > 0 such that $c_{n,p}\le M{n}_0^{1∕2}{\left(\ln p\right)}^{-3∕2}$ for some constant M < 0, conditioning on S we have

$$\underset{\widehat{\Sigma }}{\inf }\underset{\begin{array}{c}\hfill \Sigma \in \mathcal{H}\left(c_{n,p}\right)\hfill \\ \hfill \mathbf{S}:n_{\min }^{\ast }\ge n_0\hfill \end{array}}{\sup }\mathrm{E}\left({\Vert \widehat{\Sigma }-\Sigma \Vert }^2\right)\ge C{c}_{n,p}^2\frac{\ln p}{n_0}.$$

Remark 3.1 (ℓ_q norm loss) We focus in this paper on estimation under the spectral norm loss. The results given in Theorem 3.1 can be easily generalized to the general matrix ℓ_q norm for 1 ≤ q ≤ ∞. The results given in Equations (18) and (19) remain valid when the spectral norm is replaced by the matrix ℓ_q norm for 1 ≤ q ≤ ∞.

Remark 3.2 (Positive definiteness) Under mild conditions on Σ, the estimator ${\widehat{\Sigma }}^{\mathrm{at}}$ is positive definite with high probability. However, ${\widehat{\Sigma }}^{\mathrm{at}}$ is not guranteed to be positive definite for a given data set. Whenever ${\widehat{\Sigma }}^{\mathrm{at}}$ is not positive semi-definite, a simple extra step can make the final estimator ${\widehat{\Sigma }}_{+}^{\mathrm{at}}$ positive definite and also rate-optimal.

Write the eigen-decomposition of ${\widehat{\Sigma }}^{\mathrm{at}}$ as ${\widehat{\Sigma }}^{\mathrm{at}}={\sum }_{i=1}^p{\widehat{\lambda }}_i{\widehat{\mathbf{v}}}_i{\widehat{\mathbf{v}}}_i^{\mathsf{T}}$, where ${\widehat{\lambda }}_1\ge \cdots \ge {\widehat{\lambda }}_p$ are the eigenvalues and v̂_i are the corresponding eigenvectors. Define the final estimator

$${\widehat{\Sigma }}_{+}^{\mathrm{at}}={\widehat{\Sigma }}^{\mathrm{at}}+\left(\mid {\widehat{\lambda }}_p\mid +\frac{\ln p}{n_{\min }^{\ast }}\right)I\{{\widehat{\lambda }}_p<0\}\cdot {\mathbf{I}}_{p\times p},$$

where Ip×p is the p × p identity matrix. Then ${\widehat{\Sigma }}_{+}^{\mathrm{at}}$ is a positive definite matrix with the same structure as that of ${\widehat{\Sigma }}^{\mathrm{at}}$. It is easy to show that ${\widehat{\Sigma }}_{+}^{\mathrm{at}}$ and ${\widehat{\Sigma }}^{\mathrm{at}}$ attains the same rate of convergence over $\mathcal{H}\left(c_{n,p}\right)$. See Cai, Ren and Zhou [7] for further discussions.

Remark 3.3 (Exclusion of the Rademacher Distribution) To guarantee that ${\widehat{\theta }}_{ij}^{\ast }$ is a good estimate of θ_ij, one important condition needed in the theoretical analysis is that ${\theta }_{ij}∕\sqrt{{\sigma }_{ii}{\sigma }_{jj}}$ is bounded from below by a positive constant. Howver when the components of Z_k in (8) are i.i.d. Rademacher, it is possible that ${\theta }_{ij}∕\sqrt{{\sigma }_{ii}{\sigma }_{jj}}=0$. For example, If Z₁ and Z₂ are i.i.d. Rademacher and X_i = Z₁ – Z₂, then $\mathrm{var}\left(X_i{X}_j\right)=\mathrm{var}(Z_1^2-Z_2^2)=0$, and this implies ${\theta }_{ij}∕\sqrt{{\sigma }_{ii}{\sigma }_{jj}}=0$.

A key technical tool in the analysis of the adaptive thresholding estimator is a large deviation result for the self-normalized entries of the generalized sample covariance matrix. The following lemma, proved in Section 6, plays a critical role in the proof of Theorem 3.1 and can be useful for other high-dimensional inference problems with missing data.

Lemma 3.1 Suppose ln $p=o\left({\left(n_{\min }^{\ast }\right)}^{1∕3}\right)$ and Assumptions 2.1 and 2.2 hold. For any constants δ ≥ 2, ε > 0, M > 0, conditioning on S, we have

$$\mathrm{Pr}\left(\frac{\mid {\widehat{\sigma }}_{ij}^{\ast }-{\sigma }_{ij}\mid }{{\left({\widehat{\theta }}_{ij}^{\ast }\right)}^{1∕2}}\ge \delta \sqrt{\frac{\ln p}{n_{ij}^{\ast }}},\forall 1\le i,j\le p\right)=O\left\{{\left(\ln p\right)}^{-1∕2}{p}^{-\delta +2}\right\},$$ (20)

$$\mathrm{Pr}\left(\underset{ij}{\max }\frac{\mid {\widehat{\theta }}_{ij}^{\ast }-{\theta }_{ij}\mid }{{\sigma }_{ii}{\sigma }_{jj}}\ge \epsilon \right)=O\left(p^{-M}\right).$$ (21)

In addition to optimal estimation of a sparse covariance matrix Σ under the spectral norm loss, it is also of significant interest to recover the support of Σ, i.e., the locations of the nonzero entries of Σ. The problem has been studied in the case of complete data in, e.g., Cai and Liu [5] and Rothman et al. [24]. With incomplete data, the support can be similarly recovered through adaptive thresholding. Specifically, define the support of Σ = (σ_ij)_1≤i,j≤p by supp(Σ) = {(i, j) : σ_ij ≠ 0}. Under the condition that the non-zero entries of Σ are sufficiently bounded away from zero, the adaptive thresholding estimator ${\widehat{\Sigma }}^{\mathrm{at}}$ recovers the support supp(Σ) consistently. It is noteworthy that in the support recovery analysis, the sparsity assumption is not directly needed.

Theorem 3.2 (Support Recovery) Suppose ln $p=o\left({\left(n_{\min }^{\ast }\right)}^{1∕3}\right)$ and Assumptions 2.1 and 2.2 hold. Let γ be any positive constant. Suppose Σ satisfies

$$\mid {\sigma }_{ij}\mid >(4+\gamma )\sqrt{\frac{{\theta }_{ij}\ln p}{n_{ij}^{\ast }}},forall(i,j)\in \mathrm{supp}\left(\Sigma \right).$$ (22)

Let ${\widehat{\Sigma }}^{\mathrm{at}}$ be the adaptive thresholding estinmator with δ = 2, then, conditioning on S, we have

$$\mathrm{Pr}\left\{\mathrm{supp}\left({\widehat{\Sigma }}^{\mathrm{at}}\right)=\mathrm{supp}\left(\Sigma \right)\right\}\to 1asn,p\to \infty .$$ (23)

4 Numerical Results

We investigate in this section the numerical performance of the proposed estimators through simulations. The proposed adaptive thresholding procedure is also illustrated with an estimation of the covariance matrix based on data from four ovarian cancer studies.

The estimators ${\widehat{\Sigma }}^{\mathrm{bt}}$ and ${\widehat{\Sigma }}^{\mathrm{at}}$ introduced in the previous sections all require specification of the tuning parameters (k or δ). Cross-validation is a simple and practical data-driven method for the selection of these tuning parameters. Numerical results indicate that the proposed estimators with the tuning parameter selected by cross-validation perform well empirically. We begin by introducing the following K-fold cross-validation method for the empirical selection of the tuning parameters.

4.1 Cross-validation

For a pre-specified positive integer N, we construct a grid T of non-negative numbers. For bandable covariance matrix estimation, we set T = {1, ⌈p^1/Nl⌉,..., ⌈p^N/N⌉}, and for sparse covariance matrix estimation, we let T = {0,1/N,..., 4N/N}.

Given n samples ${\mathbf{X}}^{\ast }\in {\mathbb{R}}^{p\times n}$ with mising values, for a given positive integer K, we randomly divide them into two groups of size n₁ ≈ n(K – 1)/K, n₂ ≈ n/K for H times. For h = 1, . . . , H, we denote by $J_1^h$ and $J_2^h\subseteq \{1,\dots ,n\}$ the index sets of the two groups for the h-th split. The proposed estimator, ${\widehat{\Sigma }}^{\mathrm{bt}}$ for bandable covariance matrices, or ${\widehat{\Sigma }}^{\mathrm{at}}$ for sparse covariance matrices, is then applied to the first group of data ${\mathbf{X}}_{J_1^h}^{\ast }$ with each value of the tuning parameter t ∈ T and denote the result by ${\widehat{\Sigma }}_h^{\mathrm{bt}}\left(t\right)$ or ${\widehat{\Sigma }}_h^{\mathrm{at}}\left(t\right)$ respectively. Denote the generalized sample covariance matrix of the second group of data ${\mathbf{X}}_{J_2^h}^{\ast }$ by ${\widehat{\Sigma }}_h^{\ast }$ and set

$$\widehat{R}\left(t\right)=\frac{1}{H}\sum _{h=1}^H{\Vert {\widehat{\Sigma }}_h\left(t\right)-{\widehat{\Sigma }}_h^{\ast }\Vert }_F^2,$$ (24)

where ${\widehat{\Sigma }}_h\left(t\right)$ is either ${\widehat{\Sigma }}^{\mathrm{bt}}\left(t\right)$ for bandable covariance matrices, or ${\widehat{\Sigma }}^{\mathrm{at}}\left(t\right)$ for sparse covariance matrices.

The final tuning parameter is chosen to be

$$t_{\ast }=\underset{T}{\arg \min }\widehat{R}\left(t\right)$$

and the final estimator ${\widehat{\Sigma }}^{\mathrm{bt}}$ (or ${\widehat{\Sigma }}^{\mathrm{at}}$) is calculated using this choice of the tuning parameter t_*. In the following numerical studies, we will use 5-fold cross-validation (i.e., K = 5) to select the tuning parameters.

Remark 4.1 The Frobenius norm used in (24) can be replaced by other losses such as the spectral norm. Our simulation results indicate that using the Frobenius norm in (24) works well, even when the true loss is the spectral norm loss.

4.2 Simulation Studies

In the simulation studies, we consider the following two settings for the missingness. The first is MUCR where each entry X_ik is observed with probability 0 < ρ ≤ 1, and the second is missing not uniformly but completely at random (MCR) where the complete data matrix X is divided into four equal-size blocks,

$$\mathbf{X}=\left[\begin{array}{cc}\hfill {\mathbf{X}}_{\left(11\right)}\hfill & \hfill {\mathbf{X}}_{\left(12\right)}\hfill \\ \hfill {\mathbf{X}}_{\left(21\right)}\hfill & \hfill {\mathbf{X}}_{\left(22\right)}\hfill \end{array}\right],{\mathbf{X}}_{\left(11\right)},{\mathbf{X}}_{\left(12\right)},{\mathbf{X}}_{\left(21\right)},{\mathbf{X}}_{\left(22\right)}\in {\mathbb{R}}^{\frac{p}{2}\times \frac{n}{2}},$$

and each entry of X₍₁₁₎ and X₍₂₂₎ is observed with probability ρ⁽¹⁾ and each entry of X₍₁₂₎ and X₍₂₁₎ is observed with probability ρ⁽²⁾, for some 0 < ρ⁽¹⁾, ρ⁽²⁾ ≤ 1.

As mentioned in the introduction, high-dimensional inference for missing data has been studied in the case of MUCR and we would like to compare our estimators with the corresponding estimators based on a different sample covariance matrix designed for the MUCR case. Under the assumption that EX = 0 and each entry of X is observed independently with probability ρ, Wainwright [21] and Lounici [23] introduced the following substitute of the usual sample covariance matrix

$${\widehat{\Sigma }}^{•}={\left({\sigma }_{ij}^{•}\right)}_{1\le i,j\le p}\text{with}{\widehat{\sigma }}_{ij}^{•}=\{\begin{array}{cc}\frac{1}{n{(1-\rho )}^2}{\sum }_{k=1}^n{X}_{ik}^{\ast }{X}_{jk}^{\ast },\hfill & i\ne j\hfill \\ \frac{1}{n(1-\rho )}{\sum }_{k=1}^n{X}_{ik}^{\ast }{X}_{jk}^{\ast },\hfill & i=j\hfill \end{array}\phantom{\}}$$ (25)

where the missing entries of X* are replaced by 0's. It is easy to show that ${\widehat{\Sigma }}^{•}$ is a consistent estimator of under MUCR and could be used similarly as the sample covariance matrix in the complete data setting.

For more general settings where EX ≠ 0 and the coordinates X₁, X₂, . . ., X_p are observed with different probabilities ρ₁, . . . , ρ_p, ${\widehat{\Sigma }}^{•}$ can be generalized as

$${\widehat{\Sigma }}^{•}={\left({\widehat{\sigma }}_{ij}^{•}\right)}_{1\le i,j\le p}\text{with}{\widehat{\sigma }}_{ij}^{•}=\{\begin{array}{cc}\frac{1}{n(1-{\widehat{\rho }}_i)(1-{\widehat{\rho }}_j)}{\sum }_{k=1}^n{X}_{ik,c}^{\ast }{X}_{jk,c}^{\ast },\hfill & i\ne j\hfill \\ \frac{1}{n(1-{\widehat{\rho }}_i)}{\sum }_{k=1}^n{X}_{ik,c}^{\ast }{X}_{jk,c}^{\ast },\hfill & i=j\hfill \end{array}\phantom{\}}$$ (26)

where for i = 1, . . . , p and k = 1, . . . , n, ${\widehat{\rho }}_i=\frac{1}{n}{\sum }_{k=1}^n{S}_{ik}$ and $X_{ik,c}^{\ast }=X_{ik}^{\ast }-{\stackrel{‒}{X}}_i^{\ast }$.

Based on ${\widehat{\Sigma }}^{•}$, we can analogously define the corresponding blockwise tridiagonal estimator ${\widehat{\Sigma }}^{\mathrm{bt}•}$ for bandable covariance matrices, and adaptive thresholding estimator ${\widehat{\Sigma }}^{\mathrm{at}•}$ for sparse covariance matrices.

We first consider estimation of bandable covariance matrices and compare the proposed blockwise tridiagonal estimator ${\widehat{\Sigma }}^{\mathrm{bt}}$ with the corresponding estimator ${\widehat{\Sigma }}^{\mathrm{bt}•}$. For both methods, the tuning parameter k is selected by 5-fold cross-validation with N varying from 20 to 50. The following bandable covariance matrices are considered:

1. (Linear decaying bandable model) Σ = (σ_ij)_1≤i,j≤p with σ_ij = max{0, 1 – |i – j|/5}.

2. (Squared decaying bandable model) Σ = (σ_ij)_1≤i,j≤p with σ_ij = (|i – j| + 1)⁻².

For missingness, both MUCR and MCR are considered and (25) and (26) are used to calculate ${\widehat{\Sigma }}^{•}$ respectively. The proposed procedure ${\widehat{\Sigma }}^{\mathrm{bt}}$ is compared with the estimator ${\widehat{\Sigma }}^{\mathrm{bt}•}$, which is based on ${\widehat{\Sigma }}^{•}$. The results for the spectral norm, ℓ₁ norm and Frobenius norm losses are reported in Table 1. It is easy to see from Table 1 that the proposed estimator ${\widehat{\Sigma }}^{\mathrm{bt}}$ generally outperforms ${\widehat{\Sigma }}^{\mathrm{bt}•}$, especially in the fast decaying setting.

Table 1.

Comparsion between ${\widehat{\Sigma }}^{\mathrm{bt}}$ and ${\widehat{\Sigma }}^{\mathrm{bt}•}$ in different settings of bandable covariance matrix estimation.

	Spectral norm		ℓ1 norm		Frobenius norm
(p, n)	${\widehat{\Sigma }}^{\mathrm{bt}}$	${\widehat{\Sigma }}^{\mathrm{bt}•}$	${\widehat{\Sigma }}^{\mathrm{bt}}$	${\widehat{\Sigma }}^{\mathrm{bt}•}$	${\widehat{\Sigma }}^{\mathrm{bt}}$	${\widehat{\Sigma }}^{\mathrm{bt}•}$
Linear Decay Bandable Model, MUCR ρ = .5
(50, 50)	2.78(0.17)	2.88(0.18)	4.37(0.57)	4.57(0.76)	7.73(0.85)	7.85(0.80)
(50, 200)	1.44(0.06)	1.56(0.07)	2.52(0.17)	2.71(0.19)	3.91(0.18)	4.16(0.16)
(200, 100)	2.25(0.13)	2.44(0.16)	3.83(0.32)	4.22(0.46)	10.27(0.29)	10.89(0.29)
(200, 200)	1.67(0.07)	1.82(0.08)	2.81(0.19)	3.08(0.22)	7.19(0.19)	7.68(0.14)
(500, 200)	2.00(0.07)	2.18(0.10)	3.45(0.16)	3.74(0.27)	12.10(0.36)	12.87(0.42)
Squared Decay Bandable Model, MUCR ρ = .5
(50, 50)	1.34(0.08)	1.40(0.11)	2.28(0.16)	2.37(0.21)	3.78(0.19)	3.91(0.18)
(50, 200)	0.82(0.01)	0.84(0.01)	1.47(0.03)	1.49(0.02)	2.24(0.02)	2.30(0.02)
(200, 100)	1.13(0.01)	1.17(0.02)	2.12(0.05)	2.18(0.07)	5.74(0.04)	5.91(0.05)
(200, 200)	0.92(0.00)	0.94(0.00)	1.66(0.02)	1.72(0.03)	4.49(0.02)	4.61(0.01)
(500, 200)	0.97(0.00)	0.98(0.00)	1.80(0.02)	1.86(0.02)	7.15(0.01)	7.35(0.01)
Linear Decay Bandable Model, MCR ρ(1) = .8, ρ(2) = .2
(50, 50)	2.76(0.26)	3.46(1.43)	4.24(0.73)	5.87(2.91)	7.03(1.25)	8.47(1.29)
(50, 200)	1.51(0.11)	2.64(0.40)	2.52(0.30)	4.29(0.99)	3.62(0.30)	5.77(0.45)
(200, 100)	2.32(0.22)	3.93(0.67)	3.73(0.47)	6.21(1.11)	9.04(0.48)	13.47(0.84)
(200, 200)	1.67(0.10)	3.23(0.27)	2.71(0.26)	4.91(0.49)	6.32(0.11)	11.32(0.49)
(500, 200)	1.98(0.09)	3.78(0.20)	3.19(0.20)	5.70(0.42)	10.39(0.12)	18.48(0.49)
Squared Decay Bandable Model, MCR ρ(1) = .8, ρ(2) = .2
(50, 50)	1.26(0.08)	1.49(0.13)	2.21(0.23)	2.60(0.28)	3.48(0.14)	4.18(0.23)
(50, 200)	0.82(0.01)	0.88(0.04)	1.47(0.05)	1.77(0.11)	2.18(0.04)	2.68(0.11)
(200, 100)	1.06(0.01)	1.30(0.04)	1.96(0.04)	2.44(0.07)	5.32(0.02)	6.51(0.06)
(200, 200)	0.90(0.00)	0.96(0.03)	1.60(0.02)	1.99(0.06)	4.27(0.02)	5.26(0.15)
(500, 200)	0.93(0.00)	1.03(0.01)	1.69(0.01)	2.11(0.03)	6.73(0.01)	8.25(0.04)

Now we consider estimation of sparse covariance matrices with missing values under the following two models.

1. (Permutation Bandable Model) Σ = (σ_ij)_1≤i,j≤p, where σ_ij = max(0, 1–0.2·|s(i)–s(j)|) and s(i), i = 1, . . ., p is a random permutation of {1, . . ., p}.

2. (Randomly Sparse Model) Σ = I_p + (D + D⊤)/(∥D + D≊∥ + 0.01), where D is randomly generated as

   $$\mathbf{D}={\left(d_{ij}\right)}_{1\le i,j\le p},d_{ij}=\{\begin{array}{cc}1\hfill & \mathrm{w}.\mathrm{p}.0.1\hfill \\ 0\hfill & \mathrm{w}.\mathrm{p}.0.8\hfill \\ -1\hfill & \mathrm{w}.\mathrm{p}.0.1\hfill \end{array}\phantom{\}}\text{for}i\ne j;d_{ii}=0.$$

   Similar to the sparse covariance matrix estimation, for missingness, we consider both MUCR and MCR. The results for the spectral norm, matrix ℓ₁ norm and Frobenius norm losses are summarized in Table 2. It can be seen from Table 2 that, even under the MUCR setting, the proposed estimator ${\widehat{\Sigma }}^{\mathrm{at}}$ based on the generalized sample covariance matrix is uniformly better than the one based on ${\widehat{\Sigma }}^{•}$. In the more general MCR setting, the difference in the performance between the two estimators is even more significant.

Table 2.

Comparsion between ${\widehat{\Sigma }}^{\mathrm{at}}$ and ${\widehat{\Sigma }}^{\mathrm{at}•}$ in different settings of sparse covariance matrix estimation.

	Spectral norm		ℓ1 norm		Frobenius norm
(p, n)	${\widehat{\Sigma }}^{\mathrm{at}}$	${\widehat{\Sigma }}^{\mathrm{at}•}$	${\widehat{\Sigma }}^{\mathrm{at}}$	${\widehat{\Sigma }}^{\mathrm{at}•}$	${\widehat{\Sigma }}^{\mathrm{at}}$	${\widehat{\Sigma }}^{\mathrm{at}•}$
Permutation Bandable Model, MUCR ρ = .5
(50, 50)	4.26(0.24)	4.45(0.41)	5.58(0.58)	6.19(7.54)	11.34(0.79)	11.73(1.08)
(50, 200)	1.70(0.05)	1.74(0.06)	3.31(0.32)	3.42(0.38)	4.93(0.09)	5.07(0.16)
(200, 100)	3.48(0.07)	3.66(0.58)	5.80(0.39)	6.23(14.89)	18.34(0.81)	19.37(5.50)
(200, 200)	2.12(0.04)	2.20(0.03)	4.17(0.29)	4.44(0.32)	11.46(0.14)	11.94(0.13)
(500, 200)	2.28(0.03)	3.51(0.17)	4.17(0.15)	6.55(0.72)	16.85(0.10)	21.96(0.49)
Randomly Sparse Model, MUCR ρ = .5
(50, 50)	1.76(0.07)	1.96(0.62)	3.69(0.24)	4.20(5.89)	5.75(0.51)	6.27(2.95)
(50, 200)	1.05(0.00)	1.06(0.00)	2.73(0.04)	2.74(0.05)	3.75(0.03)	3.77(0.04)
(200, 100)	1.40(0.01)	1.45(0.01)	4.88(0.08)	4.94(0.09)	8.34(0.07)	8.50(0.07)
(200, 200)	1.07(0.00)	1.09(0.01)	4.44(0.03)	4.46(0.03)	7.42(0.02)	7.43(0.02)
(500, 200)	1.14(0.01)	1.31(0.01)	6.39(0.04)	6.65(0.08)	11.73(0.01)	12.23(0.05)
Permutation Bandable Model, MCR ρ(1) = .8, ρ(2) = .2
(50, 50)	4.23(0.38)	4.71(1.17)	6.67(2.30)	7.46(8.92)	11.22(1.34)	11.71(2.01)
(50, 200)	1.64(0.05)	2.79(0.39)	2.94(0.21)	4.52(0.95)	4.41(0.13)	6.29(0.46)
(200, 100)	3.17(0.06)	4.16(0.57)	5.73(0.66)	8.11(1.87)	15.93(0.53)	18.03(0.77)
(200, 200)	2.00(0.03)	3.22(0.18)	3.65(0.16)	5.70(0.60)	9.83(0.11)	13.29(0.55)
(500, 200)	2.22(0.03)	3.45(0.17)	4.09(0.17)	6.44(0.96)	16.80(0.14)	21.93(0.45)
Randomly Sparse Model, MCR ρ(1) = .8, ρ(2) = .2
(50, 50)	2.15(0.46)	2.19(0.49)	4.21(0.94)	4.47(4.65)	6.36(0.96)	7.25(1.57)
(50, 200)	1.09(0.02)	1.16(0.04)	2.82(0.19)	2.99(0.32)	3.83(0.10)	4.00(0.20)
(200, 100)	1.46(0.02)	1.82(0.03)	4.96(0.12)	5.61(0.21)	8.45(0.07)	10.10(0.14)
(200, 200)	1.08(0.00)	1.20(0.01)	4.46(0.04)	4.57(0.05)	7.43(0.02)	7.66(0.04)
(500, 200)	1.12(0.01)	1.33(0.01)	6.35(0.04)	6.60(0.07)	11.71(0.02)	12.20(0.06)

4.3 Comparison with Complete Samples

For covariance matrix estimation with missing data, an interesting question is: what is the “effective sample size”? That is, for samples with missing values, we would like to know the equivalent size of complete samples such that the accuracy for covariance matrix estimation is approximately the same. We now compare the performance of the proposed estimator based on the incomplete data with the corresponding estimator based on the complete data for various sample sizes. We fix the dimension p = 100. For the incomplete data, we consider n = 1000 and MUCR with ρ = .5. The covariance matrix Σ is chosen as

• Linear Decaying Bandable Model (in Bandable Covariance Matrix Estimation);

• Permutation Bandable Model (in Sparse Covariance Matrix Estimation);

Correspondingly, we consider the similar settings for the complete data with the same Σ and p, but different sample size n_c, where n_c can be one of the following three values,

1. $\overline{n_{\mathrm{pair}}^{\ast }}={\sum }_{i,j=1}^n{n}_{ij}^{\ast }∕p^2$: the average number of pairs of (x_i, x_j)'s that can be observed within the same sample;

2. $\overline{n_{\mathrm{s}}^{\ast }}={\sum }_{i=1}^n{n}_i^{\ast }∕p$: the average number of single x_i's can be observed;

3. n: the same number of samples with the missing values.

The results for all the settings are summarized in Table 3. It can be seen that the equivalent sample size depends on the loss function and in general it is between $\overline{n_{\mathrm{pair}}^{\ast }}$ and $\overline{n_{\mathrm{s}}^{\ast }}$. Overall, the average risk under the missing data setting is most comparable to that under the complete data setting for the sample size of $n_c=\overline{n_{\mathrm{pair}}^{\ast }}$, the average number of observed pairs.

Table 3.

Comparison between incomplete samples and complete samples.

Setting	sample size	Spectral norm	ℓ1 norm	Frobenius norm
Bandable Covariance Matrix Estimation
Missing Data	n = 1000	0.72(0.01)	1.25(0.03)	2.40(0.01)
Complete Data	$n_c=\overline{n_{\text{pair}}^{\ast }}$	0.97(0.03)	1.49(0.05)	2.48(0.04)
Complete Data	$n_c=\overline{n_{\mathrm{s}}^{\ast }}$	0.65(0.01)	1.01(0.03)	1.69(0.03)
Complete Data	nc = n	0.48(0.01)	0.73(0.01)	1.22(0.01)
Sparse Covariance Matrix Estimation
Missing Data	n = 1000	0.75(0.01)	1.37(0.04)	2.90(0.02)
Complete Data	$n_c=\overline{n_{\text{pair}}^{\ast }}$	0.83(0.02)	1.31(0.05)	2.94(0.04)
Complete Data	$n_c=\overline{n_{\mathrm{s}}^{\ast }}$	0.65(0.01)	1.01(0.03)	1.86(0.04)
Complete Data	nc = n	0.45(0.01)	0.64(0.01)	1.12(0.01)

4.4 Analysis of Ovarian Cancer Data

In this section, we illustrate the proposed adaptive thresholding procedure with an application to data from four ovarian cancer genomic studies, Cancer Genome Atlas Research Network [11] (TCGA), Bonome et al. [4] (BONO), Dressman et al. [15] (DRES) and Tothill et al. [27] (TOTH). The method introduced in Sections 3 enables us to estimate the covariance matrix by integrating data from all four studies and thus yields a more accurate estimator. The data structure is illustrated in Figure 2. The gene expression markers (the first 426 rows) are observed in all four studies without any missingness (the top black block in Figure 2). The miRNA expression markers are observed in 552 samples from the TCGA study (the bottom left block in Figure 2) and completely missing in the 881 samples from the TOTH, DRES, BONO and part of TCGA studies (the white block in Figure 2).

Figure 2.

Illustration of the ovarian cancer dataset. Black block = completely observed; White block = completely missing.

Our goal is to estimate the covariance matrix Σ of the 1225 variables with the particular interest in the cross-covariances between the gene and miRNA expression markers. It is clear that the missingness here is not uniformly at random. On the other hand, it is reasonable to assume the missingness does not depend on the value of the data and thus missing completely at random (Assumption 2.1) can be assumed. We apply the adaptive thresholding procedure with δ = 2 to estimate the covariance matrix and recover its support based on all the observations. The support of the estimate is shown in a heatmap in Figure 3. The left panel is for the whole covariance matrix and the right panel zooms into the cross-covariances between the gene and miRNA expression markers.

Figure 3.

Heatmaps of the covariance matrix estimate with all the observed data.

It can be seen from Figure 3 that the two diagonal blocks, with 12.24% and 8.39% nonzero off-diagonal entries respectively, are relatively dense, indicating that the relationships among the gene expression markers and those among the miRNA expression markers, as measured by their covariances, are relatively close. In contrast, the cross-covariances between gene and miRNA expression markers are very sparse with only 0.38% of significant gene-miRNA pairs. The gene and miRNA expression markers a ect each other through different mechanisms, the cross-covariances between the gene and miRNA markers are of significant interest (see Ko et al. [19]). It is worthwhile to take a closer look at the cross-covariance matrix displayed on the right panel in Figure 3. For each given gene, we count the number of miRNAs whose covariances with this gene are significant, and then rank all the genes by the counts. Similarly, we rank all the miRNAs. The top 5 genes and the top 5 miRNA expression markers are shown in Table 4.4.

Table 4.

Genes and miRNA's with most selected pairs

Gene Expression Marker	Counts	miRNA Expression Marker	Counts
ACTA2	61	hsa-miR-142-5p	31
INHBA	57	hsa-miR-142-3p	29
COL10A1	53	hsa-miR-22	26
BGN	46	hsa-miR-21*	24
NID1	41	hsa-miR-146a	21

Many of these gene and miRNA expression markers have been studied before in the literature. For example, the miRNA expression markers hsa-miR-142-5p and hsa-miR-142-3p have been demonstrated in Andreopoulos and Anastassiou [1] as standing out among the miRNA markers as having higher correlations with more genes, as well as methylation sites. Carraro et al. [12] finds that inhibition of miR-142-3p leads to ectopic expression of the gene marker ACTA2. This indicates strong interaction between miR-142-3p and ACTA2.

To further demonstrate the robustness of our proposed procedure against missingness, we consider a setting with additional missing observations. We first randomly select half of the 552 complete samples (where both gene and miRNA expression markers are observed) and half of the 881 incomplete samples (where only gene expression markers are observed), and then independently mask each entry of the selected samples with probability 0.05. The proposed adaptive thresholding procedure is then applied to the data with these additional missing values. The estimated covariance matrix is shown in heatmaps in Figure 4. These additional missing observations do not significantly a ect the estimation accuracy. Figure 4 is visually very similar to Figure 3. To quantify the similarity between the two estimates, we calculate the Matthews correlation coefficient (MCC) between them. The value of MCC is equal to 0.9441, which indicates that the estimate based on the data with the additional missingness is very close to the estimate based on the original samples. We also pay close attention to the cross-covariance matrix displayed on the right panel in Figure 4 and rank the gene and miRNA expression markers in the same way as before. The top 5 genes and the top 5 miRNA expression markers, listed in Table 5, are nearly identical to those given in Table 4.4, which are based on the original samples. These results indicate that the proposed method is robust against additional missingness.

Figure 4.

Heatmaps of the covariance matrix estimate with additional missing values.

Table 5.

Genes and miRNA's with most selected pairs after masking

Gene Expression Marker	Counts	miRNA Expression Marker	Counts
ACTA2	60	hsa-miR-142-3p	31
INHBA	56	hsa-miR-142-5p	30
COL10A1	50	hsa-miR-146a	21
BGN	43	hsa-miR-150	21
NID1	40	hsa-miR-21*	21

5 Discussions

We considered in the present paper estimation of bandable and sparse covariance matrices in the presence of missing observations. The pivotal quantity is the generalized sample covariance matrix defined in (7). The technical analysis is more challenging due to the missing data. We have mainly focused on the spectral norm loss in the theoretical analysis. Performance under other losses such as the Frobenius norm can also be analyzed.

To illustrate the proposed methods, we integrated four ovarian cancer studies. These methods for high-dimensional covariance matrix estimation with missing data are also useful for other types of data integration. For example, linking multiple data sources such as electronic data records, medicare data, registry data and patient reported outcomes could greatly increase the power of exploratory studies such as phenome-wide association studies (Denny et al. [14]). However, missing data inevitably arises and may hinder the potential of integrative analysis. In addition to random missingness due to unavailable information on a small fraction of patients, many variables such as the genetic measurements may only exist in one or two data sources and are hence structurally missing for other data sources. Our proposed methods could potentially provide accurate recovery of the covariance matrix in the presence of missingness.

In this paper, we allowed the proportion of missing values to be non-negligible as long as the minimum number of occurrences of any pair of variables $n_{\min }^{\ast }$ is of order n. An interesting question is what happens when the number of observed values is large but $n_{\min }^{\ast }$ is small (or even zero). We believe that the covariance matrix Σ can still be well estimated under certain global structural assumptions. This is out of the scope of the present paper and is an interesting problem for future research.

The key ideas and techniques developed in this paper can be used for a range of other related problems in high-dimensional statistical inference with missing data. For example, the same techniques can also be applied to estimation of other structured covariance matrices such as Toeplitz matrices, which have been studied in the literature in the case of complete data. When there are missing data, we can construct similar estimators using the generalized sample covariance matrix. The large deviation bounds for a sub-matrix and self-normalized entries of the generalized sample covariance matrix developed in Lemmas 3.1 and 2.1 would be helpful for analyzing the properties of the estimators.

The techniques can also be used on two-sample problems such as estimation of differential correlation matrices and hypothesis testing on the covariance structures. The generalized sample covariance matrix can be standardized to form the generalized sample correlation matrix which can then be used to estimate the differential correlation matrix in the two-sample case. It is also of significant interest in some applications to test the covariance structures in both one- and two-sample settings based on incomplete data. In the one-sample case, it is of interest to test the hypothesis {H₀ : Σ = I} or {H₀ : R = I}, where R is the correlation matrix. In the two-sample case, one wishes to test the equality of two covariance matrices {H₀ : Σ₁ = Σ₂}. These are interesting problems for further exploration in the future.

6 Proofs

We prove Theorem 2.1 and the key technical result Lemma 6.1 for the bandable covariance matrix estimation in this section.

6.1 Proof of Lemma 2.1

To prove this lemma, we first introduce the following technical tool for the spectral norm of the sub-matrices.

Lemma 6.1 Suppose $\Sigma \in {\mathbb{R}}^{p\times p}$ is any positive semi-definite matrix, A, B ∈ {1, . . ., p}, then

$$\Vert {\Sigma }_{A\times B}\Vert \le {\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}.$$ (27)

The proof of Lemma 6.1 is provided later and now we move back to the proof of Lemma 2.1. Without loss of generality, we assume that μ = EX = 0. We further define

$${\breve{\Sigma }}^{\ast }={\left({\breve{\sigma }}_{ij}^{\ast }\right)}_{1\le i,j\le p},{\breve{\sigma }}_{ij}^{\ast }=\frac{1}{n_{ij}^{\ast }}\sum _{k=1}^n{X}_{ik}{X}_{jk}{S}_{ik}{S}_{jk}.$$ (28)

Also for convenience of presentation, we use C, C₁, c, . . . to denote uniform constants, whose exact values may vary in different senarios. The lemma is now proved in the following steps:

1. We first consider for fixed unit vectors a, $\mathbf{b}\in {\mathbb{R}}^p$ with $\mathrm{supp}\left(\mathbf{a}\right)\subseteq A$, $\mathrm{supp}\left(\mathbf{b}\right)\subseteq B$, the tail bound of ${\mathbf{a}}^{\mathsf{T}}({\widehat{\Sigma }}^{\ast }-\Sigma )\mathbf{b}$. We would like to show that there exist uniform constants C_{1, c} > 0 such that for all x < 0,

   $$\mathrm{Pr}\left\{\mid {\mathrm{a}}^{\mathsf{T}}\left({\widehat{\Sigma }}^{\ast }-\Sigma \right)\mathbf{b}\mid \ge x\right\}\le C_1\exp \left\{-c{n}_{\min }^{\ast }\min \left(\frac{x^2}{{\tau }^4\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert },\frac{x}{{\tau }^2{\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}}\right)\right\}.$$ (29)

   Specifically, we will bound ${\mathbf{a}}^{\mathsf{T}}({\breve{\Sigma }}^{\ast }-\widehat{\Sigma })\mathbf{b}$ and ${\mathbf{a}}^{\mathsf{T}}({\breve{\Sigma }}^{\ast }-\Sigma )\mathbf{b}$ separately in the next two steps.
2. We consider ${\mathbf{a}}^{\mathsf{T}}({\breve{\Sigma }}^{\ast }-\widehat{\Sigma })\mathbf{b}$ first. Since

   $${\breve{\sigma }}_{ij}^{\ast }-{\widehat{\sigma }}_{ij}^{\ast }=\frac{1}{n_{ij}^{\ast }}\sum _{k=1}^n(X_{jk}{\stackrel{‒}{X}}_i^{\ast }+X_{ik}{\stackrel{‒}{X}}_j^{\ast })S_{ik}{S}_{jk}-{\stackrel{‒}{X}}_i^{\ast }{\stackrel{‒}{X}}_j^{\ast },$$

   ${\mathbf{a}}^{\mathsf{T}}({\breve{\Sigma }}^{\ast }-{\widehat{\Sigma }}^{\ast })\mathbf{b}$ can be written as

   $${\mathbf{a}}^{\mathsf{T}}({\breve{\Sigma }}^{\ast }-{\widehat{\Sigma }}^{\ast })\mathbf{b}=\sum _{i,j=1}^p{a}_i{b}_j({\breve{\sigma }}_{ij}^{\ast }-{\widehat{\sigma }}_{ij}^{\ast })=\sum _{i,j=1}^p{a}_i{b}_j\left(\frac{{\sum }_{k=1}^n{X}_{ik}{S}_{ik}}{n_i^{\ast }}\cdot \frac{{\sum }_{l=1}^n{X}_{jl}{S}_{il}{S}_{jl}}{n_{ij}^{\ast }}+\frac{{\sum }_{k=1}^n{X}_{ik}{S}_{ik}{S}_{jk}}{n_{ij}^{\ast }}\cdot \frac{{\sum }_{l=1}^n{X}_{jl}{S}_{jl}}{n_j^{\ast }}-\frac{{\sum }_{k=1}^n{X}_{ik}{S}_{ik}}{n_i^{\ast }}\cdot \frac{{\sum }_{l=1}^n{X}_{jl}{S}_{jl}}{n_j^{\ast }}\right)=\sum _{i,j=1}^p\sum _{k,l=1}^n{X}_{ik}{X}_{jl}{a}_i{b}_j\left(\frac{S_{ik}{S}_{il}{S}_{jl}}{n_i^{\ast }{n}_{ij}^{\ast }}+\frac{S_{ik}{S}_{jk}{S}_{jl}}{n_{ij}^{\ast }{n}_j^{\ast }}-\frac{S_{ik}{S}_{jl}}{n_i^{\ast }{n}_j^{\ast }}\right).$$ (30)

   We can calculate from (30) that

   $$\begin{array}{cc}\hfill & \mid \mathrm{E}{\mathbf{a}}^{\mathsf{T}}({\breve{\Sigma }}^{\ast }-{\widehat{\Sigma }}^{\ast })\mathbf{b}\mid \hfill \\ \hfill =& \mid \sum _{i,j=1}^p\sum _{k=1}^n{\sigma }_{ij}{a}_i{b}_j\left(\frac{S_{ik}{S}_{ik}{S}_{jk}}{n_i^{\ast }{n}_{ij}^{\ast }}+\frac{S_{ik}{S}_{jk}{S}_{jk}}{n_{ij}^{\ast }{n}_j^{\ast }}-\frac{S_{ik}{S}_{jk}}{n_i^{\ast }{n}_j^{\ast }}\right)\mid \hfill \\ \hfill =& \mid \sum _{i,j=1}^p{\sigma }_{ij}\frac{a_i}{n_i^{\ast }}{b}_j+\sum _{i,j=1}^p{a}_i\frac{b_j}{n_j^{\ast }}{\sigma }_{ij}-\sum _{k=1}^n\sum _{i,j=1}^p\frac{S_{ik}{a}_i}{n_i^{\ast }}\frac{S_{jk}{b}_j}{n_j^{\ast }}{\sigma }_{ij}\mid \hfill \\ \hfill \le & \mid \left(\frac{a_1}{n_1^{\ast }},\dots ,\frac{a_p}{n_p^{\ast }}\right)\Sigma \mathbf{b}\mid +\mid {\mathbf{a}}^{\mathsf{T}}\Sigma {\left(\frac{b_1}{n_1^{\ast }},\dots ,\frac{b_p}{n_p^{\ast }}\right)}^{\mathsf{T}}\mid +\sum _{k=1}^n\mid \left(\frac{S_{1k}{a}_1}{n_1^{\ast }},\dots ,\frac{S_{pk}{a}_p}{n_p^{\ast }}\right)\Sigma {\left(\frac{S_{1k}{b}_1}{n_1^{\ast }},\dots ,\frac{S_{pk}{b}_p}{n_p^{\ast }}\right)}^{\mathsf{T}}\mid \hfill \\ \hfill \le & \Vert {\Sigma }_{A\times B}\Vert \frac{{\Vert \mathbf{a}\Vert }_2{\Vert \mathbf{b}\Vert }_2}{n_{\min }^{\ast }}+\Vert {\Sigma }_{A\times B}\Vert \frac{{\Vert \mathbf{a}\Vert }_2{\Vert \mathbf{b}\Vert }_2}{n_{\min }^{\ast }}+\sum _{k=1}^n\Vert {\Sigma }_{A\times B}\Vert \cdot \frac{1}{2}\left\{{\Vert \left(\frac{S_{1k}{a}_1}{n_1^{\ast }},\dots ,\frac{S_{pk}{a}_p}{n_p^{\ast }}\right)\Vert }_2^2+{\Vert \left(\frac{S_{1k}{b}_1}{n_1^{\ast }},\dots ,\frac{S_{pk}{b}_p}{n_p^{\ast }}\right)\Vert }_2^2\right\}.\hfill \end{array}$$ (31)

   For the last term in (31), we have the following bound,

   $$\begin{array}{cc}\hfill & \sum _{k=1}^n\Vert {\Sigma }_{A\times B}\Vert \cdot \frac{1}{2}\left\{{\Vert \left(\frac{S_{1k}{a}_1}{n_1^{\ast }},\dots ,\frac{S_{pk}{a}_p}{n_p^{\ast }}\right)\Vert }_2^2+{\Vert \left(\frac{S_{1k}{b}_1}{n_1^{\ast }},\dots ,\frac{S_{pk}{b}_p}{n_p^{\ast }}\right)\Vert }_2^2\right\}\hfill \\ \hfill =& \Vert {\Sigma }_{A\times B}\Vert \sum _{k=1}^n\sum _{i=1}^p\frac{1}{2}\left(\frac{S_{ik}{a}_i^2}{n_i^{\ast 2}}+\frac{S_{ik}{b}_i^2}{n_i^{\ast 2}}\right)\hfill \\ \hfill =& \Vert {\Sigma }_{A\times B}\Vert \sum _{i=1}^p\frac{1}{2}\left(\frac{a_i^2+b_i^2}{n_i^{\ast }}\right)\hfill \\ \hfill \le & \Vert {\Sigma }_{A\times B}\Vert \sum _{i=1}^p\frac{a_i^2+b_i^2}{2n_{\min }^{\ast }}\le \frac{{\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}}{n_{\min }^{\ast }}.\hfill \end{array}$$

   Thus, by (31) and the inequality above, we have

   $$\mid \mathrm{E}{a}^{\mathsf{T}}({\breve{\Sigma }}^{\ast }-{\widehat{\Sigma }}^{\ast })b\mid \le \frac{3{\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}}{n_{\min }^{\ast }}.$$ (32)

   The last term of (30) can be treated as a quadratic form of the vectorization of $\mathbf{X}:\mathrm{vec}\left(\mathbf{X}\right)\in {\mathbb{R}}^{pn}$. We note the last term as vec(X)^⊤Qvec(X), where $\mathbf{Q}\in {\mathbb{R}}^{pn\times pn}$ and

   $${\mathbf{Q}}_{(i,k),(j,l)}=a_i{b}_j\left(\frac{S_{ik}{S}_{il}{S}_{jl}}{n_i^{\ast }{n}_{ij}^{\ast }}+\frac{S_{ik}{S}_{jk}{S}_{jl}}{n_{ij}^{\ast }{n}_j^{\ast }}-\frac{S_{ik}{S}_{jl}}{n_i^{\ast }{n}_j^{\ast }}\right),1\le i,j\le p,1\le k,l\le n.$$

   Q has the following properties,

   $$\begin{array}{cc}\hfill {\Vert \mathbf{Q}\Vert }_F^2=& \sum _{i,j=1}^p\sum _{k,l=1}^n{a}_i^2{b}_j^2{\left(\frac{S_{ik}{S}_{il}{S}_{jl}}{n_i^{\ast }{n}_{ij}^{\ast }}+\frac{S_{ik}{S}_{jk}{S}_{jl}}{n_{ij}^{\ast }{n}_j^{\ast }}-\frac{S_{ik}{S}_{jl}}{n_i^{\ast }{n}_j^{\ast }}\right)}^2\hfill \\ \hfill & \le \sum _{i,j=1}^p{a}_i^2{b}_j^2\sum _{k,l=1}^n\left(2\frac{S_{ik}{S}_{il}{S}_{jl}}{n_i^{\ast 2}{n}_{ij}^{\ast 2}}+2\frac{S_{ik}{S}_{jk}{S}_{jl}}{n_{ij}^{\ast 2}{n}_j^{\ast 2}}+\frac{S_{ik}{S}_{jl}}{n_i^{\ast 2}{n}_j^{\ast 2}}\right),\text{since}{S}_{ik}\in \{0,1\};\hfill \\ \hfill & \le \sum _{i,j=1}^p{a}_i^2{b}_j^2\frac{5}{n_{\min }^{\ast 2}}=\frac{5{\Vert \mathbf{a}\Vert }_2^2{\Vert \mathbf{b}\Vert }_2^2}{n_{\min }^{\ast 2}}=\frac{5}{n_{\min }^{\ast 2}};\hfill \end{array}$$ (33)

   $$\Vert \mathbf{Q}\Vert \le {\Vert \mathbf{Q}\Vert }_F\le \frac{\sqrt{5}{\Vert \mathbf{a}\Vert }_2{\Vert \mathbf{b}\Vert }_2}{n_{\min }^{\ast }}\le \frac{\sqrt{5}}{n_{\min }^{\ast }}.$$ (34)

   For $\mathrm{vec}\left(\mathbf{X}\right)\in {\mathbb{R}}^{pn}$, since its segments {X_k, k = 1, . . ., p} are independent and X_k = ΓZ_k we can further write vec(X) = D_Γvec(Z), where ${\mathbf{D}}_{\Gamma }\in {\mathbb{R}}^{pn\times qn}$ is with n diagonal blocks of Γ, vec(Z) is a (qn)-dimensional i.i.d. sub-Gaussian random vector. Based on Hanson-Wright's inequality (Theorem 1.1 in Rudelson and Vershynin [25]),

   $$\begin{array}{cc}\hfill & \mathrm{Pr}\left\{\mid {\mathbf{a}}^{\mathsf{T}}\left({\breve{\Sigma }}^{\ast }-{\widehat{\Sigma }}^{\ast }\right)\mathbf{b}-\mathrm{E}{\mathbf{a}}^{\mathsf{T}}\left({\breve{\Sigma }}^{\ast }-{\widehat{\Sigma }}^{\ast }\right)\mathbf{b}\mid \ge x\right\}\hfill \\ \hfill =& \mathrm{Pr}\left\{\mid \mathrm{vec}{\left(\mathbf{X}\right)}^{\mathsf{T}}\mathbf{Q}\mathrm{vec}\left(\mathbf{X}\right)-\mathrm{Evec}{\left(\mathbf{X}\right)}^{\mathsf{T}}\mathbf{Q}\mathrm{vec}\left(\mathbf{X}\right)\mid \ge x\right\}\hfill \\ \hfill =& \mathrm{Pr}\left[\mid \mathrm{vec}{\left(\mathbf{Z}\right)}^{\mathsf{T}}{\mathbf{D}}_{\Gamma }^{\mathsf{T}}{\mathbf{QD}}_{\Gamma }\mathrm{vec}\left(\mathbf{Z}\right)-\mathrm{E}\left\{\mathrm{vec}{\left(\mathbf{Z}\right)}^{\mathsf{T}}{\mathbf{D}}_{\Gamma }^{\mathsf{T}}{\mathbf{QD}}_{\Gamma }\mathrm{vec}\left(\mathbf{Z}\right)\right\}\mid \ge x\right]\hfill \\ \hfill \le & 2\exp \left\{-c\min \left(\frac{x^2}{{\tau }^4{\Vert {\mathbf{D}}_{\Gamma }^{\mathsf{T}}{\mathbf{QD}}_{\Gamma }\Vert }_F^2},\frac{x}{{\tau }^2\Vert {\mathbf{D}}_{\Gamma }{\mathbf{QD}}_{\Gamma }\Vert }\right)\right\}.\hfill \end{array}$$ (35)

   Here c > 0 is a uniform constant. Since Q is supported on {(i, k), (j, l) : i ∈ A, J ∈ B}, we have ${\mathbf{D}}_{\Gamma }^{\mathsf{T}}{\mathbf{QD}}_{\Gamma }={\mathbf{D}}_{{\Gamma }_A}^{\mathsf{T}}{\mathbf{Q}}_{A\times B}{\mathbf{D}}_{{\Gamma }_B}$. Here ${\mathbf{D}}_{{\Gamma }_A}\in {\mathbb{R}}^{\mid A\mid n\times qn}$, ${\mathbf{D}}_{{\Gamma }_B}\in {\mathbb{R}}^{\mid B\mid n\times qn}$ are with n diagonal block Γ_A×[q] and Γ_B×[q], respectively, where [q] = {1, . . ., q}. Since ${\Gamma }_{A\times \left[q\right]}{\Gamma }_{A\times \left[q\right]}^{\mathsf{T}}={\Sigma }_A$, ${\Gamma }_{B\times \left[q\right]}{\Gamma }_{B\times \left[q\right]}^{\mathsf{T}}={\Sigma }_B$, we know

   $$\Vert {\mathbf{D}}_{{\Gamma }_A}\Vert =\Vert {\Gamma }_{A\times \left[q\right]}\Vert \le {\Vert {\Sigma }_A\Vert }^{1∕2},\Vert {\Gamma }_{B\times \left[q\right]}\Vert \le \Vert {\mathbf{D}}_{{\Gamma }_B}\Vert \le {\Vert {\Sigma }_B\Vert }^{1∕2}.$$

   Then we further have

   $$\begin{array}{cc}\hfill & \mathrm{Pr}\left\{\mid {\mathbf{a}}^{\mathsf{T}}\left({\breve{\Sigma }}^{\ast }-{\widehat{\Sigma }}^{\ast }\right)\mathbf{b}-\mathrm{E}{\mathbf{a}}^{\mathsf{T}}\left({\breve{\Sigma }}^{\ast }-{\widehat{\Sigma }}^{\ast }\right)\mathbf{b}\mid \ge x\right\}\hfill \\ \hfill \le & 2\exp \left\{-c\min \left(\frac{x^2}{{\tau }^4{\Vert {\mathbf{D}}_{{\Gamma }_A}^{\mathsf{T}}{\mathbf{Q}}_{A\times B}{\mathbf{D}}_{{\Gamma }_B}\Vert }_F^2},\frac{x}{{\tau }^2\Vert {\mathbf{D}}_{{\Gamma }_A}^{\mathsf{T}}{\mathbf{Q}}_{A\times B}{\mathbf{D}}_{{\Gamma }_B}\Vert }\right)\right\}\hfill \\ \hfill \le & 2\exp \left\{-c\min \left(\frac{x^2}{{\tau }^4{\Vert {\mathbf{D}}_{{\Gamma }_B}\Vert }^2{\Vert {\mathbf{D}}_{{\Gamma }_A}^{\mathsf{T}}\Vert }^2{\Vert \mathbf{Q}\Vert }_F^2},\frac{x}{{\tau }^2\Vert {\mathbf{D}}_{{\Gamma }_B}\Vert \Vert {\mathbf{D}}_{{\Gamma }_A}^{\mathsf{T}}\Vert \Vert \mathbf{Q}\Vert }\right)\right\}\hfill \\ \hfill \le & 2\exp \left[-c\min \left\{\frac{x^2}{{\tau }^4\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert {\Vert \mathbf{Q}\Vert }_F^2},\frac{x}{{\tau }^2{\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}\Vert \mathbf{Q}\Vert }\right\}\right]\hfill \\ \hfill \le & 2\exp \left[-c\min \left\{\frac{x^2{n}_{\min }^{\ast 2}}{{\tau }^4\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert },\frac{x{n}_{\min }^{\ast }}{{\tau }^2{\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}}\right\}\right].\hfill \end{array}$$ (36)

   We define $x^{\prime }=\max \{x-3{\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}∕n_{\min }^{\ast },0\}$, combining the inequality above and (32), we have

   $$\begin{array}{cc}\hfill & \mathrm{Pr}\left\{\mid {\mathbf{a}}^{\mathsf{T}}\left({\breve{\Sigma }}^{\ast }-{\widehat{\Sigma }}^{\ast }\right)\mathbf{b}\mid \ge x\right\}\le \mathrm{Pr}\left\{\mid {\mathbf{a}}^{\mathsf{T}}\left({\breve{\Sigma }}^{\ast }-{\widehat{\Sigma }}^{\ast }\right)\mathbf{b}-\mathrm{E}{\mathbf{a}}^{\mathsf{T}}{\widehat{\Sigma }}^{\ast }\mathbf{b}\mid \ge x^{\prime }\right\}\hfill \\ \hfill \le & 2\exp \left[-c\min \left\{\frac{{\left(x^{\prime }\right)}^2{n}_{\min }^{\ast 2}}{{\tau }^4\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert },\frac{x^{\prime }{n}_{\min }^{\ast }}{{\tau }^2{\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}}\right\}\right]\hfill \\ \hfill \le & 2\exp \left[-c^{\prime }\min \left\{\frac{x^2{n}_{\min }^{\ast 2}}{{\tau }^4\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert },\frac{x{n}_{\min }^{\ast }}{{\tau }^2{\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}}\right\}+C\max \left(\frac{1}{{\tau }^4},\frac{1}{{\tau }^2}\right)\right]\hfill \\ \hfill \le & C\exp \left[-c^{\prime }\min \left\{\frac{x^2{n}_{\min }^{\ast 2}}{{\tau }^4\Vert {\Sigma }_4\Vert \Vert {\Sigma }_B\Vert },\frac{x{n}_{\min }^{\ast }}{{\tau }^2{\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}}\right\}\right].\hfill \end{array}$$ (37)

   In the last inequality above, we used a fact that τ is lower bounded by a uniform constant. This is due to Assumption 2.2 that E(Z) = 0, var(Z) = 1, Eexp(tZ) ≤ exp(t²τ²/2). Then,

   $$\exp (4{\tau }^2∕2)\ge \frac{1}{2}\{\mathrm{E}\exp \left(2Z\right)+\mathrm{E}\exp (-2Z)\}=\sum _{k=0}^{\infty }\frac{2^{2k}\mathrm{E}{Z}^{2k}}{\left(2k\right)!}\ge 2\mathrm{E}{Z}^2=2,$$

   which implies τ² ≤ ½ ln(2).

   $$\begin{array}{cc}\hfill & a^{\mathsf{T}}({\breve{\Sigma }}^{\ast }-\Sigma )b\hfill \\ \hfill =& \sum _{i,j=1}^p{a}_i{b}_j\left(\frac{1}{n_{ij}^{\ast }}\sum _{k=1}^n{X}_{ik}{X}_{jk}{S}_{ik}{S}_{jk}\right)-\mathrm{E}\sum _{i,j=1}^p{a}_i{b}_j\left(\frac{1}{n_{ij}^{\ast }}\sum _{k=1}^n{X}_{ik}{X}_{jk}{S}_{ik}{S}_{jk}\right)\hfill \\ \hfill =& \sum _{k=1}^n\sum _{i,j=1}^p\left(\frac{a_i{b}_j{S}_{ik}{S}_{jk}}{n_{ij}^{\ast }}{X}_{ik}{X}_{jk}-\mathrm{E}\frac{a_i{b}_j{S}_{ik}{S}_{jk}}{n_{ij}^{\ast }}{X}_{ik}{X}_{jk}\right)\hfill \\ \hfill \triangleq & \sum _{k=1}^n({\mathbf{X}}_k^{\mathsf{T}}{\mathbf{C}}^k{\mathbf{X}}_k-\mathrm{E}{\mathbf{X}}_k^{\mathsf{T}}{\mathbf{C}}^k{\mathbf{X}}_k)\hfill \\ \hfill =& \sum _{k=1}^n({\mathbf{Z}}_k^{\mathsf{T}}{\Gamma }^{\mathsf{T}}{\mathbf{C}}^k\Gamma {\mathbf{Z}}_k-\mathrm{E}{\mathbf{Z}}_k^{\mathsf{T}}{\Gamma }^{\mathsf{T}}\Gamma {\mathbf{Z}}_k).\hfill \end{array}$$ (38)
3. It is easy to see that $\mathrm{E}{\breve{\Sigma }}^{\ast }=\Sigma$, so $\mathrm{E}{\mathbf{a}}^{\mathsf{T}}({\breve{\Sigma }}^{\ast }-\Sigma )\mathbf{b}=0$. Then Here ${\mathbf{C}}^k\in {\mathbb{R}}^{p\times p}$ is a matrix such that $C_{ij}^k=a_i{b}_j{S}_{ik}{S}_{jk}∕n_{ij}^{\ast }$. Note that Ck is supported on A × B, we can prove the following properties of C^k.

   $$\begin{array}{cc}\hfill {\Vert {\Gamma }^{\mathsf{T}}{\mathbf{C}}^k\Gamma \Vert }_F& =\sqrt{\mathrm{tr}\left({\mathbf{C}}^k\Gamma {\Gamma }^{\mathsf{T}}{\mathbf{C}}^{k\mathsf{T}}\Gamma {\Gamma }^{\mathsf{T}}\right)}\hfill \\ \hfill & =\sqrt{\mathrm{tr}\left({\mathbf{C}}_{A\times B}^k{\Gamma }_{B\times \left[q\right]}{\Gamma }_{B\times \left[q\right]}^{\mathsf{T}}{\mathbf{C}}_{A\times B}^{k\mathsf{T}}{\Gamma }_{A\times \left[q\right]}{\Gamma }_{A\times \left[q\right]}^{\mathsf{T}}\right)}\hfill \\ \hfill & \le \Vert {\Gamma }_{B\times \left[q\right]}\Vert \Vert {\Gamma }_{A\times \left[q\right]}\Vert \sqrt{\mathrm{tr}\left({\mathbf{C}}^k{\mathbf{C}}^{k\mathsf{T}}\right)}\hfill \\ \hfill & \le {\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}\sqrt{\mathrm{tr}\left({\mathbf{C}}^k{\mathbf{C}}^{k\mathsf{T}}\right)};\hfill \end{array}$$ (39)

   $$\begin{array}{cc}\hfill & \sum _{k=1}^n{\Vert {\Gamma }^{\mathsf{T}}{\mathbf{C}}^k\Gamma \Vert }_F^2\le \Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert {\Vert {\mathbf{C}}^k\Vert }_F^2=\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \sum _{k=1}^n\sum _{i,j=1}^p{\left(\frac{a_i{b}_j{S}_{ik}{S}_{jk}}{n_{ij}^{\ast }}\right)}^2\hfill \\ \hfill =& \Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \sum _{k=1}^n\sum _{i,j=1}^p\frac{S_{ik}{S}_{jk}{a}_i^2{b}_j^2}{n_{ij}^{\ast 2}}=\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \sum _{i,j=1}^p\frac{a_i^2{b}_j^2}{n_{ij}^{\ast }}\hfill \\ \hfill \le & \Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \frac{{\Vert \mathbf{a}\Vert }_2^2{\Vert \mathbf{b}\Vert }_2^2}{n_{\min }^{\ast }}=\frac{\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert }{n_{\min }^{\ast }};\hfill \end{array}$$ (40)

   $$\begin{array}{cc}\hfill \Vert {\Gamma }^{\mathsf{T}}{\mathbf{C}}^k\Gamma \Vert & \le {\Vert {\Gamma }^{\mathsf{T}}{\mathbf{C}}^k\Gamma \Vert }_F\le {\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}\sqrt{\mathrm{tr}\left({\mathbf{C}}^k{\mathbf{C}}^{k\mathsf{T}}\right)}\hfill \\ \hfill & \le {\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}\sqrt{\sum _{i,j=1}^p{\left(\frac{a_i{b}_j{S}_{ik}{S}_{jk}}{n_{ij}^{\ast }}\right)}^2}\hfill \\ \hfill & \le {\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}\sqrt{\sum _{i,j=1}^p\frac{a_i^2{b}_j^2}{n_{\min }^{\ast 2}}}\le \frac{\sqrt{\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert }}{n_{\min }^{\ast }}.\hfill \end{array}$$ (41)

   Now, note that the last line of (38) can be also equivalently written as

   $$\begin{array}{cc}\hfill & \mathrm{vec}{\left(\mathbf{Z}\right)}^{\mathsf{T}}{\mathbf{C}}^{con}\mathrm{vec}{\left(\mathbf{Z}\right)}^{\mathsf{T}}-\mathrm{Evec}{\left(\mathbf{Z}\right)}^{\mathsf{T}}{\mathbf{C}}^{con}\mathrm{vec}{\left(\mathbf{Z}\right)}^{\mathsf{T}},\hfill \\ \hfill & {\mathbf{C}}^{con}=\left[\begin{array}{ccc}\hfill {\Gamma }^{\mathsf{T}}{\mathbf{C}}^1\Gamma \hfill & \hfill \hfill & \hfill \hfill \\ \hfill \hfill & \hfill \ddots \hfill & \hfill \hfill \\ \hfill \hfill & \hfill \hfill & \hfill {\Gamma }^{\mathsf{T}}{\mathbf{C}}^n\Gamma \hfill \end{array}\right]\in {\mathbb{R}}^{\left(qn\right)\times \left(qn\right)},\hfill \end{array}$$

   where vec(Z) is the vectorization of Z, which is an qn-dimensional i.i.d. sub-Gaussian vector. Based on the properties of C^k above, we have

   $$\begin{array}{c}\hfill {\Vert {\mathbf{C}}^{con}\Vert }_F^2=\sum _{k=1}^n{\Vert {\Gamma }^{\mathsf{T}}{\mathbf{C}}^k\Gamma \Vert }_F^2\stackrel{\left(40\right)}{\le }\frac{\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert }{n_{\min }^{\ast }},\hfill \\ \hfill \Vert {\mathbf{C}}^{con}\Vert \le \underset{1\le k\le n}{\max }\Vert {\Gamma }^{\mathsf{T}}{\mathbf{C}}^k\Gamma \Vert \stackrel{\left(41\right)}{\le }\frac{{\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}}{n_{\min }^{\ast }}.\hfill \end{array}$$

   Now applying Hanson-Wright's inequality (Theorem 1.1 in Rudelson and Vershynin [25]), we have

   $$\begin{array}{cc}\hfill & \mathrm{Pr}\{\mid \mathrm{vec}{\left({\mathbf{Z}}_k\right)}^{\mathsf{T}}{\mathbf{C}}^{con}\mathrm{vec}\left({\mathbf{Z}}_k\right)-\mathrm{Evec}{\left({\mathbf{Z}}_k\right)}^{\mathsf{T}}{\mathbf{C}}^{con}\mathrm{vec}\left({\mathbf{Z}}_k\right)\mid \ge x\}\hfill \\ \hfill \le & 2\exp \left\{-c\min \left(\frac{x^2}{{\tau }^4{\Vert {\mathbf{C}}^{con}\Vert }_F^2},\frac{x}{{\tau }^2\Vert {\mathbf{C}}^{con}\Vert }\right)\right\}\hfill \\ \hfill \le & 2\exp \left\{-c{n}_{\min }^{\ast }\min \left(\frac{x^2}{{\tau }^4\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert },\frac{x}{{\tau }^2{\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}}\right)\right\}.\hfill \end{array}$$ (42)

   Thus,

   $$\begin{array}{cc}\hfill & \mathrm{Pr}\left\{\mid {\mathbf{a}}^{\mathsf{T}}({\breve{\Sigma }}^{\ast }-\Sigma )\mathbf{b}\mid \ge x\right\}\hfill \\ \hfill \le & 2\exp \left\{-c{n}_{\min }^{\ast }\min \left(\frac{x^2}{{\tau }^4\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert },\frac{x}{{\tau }^2{\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}}\right)\right\}.\hfill \end{array}$$ (43)

   Here c is a uniform constant. Combining (43) and (37), we have (29).
4. Next, we use the ε-net technique to give the bound on $\Vert {\widehat{\Sigma }}_{A\times B}^{\ast }-{\Sigma }_{A\times B}\Vert$. Denote ${\mathbf{D}}^{\ast }={\widehat{\Sigma }}_{A\times B}^{\ast }-{\Sigma }_{A\times B}$. Suppose $S_{1∕3}^A$ is the (1/3)-net for all unit vectors in ${\mathbb{R}}^{\mid A\mid }$; similarly $S_{1∕3}^B$ is the (1/3)- net for all unit vectors in ${\mathbb{R}}^{\mid B\mid }$. Based on the proof of Lemma 3 in Cai et al. [9], we can let $\mathrm{Card}\left(S_{1∕3}^A\right)\le 7^k$, $\mathrm{Card}\left(S_{1∕3}^B\right)\le 7^k$. Since all a, ${\mathbf{a}}_0\in {\mathbb{R}}^{\mid A\mid }$, b, ${\mathbf{b}}_0\in {\mathbb{R}}^{\mid B\mid }$,

   $$\begin{array}{cc}\hfill \mid {\mathbf{a}}^{\mathsf{T}}{\mathbf{D}}^{\ast }\mathbf{b}\mid -\mid {\mathbf{a}}_0^{\mathsf{T}}{\mathbf{D}}^{\ast }{\mathbf{b}}_0\mid & \le \mid {\mathbf{a}}^{\mathsf{T}}{\mathbf{D}}^{\ast }\mathbf{b}-{\mathbf{a}}_0^{\mathsf{T}}{\mathbf{D}}^{\ast }{\mathbf{b}}_0\mid \le \mid {(\mathbf{a}-{\mathbf{a}}_0)}^{\mathsf{T}}{\mathbf{D}}^{\ast }\mathbf{b}\mid +\mid {\mathbf{a}}_0^{\mathsf{T}}{\mathbf{D}}^{\ast }(\mathbf{b}-{\mathbf{b}}_0)\mid \hfill \\ \hfill & \le ({\Vert \mathbf{a}-{\mathbf{a}}_0\Vert }_2+{\Vert \mathbf{b}-{\mathbf{b}}_0\Vert }_2)\Vert {\mathbf{D}}^{\ast }\Vert ,\hfill \end{array}$$ (44)

   we have for all $\mathbf{a}\in {\mathbb{R}}^{\mid A\mid }$, $\mathbf{b}\in {\mathbb{R}}^{\mid B\mid }$, ${\Vert \mathbf{a}\Vert }_2={\Vert \mathbf{b}\Vert }_2=1$, we can find ${\mathbf{a}}_0\in S_{1∕3}^A$, ${\mathbf{b}}_0\in S_{1∕3}^B$, such that ∥a₀ – a∥₂ ≤ 1/3, ∥b₀ – _b∥₂ ≤ 1/3, then

   $$\begin{array}{c}\hfill \mid {\mathbf{a}}^{\mathsf{T}}{\mathbf{D}}^{\ast }\mathbf{b}\mid \le \mid {\mathbf{a}}_0^{\mathsf{T}}{\mathbf{D}}^{\ast }{\mathbf{b}}_0\mid +\frac{2}{3}\Vert {\mathbf{D}}^{\ast }\Vert \le \underset{{\mathbf{a}}_0\in S_{1∕3}^A,{\mathbf{b}}_0\in S_{1∕3}^B}{\sup }\mid {\mathbf{a}}_0^{\mathsf{T}}{\mathbf{D}}^{\ast }{\mathbf{b}}_0\mid +\frac{2}{3}\Vert {\mathbf{D}}^{\ast }\Vert ,\hfill \\ \hfill \Vert {\mathbf{D}}^{\ast }\Vert =\underset{\begin{array}{c}\hfill \mathbf{a}\in {\mathbb{R}}^{\mid A\mid },\mathbf{b}\in {\mathbb{R}}^{\mid B\mid },\hfill \\ \hfill {\Vert \mathbf{a}\Vert }_2={\Vert \mathbf{b}\Vert }_2=1\hfill \end{array}}{\sup }\mid {\mathbf{a}}^{\mathsf{T}}{\mathbf{D}}^{\ast }\mathbf{b}\mid \le \underset{{\mathbf{a}}_0\in S_{1∕3}^A,{\mathbf{b}}_0\in S_{1∕3}^B}{\sup }\mid {\mathbf{a}}_0^{\mathsf{T}}{\mathbf{D}}^{\ast }{\mathbf{b}}_0\mid +\frac{2}{3}\Vert {\mathbf{D}}^{\ast }\Vert ,\hfill \end{array}$$

   which yields

   $$\Vert {\widehat{\Sigma }}_{A\times B}^{\ast }-{\Sigma }_{A\times B}\Vert =\Vert {\mathbf{D}}^{\ast }\Vert \le 3\underset{{\mathbf{a}}_0\in S_{1∕3}^A,{\mathbf{b}}_0\in S_{1∕3}^B}{\sup }\mid {\mathbf{a}}_0^{\mathsf{T}}{\mathbf{D}}^{\ast }{\mathbf{b}}_0\mid .$$ (45)

   Finally, by combining (29) and the inequality above, we know there exist uniform constants C₁, c > 0 such that for all t > 0,

   $$\begin{array}{c}\hfill \mathrm{Pr}\left(\Vert {\widehat{\Sigma }}_{A\times B}^{\ast }-{\Sigma }_{A\times B}\Vert \right)\le \mathrm{Pr}\left(\underset{{\mathbf{a}}_0\in S_{1∕3}^A,{\mathbf{b}}_0\in S_{1∕3}^B}{\sup }\mid {\mathbf{a}}_0^{\mathsf{T}}{\mathbf{D}}^{\ast }{\mathbf{b}}_0\mid \ge \frac{x}{3}\right)\hfill \\ \hfill \le C_1{\left(7\right)}^{\mid A\mid +\mid B\mid }\exp \left[-c{n}_{\min }^{\ast }\min \left\{\frac{x^2}{{\tau }^4\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert },\frac{x}{{\tau }^2{\left(\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert \right)}^{1∕2}}\right\}\right].\hfill \end{array}$$ (46)

   Since $\mid A\mid +\mid B\mid \le 2\mid A\cup B\mid$, we have finished the proof of Lemma 2.1.

Proof of Lemma 6.1. Since Σ is positive semi-definite, we can find the Cholesky decomposition such that Σ = VV^Ȫ4. Then ${\Sigma }_{A\times B}={\mathbf{V}}_{A\times \left[p\right]}{\mathbf{V}}_{B\times \left[p\right]}^{\mathsf{T}}$ and

$$\begin{array}{cc}\hfill \Vert {\Sigma }_{A\times B}\Vert & =\underset{\begin{array}{c}\hfill \mathbf{x}\in {\mathbb{R}}^{\mid A\mid },\mathbf{y}\in {\mathbb{R}}^{\mid B\mid }\hfill \\ \hfill {\Vert \mathbf{x}\Vert }_2={\Vert \mathbf{y}\Vert }_2=1\hfill \end{array}}{\max }{\mathbf{x}}^{\mathsf{T}}{\mathbf{V}}_{A\times \left[p\right]}{\mathbf{V}}_{B\times \left[p\right]}^{\mathsf{T}}\mathbf{y}\hfill \\ \hfill & \le \underset{\begin{array}{c}\hfill \mathbf{x}\in {\mathbb{R}}^{\mid A\mid },\mathbf{y}\in {\mathbb{R}}^{\mid B\mid }\hfill \\ \hfill {\Vert \mathbf{x}\Vert }_2={\Vert \mathbf{y}\Vert }_2=1\hfill \end{array}}{\max }{\left({\mathbf{x}}^{\mathsf{T}}{\mathbf{V}}_{A\times \left[p\right]}{\mathbf{V}}_{A\times \left[p\right]}^{\mathsf{T}}\mathbf{x}\right)}^{1∕2}{\left({\mathbf{y}}^{\mathsf{T}}{\mathbf{V}}_{B\times \left[p\right]}{\mathbf{V}}_{B\times \left[p\right]}^{\mathsf{T}}\mathbf{y}\right)}^{1∕2}\hfill \\ \hfill & =\underset{\begin{array}{c}\hfill \mathbf{x}\in {\mathbb{R}}^{\mid A\mid },\mathbf{y}\in {\mathbb{R}}^{\mid B\mid }\hfill \\ \hfill {\Vert \mathbf{x}\Vert }_2={\Vert \mathbf{y}\Vert }_2=1\hfill \end{array}}{\max }{\left({\mathbf{x}}^{\mathsf{T}}{\Sigma }_A\mathbf{x}\right)}^{1∕2}{\left({\mathbf{y}}^{\mathsf{T}}{\Sigma }_B\mathbf{y}\right)}^{1∕2}=\Vert {\Sigma }_A\Vert \Vert {\Sigma }_B\Vert .\hfill \end{array}$$

Here we have used the Cauchy-Schwarz inequality.

6.2 Proof of Theorem 2.1

Define B = (b_ij)_1≤i,j≤p such that b_ij = σ_ij if i ∈ I_s, j ∈ I_s′ and |s – s′| ≤ 1, and 0 otherwise. Let Δ = Σ – B. Then

$$\Vert {\widehat{\Sigma }}^{\mathrm{bt}}-\Sigma \Vert \le \Vert {\widehat{\Sigma }}^{\mathrm{bt}}-\mathbf{B}\Vert +\Vert \Delta \Vert .$$

It is easy to see that

$$\Vert \Delta \Vert \le {\Vert \Delta \Vert }_{{\ell }_1}\le \underset{i}{\max }\sum _{j:\mid i-j\mid \ge k}\mid {\sigma }_{ij}\mid \le M{k}^{-\alpha }.$$

To bound $\Vert {\widehat{\Sigma }}^{\mathrm{bt}}-\mathbf{B}\Vert$, note that

$$\Vert {\widehat{\Sigma }}^{\mathrm{bt}}-\mathbf{B}\Vert =\underset{u\in {\mathbb{R}}^p:{\Vert u\Vert }_2=1}{\sup }\mid \langle u,({\widehat{\Sigma }}^{\mathrm{bt}}-\mathbf{B})u\rangle \mid .$$

For any $u\in {\mathbb{R}}^p,{\Vert u\Vert }_2=1$, we have

$$\begin{array}{cc}\hfill \mid \langle u,({\widehat{\Sigma }}^{\mathrm{bt}}-\mathbf{B})u\rangle \mid & \le \sum _{s,s^{\prime }:\mid s-s^{\prime }\mid \le 1}\mid \langle u_{I_s},({\widehat{\Sigma }}_{I_s\times I_{s^{\prime }}}^{\ast }-{\Sigma }_{I_s\times I_{s^{\prime }}})u_{I_{s^{\prime }}}\rangle \mid \hfill \\ \hfill & \le \sum _{s,s^{\prime }:\mid s-s^{\prime }\mid \le 1}{\Vert u_{I_s}\Vert }_2{\Vert u_{I_{s^{\prime }}}\Vert }_2\Vert {\widehat{\Sigma }}_{I_s\times I_{s^{\prime }}}^{\ast }-{\Sigma }_{I_s\times I_{s^{\prime }}}\Vert \hfill \\ \hfill & \le \left(\sum _{s,s^{\prime }:\mid s-s^{\prime }\mid \le 1}{\Vert u_{I_s}\Vert }_2{\Vert u_{I_{s^{\prime }}}\Vert }_2\right)\left(\underset{\mid s-s^{\prime }\mid \le 1}{\max }\Vert {\widehat{\Sigma }}_{I_s\times I_{s^{\prime }}}^{\ast }-{\Sigma }_{I_s\times I_{s^{\prime }}}\Vert \right).\hfill \end{array}$$

The Cauchy-Schwarz inequality yields

$$\sum _{s,s^{\prime }:\mid s-s^{\prime }\mid \le 1}{\Vert u_{I_s}\Vert }_2{\Vert u_{I_{s^{\prime }}}\Vert }_2\le \frac{1}{2}\sum _{s,s^{\prime }:\mid s-s^{\prime }\mid \le 1}({\Vert u_{I_s}\Vert }_2^2+{\Vert u_{I_{s^{\prime }}}\Vert }_2^2)\le 3\sum _{s=1}^N{\Vert u_{I_s}\Vert }_2^2=3.$$ (47)

Therefore,

$$\Vert {\widehat{\Sigma }}^{\mathrm{bt}}-\Sigma \Vert \le \Vert {\widehat{\Sigma }}^{\ast }-\mathbf{B}\Vert +\Vert \Delta \Vert \le 3\underset{\mid s-s^{\prime }\mid \le 1}{\max }\Vert {\widehat{\Sigma }}_{I_s\times I_{s^{\prime }}}^{\ast }-{\Sigma }_{I_s\times I_{s^{\prime }}}\Vert +M{k}^{-\alpha },$$

which yields

$$\mathrm{E}{\Vert {\widehat{\Sigma }}^{\mathrm{bt}}-\Sigma \Vert }^2\le 18\mathrm{E}{\left(\underset{\mid s-s^{\prime }\mid \le 1}{\max }\Vert {\widehat{\Sigma }}_{I_s\times I_{s^{\prime }}}^{\ast }-{\Sigma }_{I_s\times I_{s^{\prime }}}\Vert \right)}^2+2M^2{k}^{-2\alpha }.$$

According to lemma 2.1, there exists constant C, c > 0 which only depend on τ such that for all x > 0,

$$\mathrm{Pr}\left(\underset{\mid s-s^{\prime }\mid \le 1}{\max }\Vert {\widehat{\Sigma }}_{I_s\times I_{s^{\prime }}}-{\Sigma }_{I_s\times I_{s^{\prime }}}\Vert \ge x\right)\le C\lceil \frac{p}{k}\rceil {\left(49\right)}^k\exp \left\{-c{n}_{\min }^{\ast }\min \left(\frac{x^2}{{\Vert \Sigma \Vert }^2},\frac{x}{\Vert \Sigma \Vert }\right)\right\}.$$ (48)

Now we set $t=C^{\prime }(k+\ln p)∕n_{\min }^{\ast }$ for C′ large enough. The spectral norm risk satisfies

$$\begin{array}{cc}\hfill \mathrm{E}{\Vert {\widehat{\Sigma }}^{\mathrm{bt}}-\Sigma \Vert }^2& \le 18\mathrm{E}\underset{\mid s-s^{\prime }\mid \le 1}{\max }\Vert {\widehat{\Sigma }}_{I_s\times I_{s^{\prime }}}^{\ast }\Vert +2M^2{k}^{-2\alpha }\hfill \\ \hfill & \le 18{\int }_0^{\infty }\mathrm{Pr}\left(\underset{\mid s-s^{\prime }\mid \le 1}{\max }{\Vert {\widehat{\Sigma }}_{I_s\times I_{s^{\prime }}}-{\Sigma }_{I_s\times I_{s^{\prime }}}\Vert }^2\ge x\right)dx+2M^2{k}^{-2\alpha }\hfill \\ \hfill & \le 18t+18{\int }_t^{\infty }\mathrm{Pr}\left(\underset{\mid s-s^{\prime }\mid \le 1}{\max }{\Vert {\widehat{\Sigma }}_{I_s\times I_{s^{\prime }}}-{\Sigma }_{I_s\times I_{s^{\prime }}}\Vert }^2\ge x\right)dx+2M^2{k}^{-2\alpha }\hfill \\ \hfill & \le 18t+C\lceil \frac{p}{k}\rceil {\left(49\right)}^k{\int }_t^{\infty }\exp \left\{-c^{\prime }{n}_{\min }^{\ast }\min \left(x,x^{\frac{1}{2}}\right)\right\}dx+2M^2{k}^{-2\alpha }\hfill \\ \hfill & \le 18t+C\lceil \frac{p}{k}\rceil {\left(49\right)}^k\frac{1}{n_{\min }^{\ast }}\exp (-c^{\prime }{n}_{\min }^{\ast }t)+2M^2{k}^{-2\alpha },\hfill \end{array}$$ (49)

then (49) yields

$$\mathrm{E}{\Vert {\widehat{\Sigma }}^{\mathrm{bt}}-\Sigma \Vert }^2\le C\left(\frac{k+\ln p}{n_{\min }^{\ast }}+k^{-2\alpha }\right),$$ (50)

where C only depends on τ, M, M₀. We can finally finish the proof of Theorem 2.1 by taking $k={\left(n_{\min }^{\ast }\right)}^{1∕(2\alpha +1)}$.