Foundational Principles for Large-Scale Inference: Illustrations Through Correlation Mining

Abstract

When can reliable inference be drawn in fue “Big Data” context? This paper presents a framework for answering this fundamental question in the context of correlation mining, wifu implications for general large scale inference. In large scale data applications like genomics, connectomics, and eco-informatics fue dataset is often variable-rich but sample-starved: a regime where the number n of acquired samples (statistical replicates) is far fewer than fue number p of observed variables (genes, neurons, voxels, or chemical constituents). Much of recent work has focused on understanding the computational complexity of proposed methods for “Big Data”. Sample complexity however has received relatively less attention, especially in the setting when the sample size n is fixed, and the dimension p grows without bound. To address fuis gap, we develop a unified statistical framework that explicitly quantifies the sample complexity of various inferential tasks. Sampling regimes can be divided into several categories: 1) the classical asymptotic regime where fue variable dimension is fixed and fue sample size goes to infinity; 2) the mixed asymptotic regime where both variable dimension and sample size go to infinity at comparable rates; 3) the purely high dimensional asymptotic regime where the variable dimension goes to infinity and the sample size is fixed. Each regime has its niche but only the latter regime applies to exa cale data dimension. We illustrate this high dimensional framework for the problem of correlation mining, where it is the matrix of pairwise and partial correlations among the variables fua t are of interest. Correlation mining arises in numerous applications and subsumes the regression context as a special case. we demonstrate various regimes of correlation mining based on the unifying perspective of high dimensional learning rates and sample complexity for different structured covariance models and different inference tasks.

I. Introduction

The increasing availability of large scale and high dimensional data is driving a major resurgence of data science, recently rebranded under the moniker “Big Data” [90]. There has been a preponderance of catch phrases such as “big-data-biology, ” “ecoinformatics,” “precision medicine,” “data-driven decision-making,” “Big Data business analytics” in scientific publications and the media. However, until recently, most of the research in Big Data has concentrated on issues of data management, data warehousing, computational data analysis, and end-user data utilization [129], [93], [96]. While the data management research community has made progress on the problem of quality assurance, e.g., associated with provenance and computer errors [80], the issue of limited sample size and statistical reproducibility remains largely open. This issue has been recognized as one of the principal hurdles that stand in the way of success of the scientific enterprise [154], [116]. The negative consequences of insufficient samples can be especially dire when the data is high dimensional, heterogenous and uncalibrated [26], [91]. It is therefore both important and timely to address the problem of inference on Big Data from the point of view of statistical reproducibility.

The statistical reproducibility point of view is founded on a non-monolithic notion of Big Data: the data should be considered as a matrix with p columns and n rows indexed by, respectively, p variables over each of the data fields and n independent samples of these variables. If there is only a single sample then the matrix collapses to a vector and no statistical analysis of reproducibility can be performed. With larger sample size, the probability of reproducibility can be studied in the context of the statistical theory of random matrices.

Here we develop this perspective for a particular Big Data problem: correlation mining in high dimension where the number of samples is much smaller than the dimension, a setting that we call “sampled starved.” Correlation mining is an area of data mining where the objective is to discover patterns of correlation between a large number (p) of obse1ved variables based on a limited number (n) of samples. Correlation mining can be framed as the mathematical problem of reliably reconstructing different attributes of the correlation matrix of the population from the sample covariance matrix that is empirically constructed from the p × n data matrix. The inference task depends on the attributes of the correlation that are of interest, while the performance of a correlation mining algorithm for a particular task depends on the number of samples and the underlying structure of the population covariance. In high dimensional settings where p >> n, correlation mining presents significant challenges to the practitioner, both in terms of unavailability of computationally tractable algorithms and in terms of lack theory that could be used to specify sample size requirements. This paper will provide some perspective on the latter challenges.

Covariance matrices arise in a very diverse set of Big Data applications including: empirical finance and econometrics [84], [85], [86], [127], [14], [44]; MIMO radar and communications [54], [51], [77], [13], [48], [89],[62], [12]; image analysis [49],[92],[18],[152]; network sensing [137], [17], [102], [103], [19], [21]; life and biomedical sciences [94],[136], [2], [118], [79], [156], [106]; and climate science [55], [145], [119], [122], to name just a few. However, covariance matrices are used differently depending on the application and the task. For tracking of targets from space-time-adaptive-radar (STAP) the task is to estimate the entire covariance matrix in so far as it yields estimates of eigenvectors that span the signal (target) subspace [53]. For exploring functional gene regulation networks the task is to determine the matrix locations of the largest pairwise correlations or inverse correlations, often obtained by thresholding the sample correlation matrix [61] or by performing sparsity constrained optimization [7], [47]. In linear discriminant analysis the task is to estimate a quadratic form of the inverse covariance matrix [19], [97]. In anomaly detection, it is the Schur complement of the covariance matrix that is of interest as it is related to residual prediction error covariance [150]. In independence screening one is interested in the support of the non-zero rows of the covariance [36]. In variable selection for prediction it is the support of the regression coefficient vector that is of interest [37] (a setting in which correlation mining subsumes the regression context as a special case).

Of central importance for all of these applications are the sample size requirements, which can differ from task to task. Reliably performing some of these correlation mining tasks might require relatively few samples, e.g., screening for the presence of variables that are hubs of high correlation in a sparsely correlated population, while other tasks might require many more samples, e.g., accurately estimating all entries of the inverse covariance matrix in a densely correlated population. A theoretical framework has been emerging for predicting these sampling requirements as a function of the population correlation structure and as a function of the inference task. The principal aim of this paper is to present building blocks of this framework with an emphasis on the high dimensional setting where the number p of variables is much larger than the number n of samples (p >> n), a setting relevant to correlation mining in massive data sets.

Several real-world examples of this high dimensional setting are given below:

• In studies of correlation networks of the annotated human genome p is on the order of tens of thousands of genes while n is typically fewer than a hundred samples, e.g., corresponding to a population of human subjects. In these studies correlation levels of magnitude as low as 0.2 are sometimes considered significant [30], though some of them may actually be spurious.

• In space-time-adaptive-processing (STAP) radar a spatiotemporal covariance matrix is used to filter out clutter in order to better detect a moving target at a particular range and Doppler frequency. For full degrees-of-freedom (DOF) STAP the estimator of the spatio-temporal clutter covariance can have dimension p = rq on the order of hundreds of thousands and the number of samples n under a hundred. Here r is the number of radar pulses (time), q is the number of elements in the radar array (space), and n is the number of range bins in the vicinity of the target [53],[89].

• For spambot network discovery from honeypot data, correlation of spamming profiles is performed between p = hundreds of thousands of known IP addresses while the number of time points in each profile may only be on the order of n = 30 [153].

• In recommender systems preference vectors of p = millions of subscribers are correlated with other subscribers on the basis of n = a few hundred preference categories, e.g., movies or music, to predict future user preferences [78], [64].

• In fMRI connectomics a long term goal is to correlate brain activations across individual brain neurons, i.e., p = 10¹¹. Currently connectome researchers might parcel the brain into p regions, with p = several thousand, and estimate correlation by averaging over n = hundreds of repeated activation stimuli. The objective is to use the sample correlation matrix to study patterns of activation in order to reveal the “brain network” [125], [133].

Therefore, it is not an exaggeration to say that correlation mining practitioners face a deluge of variables with very limited sample size. With so few samples one is bound to find spurious correlations between some pairs of the many variables. It is therefore essential to understand the intrinsic sampling requirements of such statistical inference problems. The study of sampling requirements falls into several different asymptotic regimes. Classical statistical error prediction and control methods are based on an asymptotic regime where the dimension p is fixed and the sample size n goes to infinity. Such a regime is obviously inapplicable to the case of n < p. Recently, statisticians have developed theory that applies to the regime where p goes to infinity. This theory is in the realm of high dimensional statistical inference, often called the “large p small n regime” [148], [15]. This theory covers the case where both n and p go to infinity, which, as contrasted to the classical fixed p regime, is a setting that we called the mixed asymptotic regime and includes the so-called “high dimensional,” “veryhigh dimensional” or “ultra-high dimensional” settings [15], [36]. Each of these denote regimes where the speed at which n goes to infinity as a function of p becomes progressively slower. However, since the theory requires that both p and n go to infinity it may not be very useful in applications where the availability of samples is limited and finite. In such a case a more useful and relevant regime is the “purely-high dimensional” setting where the number of samples n remains fixed while the number of variables p goes to infinity. This setting applies, for example, to the correlation screening application [61], discussed in Sec. V, where the correlation threshold ρ approaches one as p becomes large. This regime is in fact the highest possible dimensional regime, short of having no samples at all. Thus it is appropriate to call this purely-high dimensional regime the “ultimately-high dimensional regime,” and we shall use these two terms interchangeably in the sequel. A table comparing these asymptotic regimes is given below (Table I).

TABLE I.

Overview of different asymptotic regimes of statistical data analysis. These regimes are determined by the relation between the number n of samples drawn from the population and the number p, called the dimension, of variables. In the classical asymptotic regime the number p is fixed while n goes to infinity. This is the regime where most of the well known classical statistical testing procedures, such as student t tests of the mean, Fisher F tests of the variance, and Pearson tests of the correlation, can be applied reliably. Mixed asymptotic regimes where n and p both go to infinity have received much attention in modern statistics. However, in this era of Big Data where p is exceedingly large, the mixed asymptotic regime is inadequate since it still requires that n go to infinity. The recently introduced “purely high dimensional regime” [61], [58] addressed in this paper, is more suitable to Big Data problems where n is limited and finite (adapted from [63]).

Asymptotic framework	Terminology (“setting”)	Sample size n	Model dimension p	Application domain	References (selected)
Large sample asymptotics (Classical)	small dimensional	→ ∞	fixed	“small data”	Fisher [42], [43], Rao [108], [109], Neyman and Pearson [99], Wilks [151], Wald [141], [142], [143], [144], Cramér [24], [23], Le Cam [82], [83], Chernoff [20], Kiefer and Wolfowitz[76], Bahadur [6], Efron [35]
Mixed asymptotics	high to ultra high dimensional	→ ∞	→ ∞	“medium sized” data (mega or giga scales)	Donoho [33], Zhao and Yu [155], Meinshausen and Bühlmann [95], Candès and Tao [16], Wainwright [138], [139], Bickel, Ritov, and Tsybakov[10], Peng, Wang, Zhou, and Zhu [104], Khare, Oh, and Rajaratnam, [72], Fan and Lv [36]
Large dimension asymptotics	purely high dimensional	fixed	→ ∞	Sample Starved “Big Data” (tera to exascales)	Hero and Rajaratnam [61] Hero and Rajaratnam [58] Firouzi, Hero and Rajaratnam [38]

The purely-high dimensional regime of large p and fixed n is a mathematical characterization of the extremely big data problem. Evidently, this regime poses several challenges to correlation mining practitioners. These include both computational challenges and the challenge of error control and performance prediction. Yet this regime also holds some rather pleasant surprises. Remarkably, there are surprising benefits to having few samples in terms of computation and scaling laws. There is a scalable computational complexity advantage relative to other high dimensional regimes where both n and p are large. In particular, correlation mining algorithms can take advantage of numerical linear algebra shortcuts and approximate k nearest neighbor search to compute large sparse correlation or partial correlation networks. Another benefit of purely-high dimensionality is an advantageous scaling law of the false positive rates as a function of n and p. Even small increases in sample size can provide significant gains in this regime. For example, when the dimension is p = 10,000 and the number of samples is n = 100, the experimenter only needs to double the number of samples in order to accommodate an increase in dimension by six orders of magnitude (p = 10,000,000,000) without increasing the false positive rate [58].

The unifying comparative tool used in this paper is sample complexity analysis. We will use this analysis to place different statistical models and different inference tasks into the various asymptotic regimes shown Table I (see Tables II-III). Sample complexity analysis has been widely applied to study the performance of inference procedures. In supervised computational learning sample complexity is defined as the number of training samples necessary to maintain a given level of generalization error as a function of the complexity of the statistical model or algorithm [9] [57], [100], [28], [67]. Similar notions arise in the context of high dimensional convergence rates [15], Bayesian model complexity [124], and stochastic complexity (minimum description length - MDL) [111]^c1. There is always a tradeoff between statistical sample complexity and statistical model complexity. The nature of this tradeoff is specific to the model and to the task. This specificity forms the basis for the unifying the theory of large scale inference that is developed in this paper.

TABLE II.

Expressions for performance of MAP estimators (4th row) and the associated regimes of asymptotic sample complexity (5th row) for estimating the qr × qr spatio-temporal inverse correlation matrix Ω = ∑⁻¹ = (cov(X))⁻¹, associated with the q × r space-time Gaussian random matrix $\mathbb{X}$, for different types of models of Ω representing prior contextual information (2nd and 3rd rows). The contextual information specifies patterns of sparsity and/or dependency between the q spatial coordinates and the r temporal coordinates. The bound is the asymptotic (large q, r and n) log Frobenius norm error of the Bayes-optimal MAP estimator of Ω, for the different types of contextual information (1st row). The four rightmost columns of the table correspond respectively to: no contextual information about Ω (None); information that Ω is sparse, corresponding to $\mathbb{X}$ being a Gauss Markov random field (GMRF), with λ > 0 sufficiently large so that the sparsity factor is of order o(qr); information that Ω has (rank k = 1) Kronecker product structure (Kronecker), corresponding to decoupled rows and columns of $\mathbb{X}$; and information that Ω has (rank k = 1) Kronecker product structure with sparse Kronecker factors (Kronecker GMRF), with λ₁, λ₂ > 0 ufficicently large so that sparsity factors are of order o (max{q, r}). Comparing the sample complexity regimes (4th row) between the various types of contextual information quantifies the value of taking additional samples (See Fig. 4).

Information	None	Sparse	Kronecker	Kronecker+sparse
Model	saturated Ω	sparse Ω	Ω = A ⊗ B	sparse Ω = A ⊗ B
log f(Ω)	constant	λ ∥ Ω ∥ 1	δ (rankR(Ω) − 1)	δ (rankR(Ω) − 1) + λ1 ∥A∥1 + λ2 ∥B∥2
Bound	$\frac{1}{2}\text{log}\left(\frac{q^2{r}^2}{n}\right)$	$\frac{1}{2}\text{log}\left(\frac{qr\text{log}qr}{n}\right)$	$\frac{1}{2}\text{log}\left(\frac{(q^2+r^2)\text{log}M}{n}\right)$	$\frac{1}{2}\text{log}\left(\frac{(q+r)\text{log}M}{n}\right)$
Regime	$\frac{q^2{r}^2}{n}\to \alpha$	$\frac{qr\text{log}qr}{n}\to \alpha$	$\frac{(q^2+r^2)\text{log}M}{n}\to \alpha$	$\frac{(q+r)\text{log}M}{n}\to \alpha$

TABLE III.

Different sample complexity regimes characterize the difficulty of performing different inference tasks on the inverse covariance Ω. A similar table would hold for inference on the covariance Σ. Task (1st row) specified risk function (2nd row) for which we can use the upper bound (3rd row). The sample complexity regimes (4th row) for which the bound remains constant depends on the task: increasingly large sample sizes are required as the complexity of the task increases from left to right. The limiting value of the critical threshold ρ_c, defined in (9), for screening (5th row) is shown for each regime. For the screening task (detection of existence of large partial correlations) the bound is the purely high dimensional (large p and fixed n) asymptotic limit of the probability of false positives associated with the test that at least one variable is highly correlated (given by Theorem 1). For the detection task (detection of existence of partial correlation of magnitude greater than ρ ∈ (0, 1)) the bound is the mixed high dimensional (large p and large n) asymptotic limit of the probability of false positives. In this regime, the critical threshold converges to ρ* where 0 < ρ* < ρ. For the support recovery task (model selection) the bound is the mixed high dimensional asymptotic probability of misclassification of the set of partial correlations using CONCORD. For this task the parameter ν is specified by a priori knowledge on sparsity of the support set: ν ∈ (0, 1] and as the sparsity increases ν → 0. For the estimation task the bound is the mixed high dimensional asymptotic squared Frobenius norm error on the MAP estimator of sparse inverse covariance. Finally, for the performance estimation task, the bound is the mixed high dimensional asymptotic minmax bound on estimation MSE of the probability of any (Borel) uncertainty set. The values of the constants α and β are not necessarily the same over different columns of the table.

Task	Screening	Detection	Support recovery	Param. estimation	Perform. estimation
Risk	P(Ne > 0)	P(Ne > 0)	P(card{SΔŜ} = φ)	$E[\parallel \Omega -\widehat{\Omega }{\parallel }_F^2]$	$\int E[f_{\Omega }\left(\text{x}\right)-\widehat{f}{\left(\text{x}\right)}^2]d\text{x}$
Bound	1 − e−κn	pe−nβ	2pν e−nβ	$\frac{p\text{log}}{n}\beta$	n−2/(1+p) β
Regimes	$\frac{\text{log}p}{n}\to \infty$	$\frac{\text{log}p}{n}\to \alpha$	$\frac{p^{\nu }}{n}\to \alpha$	$\frac{\text{p log}p}{n}\to \alpha$	$\frac{p}{\text{log}n}\to \alpha$
Threshold	ρc → 1	ρc → ρ*	ρc → 0	ρc → 0	ρc → 0

The outline of the paper is as follows. Sec. II reviews the definitions of covariance, precision, correlation, partial correlation, and Gaussian graphical models. Section III treats the problem of covariance estimation. Section IV develops the problem of model selection and support estimation. In Sec. V the problem of correlation screening is discussed. In Sec. VI sample complexity comparisons of various types of correlation inference tasks are presented. Concluding remarks are given in Sec. VII.

II. Correlation Matrices and Correlation Networks

A. Covariance and precision matrices

Let $\mathbf{X}={[x_1,\dots ,x_p]}^T\in {\mathbf{R}}^p$ be a real-valued random (column) vector having a distribution whose second order moments exist. The mean vector $\mu =E\left[\mathbf{X}\right]\in {\mathbf{R}}^p$ and the covariance matrix $\sum =\text{cov}\left(\mathbf{X}\right)=E\left[(\mathbf{X}-\mu ){(\mathbf{X}-\mu )}^T\right]\in {\mathbf{R}}^{p\times p}$ are defined in terms of the statistical expectation operator E[·] associated with the distribution of X. The matrix Σ is symmetric positive semidefinite and its ij’th entry is the covariance $\text{cov}(X_i,X_j)=E\left[(X_i-{\mu }_i)(X_j-{\mu }_j)\right]$ with μ_i the i-th element of μ. It will be assumed throughout that Σ is in fact positive definite, which is generally true when the p components of X are non-redundant random variables, i.e., no variable can be predicted without error using linear combinations of the other variables. The inverse of the covariance matrix is called the precision matrix Ω = Σ⁻¹. The covariance and precision matrices capture marginal dependency and conditional dependency, respectively, in the joint distribution of X. In particular, when Σ (Ω) is sparse, i.e., it has only a few non-zero elements, there are many variables that are marginally (conditionally) independent of the other variables. More will be said about this in Sec. II-B.

The matrices Σ and Ω are not invariant to scaling of the component variables of X. Scale invariance is important when one is interested in a measure of dependency that is not a function of the units used to represent different variables. The correlation matrix R and partial correlation matrix P are respective versions of Σ and Ω that are scale invariant

$$\mathbf{R}=\text{diag}{(\Sigma )}^{-1∕2}\Sigma \text{diag}{(\Sigma )}^{-1∕2}$$ (1)

$$\mathbf{P}=\text{diag}{(\Omega )}^{-1∕2}\Omega \text{diag}{(\Omega )}^{-1∕2}$$ (2)

where, for a p × p matrix B, diag(B) is the p × p diagonal matrix formed from the diagonal elements of B and, for a diagonal matrix D with non-zero valued entries ${\left\{d_{ii}\right\}}_{i=1}^p$ along the diagonal, D^−1/2 denotes the diagonal matrix with entries ${\{1∕\sqrt{d_{ii}}\}}_{i=1}^p$ along the diagonal. The diagonal elements of R and P are equal to 1 and the off-diagonal elements are between −1 and 1. An important property of R and P is that they retain any zero patterning in Σ and Ω, respectively, if Σ or Ω are sparse.

Let ${\left\{{\text{X}}_k\right\}}_{k=1}^n$ be a set of n i.i.d. realizations from the distribution of X. The sample covariance matrix is the p × p positive semidefinite matrix

$${\mathbf{S}}_n=\frac{1}{n-1}\sum _{k=0}^n({\mathbf{X}}_k-\widehat{\mu })-{({\mathbf{X}}_k-\widehat{\mu })}^T$$ (3)

where $\mu =n^{-1}{\Sigma }_{k=0}^n{\text{X}}_k$ is the sample mean vector. The sample covariance S_n is a statistically consistent estimator of Σ, i.e., it converges to Σ with probability one as $n\to \infty$ if p is fixed. Furthermore, the plug-in estimator $\widehat{R}=\text{diag}{\left({\text{S}}_n\right)}^{-1∕2}{\text{S}}_n\text{diag}{\left({\text{S}}_n\right)}^{-1∕2}$ can be used to estimate R. The matrix S_n is positive definite with probability one if n ≥ p and if the distribution of X is Lebesgue continuous. In this case, plug-in estimators $\widehat{\Omega }={\mathbf{S}}_n^{-1}$ and $\widehat{\mathrm{P}}=\text{diag}{\left(\widehat{\Omega }\right)}^{-1∕2}\widehat{\Omega }\text{diag}{\left(\widehat{\Omega }\right)}^{-1∕2}$ can be used to consistently estimate Ω and P. In the case n < p, which is the main focus of this paper, alternative estimation strategies must be adopted.

B. The Gaussian graphical model

Assume that the data is a set of n i.i.d. realizations ${\left\{{\mathrm{X}}_k\right\}}_{k=1}^n$ of a random vector $\mathbf{X}\in {\mathbf{R}}^p$ that has a multivariate Gaussian distribution with positive definite covariance matrix Σ. When the precision matrix Ω is sparse such data is said to be a Gauss Markov random field (GMRF) or, equivalently, is said to be distributed according to a Gaussian graphical model (GGM). The log-likelihood function for Σ and Ω can be expressed as

$$l(\Sigma )=-\frac{n}{2}\text{tr}\left\{{\Sigma }^{-1}{\mathbf{S}}_n\right\}-\frac{n}{2}\text{logdet}(\Sigma ),$$ (4)

$$l(\Omega )=-\frac{n}{2}\text{tr}\{\Omega {\mathbf{S}}_n\}+\frac{n}{2}\text{logdet}(\Omega ),$$ (5)

up to unimportant additive constants. In the GGM Ω is a more natural parameterization than Σ in the sense that it is sparse while Σ may not be sparse. In addition, the Gaussian Graphical model corresponds to a natural exponential family with Ω as the canonical parameter. Furthermore, the sparse structure of Ω is directly related to conditional independencies specified by the pairwise, local and global Markov properties of undirected graphical models and also by the factorization of the joint density of X [81],[140].

The GGM is a “graphical model” in the following sense. Define a graph G whose p vertices (or nodes) correspond to the p variables X₁,…, X_p in X. Assign an edge between node i and node j if and only if the ij-th entry of the sparse precision matrix Ω is non-zero. The edges of the so-constructed graph G, often called a correlation or partial correlation graph, are specified by an adjacency matrix A. A is a binary matrix that indicates the support of the matrix Ω; specifically the non-zero entries of A indicate the locations the non-zero entries of Ω. The degree of node i is the number of neighboring nodes in the graph and corresponds to the number of non-zero entries in the i-th row of Ω. In GMRFs it is typically assumed that the graph $\mathcal{G}$ has only a small number (O(p)) of edges, corresponding to the property Ω is row sparse^c1 [81].

Observe that in a GGM it is the inverse covariance Ω and not the covariance itself that is important. There are many other reasons for the wide interest in Ω and its scaled version P, the partial correlation matrix (2) [98], [3], [81], [15],[128], [70], [69]. First, given predictor variables X = [X₁,…, X_n]^T, the optimal linear predictor of a response variable Y depends on cov(X) only through its inverse. Second, when X is Gaussian, the Bayes-optimal test between two hypothesized covariances similarly depends only on the inverse covariances. Third, many classical multivariate procedures such as linear discriminant analysis (LDA) and multivariate analysis of variance (MANOVA) depend directly on the inverse covariance matrix. Fourth, the ij-th entry of P is equal to the conditional correlation between X_i and X_j given the rest of the variables ${\left\{X_k\right\}}_{k\ne i,j}$. Fifth, if ${\widehat{X}}_i$ is the optimal linear predictor of X_i given ${\left\{X_k\right\}}_{k\ne i}$, the ij-th entry of the partial correlation matrix P is equal to the correlation coefficient between the prediction error residuals associated with ${\widehat{X}}_i$ and ${\widehat{X}}_j$. Finally, many physical random processes have a sparse inverse covariance matrix while the covariance matrix is not sparse. We illustrate this last point by the following example.

C. Illustrative example of a GGM

We illustrate the differences between the covariance and inverse covariance matrices in a GGM using an example motivated by statistical physics. Let X(u₁, u₂) be a function on the unit square [0, 1]² ⊂ R² that satisfies the Poisson partial differential equation

$$\begin{array}{cc}\hfill \Delta X(u_1,u_2)=W(u_1,u_2),& u_1,u_2\in {(0,1)}^2\hfill \\ \hfill X(u_1,u_2)=0,& u_1,u_2\notin {(0,1)}^2\hfill \end{array}$$

where W (u₁, u₂) is a fixed integrable function. Here $\Delta X=\frac{{\partial }^{2}X}{\partial u_1^2}+\frac{{\partial }^{2}X}{\partial u_2^2}$ is the Laplacian differential operator. Solutions X to the Poisson equation arise in heat transfer, electromagnetics, and fluid dynamics [123].

We extract a Gaussian graphical model from the Poisson equation by discretizing it to a finite difference equation and setting the discretized W to be a Gaussian random matrix. This will convert the continuous domain function X over [0, 1]² to a vector X with p = N₁N₂ elements, where N₁ = 1/δ₁ and N₂ = 1/δ₂ and δ₁, δ₂ are the u₁ and u₂ increments used in the finite difference approximation. Specifically, discretize u₁ and u₂ to the sets $u_1\in {\left\{k{\delta }_1\right\}}_{k=0}^{N_1}$ and $u_2\in {\left\{k{\delta }_2\right\}}_{k=0}^{N_2}$ Then, assuming smooth W, the discretized X satisfies the finite difference equation:

$$X_{i,j}=\frac{(X_{i+1,j}+X_{i-1,j}){\delta }_2^2+(X_{i,j+1}+X_{i,j-1}){\delta }_1^2-W_{i,j}{\delta }_1{\delta }_2}{2({\delta }_1^2+{\delta }_2^2)}$$ (6)

Second, arrange the ${\left\{X_{i,j}\right\}}_{i,j=1}^{N_1,N_2}$ into the vector $\mathbf{X}\in {\mathbf{R}}^{N^2}$ in lexicographic order. This results in an equivalent matrix form of the equation (6)

$$[\mathbf{I}-\mathbf{A}]\mathbf{X}=\mathbf{W}$$

where A is a sparse tri-diagonal matrix.

Under the assumption that W is a i.i.d. zero mean spatio temporal Gaussian driving noise, i.e., $\text{cov}\left(\mathbf{W}\right)={\sigma }_w^2\mathrm{I}$ the covariance Σ and the inverse covariance Ω of X have the from

$$\Sigma ={\Omega }^{-1},\Omega =[\mathbf{I}-\mathbf{A}]{[\mathbf{I}-\mathbf{A}]}^T∕{\sigma }_W^2.$$

As A is sparse tri-diagonal, the inverse covariance Ω and partial correlation P are both sparse penta-diagonal matrices with support set shown in Fig. 1 for N₁ = N₂ = 5. Therefore, the random vector X is a Gauss Markov random field with GGM spatial dependency structure specified by the non-zero entries of Ω, equivalently of P. Note that the covariance matrix Σ not sparse as the inverse of the penta-diagonal matrix is a full matrix. Thus, as is usually the case for a GGM, the inverse covariance is a more parsimonious model description than is the covariance.

Fig. 1.

The Gaussian graphical model obtained from a finite difference approximation to the Poisson equation has only local spatial dependency (left panel) resulting in a pentadiagonal inverse covariance matrix (right panel).

For visualization, two realizations of X are shown in Fig. 2 for a simulation of the discretized Poisson equation (6) on a 30 × 30 spatial grid (p = 900). While general methods of support set estimation are discussed in Sec. IV, Fig. 3 illustrates a simple empirical estimator of the support set of Ω based on n = 1500 samples of X. This simple estimator is constructed by thresholding an estimate $\widehat{\mathrm{P}}$ of the partial correlation matrix P at a level $\rho \in [0,1]$, which is user defined. The non-zero entries of the adjacency matrix associated with the thresholded $\widehat{\mathrm{P}}$ specifies the support set estimator. Here $\widehat{\mathrm{P}}=\text{diag}{\left(\widehat{\Omega }\right)}^{-1∕2}\widehat{\Omega }\text{diag}{\left(\widehat{\Omega }\right)}^{-1∕2}$, where $\widehat{\Omega }={\widehat{\mathrm{S}}}_n^{-1}$ is the inverse of the sample covariance Sn defined in (3). Figure 3 shows the support set estimator for six different values of the applied threshold ρ. Note that when ρ = 0.26114 the true support set, illustrated in Fig. 1 for a 5 × 5 grid, is recovered. When ρ decreases below a critical value there is an abrupt increase in the number of false edges. Below this critical value (top left panel of Fig. 3) the number of false edges approaches the number $\left(\begin{array}{c}\hfill p\hfill \\ \hfill 2\hfill \end{array}\right)$ of edges in the complete graph.

Fig. 2.

Two (n = 2) independent realizations of the discretized Poisson random field obtained from the finite difference approximation (6) with N₁ = N₂ = 90. Here the driving process W is a zero mean spatio-temporally white Gaussian noise.

Fig. 3.

The support of the thresholded sample partial correlation matrix, rendered as a graph over spatial locations in the plane, for a Gaussian random field generated by the Poisson equation on a 30 × 30 grid (p = 900). The sample partial correlation $\widehat{\mathbf{P}}=\text{diag}{\left(\widehat{\Omega }\right)}^{-1∕2}\widehat{\Omega }\text{diag}{\left(\widehat{\Omega }\right)}^{-1∕2}\text{where}\widehat{\Omega }={\mathbf{S}}_n^{-1}$ the inverse of the sample covariance matrix, was computed using n = 1500 samples. The graph is shown for six different threshold levels p applied to $\widehat{\mathrm{P}}$ The true cartesian support set is recovered for ρ = 0.26114. There is a sharp increase in the number of false edges as the threshold ρ decreases below a certain threshold, somewhere between 0.0791 and 0.13978. The location of this threshold appears well approximated by the theory [61] that predicts a critical threshold value 0.091 (See equation (9) in Sec. V of this paper).

III. Correlation Mining: Estimation

An ambitious correlation mining objective is to empirically estimate each and every one of the entries of the covariance Σ or inverse covariance matrix Ω from the n samples. The accuracy of these estimators is often measured by the Frobenius norm of the estimation error. Other common criteria for gauging estimator accuracy are the matrix ℓ₂ norm, also called the operator norm, and the matrix ℓ₁ norm. For example, in estimation of Ω the Frobenius norm difference between Ω and its empirical estimate $\widehat{\Omega }$ is:

$$\parallel \Omega -\widehat{\Omega }{\parallel }_F=\sqrt{\sum _{i,j=1}^p{({\omega }_{ij}-{\widehat{\omega }}_{ij})}^2,}$$

where ω_ij denotes the ij-th entry of Ω. When the objective is to estimate the entries of the covariance, the correlation or the partial correlation matrix, the Frobenius norm error is defined similarly. Covariance and inverse covariance estimation arise in many applications including: finance [86], [50]; gene expression modeling [157], [115]; sensor array processing [60],[71], [66]; space-time-adaptive processing (STAP) radar [149], [53] [126]; spatio-temporal classification [52] and prediction [130]; brain connectomics [22],[134]; and sensor network anomaly detection [19].

When the data are multivariate Gaussian, the maximum likelihood (ML) estimator of Ω maximizes the log-likelihood function (5). In the case when n ≥ p the ML estimator of the covariance Σ is equal to $\widehat{\Sigma }=\frac{n-1}{n}{\mathrm{S}}_n$ where S_n is the sample covariance matrix defined in (3), and the ML estimator of Ω is $\widehat{\Omega }={\widehat{\Sigma }}^{-1}$ In high dimensional situations where n < p, S_n is not positive definite and these ML estimators do not exist. Estimator regularization is commonly used in this situation in order to yield a positive definite solution. Model-based regularization methods add a suitable penalty function c(Ω) to the log-likelihood function resulting in the so-called penalized ML estimator. When the penalty is interpreted as a log prior on Ω the penalized ML estimator is equivalent to the maximum a posteriori (MAP) estimator, also known as the posterior mode estimator. Commonly used regularization penalties are the quadratic penalty $\lambda \parallel \Omega {\parallel }_F^2$ and the non-sparsity penalty $\lambda \parallel \Omega {\parallel }_1=\lambda {\sum }_{i\ne j}^p\mid {\omega }_{ij}\mid$, where λ > 0 is called the regularization parameter, and pushes the penalized ML solution towards a diagonal matrix as λ increases to ∞ Other penalties shrink the penalized ML solution towards more highly structured matrices, e.g., block sparse, Toeplitz, banded, circulant, or Kronecker matrix structure. We call the latter structured penalties as contrasted to the un-structured penalties that encourage the estimator to be diagonal. When the true inverse covariance Ω has the same structure as the structure induced by the penalty we say that the penalty of the penalized ML estimator is matched to the model.

The effect of different structured models for Ω on the asymptotic convergence rate of the penalized ML error can be of interest to practitioners. In large scale settings the high dimensional convergence rate is a natural comparative performance measure. High dimensional rates of convergence have been obtained under a large range of structured and unstructured penalties (and matched models). Often, when the penalty is matched to the model, the Frobenius norm estimation error decreases at asymptotic rate of the form $\sqrt{P\left(\text{log}p\right)∕n}\text{as}p,n\to \infty$ where P is the (effective) number of free parameters in the model. We illustrate for the case of spatio-temporal multivariate Gaussian graphical models; a poster child for high dimensional inference.

A. Illustration: estimation of a spatio-temporal precision matrix

Consider the case where a set of q sensors acquires r time samples (a snapshot) of a spatio-temporal random field, e.g., the STAP radar example [53],[89] discussed in the introduction. There are n snapshots of this random field. Each snapshot generates a multivariate Gaussian distributed random matrix $\mathbb{X}$ with q rows (spatial) and r columns (temporal). Define the lexicographically ordered vectorization $\mathbf{X}=\text{vec}\left(\mathbb{X}\right)\in {\mathbf{R}}^{qr}\text{of}\mathbb{X}$. The spatio-temporal covariance cov(X) = Σ of a snapshot is an unknown p × p matrix where p = qr. The task is to estimate the precision matrix Ω = Σ⁻¹. When there is no additional information about Σ, beyond that it is symmetric and positive definite, the precision matrix Ω is completely unknown and there are $P=\left(\begin{array}{c}\hfill p\hfill \\ \hfill 2\hfill \end{array}\right)=O\left(p^2\right)$ unknown parameters to be estimated, which is the number of distinct entries of the matrix. The base case of no information is often called the saturated model. If the experimenter is given additional contextual information about the structure of the precision matrix the value of this information can often be quantified in terms of sample complexity. We give several examples below.

Spatio-temporal contextual information: sparse precision matrix

Contextual information specifying that $\mathbb{X}$ is a GGM with sparse inverse covariance matrix results in a significant reduction in the number of free parameters in Ω: from P = O(p²) to P = O(p). The problem of sparse covariance estimation has been of significant interest [95], [8], [47]. When the spatio-temporal matrix $\mathbb{X}$ represents the outputs of sensor network, an example of contextual information is the knowledge that the physical environment is a time-varying random field whose amplitudes obey the laws of Lagrangian classical mechanics, e.g., fluid flow, heat flow, or electromagnetic wave fields that satisfy Poisson or Navier-Stokes partial differential equations [117]. The contextual information may simply be an upper bound on the sparsity parameter s or it may actually specify the sparsity pattern, i.e., fix the graph G specifying the support of Ω.

Spatio-temporal contextual information: Kronecker covariance

Any spatio-temporal covariance has some degree of Kronecker structure that can be captured by the Kronecker sum decomposition of Pitsianis and Van Loan [135], [130]. When applied to the inverse covariance, this decomposition is $\Omega ={\sum }_{k=1}^k{\mathbf{A}}_i\otimes {\mathbf{B}}_i$ where k is the Kronecker rank and A_i,B_i are linearly independent q × q and r × r matrices, called Kronecker factors. When the contextual information is that Ω has Kronecker rank k the number of free parameters in Ω is reduced from Ω(p²) to $O\left(p^2\right)\text{to}O\left(k(q^2+r^2)\right)$. The simplest case occurs when Ω is known to have Kronecker rank 1: $\Omega =\mathbf{A}\otimes \mathbf{B}$ where A and B are symmetric positive definite q × q (spatial) and r × r (temporal) covariance matrices. Covariance estimation in this Kronecker model, also known as the matrix normal model, has been widely studied [34], [147], [131]. A physical example of Kronecker-structured covariance is the case that $\mathbb{X}$ are outputs of a passive antenna array that is sensing the EM emissions of k wide-sense-stationary nonmoving targets in the far field of the array. In particular, if k = 1 there is a single target and the spatial and temporal components of the covariance are completely decoupled. In this case, the contextual information could be the number of targets.

Spatio-temporal contextual information: Kronecker+sparse precision matrix

Consider the case that the contextual information specifies that $\mathbb{X}$ has a precision matrix Ω that is both Kronecker structured and has sparse Kronecker factors. When the Kronecker rank is k = 1 then Ω factors into A ⊗ B and there are P = O(qs_A + rs_B) unknown parameters, where s_A and s_B specify the sparsity in each of the Kronecker factors A and B [131], [1].

The value of the different types of contextual information can be studied in terms of estimation of Ω using theory developed recently in [131], which discusses estimators and estimator convergence rates for the cases that Ω is, respectively, sparse, Kronecker, and sparse-Kronecker structured. These structural constraints are incorporated into the estimator $\widehat{\Omega }$ of Ω using penalized maximum likelihood methods, which can be interpreted as MAP covariance estimators when the prior distributions on the entries of Ω are specified by the penalty functions. For each case denote the mean-squared error of the MAP estimator as the expectation of the Frobenius norm difference squared: $\text{MSE}=E[\parallel \Omega -\widehat{\Omega }{\parallel }_F^2]$ The relative decrease in the associated MSE due to any of the above pieces of contextual information is a function of the sample complexity for the inverse covariance estimation task. The forms of these MAP estimators, their asymptotic logMSE, and their asymptotic sample complexity can be obtained directly from [131], [130] and are summarized below and in Table II.

For the saturated model (no side information about Ω) the maximum likelihood estimator is equal to the inverse of the maximum likelihood estimator is equal to the inverse of the is the sample covariance $\widehat{\Sigma }=n^{-1}{\sum }_{i=1}^n({\text{x}}_i-\stackrel{‒}{x}){({\text{x}}_i-\stackrel{‒}{\mathbf{x}})}^T$, where $\stackrel{‒}{\mathbf{x}}$ is the sample mean of ${\left\{{\text{x}}_i\right\}}_{i=1}^n$ [97]. This is the MAP estimator under the uniform estimator-loss function and a uninformative (constant) prior on Ω. When contextual information specifies that Ω is in fact sparse, the Glasso penalized ML estimator [47] adds an l1-norm penalty on Ω to the log-likelihood function. The Glasso precision estimator is the MAP estimator under a Laplacian-like prior on Ω and it can be determined using an iterative maximization algorithm. When contextual information is that Ω has Kronecker structure, again an iterative algorithm must be used to find the maximum likelihood estimator under the Kronecker model [146]. This is the Bayes optimal estimator under a Kronecker covariance model and non-informative priors on the Kronecker factors. Finally, when the contextual information is that Ω is both Kronecker and sparse the Kronecker log-likelihood model can be penalized by l1-norms on each of the Kronecker factors, resulting in a MAP estimator under a Laplacian-type prior on each of the factors [1], [131].

The value-added brought to estimator performance by each of these contextual information sources can be assessed by studying the asymptotic sample complexity. The asymptotic sample complexity is defined by the number n = n_p of samples, as a function of the dimension p, required to maintain a given level of performance as p goes to infinity. Under general conditions on the MAP spatio-temporal covariance estimators, the log MSE takes the high dimensional (large q, r and large n) asymptotic form shown in the 3rd row of Table II [131]. The second row of the table gives the form for the prior on Ω as (from left to right): 1) uniform prior in the case of no contextual information; 2) sparse ℓ₁ prior in the case of sparsity information on Ω; 3) Kronecker structured Ω in the case of contextual information that decouples space and time (the prior δ(rankR(Ω) − 1) is a delta function that forces kronecker structure Ω = A ⊗ B)^c1; 4) Kronecker plus sparse Ω.

By quantifying the change in log MSE associated with different types of contextual information (sparse, Kronecker, Kronecker+sparse) the value of taking additional samples can be determined across the contextual information regimes shown in Table II. For the purpose of comparison we assume that q = r = ^√p, i.e., the spatial and temporal dimensions are identical. For the different categories of contextual information we fix the number of variables p and the log MSE. Figure 4 plots the level sets of constant log MSE over the number of samples and the number of variables. These curves indicate that the knowledge of both sparse and Kronecker structure is more valuable than knowledge of either sparse or Kroenecker structure alone. To illustrate, assume that X is a 100×10 matrix corresponding to 10 shapshots of 100 sensors. Then p = 1000 and, from the right panel of Fig. 4, if the contextual information specified that the inverse covariance has Kronecker and sparse structure the MAP estimator requires only n = 75 samples as compared to 4000 or 8000 samples if Kronecker or sparse structure is specified, and n = 1, 000, 000 samples if no information were available. The value of the information that the covariance is both sparse and Kronecker structure is that it decreases sampling requirements by more than 4 orders of magnitude relative to no contextual information!

Fig. 4.

Sample complexity for estimating the qr × qr spatio-temporal correlation matrix for different types of prior contextual information shown in the 1st row of Table II for the case q = r = ^√p, where ^√p is a positive integer. The curves show asymptotic sample complexity (5th row in the Table), which are constant contours of the proxies (4th row of the Table) over the plane of the number p of variables and the number of samples n. The asymptotic proxies are equal over all the curves and along each curve. Curves to the left represent lower sample complexity and indicate the reduction in the required number of samples to attain a given level of log MSE for specified number of parameters. Here the contextual information represents knowledge that the inverse covariance has sparse structure alone (curve labeled “GMRF”), Kronecker structure alone (curve labeled “Kronecker”), vs Kronecker+GMRF structure. The curve labeled “No information” represents the case where there is no contextual side information about the inverse covariance. The curve labeled Kronecker GMRF dominates all of the others since information that the inverse covariance has both Kronecker and GMRF structure achieves maximal reduction in the number of free parameters and provides the highest value per sample.

IV. Correlation Mining: Model Selection

One of the primary goals of model selection in the correlation mining setting is to identify the support of the correlation or partial correlation matrix, i.e., identify the pairs of variables with non-zero correlations or partial correlations, from n measurements of the set of p variables [8], [47], [65], [56], [27], [95], [104], [113], [46], [87], [72], [101]. Model selection should be easier than estimating the values of all the correlations, discussed in Sec. III. The original covariance selection problem [31] considers estimating inverse covariance matrices with zero entries under a multivariate Gaussian model for the observations. In recent years, the problem of estimating the support of sparse inverse covariance matrices has become a popular topic in the high dimensional statistics and machine learning literature.

As discussed in Sec. II-A, the problem of identifying the sparse patterning structure in the inverse covariance matrix Ω is equivalent to identifying the graph $\mathcal{G}$ associated with the non-zero entries of Ω, and is thus also popularly known as graphical model selection. Various approaches have been proposed for identifying graphical models from high dimensional data. They can be categorized broadly into penalized likelihood methods and Bayesian methods. Popular Bayesian methods entail specifying priors on the space of sparse covariance or inverse covariance matrices [29], [88], [107], [75] and using Bayesian scoring rules to undertake model selection. Penalized likelihood methods in the context of (partial) correlation mining can be further categorized into a) Gaussian-based penalized likelihood methods [8], [47], [65], [56], [27], and b) Pseudo-likelihood based or regression based methods [95], [104], [113], [46], [87], [72], [101].

Recent work on penalized likelihood methods have focused on understanding both the computational complexity and sample complexity of model selection approaches. The former entails computing the sparse inverse covariance estimate using ℓ₁-penalized likelihood approaches and identifying the corresponding graph. This includes among others developing fast iterative algorithms for maximizing ℓ₁-penalized likelihoods in order to obtain sparse inverse covariance estimates, quantifying the computational complexity of these algorithms, and deriving convergence rates of the iterative algorithms. In line with the main theme of this paper, the focus of this section will be to understand the sample complexity of model selection problems in the correlation mining context.

Sample complexity of model selection approaches are generally stated in terms of sign consistency and estimation consistency (see [104], [72]). As in covariance estimation, the set-up is to let both the sample size n, and the dimension p = p_n tend to infinity and establish large sample properties for a sequence of covariance parameters that is growing in dimension. We shall illustrate the sample complexity of model selection via a concrete example below. In particular, we present below a sample complexity result of a recently proposed graphical model selection method proposed in [72] called CONCORD.

CONCORD seeks to maximize the following jointly convex objective function, called the pseudo-likelihood, as proposed in [72]:

$$\begin{array}{cc}\hfill Q_{\text{con}}(\Omega )=:& -\sum _{i=1}^{p}n\text{log}{\omega }_{ii}+\frac{1}{2}\sum _{i=1}^p\parallel {\omega }_{ii}{\mathbf{Y}}_i\hfill \\ \hfill & +\sum _{j\ne i}{\omega }_{ij}{\mathbf{Y}}_j{\parallel }_2^2+\lambda \sum _{1\le i<j\le p}\mid {\omega }_{ij}\mid ,\hfill \end{array}$$ (7)

where ω_ij is the ij-th element of the p × p matrix Ω, and Y_i denotes the i-th feature vector. Iterative optimization algorithms, along with aspects of computational complexity and algorithm convergence, are covered in [72], [74], [101]. For sample complexity results, we shall follow very closely large sample results of the CONCORD graphical model selection approach as given in [72]. Both estimation consistency and oracle properties under suitable regularity conditions are stated below. The reader is referred to [72] for further technical details.

Let ${\left\{{\stackrel{‒}{\Omega }}_n\right\}}_{n\ge 1}$ denote the sequence of true underlying inverse covariance matrices and let the dimension p = p_n vary with the sample size n. As in [104], assume the existence of the non-zero entries of Ω, and is thus also popularly known as graphical model selection. Various approaches have been accurate estimates of the diagonal entries ${{\{{\widehat{\alpha }}_{n,ii},\}}_{1\le i}}_{\le pn}$ such that for any $\eta >0$ there exists a constant C > 0 such that

$$\underset{1\le i\le p_n}{\text{max}}\mid {\widehat{\alpha }}_{n,ii}-{\stackrel{‒}{\omega }}_{ii}\mid \le C\left(\sqrt{\frac{\text{log}n}{n}}\right),$$

holds with probability larger than $1-O\left(n^{-\eta }\right)$.

For vectors ${\omega }^o\in \mathbb{R}\frac{p_n(p_n-1)}{2}$ and ${\omega }^d\in {\mathbb{R}}_{+}^{p_n}$, let the notation ${\mathcal{L}}_n({\omega }^o,{\omega }^d)\text{denote}\frac{\mathcal{L}\text{con}}{n}$ evaluated at a matrix with off-diagonal entries ω^o and diagonal entries ω^d. Let ${\stackrel{‒}{\omega }}_n^o={\left(\left({\stackrel{‒}{\omega }}_{n,ij}\right)\right)}_{1\le i<j\le p_n}$ denote the vector of off-diagonal entries of ${\stackrel{‒}{\Omega }}_n$, and ${\widehat{\alpha }}_{pn}\in {\mathbb{R}}_{+}^{p_n}$ denotes the vector with entries ${{\{{\widehat{\alpha }}_{n,ii},\}}_{1\le i}}_{\le pn}$. Let the sequence A_n denote the set of non-zero entries in the vector ${\stackrel{‒}{\omega }}_n^o$ and let $q_n=\mid A_n\mid$. Let ${\stackrel{‒}{\theta }}_{n,ij}=\frac{{\stackrel{‒}{\omega }}_{n,ij}}{\sqrt{{\widehat{\alpha }}_{n,ii}}{\widehat{\alpha }}_{a,jj}}$ for $1\le i<j\le p_n$ and define ${\stackrel{‒}{\theta }}_n^o={\left(\left({\stackrel{‒}{\theta }}_{n,ij}\right)\right)}_{1\le i<j\le p_n}\in {\mathbb{R}}^{p_n}(p_n-1){∕}^2$. Also, let s_n = min_(i,j)${\stackrel{‒}{\omega }}_{n,ij}$. Assume furthermore that the following three regularity conditions are met (i) the spectrum of ${\stackrel{‒}{\Omega }}_n$ is uniformly bounded from above and below, (ii) sub-Gaussianity of the data, (iii) the incoherence condition [95].

Under the above assumptions the following model selection result is established in [72].

Theorem 1: [72] Suppose that assumptions (i), (ii), (iii) are satisfied. Suppose $p_n=O\left(n^k\right)$ for some $k>0,q_n=o\left(\sqrt{n∕\text{log}n}\right),\sqrt{\frac{qn\text{log}n}{n}}=o\left({\lambda }_n\right),{\lambda }_n\sqrt{n∕\text{log}n}\to \infty ,\frac{s_n}{\sqrt{q_n{\lambda }_n}}\to \infty \text{and}\sqrt{q_n{\lambda }_n}\to 0,\text{as}n\to \infty$. Then there exists a constant C such that for any $\eta >0$, the following events hold with probability at least $1-O\left(n^{-\eta }\right)$.

• There exists a minimizer ${\widehat{\omega }}_n^o={\left(\left({\widehat{\omega }}_{n,ij}\right)\right)}_{1\le i<j\le p_n}$ of the CONCORD objective function $Q_{\text{con}}({\omega }^o,{\widehat{\alpha }}_n)$.

• Any minimizer ${\widehat{\omega }}_n^o\text{of}{Q}_{\text{con}}({\omega }^o,{\widehat{\alpha }}_n)$ satisfies $\parallel {\widehat{\omega }}_n^o{\stackrel{‒}{\omega }}_n^o{\parallel }_2\le C\sqrt{q_n}{\lambda }_n$ and $\text{sing}\left({\widehat{\omega }}_{n,ij}\right)=\text{sign}\left({\stackrel{‒}{\omega }}_{n,ij}\right),\forall 1\le i<\le p_n$

The above result in [72] establishes model selection consistency (or rather sign consistency to be precise), and is in spirit similar to other model selection consistency results in the literature (see also [95] and [104]). A few remarks are in order with regards to sample complexity. First, note that model selection consistency requires that both the sample n and dimension p_n tend to infinity. Hence asymptotic guarantees require large sample sizes and thus model selection consistency may not be valid in sample starved settings. Second, results in model selection consistency are often proved under the assumption of sub-Gaussianity of the tails, and thus may be restrictive in many applications with heavy-tailed data. Third, note that the dimension p_n can grow faster than the sample size n, but cannot grow faster than a polynomial rate.

V. Correlation Mining: Screening

In correlation screening one seeks to discover patterns of high correlation or partial correlation between p variables based on a set of n observations [61], [58]. Stated in terms of a Gaussian graphical model, the objective is to infer topological characteristics of the graph G associated with the zeros in the precision matrix Ω. In [61], we treated the problem of screening for the presence of variables with high correlations to other variables. In [58], we considered the setting of screening for the presence of connected nodes and hubs in G with high partial correlation. Screening for such topology characteristics of $\mathcal{G}$ should be easier than model selection or covariance estimation. Similarly to what was demonstrated for covariance estimation (recall Table II), contextual information can be of high value for screening. For example, one may be given information that specifies a certain sparsity level or block diagonal structure of the inverse covariance, or information on the minimum level of correlation that exists among the active variables in a block.

The correlation screening method of [58] finds edges, hubs, and other subgraph structures of G by performing hypothesis testing. The method applies a threshold to an empirical estimate $\widehat{P}$ of the partial correlation matrix P, defined in (2), placing an edge in G where the magnitude of the entry of $\widehat{P}$ exceeds the threshold. When n ≥ p, $\widehat{P}$ may be the simple plugin estimator of the matrix inverse of the sample correlation estimator, while if n < p the correlation screening methods developed in [58] uses the Moore-Penrose generalized inverse of the sample correlation estimator. Correlation screening has been studied and applied to hub discovery [58], edge discovery [63] and classification of local node degree [41] in a variety of graphical model applications including: stationary Gaussian spatio-temporal processes models [39], [40]; sparse regression models [37]; and multiple model testing for common sparsity patterns [61].

The computational complexity of correlation screening is much lower than model selection or covariance estimation, only of the order of O(n²), in the sample starved case of n << p. To illustrate, the sample complexity of screening edges in G can be determined from the following theorem (adapted from [58, Prop. 2], see also [63]):

Theorem 2: Assume that the n samples ${\left\{{\text{X}}_k\right\}}_{k=1}^n$ are i.i.d. random vectors in ${\mathbb{R}}^p$ with bounded elliptically contoured density and block sparse p × p covariance matrix. Let ρ be the threshold applied to the sample partial correlation matrix $\widehat{P}$. Assume that p goes to ∞ and ρ = ρ_p goes to one at a rate specified by the relation ${\text{lim}}_{p\to \infty }p(p-1){(1-{\rho }_p^2)}^{(n-2)∕2}=e_n$, where e_n ∈ (0, ∞) is given. Then the probability P_e that there exists at least one false edge in G satisfies

$$\underset{p\to \infty }{\text{lim}}{P}_e=1-\text{exp}(-{\kappa }_n∕2),$$ (8)

where

$$k_n=e_n{a}_n{(n-2)}^{-1}$$

where a_n is the volume of the n − 2 dimensional unit sphere in Rⁿ⁻¹ and is given by $a_n=\frac{\Gamma ((n-1)∕2)}{\sqrt{\pi }\Gamma \left((n-2)∕2\right)}$.

As in other types of hypothesis testing problems, two types of correlation screening errors can occur: false positives (Type I) and false negatives (Type II). It is common for an experimenter to constrain the false positive rate to ensure a certain level of Type I error control. Remarkably, Thm. 2 asserts that, under the stated conditions, the large p false positive rate does not depend on the true covariance Σ. In this large p case the correlation threshold ρ can be set to attain a given level of false positive control. This fortunate situation is analogous to constant false alarm rate (CFAR) signal detection in radar processing [120], [112],[110]. In [61] it was shown that, when screening a large number p of variables, the false positive rate undergoes a fundamental phase transition as a function of the applied threshold: the rate precipitously increases from almost to zero to one as the correlation threshold is decreased beyond a critical threshold ρ_c . A direct consequence of Thm. 2 is that ρ_c has the form [58, Eq. (10)]:

$${\rho }_c=\sqrt{1-{\left(a_n(p-1)\right)}^{-2∕(n-4)}},$$ (9)

When the applied threshold is greater than ρ_c there will be few false positives while when it is below ρ_c the system will be inundated by false positives.

Using (9), and a large p approximation [58, Eq. (22)) to the false positive rate following directly from Thm. 2, we can quantify the intrinsic value of taking an additional sample when the task is to detect variables that have true correlations exceeding a given threshold ρ. Assume that p is fixed but large. Figures 5 and 6 gives a family of design curves that can be used by the system designer to right-size n for given p and given desired correlation level ρ

Fig. 5.

Correlation screening curves quantifying value of information associated with acquiring more samples n for different parameter dimensions (p = 100, 10,000, and 10,000,000,000) in terms of the minimum detectable correlation value ρ. The screening task is to detect variables that have high correlation (greater than ρ) to at least one other variable. These curves specify the minimum required number of samples (n) for reliable detection of such variables at given family wise false positive error rates. For example, for ten billion (10¹⁰) variables at least 200 samples are required to reliably detect a variable having correlation greater than ρ = 0.6, while fewer than half the number of samples would be needed to detect the same level of correlation if there were only ten thousand variables. Thus, for the correlation screening task, the value of a sample is much higher when there are fewer variables, and the displayed curves quantify this value. The curves in left panel are isoclines on the probability of error surface for fixed family wise error rate (FWER) equal to 0.0001. The curves in the right panel are similar except that they are isoclinal for fixed mean false positive rate of 1 (only 1 false positive node detected out of p nodes). The respective FWER's of false positive probability are designated on each curve in the right panel. The curves in the left and right panel are very similar since the probability of error surface undergoes an abrupt phase transition from 0 to 1.

Fig. 6.

Correlation screening curves quantifying value of information associated with acquiring more samples n for different minimal detectable correlation levels (ρ = 0.3,0.4,0.5,0.6,0.7, 0.8) in terms of the parameter dimension ρ. The screening task is the same as in Fig. 5 but the phase transition is plotted differently to reveal the value of information for detecting different fixed levels of correlation for varying numbers of parameters p. Note that the number of samples n required for reliably detecting variables with high correlations, e.g., ρ = 0.8, increases much more slowly as p increases than it does for small correlations, e.g., ρ = 0.3. Thus, as the desired correlation level increases, there is a diminishing return in the value of information delivered by acquiring additional samples.

VI. Correlation Mining: Intrinsic Sample Complexity Regimes

The experimenter is often faced with several correlation mining tasks, possibly performed in sequence. For example, detection of existence of high correlations among p variables may be followed by identification of the set of highly correlated variables, followed by estimation of the values of their correlations, followed by specification of the uncertainty (confidence intervals) associated with these estimates. Reliable accomplishment of each task becomes more difficult as one progresses from detection to uncertainty quantification, requiring progressively larger numbers of samples and progressively smaller critical phase transition thresholds ρ_c. Establishing the sampling regimes associated with each one of these tasks is one of the fundamental problems of large scale inference and data science.

Recall that the asymptotic sample complexity associated with an inference task is the number n = n_p of samples, as a function of the dimension p, required to maintain a given value of risk as p goes to infinity. Table III summariz’s the sample complexity regimes (3rd row) and critical phase transition threshold regimes (4th row) for tasks relevant to co1relation mining. These are discussed below in order of increasing sample complexity.

• Screening: The screening task is to use the sample correlation to detect the existence of high correlations, by which we mean large values in either the population correlation or partial correlation matrix; equivalently to detect the existence of an edge in the correlation or partial correlation network, as discussed in Sec. V. As such, it is a binary hypothesis testing problem having risk function equal to the false positive probability under the null hypothesis H₀ of a sparse and invertible population covariance matrix Σ. For the threshold-based correlation screening method of [61] the false positive probability is P (N_e > 0), where N_e is the number of entries (edges) of the sample partial correlation matrix that exceed a threshold ρ. The bound is the asymptotic limiting value of the false positive edge probability specified in Thm. 2. The sample complexity regime is: n=fixed (not a function of p) while p → ∞, denoted in Table III as $\frac{\text{log}p}{n}\to \infty$. The critical threshold (9) converges to 1 as p → ∞.

• Detection: The detection task is the same as the screening task except that both high and low correlations are of interest. The experimenter specifies a threshold ρ ∈ (0, 1) and the objective is to find correlations of magnitude at least ρ. The sample complexity regime for this problem is: n increases to infinity with p at the asymptotic rate $\frac{\text{log}p}{n}=\alpha$, where where α ∈ (0, ∞) is a constant. The critical threshold (9) converges to a constant ρ^* satisfying 0 < ρ^* < p as p → ∞.

• Support recovery: This is the problem of model selection discussed in Sec. IV where the objective is to identify the support set S ⊂ {1,… , p} of the population inverse covariance Ω. If $\widehat{S}$ denotes an estimator of this support set, the risk function is the probability that indices in S are missing in $\widehat{S}$ or that indices in {1,… , p} ⊂ S are erroneously included in $\widehat{S}$, denoted $P(\text{card}\{S\Delta \widehat{S}\}=\varphi )$ where AΔB is the symmetric difference between sets A and B. Assume it is a priori known that the cardinality of S is at most k, where 1 ≤ k ≤ p. A finite sample upper bound follows by applying the union bound over the possible subsets of {1, ..., p} of cardinality at most k to obtain: $P(\text{card}\{S\Delta \widehat{S}\}=\varphi )\le {\sum }_{i=o}^k\left(\begin{array}{c}\hfill p\hfill \\ \hfill i\hfill \end{array}\right)e^{-\beta n}$, where β is the minimum Kullback-Liebler divergence between these subsets. The well known bound [4] on partial sums of binomial coefficients ${\sum }_{i=0}^k\left(\begin{array}{c}\hfill p\hfill \\ \hfill i\hfill \end{array}\right)\le 2^{H(k∕p)p}$ where $H\left(ϵ\right)\text{log}ϵ-(1-ϵ)\text{log}(1-ϵ)$, can then be used along with the representation H(k/p)p = p^ν for some ν. ε (0, 1] to obtain the risk bound: 2p^ν e^−βn. Therefore, the limiting regime of values (n, p) for which this bound is constant gives the sample complexity: n increases to infinity with p at the asymptotic rate $\frac{p^{\nu }}{n}=\alpha$, where $\nu \in (0,1)\text{and}\alpha \in (0,\infty )$. Note that this is consistent with the rate $p=O\left(n^k\right)$ of the CONCORD support recovery algorithm, given in Thm. 1, for k > 1, the regime of where the number of samples is sufficient for model selection but not for parameter estimation. In this regime, the critical threshold (9) converges to zero as p → ∞. Note that the asymptotic rates reported in Table III for support recovery (or equivalently model selection) appear to be slower than what has been derived in the statistics literature. In particular, results are available which assert that provided (log p)/n → 0, support recovery is possible with probability tending to 1 (see [114] for more details). We note that the these faster rates are a direct consequence of assuming that the variables are either Gaussian or sub-Gaussian, or by imposing some tail condition [114]. When such conditions are relaxed, the asymptotic rates coincide with the regimes given in Table III (see also [68], [11] for details on convergence in matrix norms when Gaussianity is relaxed).

• Parameter estimation: The problem of parameter estimation is to determine the individual values of the $\left(\begin{array}{c}\hfill p\hfill \\ \hfill 2\hfill \end{array}\right)$ entries in Ω. The risk function is the MSE, defined as the mean Frobenius norm squared error $E[\parallel \Omega =\widehat{\Omega }{\parallel }_F^2]$ between the population inverse covariance and an empirical estimator. The bound is the high dimensional limiting value of this MSE as n → ∞ and p → ∞ [15]. The sample complexity for this problem is: n increases to infinity with p at the asymptotic rate $\frac{p\text{log}p}{n}=\alpha$, where α ∈ (0, ∞) is a constant. Again the critical threshold (9) converges to zero as p → ∞.

• Performance estimation: We consider the most general (and stringent) setting for performance estimation where, for a specified Borel set $\mathit{B}\subset {\mathbf{R}}^p$ the probability $P(\mathbf{X}\in B)$ must be accurately estimated. For example, assuming that X has a zero mean elliptically contoured density f(x), if B is the set {x ∈ R^p : ∥x∥ > γ } for γ > 0, P (X ∈ B) is the critical region for optimally rejecting outliers and detecting anomalies [121], [59] and the value $\widehat{\gamma }=\parallel \text{X}\parallel$ can be used to estimate the p-value associated with the null hypothesis that X is not an outlier. The sample complexity of estimation of P(X ϵ B), for all B, is equivalent to that of estimation of the density function f(x). Therefore, we adopt the mean integrated squared error (MISE) [132] as the risk function: $\int E{[f_{\Omega }(\text{x)}-\widehat{f}\left(\text{x}\right))}^{2}d\text{x}\text{where}\widehat{f}$ is an empirical density estimator. It is known that if f is in the class of Lipschitz functions the minimax MISE risk is of the form [32]: $\beta n^{-2∕(1+p)}\beta >0$. Hence we use this minimax risk as a proxy for performance and the sample complexity for this problem is: n increases to infinity with p at the asymptotic rate $\frac{\text{log}p}{n}=\alpha$ where $\alpha \in (0,\infty$ is a constant. The critical threshold (9) again converges to zero as p → ∞.

We conclude this section with a comparison of computational complexity. Unlike prediction and model selection, correlation screening methods are scalable to very high dimensions both in terms of computation and memory scalability. Popular sparse optimization approaches to covariance estimation and covariance selection are iterative and include penalized likelihood methods such as Glasso [45], SPACE [105] and CONCORD [73] . The computational complexity of Glasso after t iterations is of order O(tp³). This can be reduced when using regression based methods such as SPACE and CONCORD which have a computational complexity of order min{O(tnp²), O(tp³)}. In contrast, correlation and partial correlation screening are non-iterative algorithms and the computational complexity is only of order O(np log p) which can be considerably less than its penalized likelihood counterparts. The lower order O(np log p) is due to the fact that building the thresholded sample covariance is equivalent to constructing a Euclidean ball graph over p nodes in n − 1 dimensional space, for which reliable approximate nearest neighbors (ANN) algorithms [5] can be applied. Very fast and scalable C, Python and Matlab implementations of ANN algorithms are available, e.g., FLANN, Gensim, and annoy, which have been implemented on datasets with p in the millions and n in the hundreds.

VII. Conclusions

Big data is not just lots of data. This monolithic characterization is overly simplistic and ignores the issues of inference, limited samples, and reproducibility. Big data is of limited utility without appropriate inferential tools, e.g., use of the dataset to produce empirical estimates, classifications, or decisions on the population that generated the data. Inferences in turn lack credibility without accounting for errors due to limited samples. Without credible inferences there is no reproducibility: another random sample from the sample population may produce completely different results.

This paper adopted a statistical perspective in which a large scale dataset is a set of n random samples drawn from a population of p variables and p is large. We focused on the problem of correlation mining where the objective is to infer properties of the population covariance matrix from the samples. The reliability of the inferences from limited samples can be mathematically characterized by the high dimensional learning rates and sample complexity associated with the inference problem. These specify the relative rate at which n must go to infinity as a function of p in order to ensure accurate performance. The sample complexity falls into different high dimensional regimes including the classical regime, where p is fixed and n goes to infinity, the mixed dimensional where both n and p goes to infinity, and the purely high dimensional where n is fixed and p goes to infinity.

The comparative sampling complexity analysis illustrated in this paper unifies the problem of sample sizing for large scale inference problems and, in particular, for correlation mining. Indeed, different sample complexity regimes each occupy a niche for different correlation mining tasks. In particular, screening for high correlations is governed by purely high dimensional rates while model selection, covariance estimation, and uncertainty quantification require n to go to infinity at progressively larger rates as a function of p. This implies, for example, that one can do screening with many fewer samples than are required for the other tasks. Furthermore, in situations where samples are acquired sequentially our analysis suggests that that one can adapt the inference task over time: starting with correlation screening when samples are few, and progressing on through support detection, covariance estimation, and uncertainty quantification as more and more samples are acquired. Such a strategy is explored in the context of the sequential prediction and regression criterion (SPARC) [37].