The Empirical Distribution of a Large Number of Correlated Normal Variables

Abstract

Motivated by the advent of high dimensional highly correlated data, this work studies the limit behavior of the empirical cumulative distribution function (ecdf) of standard normal random variables under arbitrary correlation. First, we provide a necessary and sufficient condition for convergence of the ecdf to the standard normal distribution. Next, under general correlation, we show that the ecdf limit is a random, possible infinite, mixture of normal distribution functions that depends on a number of latent variables and can serve as an asymptotic approximation to the ecdf in high dimensions. We provide conditions under which the dimension of the ecdf limit, defined as the smallest number of effective latent variables, is finite. Estimates of the latent variables are provided and their consistency proved. We demonstrate these methods in a real high-dimensional data example from brain imaging where it is shown that, while the study exhibits apparently strongly significant results, they can be entirely explained by correlation, as captured by the asymptotic approximation developed here.

1 Introduction

The empirical cumulative distribution function (ecdf) and its large sample properties have a long and rich history in probability and statistics. However, most of this vast literature assumes that the variables used to construct the ecdf are independent (e.g., Wasserman, 2006, Chapter 2, and references therein). Under independence, the ecdf is a consistent estimator of the true cumulative distribution function (cdf). The consistency property continues to hold under various forms of weak dependence (e.g. Dedecker and Merlevéde (2007); Wu (2006)).

Motivated by modern problems in high dimensional data, where a large number of correlated variables are measured, it is of interest to study the asymptotic behavior of the ecdf when the variables involved are arbitrarily correlated. In particular, the present paper is motivated by large-scale multiple testing problems, where each of p tests produces a z-score Z_i, i = 1, …, p. It has been pointed out by Bradley Efron that in large-scale multiple testing problems, the observed distribution of the z-scores often does not match the theoretical null distribution N(0, 1) (Efron, 2004, 2007a, b, 2008). Efron (2007a) conjectured that, even when the theoretical model is correct, the observed distribution of the test statistics can look different from the theoretical null distribution simply because of correlation between them. This interesting observation suggests that the ecdf may not always be consistent and calls for a detailed study of the ecdf of a large number of dependent variables.

Assuming that the variables Z₁, Z₂, …, Z_p are marginally standard normal simplifies the problem and allows obtaining results for their ecdf under arbitrary dependence because all the dependence is expressed through correlation. In this situation, Efron (2007a) proposed the so-called empirical null as an approximation to the observed distribution of the z-scores, parametrized as a normal distribution with mean and variance other than 0 and 1. To further understand the effect of correlation, Efron (2010) derived the covariance function of the ecdf and applied it to estimating the variance of functions of the ecdf relevant in large-scale multiple testing, such as the local and tail false discovery rate. Schwartzman (2010) proposed to approximate the ecdf by a Gram-Charlier expansion and used its coefficients to establish some constraints to the extent of the departure of the ecdf from the marginal N(0, 1). However, these approaches have not succeeded to fully characterize the behavior of the ecdf under correlation.

In this article we describe the asymptotic behavior of the ecdf of a large number of correlated standard normal variables. First, we show that in general, the ecdf need not converge to Φ, the standard normal cdf. A necessary and sufficient condition for convergence, which we call weak correlation, is that the average of the absolute pairwise correlations between the z-scores (or the average of the squares of the pairwise correlations) tends to zero with increasing dimension p. However, we show that in a wide range of strong correlation situations, the ecdf converges instead to a random distribution function. This random function can be written as a (possibly infinite) normal mixture parametrized by latent independent standard normal variables. It can be thought of as an analytic asymptotic approximation to the ecdf, where the latent variables can be consistently estimated, under some regularity conditions, from the observed data sequence. Further, it can be thought of as a dimension reduction of the p-dimensional ecdf in the sense that its inherent dimension is the number of the latent variables. We give a lower bound for this inherent dimension and show that, under certain regularity conditions, it can be achieved by a particular parametrization obtained via an eigendecomposition of the correlation matrix. This parametrization is based on the factor analysis model of Fan et al. (2012), who use it to calculate the false discovery proportion in large scale multiple testing under arbitrary correlation. Here we consider a more general framework where not only eigendecompositions are investigated.

As an illustration of the behavior of the ecdf as a random function as described in this paper, Figure 1 presents the histograms of two realizations of normal random variables under two correlation structures:

Figure 1.

Histograms of various instances of 1000 standard normal variables with a one-block correlation structure (a) and a two-independent-blocks correlation structure (b). The red line is the standard normal density and the blue dashed line is the asymptotic approximation we use.

1. One block: ${\{Z_i\}}_{i=1}^{1000}$ is a sequence of exchangeable random variables with correlation ρ = 0.9, i.e., for any i ≠ j, cor(Z_i, Z_j) = ρ

2. Two independent blocks: ${\{Z_i\}}_{i=1}^{1000}$ consists of two intercalated independent sequences of exchangeable random variables, i.e., for any i ≠ j such that |i − j| is even, cor(Z_i, Z_j) = ρ (and 0 otherwise).

It can be seen that for both structures the empirical distribution differs from the standard normal density (red line). In case (a) the histogram is shifted and narrower than the standard normal density, while in case (b) it looks like a mixture of two normal distributions. Notice that the histograms change between the two realizations, suggesting that the distribution may not be converging to a deterministic limit. We will show that in these cases the empirical distribution indeed does not converge to a deterministic limit, but can be asymptotically approximated by a normal mixture estimated from the data, represented here by the dashed blue line. Not surprisingly, in case (a) the asymptotic approximation is of dimension 1, containing one latent standard normal variable, and in case (b) is of dimension 2, containing two independent latent standard normal variables. The fitted density in case (a) can be interpreted as Efron’s empirical null model of a shifted and scaled normal, but clearly that model cannot capture the behavior of the distribution in case (b).

When considering the entire range of possible correlation structures, we show that there are essentially three regimes. The first is what we call weak correlation; while the random variables need not to be independent, the ecdf converges to Φ. The second regime is what we call finite dimensional correlation and is the main interest of this paper. Under this type of correlation, the ecdf can be approximated by a random function that depends on a finite number of latent independent standard normal random variables. Furthermore, under some regularity conditions, a representation using the smallest number of latent variables can be achieved and estimated consistently. The examples of Figure 1 belong to this regime. Finally, in the third regime, the limiting random function depends on infinite number of independent standard normal random variables, like the ecdf itself.

As an illustration of the usefulness of the results presented in this article, we present an analysis of brain imaging data obtained from a study of cortical thickness of adults who had a diagnosis of attention deficit/hyperactivity disorder (ADHD) as children (Proal et al., 2011). In this study, it had been noticed before that, when searching for brain locations whose cortical thickness is related to clinical diagnosis, the histogram of the z-scores did not follow the theoretical standard normal distribution (Reiss et al., 2012). Here we do a slightly different analysis (where the correlation structure is known) and show that, while the study exhibits apparently strongly significant results, they can be entirely explained by correlation, as captured by the asymptotic approximation mentioned above.

The rest of the paper is organized as follows. After a brief treatment of weak correlation in Section 2, the main results of the paper for general correlation are given in Section 3. Section 4 discusses how to consistently estimate the latent variables. In Section 4 we also briefly discuss the case where the correlation matrix is unknown. Several concrete examples, including those of Figure 1, are presented in detail in Section 5. A data example is analyzed in Section 6. Section 7 considers some possible extensions of this work. All the proofs are given in a supplementary material document.

2 Weak correlation

2.1 Definition

To define weak correlation some notation is needed. For a given p × p matrix R_p = (r_ij) define the following average norms: ${\Vert R_p\Vert }_1^{(p)}:=\frac{1}{p^2}{\sum }_{i,j}\mid r_{ij}\mid$ and ${\Vert R_p\Vert }_2^{(p)}:=\frac{1}{p}{\left({\sum }_{i,j}{r}_{ij}^2\right)}^{1/2}=\frac{1}{p}{({\sum }_i{\lambda }_i^2)}^{1/2}$, where the λ_i’s are the eigenvalues of R_p. The latter is simply a scaled version of the Frobenius norm, and in both cases we use the superscript (p) to denote that the norm itself, not just the matrix, changes with p. If ${\{R_p\}}_{p=1}^{\infty }$ is a sequence of correlation matrices then, as p → ∞,

$${\Vert R_p\Vert }_1^{(p)}\to 0\iff {\Vert R_p\Vert }_2^{(p)}\to 0.$$

This is because, by Jensen’s inequality, ${\Vert R_p\Vert }_1^{(p)}\le {\Vert R_p\Vert }_2^{(p)}$, while on the other hand, |r_ij| ≤ 1 and (r_ij)² ≤ |r_ij|, and so ${\Vert R_p\Vert }_1^{(p)}\ge {\left\{{\Vert R_p\Vert }_2^{(p)}\right\}}^2$. A similar argument holds if R_p is a covariance matrix and the diagonal entries of R_p, i.e., the variances, are bounded.

Definition 1

Let ${\{{\xi }_i\}}_{i=1}^{\infty }$ be a sequence of standard normal variables with joint normal distribution and denote the correlation matrix of (ξ₁, …, ξ_p) by R_p. If ${\Vert R_p\Vert }_1^{(p)}\to 0$, or equivalently, ${\Vert R_p\Vert }_2^{(p)}\to 0$, then ${\{{\xi }_i\}}_{i=1}^{\infty }$ is called weakly correlated. Otherwise it is called strongly correlated.

2.2 Convergence of the ecdf

Let ${\{Z_i\}}_{i=1}^{\infty }$ be a sequence of standard normal variables with joint normal distribution. The ecdf is

$${\widehat{F}}_p(z):=\frac{1}{p}\sum _{i=1}^{p}I(Z_i\le z),$$ (1)

where I(·) denotes the indicator function. Correlation does not affect the expectation E[F̂_p(z)] = Φ(z) of the ecdf, but it affects its covariance. The covariance function is given in Proposition 1 of Schwartzman (2010).

The following theorem establishes that a necessary and sufficient condition for consistency of the ecdf in is that ${\{Z_i\}}_{i=1}^{\infty }$ is weakly correlated.

Theorem 1

Let Z₁, Z₂, …, Z_p, … be N(0, 1) variables with ecdf (1) and let R_p denote the correlation matrix of (Z₁, …, Z_p).

1. (Sufficiency) If ${\{Z_i\}}_{i=1}^{\infty }$ is weakly correlated, then F̂_p(z) converges to Φ(z) in uniformly:

   $$\underset{z}{sup}\mathrm{E}{\left\{{\widehat{F}}_p(z)-\mathrm{\Phi }(z)\right\}}^2\le \frac{1}{4p}+C{\Vert R_p\Vert }_1^{(p)}\to 0,$$

   where C is a universal constant.

2. (Necessity) If ${\{Z_i\}}_{i=1}^{\infty }$ is not weakly correlated, i.e., ${\Vert R_p\Vert }_2^{(p)}\nrightarrow 0$, then F̂_p(z) does not converge to Φ(z) in for any z ≠ 0, i.e.:

   $$\mathrm{E}{\left\{{\widehat{F}}_p(z)-\mathrm{\Phi }(z)\right\}}^2\stackrel{p\to \infty }{\nrightarrow }0,\forall z\ne 0$$

Notice that according to Theorem 1 the convergence is either uniform or none at all; if the correlation is strong, i.e., not weak, then for any z ≠ 0 there is no convergence to Φ(z). Under a general correlation structure the ecdf may converge to a random function. In Section 3 below we aim at identifying and estimating this function.

2.3 Examples

It is easy to check that every Gaussian autoregressive moving average (ARMA) process is weakly correlated. So is every m-dependent Gaussian sequence with banded correlation matrix so that r_ij = 0 for i − j > m and fixed finite m. This includes correlation in fixed finite blocks. In all these cases the ecdf converges to the standard normal distribution.

More generally, all Gaussian stationary ergodic processes are weakly correlated. This is because ergodicity requires that the autocorrelation function ρ(ℓ) = r_i,i+ℓ satisfies |ρ(ℓ)| → 0 as ℓ → ∞, and therefore

$${\Vert R_p\Vert }_1^{(p)}=\frac{1}{p^2}\sum _{i,j}\mid r_{ij}\mid =\frac{1}{p^2}\left[p+2\sum _{\ell =1}^{p-1}(p-\ell )\mid \rho (\ell )\mid \right]\le \frac{1}{p}+\frac{2}{p}\sum _{\ell =1}^{p-1}\mid \rho (\ell )\mid \to 0.$$

It is not hard to check that even long-range correlation, defined by ${\sum }_{\ell =1}^{\infty }\mid \rho (\ell )\mid =\infty$, also implies weak correlation except in the extreme case where ${\sum }_{\ell =1}^{p-1}\mid \rho (\ell )\mid$ is of order p (the largest possible).

3 General correlation

3.1 An asymptotic approximation

To describe the asymptotic behavior of the ecdf for a general correlation structure, the main idea is to decompose the correlation into a strong correlation component and a weak correlation component. Then, the asymptotic behavior of the ecdf as a random function will be captured by the strong correlation component, while the weak component will converge as in Theorem 1.

Specifically, suppose that for every p we have the decomposition

$$R_p=A_p+B_p,$$ (2)

where A_p and B_p are symmetric positive semi-definite matrices. Write A_p as

$$A_p=L_p{(L_p)}^T,$$ (3)

where L_p is of dimension p × k(p); let ${\mathit{\ell }}_i^{(p)}$ be the i’th row of L_p. If the matrix A_p is the zero matrix then we define k(p) := 1, and L_p := (0, …, 0)^T.

For a matrix B, we define the matrix Cor(B) by

$${\{\text{Cor}(B)\}}_{ij}=\{\begin{array}{ll}\frac{B_{ij}}{\sqrt{B_{ii}{B}_{jj}}}\hfill & B_{ii}{B}_{jj}\ne 0\hfill \\ 0\hfill & B_{ii}{B}_{jj}=0\hfill \end{array}.$$

Notice that if B is a positive semidefinite matrix, then |{Cor(B)}_ij| ≤ 1. Obviously, if B is a correlation matrix, then Cor(B) = B. The following theorem states the main result.

Theorem 2

Under the previous setting and notation,

1. For every p there exists a (non-unique) random vector ${\mathbf{W}}_{k(p)}^{(p)}~N(0,I_{k(p)})$ such that

   $$\underset{z}{sup}E{\left\{{\widehat{F}}_p(z)-{\overline{F}}_p(z)\right\}}^2\le \frac{1}{4p}+C{\Vert \text{Cor}(B_p)\Vert }_1^{(p)},$$ (4)
   where

   $${\overline{F}}_p(z):=E[{\widehat{F}}_p(z)\mid {\mathbf{W}}_{k(p)}^{(p)}]=\frac{1}{p}\sum _{i=1}^p\mathrm{\Phi }\left(\frac{z-{\mu }_i^{(p)}}{{\sigma }_i^{(p)}}\right)$$ (5)

   with ${\mu }_i^{(p)}:={\mathit{\ell }}_i^{(p)}{\mathbf{W}}_{k(p)}^{(p)},{\sigma }_i^{(p)}:={\{{(B_p)}_{ii}\}}^{1/2}$, and C is a universal constant. If ${\sigma }_i^{(p)}=0$ we define $\mathrm{\Phi }\left(\frac{z-{\mu }_i^{(p)}}{{\sigma }_i^{(p)}}\right):=I({\mu }_i^{(p)}\le z)$.

2. Therefore, if ${\Vert \text{Cor}(B_p)\Vert }_1^{(p)}\stackrel{p\to \infty }{\to }0$ then ${sup}_{z}E{\left\{{\widehat{F}}_p(z)-{\overline{F}}_p(z)\right\}}^2\stackrel{p\to \infty }{\to }0$.

The key idea of the proof is the following: due to decomposition (2) and equality (3), there exists a random vector W_k ~ N (0, I_k) such that

$$Z_i={\mathit{\ell }}_i^{(p)}{\mathbf{W}}_{k(p)}^{(p)}+{\xi }_i,i=1,\dots ,p,$$ (6)

where (ξ₁, …, ξ_p) ~ N(0, B_p). Therefore, the conditional random vector

$$({\stackrel{\sim }{Z}}_1,\dots ,{\stackrel{\sim }{Z}}_p):=(Z_1,\dots ,Z_p)\mid {\mathbf{W}}_{k(p)}^{(p)}={\mathbf{w}}_k$$

is normal with conditional mean μ = L_pw_k and conditional covariance matrix B_p. The distance between F̂_p(z) and ${\overline{F}}_p(z):=E[{\widehat{F}}_p(z)\mid {\mathbf{W}}_{k(p)}^{(p)}]$ can be essentially bounded uniformly by ${\Vert \text{Cor}(B_p)\Vert }_1^{(p)}$.

It is important to note that the random vector ${\mathbf{W}}_{k(p)}^{(p)}$ is not unique. Because ${\mathbf{W}}_{k(p)}^{(p)}$ has a spherically symmetric distribution, Equation (4) will hold if ${\mathbf{W}}_{k(p)}^{(p)}$ is replaced by $Q{\mathbf{W}}_{k(p)}^{(p)}$ where Q is any p × p orthonormal matrix.

Definition 2

The sequence of random functions ${\{G_p(z)\}}_{p=1}^{\infty }$ is said to converge uniformly in to the (random) function G(z) if ${sup}_{z}E{\{G_p(z)-G(z)\}}^2\stackrel{p\to \infty }{\to }0$.

Corollary 1

Suppose F̂_p(z) satisfies the conditions of Theorem 2(ii). If there exists a (random) function F̄(z) such that F̄_p(z) converges to F̄(z) uniformly in , then F̂_p(z) also converges to F̄_p(z) uniformly in .

Theorem 2 holds for all decompositions of the form (2). We call every corresponding F̄_p(z) an asymptotic approximation of F̂_p(z). However, some decompositions may need a smaller number of latent variables k(p) than others. In the next section we characterize the best decomposition in the sense of giving the asymptotic approximation with the smallest number of latent variables. If F̄_p(z) converges to some F̄(z), then we call the latter the asymptotic representation of F̄_p(z).

3.2 Dimension reduction

Theorem 2 approximates F̂_p(z) by F̄_p(z), which is the projection of F̂_p(z) onto a space generated by ${\mathbf{W}}_{k(p)}^{(p)}$. Thus, as explained below, F̂_p(z), which has dimension p, is approximated by F̄_p(z) with dimension rank (A_p). Hence, Theorem 2 can be regarded as a dimension reduction of the empirical distribution function as stated in the following proposition.

Proposition 1

Let be the collection of all one-dimensional distribution functions, where convergence is defined in the sense of weak convergence of the corresponding random variables.

1. F̂_p(z) has dimension p: define the mapping (Z₁, ···, Z_p) = F̂_p(z), from ℝ^p to . Then (ℝ^p) ⊆ is homeomorphic to ℝ^p.

2. F̄_p(z) has dimension rank(A_p): define the mapping $\mathcal{B}({\mathbf{W}}_{k(p)}^{(p)})={\overline{F}}_p(z)$, from ℝ^k(p) to . Then (ℝ^k(p)) ⊆ is homeomorphic to ℝ^RANK(A_p).

In order to achieve dimension reduction, we are interested in knowing whether there exist decompositions of the form (2) such that the approximation of Theorem 2 holds and the dimension of the approximation rank(A_p) is finite. Consider the collection of decompositions of R_p that satisfy the conditions of Theorem 2 Part (ii): D ∈ if $D={\{(A_p,B_p)\}}_{p=1}^{\infty }$ such that A_p, B_p satisfy (2) and ${\Vert \text{Cor}(B_p)\Vert }_1^{(p)}\stackrel{p\to \infty }{\to }0$. Clearly, is a large collection and some notion of optimality is required in order to decide which D ∈ to choose. Given Proposition 1 (ii), we are interested in decompositions where rank(A_p) is the smallest. Theorem 3 below states a lower bound on the limiting rank of A_p and presents a decomposition that achieves it under certain conditions.

To state the result, we need to define the eigendecompositions of R_p as a special case of the decompositions in . Suppose that the eigenvalues of R_p are ${\lambda }_1^{(p)}\ge \cdots \ge {\lambda }_p^{(p)}$ and the corresponding eigenvectors are ${\mathit{\gamma }}_1^{(p)},\dots ,{\mathit{\gamma }}_p^{(p)}$ (notice that everything depends on p). The eigendecomposition of R_p is ${\sum }_{i=1}^p{\lambda }_i^{(p)}{\mathit{\gamma }}_i^{(p)}{\{{\mathit{\gamma }}_i^{(p)}\}}^T$. For k < p, define

$$A_{k,p}=\sum _{i=1}^k{\lambda }_i^{(p)}{\mathit{\gamma }}_i^{(p)}{\{{\mathit{\gamma }}_i^{(p)}\}}^T,B_{k,p}=\sum _{i=k+1}^p{\lambda }_i^{(p)}{\mathit{\gamma }}_i^{(p)}{\{{\mathit{\gamma }}_i^{(p)}\}}^T.$$ (7)

Then, the covariance matrix R_p can be expressed as R_p = A_k,p + B_k,p and A_k,p = L_k,p(L_k,p)^T where

$$L_{k,p}:=\left(\sqrt{{\lambda }_1^{(p)}}{\mathit{\gamma }}_1^{(p)},\cdots ,\sqrt{{\lambda }_k^{(p)}}{\mathit{\gamma }}_k^{(p)}\right).$$ (8)

A critical quantity is the number of “big” eigenvalues, i.e., with size of order p. Define

$$\underset{\_}{K}=\sum _{i=1}^{\infty }{\underset{\_}{K}}_i\text{with}{\underset{\_}{K}}_i=\{\begin{array}{lll}1\hfill & \text{if}\hfill & lim{inf}_p\frac{{\lambda }_i^{(p)}}{p}>0\hfill \\ 0\hfill & \text{if}\hfill & lim{inf}_p\frac{{\lambda }_i^{(p)}}{p}=0\hfill \end{array},i=1,2,\dots$$

and

$$\overline{K}=\sum _{i=1}^{\infty }{\overline{K}}_i\text{with}{\overline{K}}_i=\{\begin{array}{lll}1\hfill & \text{if}\hfill & lim{sup}_p\frac{{\lambda }_i^{(p)}}{p}>0\hfill \\ 0\hfill & \text{if}\hfill & lim{sup}_p\frac{{\lambda }_i^{(p)}}{p}=0\hfill \end{array},i=1,2,\dots$$

By definition we have that K ≤ K̄. Notice that K could be ∞ and that if K < ∞ then K_i = 1 for i ≤ K and K_i = 0 otherwise; the same for K̄. Since K and K̄ are sums of indicators, then they are either an integer or infinity. The following theorem states that K and K̄ give lower bounds for the limiting rank of A_p and presents a decomposition of the correlation matrix that achieves it under certain conditions.

Theorem 3

Under the previous setting and notation:

1. If ${\{(A_p,B_p)\}}_{p=1}^{\infty }\in \mathcal{D}$, then lim inf_p rank(A_p) ≥ K and lim sup_p rank(A_p) ≥ K̄ (both K and K̄ could be ∞).

2. For K̄ < ∞ define the decomposition $D_{\overline{K}}={\{(A_{\overline{K},p},B_{\overline{K},p})\}}_{p=1}^{\infty }$ according to (7) with k = K̄. If the nonzero diagonal terms of B_K̄,p are bounded from below, i.e.

   $$\underset{p}{liminf}\underset{1\le i\le p;{\{B_{\overline{K},p}\}}_{ii}\ne 0}{min}{\{B_{\overline{K},p}\}}_{ii}={\epsilon }_B>0$$ (9)

   then D_K̄ ∈ and obviously rank(A_K̄,p) = K̄ for all p > K̄.

The regularity condition (9) guarantees that the norm of the residual covariance in B_K̄,p goes to zero because the correlation in it goes to zero, not because the variance in it goes to zero.

To better appreciate the result given by Theorem 3, it is useful to define the following concepts.

Definition 3

1. If there exists ${\{(A_p,B_p)\}}_{p=1}^{\infty }\in \mathcal{D}$ such that lim sup_p rank(A_p) < ∞ we say that ${\{Z_i\}}_{i=1}^{\infty }$ has finite dimensional correlation.

2. If further K := K = K̄ < ∞ then we say that ${\{Z_i\}}_{i=1}^{\infty }$ has asymptotic dimension K.

With these definitions in mind, Theorem 3 implies that if ${\{Z_i\}}_{i=1}^{\infty }$ has finite dimensional correlation with asymptotic dimension K and the regularity condition (9) holds, then the decomposition ${\{(A_{K,p},B_{K,p})\}}_{p=1}^{\infty }$ is “optimal” in the sense that lim_p rank(A_K,p) = K, so that it achieves the lowest dimension among all asymptotic approximations F̄_p(z) of F̂_p(z). We shall see in Section 5 that the regularity condition (9) holds for most typical correlation structures.

It is now possible to see that the number K̂ determines three different convergence regimes. If K̂ = ∞ then ${\{Z_i\}}_{i=1}^{\infty }$ has no finite dimensional correlation. If K̄ < ∞ and (9) holds then ${\{Z_i\}}_{i=1}^{\infty }$ has finite dimensional correlation. When K̄ = 0 then (9) holds trivially since {B_0,p}_ii = {R_p}_ii = 1 and therefore a necessary and sufficient conditions for weak correlation, as in Theorem 1, is K̄ = 0. We summarize the results in the following corollary and in Figure 2.

Figure 2.

Different cases of asymptotic approximation to the ecdf. The corresponding examples from Section 5 below appear in parentheses.

Corollary 2

1. If K̄ = ∞ then ${\{Z_i\}}_{i=1}^{\infty }$ has no finite dimensional correlation.

2. If K̄ < ∞ and (9) holds then ${\{Z_i\}}_{i=1}^{\infty }$ has finite dimensional correlation.

3. If K := K = K̄ < ∞ and (9) holds then ${\{Z_i\}}_{i=1}^{\infty }$ has asymptotic dimension K and the decomposition ${\{(A_{K,p},B_{K,p})\}}_{p=1}^{\infty }$ achieves it in the sense that lim_p rank(A_K,p) = K.

4. K̄ = 0 if and only if ${\{Z_i\}}_{i=1}^{\infty }$ is weakly correlated.

The regularity condition (9) does not hold when the diagonal of B_K̄,p contains elements that are arbitrarily small. In this case, captured by the label “unknown” in Figure 2, ${\Vert B_{\overline{K},p}\Vert }_1^{(p)}$ may be small but not necessarily ${\Vert \text{Cor}(B_{\overline{K},p})\Vert }_1^{(p)}$. Since our bound (4) is based on ${\Vert \text{Cor}(B_{\overline{K},p})\Vert }_1^{(p)}$, rather than on ${\Vert B_{\overline{K},p}\Vert }_1^{(p)}$, we cannot establish the asymptotic dimension in this case. However, we could not find an example of R_p, where (9) is not satisfied. A hand-waving argument that (9) typically holds is:

$${\{B_{\overline{K},p}\}}_{ii}=\sum _{j=\overline{K}+1}^p{\lambda }_j^{(p)}{\{{\mathit{\gamma }}_j^{(p)}\}}_i^2\ge {\lambda }_{\overline{K}+1}^{(p)}\sum _{j=\overline{K}+1}^p{\{{\mathit{\gamma }}_j^{(p)}\}}_i^2\approx {\lambda }_{\overline{K}+1}^{(p)}>0,$$

where ≈ is typically true because ${\mathit{\gamma }}_j^{(p)}$ is a normalized vector and therefore ${\{{\mathit{\gamma }}_j^{(p)}\}}_i^2\approx 1/p$.

4 Estimating the asymptotic representation from the data

4.1 Estimating the latent variables

We now discuss how to estimate the underlying latent variables ${\mathbf{W}}_K^{(p)}$ when the sequence ${\{Z_i\}}_{i=1}^{\infty }$ has asymptotic dimension K := K = K̄. For ease of presentation we write everything in a matrix/vector form; thus, we write Z_p := (Z₁, …, Z_p)^T, ξ_p = (ξ₁, …, ξ_p)^T. In this notation, (6) becomes the linear regression equation ${\mathbf{Z}}_p=L_{K,p}{\mathbf{W}}_K^{(p)}+{\mathit{\xi }}_p$ and the least square estimate of ${\mathbf{W}}_K^{(p)}$ is ${\widehat{\mathbf{W}}}_K^{(p)}:={(L_{K,p}^T{L}_{K,p})}^{-1}{L}_{K,p}^T{\mathbf{Z}}_p$.

When the eigendecomposition (7) is used, then the columns of L_k(p) are orthogonal and ${(L_{K,p}^T{L}_{K,p})}^{-1}$ is a diagonal matrix whose i-th diagonal element is $1/{\lambda }_i^{(p)}$. Thus, in this case,

$${\widehat{\mathbf{W}}}_K^{(p)}=\left(\begin{array}{c}\frac{1}{\sqrt{{\lambda }_1^{(p)}}}{\left\{{\mathit{\gamma }}_1^{(p)}\right\}}^T{\mathbf{Z}}_p\\ ⋮\\ \frac{1}{\sqrt{{\lambda }_K^{(p)}}}{\left\{{\mathit{\gamma }}_k^{(p)}\right\}}^T{\mathbf{Z}}_p\end{array}\right).$$ (10)

Fan et al. (2012) and Fan and Han (2014) consider a related framework where some of the variables Z_i may have a non-zero mean, and therefore use an estimate that minimizes the distance under sparsity assumptions.

Since ${\{Z_i\}}_{i=1}^p$ has strong correlation, then $\text{Cov}({\widehat{\mathbf{W}}}_K^{(p)})$ does not converge to 0, as p goes to infinity. However, the following proposition states that ${\widehat{\mathbf{W}}}_K^{(p)}$ is still consistent in an sense.

Proposition 2

Suppose that ${\{Z_i\}}_{i=1}^{\infty }$ has asymptotic dimension K. Then

$$E\left[{\left\{{\widehat{\mathbf{W}}}_K^{(p)}-{\mathbf{W}}_K^{(p)}\right\}}^T\left\{{\widehat{\mathbf{W}}}_K^{(p)}-{\mathbf{W}}_K^{(p)}\right\}\right]\le K\frac{p}{{\lambda }_K^{(p)}}{\Vert B_{K,p}\Vert }_2^{(p)}\stackrel{p\to \infty }{\to }0.$$

4.2 Estimating the asymptotic representation

In real data problems ${\mathbf{W}}_K^{(p)}$ and, thus, F̄_p(·) are unknown. Proposition 2 suggests that if we plug-in ${\widehat{\mathbf{W}}}_K^{(p)}$ to F̄_p(·) instead of ${\mathbf{W}}_K^{(p)}$, then the distance between F̂_p(·) and the plugged-in F̄_p(·) converges to zero. Indeed, this can be proved under some additional regularity conditions.

Theorem 4

1. Define the plug-in ecdf estimate

   $${\widehat{\overline{F}}}_p(z):=\frac{1}{p}\sum _{i=1}^p\mathrm{\Phi }\left(\frac{z-{\widehat{\mu }}_i^{(p)}}{{\sigma }_i^{(p)}}\right)$$ (11)
   with ${\widehat{\mu }}_i^{(p)}:={\mathit{\ell }}_i^{(p)}{\widehat{\mathbf{W}}}_K^{(p)}$ under the convention $\mathrm{\Phi }\left(\frac{z-{\widehat{\mu }}_i^{(p)}}{0}\right):=I\left({\mu }_i^{(p)}\le z\right)$. We have that

   $$\begin{array}{l}\underset{z}{sup}E{\left\{{\widehat{F}}_p(z)-{\widehat{\overline{F}}}_p(z)\right\}}^2\le 3\underset{z}{sup}E{\left\{{\widehat{F}}_p(z)-{\overline{F}}_p(z)\right\}}^2+\frac{3(p-\mid J_p\mid )}{p}+\\ +\frac{3}{2\pi }\underset{i\in J_p}{max}\frac{1}{{\left\{{\sigma }_i^{(p)}\right\}}^2}E\left[{\left\{{\widehat{\mathbf{W}}}_K^{(p)}-{\mathbf{W}}_K^{(p)}\right\}}^T\left\{{\widehat{\mathbf{W}}}_K^{(p)}-{\mathbf{W}}_K^{(p)}\right\}\right],\end{array}$$

   where $J_p:=\{1\le i\le p:{\sigma }_i^{(p)}>0\}$ is the set of indexes for which ${\sigma }_i^{(p)}>0$ and |J_p| is the cardinality of J_p.

2. Therefore, if ${\{Z_i\}}_{i=1}^{\infty }$ has asymptotic dimension K, (9) holds and also

   $$\frac{\mid J_p\mid }{p}\stackrel{p\to \infty }{\to }1$$ (12)

   then ${sup}_{z}E{\left\{{\widehat{F}}_p(z)-{\widehat{\overline{F}}}_p(z)\right\}}^2\stackrel{p\to \infty }{\to }0$.

4.3 Approximated eigendecomposition

In this section we study the case in which the largest K eigenvalues and eigenvectors of the correlation matrix are not known exactly and an approximation is used. We show that if the distance between the approximated eigenvalues and eigenvectors and the true ones goes to zero as p goes to infinity, then the result of the previous section still holds.

Let ${\stackrel{\sim }{\lambda }}_1^{(p)}\ge \cdots \ge {\stackrel{\sim }{\lambda }}_K^{(p)}$ be an approximation to the K biggest eigenvalues, and let ${\stackrel{\sim }{\mathit{\gamma }}}_1^{(p)},\dots ,{\stackrel{\sim }{\mathit{\gamma }}}_K^{(p)}$ be norm 1 vectors that are an approximation to the corresponding eigenvectors. Define the p × K matrix ${\stackrel{\sim }{L}}_p:=\left(\sqrt{{\stackrel{\sim }{\lambda }}_1^{(p)}}{\stackrel{\sim }{\mathit{\gamma }}}_1^{(p)},\cdots ,\sqrt{{\stackrel{\sim }{\lambda }}_k^{(p)}}{\stackrel{\sim }{\mathit{\gamma }}}_k^{(p)}\right)$, and let ${\stackrel{\sim }{\mathit{\ell }}}_i^{(p)}$ be the i-th row of the matrix. Define,

$${\stackrel{\sim }{\mathbf{W}}}_K^{(p)}:=\left(\begin{array}{c}\frac{1}{\sqrt{{\stackrel{\sim }{\lambda }}_1^{(p)}}}{\left\{{\stackrel{\sim }{\mathit{\gamma }}}_1^{(p)}\right\}}^T{\mathbf{Z}}_p\\ ⋮\\ \frac{1}{\sqrt{{\stackrel{\sim }{\lambda }}_K^{(p)}}}{\left\{{\stackrel{\sim }{\mathit{\gamma }}}_K^{(p)}\right\}}^T{\mathbf{Z}}_p\end{array}\right);$$

and let ${\stackrel{\sim }{\mu }}_i^{(p)}:={\stackrel{\sim }{\mathit{\ell }}}_i^{(p)}{\stackrel{\sim }{\mathbf{W}}}_K^{(p)}$ and ${\stackrel{\sim }{\sigma }}_i^{(p)}:={\left\{1-{\sum }_{j=1}^K{\lambda }_j^{(p)}{\left\{{\mathit{\gamma }}_j^{(p)}\right\}}_i^2\right\}}^{1/2}$, where {x}_i is the i-th element of the vector x. Finally, define ${\stackrel{\sim }{F}}_p(z):=\frac{1}{p}{\sum }_{i=1}^p\mathrm{\Phi }\left(\frac{z-{\stackrel{\sim }{\mu }}_i^{(p)}}{{\stackrel{\sim }{\sigma }}_i^{(p)}}\right)$. The following proposition bounds the distance between F̃_p(z) and ${\widehat{\overline{F}}}_p(z)$.

Proposition 3

Suppose that ${\sigma }_i^{(p)},{\stackrel{\sim }{\sigma }}_i^{(p)}\ge {\epsilon }_B$, i = 1, …, p, for some ε_B > 0.

1. The following inequality holds:

   $$\begin{array}{l}E{\left\{{\stackrel{\sim }{F}}_p(z)-{\widehat{\overline{F}}}_p(z)\right\}}^2\\ \le \frac{C({\epsilon }_B)}{{\lambda }_K^{(p)}/p}\sum _{i=1}^K\left[{\left\{\frac{1}{\sqrt{{\stackrel{\sim }{\lambda }}_i^{(p)}/p}}-\frac{1}{\sqrt{{\lambda }_i^{(p)}/p}}\right\}}^2+{\left\{\sqrt{\frac{{\stackrel{\sim }{\lambda }}_i^{(p)}}{p}}-\sqrt{\frac{{\lambda }_i^{(p)}}{p}}\right\}}^2+2{\left\{{\stackrel{\sim }{\mathit{\gamma }}}_i^{(p)}-{\mathit{\gamma }}_i^{(p)}\right\}}^T\left\{{\stackrel{\sim }{\mathit{\gamma }}}_i^{(p)}-{\mathit{\gamma }}_i^{(p)}\right\}\right]\\ +\frac{C({\epsilon }_B)}{\sqrt{p}}{\left[\sum _{i=1}^K{\left\{\sqrt{\frac{{\stackrel{\sim }{\lambda }}_i^{(p)}}{p}}=\sqrt{\frac{{\lambda }_i^{(p)}}{p}}\right\}}^2+\sum _{i=1}^K{\left\{{\stackrel{\sim }{\mathit{\gamma }}}_i^{(p)}-{\mathit{\gamma }}_i^{(p)}\right\}}^T\left\{{\stackrel{\sim }{\mathit{\gamma }}}_i^{(p)}-{\mathit{\gamma }}_i^{(p)}\right\}\right]}^{1/2},\end{array}$$

   where C(ε_B) is a constant that depends on ε_B.

2. Therefore, if $\frac{{\stackrel{\sim }{\lambda }}_i^{(p)}-{\lambda }_i^{(p)}}{p}\stackrel{p\to \infty }{\to }0,{\left\{{\stackrel{\sim }{\mathit{\gamma }}}_i^{(p)}-{\mathit{\gamma }}_i^{(p)}\right\}}^T\left\{{\stackrel{\sim }{\mathit{\gamma }}}_i^{(p)}-{\stackrel{\sim }{\mathit{\gamma }}}_i^{(p)}\right\}\stackrel{p\to \infty }{\to }0$, i = 1, …, K, and the conditions of Theorem 4 (ii) hold, then ${sup}_{z}E{\left\{{\stackrel{\sim }{F}}_p(z)-{\widehat{F}}_p(z)\right\}}^2\stackrel{p\to \infty }{\to }0$.

Proposition 3 and its proof imply that if there exist consistent estimates for the first K eigenvalues and eigenvectors, then the result of Theorem 4 (ii) still holds; see also Fan and Han (2014). A systematic study of the case that the correlation matrix is unknown is left to future research.

5 Examples

We now consider some examples of correlation structures. In Examples 1–3, K = K̄ is finite, conditions (9,12) are met, and therefore the decomposition ${\{(A_{\underset{\_}{K},p},B_{\underset{\_}{K},p})\}}_{p=1}^{\infty }$ is optimal and ${\widehat{\overline{F}}}_p$ is consistent, but in Example 3, F̄_p(z) does not have a limit. In Example 4, K = 1 but K̄ > 1 and in Example 5, K = ∞.

5.1 Exchangeable correlation: asymptotic dimension 1

Suppose that ${\{Z_i\}}_{i=1}^{\infty }$ is a sequence of exchangeable random variables with correlation ρ ≥ 0, i.e., for any i ≠ j, cor(Z_i, Z_j) = ρ and

$$R_p=\left(\begin{array}{cccc}1& \rho & \dots & \rho \\ \rho & 1& \dots & \rho \\ ⋮& ⋮& \ddots & ⋮\\ \rho & \dots & \rho & 1\end{array}\right).$$ (13)

The eigenvalues of R_p are ${\lambda }_1^{(p)}=\rho p+1-\rho$ (with corresponding eigenvector equal to ${(1,\dots ,1)}^T/\sqrt{p}$) and ${\lambda }_2^{(p)}=\cdots ={\lambda }_p^{(p)}=1-\rho$, hence, K = K̄ = 1. According to (7), we have that A_1,p = [ρ + (1 − ρ)/p] and B_1,p = (1 − ρ) [I_p − (1/p) ], where I_p is the p × p identity matrix and is a p × p matrix with 1 in every entry. It is easy to check that for every i, lim_p{B_1,p}_ii = 1 − ρ > 0, and (9) holds. Also, |J_p| = p and (12) holds. Thus, in this case ${\{Z_i\}}_{i=1}^{\infty }$ has finite dimensional correlation with asymptotic dimension K = 1 and the decomposition ${\{(A_{1,p},B_{1,p})\}}_{p=1}^{\infty }$ is optimal.

To write the asymptotic representation F̄(z) in this case, it is easier to work with the asymptotically equivalent decomposition A_p = ρ , B_p = (1 − ρ)I_p, which is in D and rank(A_p) = K = 1. For this decomposition ${\mu }_i^{(p)}=\sqrt{\rho }{W}_1$ and ${\sigma }_i^{(p)}=\sqrt{1-\rho }$ do not depend on i or p and

$${\overline{F}}_p(z)=\frac{1}{p}\sum _{i=1}^p\mathrm{\Phi }\left(\frac{z-{\mu }_i^{(p)}}{{\sigma }_i^{(p)}}\right)=\frac{1}{p}\sum _{i=1}^p\mathrm{\Phi }\left(\frac{z-\sqrt{\rho }{W}_1}{\sqrt{1-\rho }}\right)=\mathrm{\Phi }\left(\frac{z-\sqrt{\rho }{W}_1}{\sqrt{1-\rho }}\right)=\overline{F}(z).$$ (14)

That is, F̂_p(z), which has dimension p, is approximated by F̄_p(z) with dimension 1. Moreover, F̄_p(z) = F̄(z) above does not depend on p and is therefore the asymptotic representation: F̂_p(z) converges to F̄(z) in the sense of Definition 2.

Given the sequence ${\{Z_i\}}_{i=1}^p$, the regression estimate (10) of W₁ is ${\widehat{W}}_1^{(p)}={\overline{Z}}_p/\sqrt{\rho }$, where ${\overline{Z}}_p:=\frac{1}{p}{\sum }_{i=1}^p{Z}_i$. An illustration of the estimated asymptotic representation ${\widehat{\overline{F}}}_p(z)$ in (11) is shown in Figure 1(a).

Notice that the distribution specified by (14) is a shifted and scaled normal distribution, corresponding to Efron’s empirical null model (Efron, 2004, 2007a,b, 2008). Fitting an empirical null in this case would amount to estimating W₁ and ρ. This example shows that Efron’s empirical null model is justified under exchangeable correlation. However, as it can be seen from the above theory and the following examples, it does not capture the effect of many other correlation structures.

When ρ ≤ 0 in (13), positive semi-definiteness of the correlation matrix requires ρ ≥ −1/(p − 1). Therefore, in the high-dimensional setting, too much negative correlation is not possible. Consequently, the histograms of the Z’s are typically narrower than N(0, 1) (see Figure 1).

5.2 Two exchangeable correlation blocks: asymptotic dimension 2

Suppose that R_p consists of two blocks of the form (13), the first one of size n₁(p) × n₁(p) and the second of size n₂(p) × n₂(p) with n₁(p) + n₂(p) = p and lim_p→∞ n₁(p)/p = π. We assume that there is a constant correlation between the two blocks ρ_B (which can be zero or negative). Thus,

$$R_p=\left(\begin{array}{cc}\underset{n_1(p)}{\underbrace{\begin{array}{ccc}1& {\rho }_1& {\rho }_1\\ {\rho }_1& \ddots & {\rho }_1\\ {\rho }_1& {\rho }_1& 1\end{array}}}& {\rho }_B\\ {\rho }_B& \underset{n_2(p)}{\underbrace{\begin{array}{ccc}1& {\rho }_2& {\rho }_2\\ {\rho }_2& \ddots & {\rho }_2\\ {\rho }_2& {\rho }_2& 1\end{array}}}\end{array}\right).$$

Let I₁(p) and I₂(p) denote the set of indexes that belong to Blocks 1 and 2. If $0\le {\rho }_B<\sqrt{{\rho }_1{\rho }_2}$, then a sequence ${\{Z_i\}}_{i=1}^{\infty }$ with the above covariance can be generated by the model

$$Z_i=\{\begin{array}{cc}\sqrt{{\rho }_1}(\sqrt{{\stackrel{\sim }{\rho }}_B}{W}_1+\sqrt{1-{\stackrel{\sim }{\rho }}_B}{W}_2)+\sqrt{1-{\rho }_1}{\epsilon }_i& i\in I_1(p)\\ \sqrt{{\rho }_2}(\sqrt{{\stackrel{\sim }{\rho }}_B}{W}_1+\sqrt{1-{\stackrel{\sim }{\rho }}_B}{W}_3)+\sqrt{1-{\rho }_2}{\epsilon }_i& i\in I_2(p)\end{array},$$

where ${\stackrel{\sim }{\rho }}_B={\rho }_B/\sqrt{{\rho }_1{\rho }_2}$, and W₁, W₂, W₃, ${\{{\epsilon }_i\}}_{i=1}^p$ are i.i.d N (0, 1). Comparing with (6), we can write

$$\begin{array}{l}{\overline{F}}_p(z)=\frac{n_1(p)}{p}\mathrm{\Phi }\left(\frac{z-\sqrt{{\rho }_1}(\sqrt{{\stackrel{\sim }{\rho }}_B}{W}_1+\sqrt{1-{\stackrel{\sim }{\rho }}_B}{W}_2)}{\sqrt{1-{\rho }_1}}\right)+\frac{n_2(p)}{p}\mathrm{\Phi }\left(\frac{z-\sqrt{{\rho }_2}(\sqrt{{\stackrel{\sim }{\rho }}_B}{W}_1+\sqrt{1-{\stackrel{\sim }{\rho }}_B}{W}_3)}{\sqrt{1-{\rho }_2}}\right)\\ \to \overline{F}(z)=\pi \mathrm{\Phi }\left(\frac{z-\sqrt{{\rho }_1}(\sqrt{{\stackrel{\sim }{\rho }}_B}{W}_1+\sqrt{1-{\stackrel{\sim }{\rho }}_B}{W}_2)}{\sqrt{1-{\rho }_1}}\right)+(1-\pi )\mathrm{\Phi }\left(\frac{z-\sqrt{{\rho }_2}(\sqrt{{\stackrel{\sim }{\rho }}_B}{W}_1+\sqrt{1-{\stackrel{\sim }{\rho }}_B}{W}_3)}{\sqrt{1-{\rho }_2}}\right).\end{array}$$

If ρ_B = 0, then the asymptotic approximation F̄(z) depends only on W₂, W₃ and has dimension K = K̄ = 2. In this case R_p has two uncorrelated blocks of the type presented in Section 5.1; the generalization of the formulas there is straightforward (see also Figure 1(b)).

For ρ_B ≠ 0, however, the above F̄(z) depends on W₁, W₂, W₃ and, thus, has dimension three; it is suboptimal. To obtain the optimal asymptotic representation in more generality, we work with the eigendecomposition. The matrix R_p has two eigenvalues

$${\lambda }_{1,2}^{(\rho )}:=\frac{{\eta }_1^{(p)}+{\eta }_2^{(p)}}{2}\pm {\left[{\left(\frac{{\eta }_1^{(p)}-{\eta }_2^{(p)}}{2}\right)}^2+n_1(p)n_2(p){\rho }_B^2\right]}^{1/2}$$

where ${\eta }_j^{(p)}=1+{\rho }_j\{n_j(p)-1\}$, j = 1, 2, and the other p − 2 eigenvalues are either 1 − ρ₁ or 1 − ρ₂. For large p, positive semi-definiteness requires that ${\rho }_B^2\le {\rho }_1{\rho }_2$. On the boundary, when ${\rho }_B^2={\rho }_1{\rho }_2,{\lambda }_2^{(p)}$ converges to a positive constant and K = K̄ = 1.

We now concentrate on the case ${\rho }_B^2<{\rho }_1{\rho }_2$. Then, ${\lambda }_{1,2}^{(p)}$ are of order p and K = K̄ = 2. The corresponding eigenvectors are

$${(\underset{n_1(p)\text{times}}{\underbrace{x_j^{(p)},x_j^{(p)},\dots ,x_j^{(p)}}},\underset{n_2(p)\text{times}}{\underbrace{y_j^{(p)},y_j^{(p)},\dots ,y_j^{(p)}}})}^T,j=1,2,$$

where $x_j^{(p)}$ and $y_j^{(p)}$ are given by

$$\begin{array}{l}{x}_1^{(p)}=\frac{-\sqrt{n_2(p)}{\rho }_B}{\sqrt{{({\eta }_1^{(p)}-{\lambda }_1^{(p)})}^2+n_1(p)n_2(p){\rho }_B^2}},y_1^{(p)}=\frac{({\eta }_1^{(p)}-{\lambda }_1^{(p)})/\sqrt{n_2(p)}}{\sqrt{{({\eta }_1^{(p)}-{\lambda }_1^{(p)})}^2+n_1(p)n_2(p){\rho }_B^2}},\\ x_2^{(p)}=\frac{({\eta }_2^{(p)}-{\lambda }_2^{(p)})/\sqrt{n_1(p)}}{\sqrt{{({\eta }_2^{(p)}-{\lambda }_2^{(p)})}^2+n_1(p)n_2(p){\rho }_B^2}},y_2^{(p)}=\frac{-\sqrt{n_1(p)}{\rho }_B}{\sqrt{{({\eta }_2^{(p)}-{\lambda }_2^{(p)})}^2+n_1(p)n_2(p){\rho }_B^2}}.\end{array}$$

We have that

$${\mu }_i^{(p)}=\{\begin{array}{cc}\sqrt{{\lambda }_1^{(p)}}{x}_1^{(p)}{W}_1+\sqrt{{\lambda }_2^{(p)}}{x}_2^{(p)}{W}_2,& i\in I_1(p)\\ \sqrt{{\lambda }_1^{(p)}}{y}_1^{(p)}{W}_1+\sqrt{{\lambda }_2^{(p)}}{y}_2^{(p)}{W}_2,& i\in I_2(p),\end{array}$$

and

$${\sigma }_i^{(p)}=\{\begin{array}{cc}\sqrt{1-[{\lambda }_1^{(p)}{\{x_1^{(p)}\}}^2+{\lambda }_2^{(p)}{\{x_2^{(p)}\}}^2]},& i\in I_1(p)\\ \sqrt{1-[{\lambda }_1^{(p)}{\{y_1^{(p)}\}}^2+{\lambda }_2^{(p)}{\{y_2^{(p)}\}}^2]},& i\in I_2(p).\end{array}$$

Moreover, ${\{B_{2,p}\}}_{ii}\stackrel{p\to \infty }{\to }1-{\rho }_1$ for i ∈ I₁(p) and ${\{B_{2,p}\}}_{ii}\stackrel{p\to \infty }{\to }1-{\rho }_2$ for i ∈ I₂(p). Thus, (9) holds with K̄ = K = 2. Also, |J_p| = p and (12) holds. The corresponding asymptotic representation can be obtained by writing F̄_p(z) in terms of the ${\mu }_i^{(p)}$ and ${\sigma }_i^{(p)}$ above and taking the limit as p → ∞.

The regression estimate of $W_j^{(p)}$ is

$${\widehat{W}}_j^{(p)}=\left\{x_j(p)\sum _{i\in I_1(p)}{Z}_i+y_j(p)\sum _{i\in I_2(p)}{Z}_i\right\}/\sqrt{{\lambda }_j^{(p)}}j=1,2.$$

5.3 Finite dimensional correlation with no asymptotic representation

In this example, correlation is finite with K = 2 and the distance between F̂_p and F̄_p goes to zero, but F̄_p itself has no limit. Let 0 < ρ < 1 and $0<{\{{\alpha }_i\}}_{i=1}^{\infty }<1$ be any sequence, W₁, W₂, ${\{{\epsilon }_i\}}_{i=1}^{\infty }$ independent standard random variables. Define

$$Z_i=\sqrt{\rho {\alpha }_i}{W}_1+\sqrt{\rho (1-{\alpha }_i)}{W}_2+\sqrt{1-\rho }{\epsilon }_i;$$

then, Z_i ~ N(0, 1) and $\text{Cov}(Z_i,Z_j)=\rho \{\sqrt{{\alpha }_i{\alpha }_j}+\sqrt{(1-{\alpha }_i)(1-{\alpha }_j)}\}$ for i ≠ j. We can write R_p = A_p + B_p, where A_p = L_p(L_p)^T,

$$L_p=\left(\begin{array}{cc}\sqrt{\rho {\alpha }_1}& \sqrt{\rho (1-{\alpha }_1)}\\ \sqrt{\rho {\alpha }_2}& \sqrt{\rho (1-{\alpha }_2)}\\ ⋮& ⋮\\ \sqrt{\rho {\alpha }_p}& \sqrt{\rho (1-{\alpha }_p)}\end{array}\right),\text{and}{B}_p=\left(\begin{array}{cccc}1-\rho & 0& \dots & 0\\ 0& 1-\rho & \dots & 0\\ ⋮& \ddots & \ddots & ⋮\\ 0& \dots & 0& 1-\rho \end{array}\right).$$

Then, ${\overline{F}}_p(z)=\mathrm{\Phi }\left(\frac{z-{\mu }_i}{\sqrt{1-\rho }}\right)$, where ${\mu }_i:=\sqrt{\rho {\alpha }_i}{W}_1+\sqrt{\rho (1-{\alpha }_i)}{W}_2$ and it is clear that for certain choices of $0<{\{{\alpha }_i\}}_{i=1}^{\infty }<1$, F̄_p(z) has no limit for any z.

5.4 Independent exchangeable correlation blocks with no asymptotic dimension

In this example there are two independent blocks similar to the example in Section 5.2 (with ρ_B = 0) and with the same notation, but here $\frac{n_1(p)}{p},\frac{n_2(p)}{p}$ change infinitely often. Consequently, there are infinitely many p’s with two “big” eigenvalues and infinitely many p’s with only one “big” eigenvalue. Therefore, in this example, K = 1 while K̄ = 2. This means that correlation is finite dimensional and (9) holds, but there is no asymptotic representation and no well defined asymptotic dimension.

Consider the subsequence m₁ = 2, $m_{k+1}=m_k^2$ for k ≥ 2; i.e., m_k = 2^(2_k−1). Suppose that Z_i belongs to the first block if i ∈ {m_k−1 + 1, …, m_k} for even k, otherwise it belongs to the second block. That is, at p = m_k for even k,

$$n_1(p)=n_1(m_{k-1})+m_k-m_{k-1}=n_1(\sqrt{p})+p-\sqrt{p},\text{and}{n}_2(p)=n_2(m_{k-1})$$

and for odd k

$$n_1(p)=n_1(m_{k-1}),\text{and}{n}_2(p)=n_2(m_{k-1})+m_k-m_{k-1}=n_2(\sqrt{p})+p-\sqrt{p}.$$

When p = m_k for even k, $m_k-\sqrt{m_k}\le n_1(m_k)\le m_k$, and therefore the eigenvalue associated with this block satisfies ${lim}_{k\text{even}}\frac{\rho n_1(m_k)+1-\rho }{m_k}=\rho$ but for odd k, $n_1(m_k)\le m_{k-1}=\sqrt{m_k}$ and ${lim}_{k\text{odd}}\frac{\rho n_1(m_k)+1-\rho }{m_k}=0$; and vice versa for the other block. The largest eigenvalue satisfies

$$\frac{{\lambda }_1^{(p)}}{p}=max\{\frac{\rho n_1(p)+1-\rho }{p},\frac{\rho n_2(p)+1-\rho }{p}\}\ge \frac{\rho p/2+1-\rho }{p}$$

and therefore $lim{inf}_p\frac{{\lambda }_1^{(p)}}{p}\ge \rho /2>0$ and K₁ = 1. For the second eigenvalue

$$\frac{{\lambda }_2^{(p)}}{p}=min\left\{\frac{\rho n_1(p)+1-\rho }{p},\frac{\rho n_2(p)+1-\rho }{p}\right\};$$

since lim inf_p of both terms is 0, then $lim{inf}_p\frac{{\lambda }_2^{(p)}}{p}=0$ and K₂ = 0. Thus, K = 1.

We next show that K̄ = 2. When p = m_k one block is larger than the other block and when p = m_k+1 the other block is larger. Therefore, for each k there exists p_k between m_k and m_k+1 such that the two blocks are equal, i.e., n₁(p_k) = n₂(p_k) = p_k/2. Thus,

$$\frac{{\lambda }_2^{(p_k)}}{p_k}=min\left\{\frac{\rho n_1(p_k)+1-\rho }{p_k},\frac{\rho n_2(p_k)+1-\rho }{p_k}\right\}=\rho /2+\frac{1-\rho }{p_k};$$

Therefore, $lim{sup}_p\frac{{\lambda }_2^{(p)}}{p}\ge {lim}_k\frac{{\lambda }_2^{(p_k)}}{p_k}=\rho /2>0$. Since the other eigenvalues are constant (1 − ρ) we have that K̄ = 2.

Moreover, in this case ${\Vert B_{1,p}\Vert }_2^{(p)}$ does not converge to 0 and therefore ${\Vert \text{Cor}(B_{1,p})\Vert }_2^{(p)}\nrightarrow 0$; hence, the decomposition {(A_1,p, B_1,p)} is not in . This is because

$${\Vert B_{1,p_k}\Vert }_2^{(p_k)}=\frac{{\left[{\{{\lambda }_2^{(p_k)}\}}^2+{\{{\lambda }_3^{(p_k)}\}}^2+\dots +{\{{\lambda }_{p_k}^{(p_k)}\}}^2\right]}^{1/2}}{p_k}\ge \frac{{\lambda }_2^{(p_k)}}{p_k}=\rho /2+\frac{1-\rho }{p_k},$$

so that $lim{inf}_k{\Vert \text{Cor}{B}_{1,p_k}\Vert }_2^{(p_k)}\ge lim{inf}_k{\Vert B_{1,p_k}\Vert }_2^{(p_k)}\ge \rho /2$.

This example can be easily generalized such that K̄ = 3, 4,.. or K̄ = ∞. Given M < ∞, to obtain K̄ = M, for even k one can divide the observations Z_mk, …, Z_mk+1 into M − 1 independent blocks with correlation ρ within blocks. To obtain K̄ = ∞, for even k one can divide the observations Z_mk, …, Z_mk+1 into k blocks.

5.5 No finite dimensional approximation

In this example, K = K̄ = ∞, and therefore, according to Theorem 3, for every decomposition in , lim_p rank(A_p) = ∞. Suppose that at p = 2ⁿ, R_p consists of n independent blocks of sizes of 2ⁿ⁻¹, 2ⁿ⁻², …, 2 and an additional block of size 2, each block of the form (13), all with the same correlation parameter ρ. At p = 2ⁿ, there are n − 1 “big” eigenvalues, ρ(p/2ⁱ − 1) + 1 for i = 1, …, n − 1, one eigenvalue 1 + ρ (from the last block of size 2) and the rest are equal to 1 − ρ. For any fixed i ∈ , for large enough the eigenvalue ${\lambda }_i^{(p)}$ is equal to ρ(p/2ⁱ − 1) + 1 and therefore for large enough ,

$$\frac{{\lambda }_i^{(p)}}{p}=\frac{\rho (p/2^i-1)+1}{p}\stackrel{p\to \infty }{\to }\rho /2^i>0.$$

Therefore, for any fixed i ∈ , K_i = 1 and $\underset{\_}{K}={\sum }_{i=1}^{\infty }{\underset{\_}{K}}_i=\infty$.

6 Data example

As a practical application, we use the methods developed above to analyze a high-dimensional data set obtained from brain imaging. The data belongs to a study of cortical thickness of adults who had a diagnosis of attention deficit/hyperactivity disorder (ADHD) as children (Proal et al., 2011). The data set consists of cortical thickness measurements for about 80000 cortical voxels, obtained from magnetic resonance imaging (MRI) scans, as well as demographic and behavioral measurements, for each of n = 139 individuals. In this study, it had been noticed by Reiss et al. (2012) that z-scores corresponding to the voxelwise relationship between cortical thickness and ADHD diagnosis did not follow the theoretical standard normal distribution. Instead, the distribution of z-scores exhibited a substantial shift away from zero, indicating a possible widespread cortical thinning over the brain for individuals with ADHD. It is unclear, however, whether those results could have been caused by correlation between voxels rather than by a real relationship with clinical diagnosis.

In order to apply the methods developed in this paper, here we perform a slightly different analysis where the correlation structure can be taken as known. Specifically, we follow the approach of Owen (2005) of performing a regression of the observed trait for the subjects on the high-dimensional predictors, one dimension at a time. Owen (2005) and also Fan et al. (2012) used this approach in the context of genomic data. In our case, the trait is a global assessment of behavior, while the predictors are the cortical thickness measurements. For ease of computation, in the following analysis we use a random sample of p = 1000 voxels, which is enough to show the effect we want to highlight.

6.1 Regression analysis

Let Y_j denote the global assessment of the j-th subject and let ${\mathbf{X}}_j={\{X_j^{(i)}\}}_{i=1}^p$ be the cortical thickness in the p voxels of the j-th subject. For each voxel i ∈ {1, …, p}, consider the simple regression model

$$Y_j={\alpha }^{(i)}+{\beta }^{(i)}{X}_j^{(i)}+{\epsilon }_{j}j=1,\dots ,n,$$

where the ε’s are i.i.d with mean 0 and variance σ². Let ${\stackrel{\sim }{Y}}_j=Y_j-\frac{1}{n}{\sum }_{j=1}^n{Y}_j$ and ${\stackrel{\sim }{X}}_j^{(i)}=X_j^{(i)}-\frac{1}{n}{\sum }_{j=1}^n{X}_j^{(i)}$ be the centered variables and define $s_{i\ell }={\sum }_{j=1}^n{\stackrel{\sim }{X}}_j^{(i)}{\stackrel{\sim }{X}}_j^{(\ell )}$. The least squares estimate of β⁽ⁱ⁾ is ${\widehat{\beta }}^{(i)}={\sum }_{j=1}^n{\stackrel{\sim }{X}}_j^{(i)}{\stackrel{\sim }{Y}}_j/s_{ii}$. According to the model, we can write ${\widehat{\beta }}^{(i)}={\beta }^{(i)}+{\sum }_{j=1}^n{\stackrel{\sim }{X}}_j^{(i)}{\epsilon }_j/s_{ii}$ and so Cov(β̂⁽ⁱ⁾, β̂^(ℓ)) = σ²s_iℓ/[s_iis_ℓℓ].

Consider the p hypotheses $H_0^{(i)}:{\beta }^{(i)}=0$ versus $H_1^{(i)}:{\beta }^{(i)}\ne 0$, i = 1, …, p. The z-score for the i-th test is $Z_i={\widehat{\beta }}^{(i)}/\text{se}[{\widehat{\beta }}^{(i)}]={\widehat{\beta }}^{(i)}\sqrt{s_{ii}}/\sigma$. Thus, under the global null hypothesis, i.e. that all $H_0^{(i)}$’s are true, we have that Z₁, …, Z_p are each approximately N (0, 1) with correlation matrix $R:={\{r_{i\ell }\}}_{i,\ell ,=1}^p$ given by the pairwise correlations $r_{i\ell }:=\text{Cor}(Z_i,Z_{\ell })=s_{i\ell }/\sqrt{s_{ii}{s}_{\ell \ell }}$. Notice that the pairwise correlations between the z-scores are precisely the pairwise correlations between the cortical thickness measurements at each voxel. Because the regression is conditional on the voxelwise measurements, we take the pairwise correlations as fixed. To compute the z-scores, we use the variance estimate ${\widehat{\sigma }}^2=\frac{1}{p(n-2)}{\sum }_{i=1}^p{\sum }_{j=1}^n{[{\stackrel{\sim }{Y}}_j-{\widehat{\beta }}^{(i)}{\stackrel{\sim }{X}}_j^{(i)}]}^2$ and ignore its negligible contribution to their variability.

6.2 The distribution of the z-scores

Consider the decomposition R_p = A_k,p + B_k,p, where A_k,p, B_k,p are of the form (7). The asymptotic dimension, if it exists, is unknown. In the next subsection we discuss the choice of k, for now we work with k = 2. After k is set, ${\mathbf{W}}_k^{(p)}$ is estimated via (10). Define

$${\widehat{\mu }}_i:={\mathit{\ell }}_i{\widehat{\mathbf{W}}}_k^{(p)},\text{and}{\sigma }_i:={\{{(B_{k,p})}_{ii}\}}^{1/2}i=1,\dots ,p,$$

where ℓ_i is the i-th row of L_k,p defined in (8). The empirical distribution is approximated by $\widehat{\overline{F}}(z)$ given by (11), and the density is approximated by $\overline{f}(z):=\frac{1}{p}{\sum }_{i=1}^p\frac{1}{{\sigma }_i}\phi \left(\frac{z-{\widehat{\mu }}_i}{{\sigma }_i}\right)$, where φ is the standard normal density. Figure 3(a) plots the histogram of the Z’s, f̄ and φ. The approximation f̄ beautifully captures the shape of the empirical distribution, even though it is based on only two latent variables.

Figure 3.

Histogram of Z₁, Z₂, …, Z₁₀₀₀ of the real data (a) and the simulation results (b). The red line illustrates φ, the standard normal density, and the blue dashed line is the approximation we use, f̄.

Recall that the approximation is done under the complete null hypothesis, i.e. that there is no effect. The validity of the complete null hypothesis is confirmed by a false discovery rate (FDR) analysis which shows no significant voxels after applying the Benjamini-Hochberg procedure (Benjamini and Hochberg, 1995) at an FDR level of 0.2. Nevertheless, the z-scores exhibit a strong shift toward positive values, indicating a possible positive correlation between cortical thickness and behavior. Our approximation indicates that the reason that many Z’s are large may be because of the correlation between them and not because of a true effect. To illustrate this point we simulated Y₁, …, Y_n ~ N (0, 1) i.i.d and independent of the X’s and repeated the same procedure as before. One such simulated instance is given in Figure 3(b). Without correlation, we would expect the histogram of the Y’s to follow the theoretical null density (red). However, the correlation is causing the impression of a strong positive effect not unlike the one seen in panel (a).

6.3 The number of latent variables

We now discuss the choice of k, the number of latent variables. Recall that the asymptotic dimension K, if it exists, is the optimal choice as discussed in Section 3.2 and is equal the number of eigenvalues of order p. A scree plot of the 50 largest eigenvalues of R_p is shown in Figure 4(a). Catell’s graphical test indicates an elbow at k = 3, and after 10 ~ 20 eigenvalues, the value is almost constant.

Figure 4.

(a) The 50 largest eigenvalues of R_P ordered according to their size. (b) Histogram of Z₁, Z₂, …, Z₁₀₀₀ and the approximation, f̄, for k = 2 (blue dashed), 10 (red dotted), 100 (solid green).

Theorem 2 states that under the decomposition R_p = A_k,p + B_k,p, the distance between F̂_p and F̄_p is bounded by $\frac{1}{4p}+C{\Vert \text{cor}(B_{k,p})\Vert }_2^{(p)}$. As k increases the bound decreases and ${\Vert \text{cor}(B_{k,p})\Vert }_2^{(p)}=0$ when k = p. However, as k increases the dimension of the representation also increases since the dimension is rank(A_k,p) (Proposition 1). Furthermore, for large k the empirical distribution and the approximation, F̄, are very close and the overfitting problem arises. To illustrate this point, we plot different choices of k in Figure 4(b). It can be seen that when k = 100 the approximation is much closer to the histogram, but such high dimension is unnecessary to capture the global behavior of the histogram. The differences between k = 2 and k = 10 are rather small suggesting that k = 2 suffices.

7 Summary and extensions

In this work we have studied the limit of the ecdf of marginally standard normal variables when strong correlation is present. As predicted by Efron (2007a), we have shown that the limit is indeed not standard normal. Specifically, we have shown that under a regime that we call finite dimensional correlation (and some regularity conditions), the limit is a finite mixture of scaled normals with random means, which reduces to Efron’s empirical null model when the correlation structure is exchangeable. Moreover, we have shown that if the correlation is not finite then the limit can still be approximated by mixture of normals but an infinite number of them may be required.

The main technique for achieving these results has been a decomposition of the correlation matrix R into two matrices A and B, where A captures the strong correlation and B captures the weakly correlated residual noise. The form of the limiting distribution of the ecdf is determined by A, while the residual noise represented by B goes to zero asymptotically. The key to achieving the asymptotic representation of the ecdf with the smallest dimension is to choose B so that it contains the largest amount of variance while remaining weakly correlated.

For future work, we consider the following extensions. First, we assumed that all random variables have variance 1. If the variances are not 1, but are bounded above and below, then the random variables could be standardized and all our results follow. If they are not bounded, then still for each finite p one can standardize and obtain inequality (4) of Theorem 2.

In Theorem 2 we proved that the distance between F̂_p and F̄_p converges to zero if the residual correlation not captured by F̄_p is weak. Could the same be said of the distance between the moments of F̂_p and F̄_p? Let ${\widehat{m}}_{n,p}:=\frac{1}{p}{\sum }_{i=1}^p{Z}_i^n$ and m̄_n,p := ∫ zⁿdF̄_p(z) be the n-th moment of F̂_p and F̄_p. For the first moment and an appropriate decomposition R_p = A_p + B_p,

$${\overline{m}}_{1,p}=\frac{1}{p}\sum _{i=1}^p\int z\frac{1}{{\sigma }_i^{(p)}}\phi \left(\frac{z-{\mu }_i^{(p)}}{{\sigma }_i^{(p)}}\right)dz=\frac{1}{p}\sum _{i=1}^p{\mu }_i^{(p)}.$$

By the definition of ${\mu }_i^{(p)}$ and equation (6), we have that $Z_i-{\mu }_i^{(p)}={\xi }_i$. Therefore,

$$E{({\widehat{m}}_{1,p}-{\overline{m}}_{1,p})}^2=\frac{1}{p^2}E{\left\{\sum _{i=1}^p\left(Z_i-{\mu }_i^{(p)}\right)\right\}}^2=\frac{1}{p^2}E{\left(\sum _{i=1}^p{\xi }_i\right)}^2={\Vert B_p\Vert }_2^{(p)},$$

and the distance converges to zero if and only if ${\Vert B_p\Vert }_2^{(p)}\to 0$. This condition is weaker than that of Theorem 2 and thus convergence of the ecdf implies convergence of the first moment. Proving convergence of higher moments is more difficult and is left for future work.

About the choice of number of components k in practice, the problem is not unlike that of determining the number of components in factor analysis or the effective dimension in principal components analysis. The connection with these techniques is worth exploring.

Finally, about the normality assumption, it is used in the proof of Theorem 2 by applying Mehler’s expansion to the joint density. Similar expansions are available for other distributions such as chi square and gamma (e.g., Koudou (1998); Schwartzman (2011)). Thus it may be possible to extend our results to those distributions as well.