Are a set of microarrays independent of each other?

Abstract

Having observed an m × n matrix X whose rows are possibly correlated, we wish to test the hypothesis that the columns are independent of each other. Our motivation comes from microarray studies, where the rows of X record expression levels for m different genes, often highly correlated, while the columns represent n individual microarrays, presumably obtained independently. The presumption of independence underlies all the familiar permutation, cross-validation, and bootstrap methods for microarray analysis, so it is important to know when independence fails. We develop nonparametric and normal-theory testing methods. The row and column correlations of X interact with each other in a way that complicates test procedures, essentially by reducing the accuracy of the relevant estimators.

1 Introduction

The formal statistical problem considered here can be stated simply: having observed an m × n data matrix X with possibly correlated rows, test the hypothesis that the columns are independent of each other. Relationships between the row correlations and column correlations of X complicate the problem’s solution.

Why are we interested in column-wise independence? The motivation in this paper comes from microarray studies, where X is a matrix of expression levels for m genes on n microarrays. In the “Cardio” study I will use for illustration there are m = 20426 genes each measured on n = 63 arrays, with the microarrays corresponding to 63 subjects, 44 healthy controls and 19 cardiovascular patients¹. We expect the gene expressions to be correlated, inducing substantial correlations within each column (Owen, 2005; Efron, 2007a; Qiu, Brooks, Klebanov and Yakovlev, 2005a), but most of the standard analysis techniques begin with an assumption of independence across microarrays, that is, across the columns of X. This can be a risky assumption: all of the familiar permutation, cross-validation and bootstrap methods for microarray analysis, such as the popular SAM program of Tusher, Tibshirani and Chu (2001), depend on column-wise independence of X; dependence can invalidate the usual choice of a null hypothesis, as discussed next, leading to flawed assessments of significance.

An immediate purpose of the Cardio study is to identify genes involved in the disease process. For gene i we compute the two-sample t-statistic “t_i” comparing sick versus healthy subjects. It will be convenient for discussion to convert these to z-scores,

$$z_i={\Phi }^{-1}\left(F_{61}\right(t_i\left)\right)i=1,2,\dots ,m,$$ (1.1)

with and Ф F₆₁ the cumulative distribution functions (cdf) of standard normal and t₆₁ distributions; under the usual assumptions, z_i will have a standard N(0, 1) null distribution, called here the “theoretical null.” Unusually large values of z_i or —z_i are used to identify non-null genes, with the meaning of “unusual” depending heavily on column-wise independence.

The left panel of Figure 1 shows the histogram of all 20426 z_i values, which is seen to be much wider than N(0, 1) near its center. An “empirical null” fit to the center as in Efron (2007b) was estimated to be N(.03, 1.57²). Null overdispersion has many possible causes (Efron, 2004, 2007a,b), one of which is positive correlation across the columns of X. Such correlations reduce the effective degrees of freedom for the t-statistic, causing (1.1) to yield overdispersed null z_is, and of course changing our assessment of significance for outlying values.

Figure 1.

Left panel: histogram of m = 20426 z-values (1.1) for Cardio study; center of histogram is much wider than N(0, 1) theoretical null. Right panel: scatterplot of microarrays 31 and 32, (x_i31, x_i32) for i = 1, 2, …, m, after removal of row-wise gene means; the scattergram seems to indicate substantial correlation between the two arrays.

The right panel of Figure 1 seems to offer a “smoking gun” for correlation: the scattergram of expression levels for microarrays 31 and 32 looks strikingly correlated, with sample correlation coefficient .805. Here X has been standardized by subtraction of its row means, so the effect is not due to so-called ecological correlations. (X is actually “doubly standardized,” as defined in Section 2.) Nevertheless the question of whether or not correlation .805 is significantly positive turns out to be surprisingly close, as discussed in Section 4, because the row-wise correlations in X drastically reduce the degrees of freedom for the scatterplot. Despite the massive appearance of 20426 points, the scattergram’s accuracy is no more than would be given by 17 independent bivariate normal pairs.

Answering the title’s question, that is, testing for column-wise independence in the presence of row-wise dependence, has both easy and difficult aspects. Section 2 introduces a class of simple permutation tests which, in the case of the Cardio data, clearly discredit column-wise independence. However these tests depend on the ordering of the n columns, and can’t be used if the initial order is lost. It is natural and desirable to look for test statistics of column-wise independence that are invariant under permutation of the columns. Classical multivariate analysis, as in Anderson (2003), develops column independence tests in terms of the eigenvalues of an n by n Wishart matrix. However, this theory depends on the assumption of row-wise independence, disqualifying it for use here.

Sections 3 through 5 consider more general classes of independence tests, both from nonpara-metric and normal theory points of view. The theorem in Section 3 illustrates a key difficulty: correlation between the rows of X (ruled out in the classic theory) can give a misleading appearance of column-wise dependence. Similarly, row-wise dependence can greatly degrade the accuracy of the usual n × n sample covariance matrix of the columns, as shown by the theorem in Section 4. Various non-permutation normal-theory tests are discussed in Section 5, some promising, but with difficulties seen for all of them. The paper ends in Section 6 with a collection of remarks and details.

2 Permutation Tests of Column-Wise Independence

Simple permutation tests can provide strong evidence against column-wise independence, as we will see for the Cardio data. Our main example concerns the 44 healthy subjects, where X is now an m × n matrix with m = 20426 and n = 44. For convenience we assume that X has been “demeaned” by the subtraction of row and column means, giving

$$\sum _i{x}_{ij}=\sum _j{x}_{ij}=0\text{for}i=1,2,\dots ,m\text{and}j=1,2,\dots ,n.$$ (2.1)

Our numerical results go further and assume “double standardization”: that in addition to (2.1),

$$\sum _j{x}_{ij}^2=n\text{and}\sum _i{x}_{ij}^2=mfori=1,\dots ,m\text{and}j=1,\dots ,n,$$ (2.2)

i.e., that each row and column of X has mean 0 and variance 1; see Remark 6.4 in Section 6.

Let $\widehat{\Delta }$ be the familiar estimate of the n × n covariance matrix between the columns of X,

$$\widehat{\Delta }=\left(X^{\prime }X\right)∕m,$$ (2.3)

Under double standardization, $\widehat{\Delta }$ is actually the sample correlation matrix, which we expect to be near the identity matrix I_n under column-wise independence. Also let v₁ denote the first eigenvector of $\widehat{\Delta }$. The left panel of Figure 2 plots the components of v₁ versus array number 1, 2, … , 44. Suppose that the columns of the original expression matrix, before standardization, are independent and identically distributed m-vectors (“i.i.d.”). Then it is easy to see, Remark 6.2 of Section 6, that all orderings of the components of v₁ are equally likely. This is not what Figure 2 shows: the components seem to increase from left to right, with a noticeable block of large values for arrays 27-32.

Figure 2.

Left panel: Components of first eigenvector of row sample correlation matrix for the 44 healthy Cardio subjects, plotted versus array number 1, 2, … , 44; dashes emphasize the block of large components for arrays 27–32. Right panel: First eigenvectors for healthy (solid line) and cancer (dashed) subjects, prostate cancer study, Singh, Febbo, et al. (2002); there was a systematic drift in expression levels as the study progressed.

Let S(v₁) be a statistic that measures structure, for instance a linear regression of v₁ versus array index. Comparing S(v₁) with a set of permuted values

$$\left\{S^{\ast l}=S\left(v^{\ast l}\right),l=1,2,\dots ,L\right\},$$ (2.4)

v*^l a random permutation of the components of v₁, provides a quick test of the i.i.d. null hypothesis.

Permutation testing was applied to v₁ for the Cardio data, using the “block” statistic

$$S\left(v_1\right)=v_1^{\prime }B{v}_1,$$ (2.5)

where B is the n × n matrix

$$B=\sum _h{\beta }_h{\beta }_h^{\prime }.$$ (2.6)

The sum in (2.6) is over all vectors β_h of the form

$${\beta }_h=(0,0,\dots ,0,1,1,\dots ,1,0,0,\dots ,0).$$ (2.7)

with the 1s forming blocks of length between 2 and 10 inclusive. A heuristic rationale for block testing appears below; intuitively, microarray experiments are prone to block disturbances because of the way they are developed and read; see Callow et al. (2000). After L = 5000 permutations, only three S* values exceeded the actual value S(v₁), p-value .0006, yielding strong evidence against the i.i.d. null hypothesis.

The right panel of Figure 2 pertains to a microarray prostate cancer study (Singh et al., 2002) discussed in Efron (2008): m = 6033 genes were measured on each of n = 102 men, 50 healthy controls and 52 prostate cancer patients. The right panel plots first eigenvectors for $\widehat{\Delta }$, (2.3), computed separately for the healthy controls and the cancer patients (the two matrices being individually doubly standardized). Both vectors increase almost linearly from left to right. Taking S(v₁) as the linear regression of v₁ versus array number, permutation testing overwhelmingly rejected the i.i.d. null hypothesis, as it also did using the block test. The prostate study appears as a favorable example of microarray technology in Efron (2008). Nevertheless, Figure 2 indicates a systematic drift in the expression level readings as the study progressed. Some genes drift up, others down (the average drift equaling 0 because of standardization), inducing a small amount of column-wise correlation.

Section 5 discusses models for X where the n × n column covariance matrix is of the “single degree of freedom” form

$$\Delta =I+\lambda \beta {\beta }^{\prime }$$ (2.8)

for some known fixed vector β, the null hypothesis of column-wise independence being H₀ : λ = 0. An obvious choice of test statistic in this situation is

$$S_{\beta }={\beta }^{\prime }\left(\widehat{\Delta }-I\right)\beta ,$$ (2.9)

a monotone increasing function of ${\beta }^{\prime }\widehat{\Delta }\beta$. If β is unknown we can replace S_β with

$$S_B=\sum _{h=1}^H{\beta }_h^{\prime }\widehat{\Delta }{\beta }_h=\mathrm{tr}\left(\widehat{\Delta }\sum _h{\beta }_h{\beta }_h^{\prime }\right)\equiv \mathrm{tr}\left(\widehat{\Delta }\mathrm{B}\right),$$ (2.10)

where {β₁, β₂, … , βH, } is a catalog of “likely prospects” as in (2.7).

Permutation test statistics such as (2.5) can be motivated from the singular value decomposition (SVD) of X,

$$\underset{m\times n}{X}=\underset{m\times K}{U}\underset{K\times K}{d}\underset{K\times n}{V^{\prime }},$$ (2.11)

where K is the rank, d the diagonal matrix of ordered singular values, and U and V orthonormal matrices of sizes m × K and n × K,

$$U^{\prime }U=V^{\prime }V=I_K,$$ (2.12)

I_K the K × K identity. The squares of the diagonal elements, say

$$e_1\ge e_2\ge \dots \ge e_K>0,(e_k=d_k^2)$$ (2.13)

are the eigenvalues of X′X = V′d²V.

S_B in (2.10) can now be written as

$$S_B=\sum _{j=1}^k\frac{e_j}{m}\left(v_j^{\prime }B{v}_j\right).$$ (2.14)

Model (2.8) suggests that most of the information against the null hypothesis H₀ of independence lies in the first eigenvector v₁, getting us back to test statistic $S\left(v_1\right)=v_1^{\prime }B{v}_1$ as in (2.5).

What should the statistician do if column-wise independence is strongly rejected, as in the Cardio example? Use of an empirical null rather than a permutation or theoretical null, N (.03, 1.57²) rather than N(0, 1) in Figure 1, removes the reliance on column-wise independence for hypothesis testing methods such as False Discovery Rates, at the expense of increased variability. Efron (2008) discusses these points.

Two objections can be raised to our permutation tests: (1) they are really testing i.i.d., not independence; (2) non-independence might not manifest itself in the order of v₁ (particularly if the order of the microarrays has been shuffied in some unknown way).

Column-wise standardization makes the column distributions more similar, mitigating objection (1). Going further, “quantile standardization” — say replacing each column’s entries by normal scores (Bolstad, Irizarry, Åstrand and Speed, 2003) — makes the marginals exactly the same. The Cardio data was reanalyzed using normal scores, with almost identical results.

Objection (2) is more worrisome from the point of view of statistical power. The order in which the arrays were obtained should be available to the statistician, and should be analyzed to expose possible trends like those in Figure 2². It would be desirable, nevertheless, to have independence tests that do not depend on order — that is, test statistics invariant under column-wise permutations. The remainder of this paper concerns both the possibilities and difficulties in the development of “non-permutation” tests.

3 Row and Column Correlations

There is an interesting relationship between the row and column correlations of the matrix X, which complicates the question of column-wise independence. For the notation of this section define the n × n matrix of sample covariances between the columns of X as

$$\widehat{\mathit{Cov}}=X^{\prime }X∕m,$$ (3.1)

called $\widehat{\Delta }$ in Section 2, and likewise

$$\widehat{\mathit{cov}}=X{X}^{\prime }∕n,$$ (3.2)

for the m × m matrix of row-wise sample covariances (having more than 400, 000, 000 entries in the Cardio example!).

Theorem 1. If X has row and column means 0, (2.1), then the n² entries of $\widehat{\mathit{Cov}}$ have empiricalmean 0 and variance c₂,

$$c_2=\sum _{k=1}^K{e}_k^2∕{\left(\mathit{mn}\right)}^2,$$ (3.3)

with e_k the eigenvalues (2.13), and so do the m² entries of $\widehat{\mathit{cov}}$. Proof. The sum of $\widehat{\mathit{Cov}}$’s entries is

$$1_n^{\prime }{X}^{\prime }X{1}_n∕m=0,$$ (3.4)

according to (2.1), while the mean of squared entries is

$$\frac{\sum _{j=1}^n\sum _{j^{\prime }=1}^n{\widehat{\mathit{Cov}}}_{j{j}^{\prime }}^2}{n^2}=\frac{\mathrm{tr}\left({\left({\mathrm{X}}^{\prime }\mathrm{X}\right)}^2\right)}{m^2{n}^2}=\frac{\mathrm{tr}\left({\mathrm{V}}^{\prime }{d}^4\mathrm{V}\right)}{m^2{n}^2}=c_2.$$ (3.5)

Replacing X′X with XX′ yields the same results for the row covariances $\widehat{\mathit{cov}}$.

Under double standardization (2.1)-(2.2), the covariances become sample correlations, say $\widehat{\mathit{cor}}$ and $\widehat{\mathit{cor}}$ for the columns and rows. Theorem 1 has a surprising consequence: whether or not the columns of X are independent, the column sample correlations will have the same mean and variance as the row correlations. In other words, substantial row-wise correlation can induce the appearance of column-wise correlation.

Figure 3 concerns the 44 healthy subjects in the Cardio study, with X an (m, n) = (20426, 44) doubly standardized matrix. All 44² column correlations are shown by the solid histogram, while the line histogram is a random sample of 10, 000 row correlations. Here c₂ = .283², so according to the Theorem both histograms have mean 0 and standard deviation .283.

Figure 3.

Left panel: solid histogram the 44² column sample correlations for X the doubly standardized matrix of healthy Cardio subjects; line histogram is sample of 10000 of the 20426² row correlations. Right panel: solid histogram the column correlations excluding diagonal 1s; line histogram the row correlations corrected for sampling overdispersion.

The 44 diagonal elements of $\widehat{\mathit{cor}}$ protrude as a prominent spike at 1. (We can’t see the spike of 20426 diagonal elements for the row correlation matrix $\widehat{\mathit{cor}}$ because they form such a small fraction of all 20426².) It is easy to remove the diagonal 1’s from consideration.

Corollary. In the doubly standardized situation, the off-diagonal elements of the column correlation matrix $\widehat{\mathit{Cor}}$ have empirical mean and variance

$$\widehat{\mu }=-\frac{1}{n-1}\text{and}{\widehat{\alpha }}^2=\frac{n}{n-1}\left(c_2-\frac{1}{n-1}\right).$$ (3.6)

For n = 44 and c₂ = .283 this gives

$$(\widehat{\mu },{\widehat{\alpha }}^2)=(-.023,{.241}^2).$$ (3.7)

.

The corresponding diagonal-removing corrections for the row correlations (replacing n by m in (3.6)) are neglible for m = 20426. However c₂ overestimates the variance of the row correlations for another reason: with only 44 points available to estimate each correlation, estimation error adds a considerable component of variance to the $\widehat{\mathit{cor}}$ histogram in the left panel, as discussed next.

Suppose now that the columns of X are in fact independent, in which case the substantial column correlations seen in Figure 3 must actually be induced by row correlations, via Theorem 1. Let cor_ii′ indicate the true correlation between rows i and i′ (that is, between X_ij and X_i′j), and define α the total correlation to be the root mean square of the cor_ii′ values,

$${\alpha }^2=\sum _{i<i^{\prime }}co{r}_{i{i}^{\prime }}^2/\left(\begin{array}{c}\hfill m\hfill \\ \hfill 2\hfill \end{array}\right).$$ (3.8)

.

Remark 6.5 of Section 6 shows that ${\widehat{\alpha }}^2$ in (3.6) is an approximately unbiased estimate of α², assuming column-wise independence. For the Cardio example $\widehat{\alpha }=.241$, similar to the size of the microarray correlation estimates in Efron (2007a), Owen (2005), and Qiu et al. (2005a). Section 4 discusses the crucial role of α in determining the accuracy of estimates based on X.

The right panel of Figure 3 compares the histogram of the column correlations ${\widehat{\mathit{cor}}}_{j{j}^{\prime }}$, now excluding cases j = j′, with the row correlation histogram corrected for sampling overdispersion via the shrinkage factor .241/.283. As predicted by Theorem 1, the similarity is striking. A possible difference lies in the long right tail of the $\widehat{\mathit{Cor}}$ distribution (including ${\widehat{\mathit{cor}}}_{31,32}$, the case illustrated in Figure 1), whose significance is examined in Section 4.

4 Normal Theory

The results of Sections 2 and 3 were developed nonparametrically. This section concerns multivariate normal theory, afterwards used in Section 5 to draw the connection with classical multivariate independence tests. We consider the matrix normal distribution for X,

(4.1)

where the Kronecker notation indicates covariance structure

(4.2)

Row x_i of X has covariance matrix proportional to Δ,

(4.3)

(not independently across rows unless is diagonal), and likewise for column x_j, x_j ~ N_m(0, Δ_jj). As in (2.1), we take all means equal 0.

Much of classical multivariate analysis focuses on the situation = I, where the rows x_i are independent replicates³,

(4.4)

in which case the sample covariance matrix $\widehat{\Delta }=X\prime X∕m$ has a scaled Wishart distribution,

$$\widehat{\Delta }\sim \mathrm{Wishart}(m,\Delta )∕m.$$ (4.5)

Distribution (4.5) has first and second moments

$$\underset{n\times n}{\widehat{\Delta }}\sim \left(\underset{n\times n}{\Delta },\underset{n^2\times n^2}{{\Delta }^{\left(2\right)}}/m\right)\text{with}{\Delta }_{jk,lh}^{\left(2\right)}={\Delta }_{jl}{\Delta }_{kh}+{\Delta }_{jh}{\Delta }_{kl}$$ (4.6)

for j, k, l, h = 1, 2, … , n; see Mardia, Kent and Bibby (1979, p. 92).

Relation (4.6) says that when = I, that is when the rows of X are independent, $\widehat{\Delta }$ unbiasedly estimates the row covariance matrix with accuracy proportional to m^-1/2. Correlation between rows reduces the accuracy of $\widehat{\Delta }$, as shown next.

Returning to the general situation (4.1)-(4.3), define

$$\tilde{\Delta }=X^{\prime }{\sigma }^{-2}X∕m,$$ (4.7)

where σ is the diagonal matrix with diagonal entries .

Theorem 2. Under model (4.1), $\tilde{\Delta }$ has first and second moments

$$\tilde{\Delta }\sim \left(\Delta ,{\Delta }^{\left(2\right)}∕\tilde{m}\right),\tilde{m}=m∕[1+(m-1\left){\alpha }^2\right],$$ (4.8)

where α is the total correlation as in (3.8),

(4.9)

and Δ⁽²⁾ is the Wishart covariance (4.6).

Comparing (4.8) with (4.6), we see that correlation between the rows reduces “effective sample size” from m to $\tilde{m}$ : for α = .241 as in (3.7), the reduction is from m=20426 to $\tilde{m}=17.2!$ (Notice that row standardization effectively makes $\tilde{\Delta }\stackrel{.}{=}\widehat{\Delta }$ (2.3), justifying the comparison.) The total correlation α shows up in other efficiency calculations; see Remark 6.7.

Proof. The row-standardized matrix X̃ = σ^-1X has matrix normal distribution

(4.10)

where has diagonal elements From (4.2) we see that is the correlation between elements X_ij and X_i′j in the same column of X;$\tilde{\Delta }={\tilde{X}}^{\prime }\tilde{X}∕m$ has entries ${\tilde{\Delta }}_{jk}={\Sigma }_i{\tilde{X}}_{ij}{\tilde{X}}_{ik}∕m$ and is unbiased for, Δ

$$E\left\{{\tilde{\Delta }}_{jk}\right\}={\Delta }_{jk},$$ (4.11)

using (4.2).

The covariance calculation for $\tilde{\Delta }$ involves expansion

$${\tilde{\Delta }}_{jk}{\tilde{\Delta }}_{lh}=\left(\sum _i{\tilde{X}}_{ij}{\tilde{X}}_{ik}∕m\right)\left(\sum _{i^{\prime }}{\tilde{X}}_{i^{\prime }l}{\tilde{X}}_{i^{\prime }h}∕m\right)$$ (4.12)

$$=\frac{1}{m^2}\left(\sum _i{\tilde{X}}_{ij}{\tilde{X}}_{ik}{\tilde{X}}_{il}{\tilde{X}}_{ih}+\sum _{i\ne i^{\prime }}{\tilde{X}}_{ij}{\tilde{X}}_{ik}{\tilde{X}}_{i^{\prime }l}{\tilde{X}}_{i^{\prime }h}\right).$$ (4.13)

Using the formula

$$E\left\{Z_1{Z}_2{Z}_3{Z}_4\right\}={\gamma }_{12}{\gamma }_{34}+{\gamma }_{13}{\gamma }_{24}+{\gamma }_{14}{\gamma }_{23}$$ (4.14)

for a normal vector (Z₁Z₂Z₃Z₄)^′ with 0 means and covariances γ_ij, (4.2) gives

$$E\left\{\sum _i{\tilde{X}}_{ij}{\tilde{X}}_{ik}{\tilde{X}}_{il}{\tilde{X}}_{ih}\right\}=m[{\Delta }_{jk}{\Delta }_{lh}+{\Delta }_{jl}{\Delta }_{kh}+{\Delta }_{jh}{\Delta }_{kl}]$$ (4.15)

and

(4.16)

Then (4.13) yields giving

$$E\left\{{\tilde{\Delta }}_{jk}{\tilde{\Delta }}_{lh}\right\}={\Delta }_{jk}{\Delta }_{lh}+({\Delta }_{jl}{\Delta }_{kh}+{\Delta }_{jh}{\Delta }_{kl})\left(\frac{1+(m-1){\alpha }^2}{m}\right),$$ (4.17)

giving

$$\mathrm{cov}({\tilde{\Delta }}_{jk},{\tilde{\Delta }}_{lh})=({\Delta }_{jl}{\Delta }_{kh}+{\Delta }_{jh}{\Delta }_{kl})∕\tilde{m}$$ (4.18)

as in (4.8).

A corollary of Theorem 2, used in Section 5, concerns bilinear functions of and Δ and $\tilde{\Delta }$,

$${\tau }^2=w^{\prime }\Delta w\text{and}{\tilde{\tau }}^2=w^{\prime }\tilde{\Delta }w,$$ (4.19)

where w is a given n-vector.

Corollary. Under model (4.1), ${\tilde{\tau }}^2$ has mean and variance

$${\tilde{\tau }}^2\sim ({\tau }^2,2{\tau }^4∕\tilde{m}).$$ (4.20)

The proof follows that for Theorem 2; see Remark 6.9.

If = I in (4.1), then $\tilde{\Delta }=\widehat{\Delta }$ and ${\tilde{\tau }}^2$ has a scaled chi-squared distribution,

$${\tilde{\tau }}^2\sim {\tau }^2\cdot {\chi }_m^2∕m,$$ (4.21)

with mean and variance ${\tilde{\tau }}^2\sim ({\tau }^2,2{\tau }^4∕m)$, so again the effect of correlation within is to reduce the effective sample size from m to $\tilde{m}$ (4.8).

We can approximate $\tilde{\Delta }$ (4.7), with

$$\widehat{\Delta }=X^{\prime }{\widehat{\sigma }}^{-2}X∕m,$$ (4.22)

where ${\widehat{\sigma }}_{ii}^2$ is an estimate of _ii based on the observed variability in row i. If the rows of X have been standardized, then ${\widehat{\sigma }}_{ii}^2=1$ and $\widehat{\Delta }$ returns to its original definition X′X/m.

Both Theorem 2 and the Corollary encourage us to think of $\widehat{\Delta }$ as, approximately, a scaled Wishart distribution based on an independent sample of size $\tilde{m}$

$$\widehat{\Delta }\stackrel{.}{\sim }\mathrm{Wishart}(\tilde{m},\Delta )∕\tilde{m}.$$ (4.23)

The dangers of this approximation are discussed in Section 5, but it is, nevertheless, an evocative heuristic, as shown below.

Figure 4 returns to the question of the seemingly overwhelming correlation .805 between arrays 31 and 32 seen in Figure 1. A one-sided p-value was calculated for each of the 946 column correlations, using as a null hypothesis the normal theory correlation coefficient distribution based on a sample size of $\tilde{m}$ = 17.2 pairs of N₂(0, I) points (the correct null if Δ = I in (4.23)). Benjamini and Hochberg’s (1995) False Discovery Rate test, level q = .1, was applied to the 946 p-values. This yielded 7 significant cases, those with sample correlation .723; all 7 were from the block of arrays 27 to 32 indicated in Figure 2. Correlation .805 does turn out to be significant, but by a much closer margin than Figure 1’s scattergram suggests.

Figure 4.

Dashed curve is normal-theory null density for correlation coefficient from $\tilde{m}=17.2$ pairs of points; see Remark 6.6. Histogram is the 946 column correlations, right panel Figure 3. FDR test, q = .1, yielded 7 significant correlations, $\widehat{\mathit{Cor}}$ ≥.723, including .805 between arrays 31 and 32, Figure 1.

The Fdr procedure was also applied using the simpler null distribution N(—.023, .241²) (3.7). This raised the significance threshold from .723 to .780, removing two of the previously significant correlations.

Theorem 1 showed that the variance of the observed column correlations is useless for testing column-wise independence, since any value at all can be induced by row correlations. The test in Figure 4 avoids this trap by looking for unusual outliers among the column correlations. It does not depend on the order of the columns, objection (2) in Section 2 for permutation tests, but pays the price of increased modeling assumptions.

5 Other Test Statistics

Theorem 2 offers a normal-theory strategy for testing column-wise independence. We begin with X ~ N_m,n(0, ⊗ Δ)(4.1), taking

(5.1)

as suggested by double standardization. The null hypothesis of column-wise independence is equivalent to the column correlation matrix equaling the identity,

$$H_0:\Delta =I,$$ (5.2)

since then (4.2) says that all pairs in different columns are independent.

To test (5.2), we estimate with Δ with $\widehat{\Delta }$, (4.22) or more simply $\widehat{\Delta }=X\prime X∕m$ after standardization, and compute a test statistic

$$S=s\left(\tilde{\Delta }\right),$$ (5.3)

where s(.) is some measure of distance between $\widehat{\Delta }$ and 1. The accuracy approximation $\widehat{\Delta }\stackrel{.}{~}(\Delta ,{\Delta }^{\left(2\right)}∕\tilde{m})$ from (4.8), with Δ = 1, is used to assess the significance level of the observed S, maybe even employing the more daring approximation $\widehat{\Delta }\stackrel{.}{~}\text{Wishart}(\tilde{m},I)∕\tilde{m}$. Strategy (5.3) looks promising but, as the examples of this section will show, it suffers from serious difficulties that are absent under the classic assumption of independent rows.

One of the difficulties stems from Theorem 1. An obvious test statistic for H₀ : Δ = I is

$$S=\sum _{j<j^{\prime }}{\tilde{\Delta }}_{j{j}^{\prime }}^2/\left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right),$$ (5.4)

the average squared off-diagonal element of $\widehat{\Delta }$. But $\widehat{\Delta }=\widehat{\mathit{Cov}}$ (3.1), so in the doubly standardized situation of (3.6), S is an increasing monotone function $\widehat{\alpha }$, the estimated total correlation. This disqualifies S as a test statistic for (5.2), since large values of $\widehat{\alpha }$ can always be attributed to row-wise correlation alone.

Similarly, the variance of the eigenvalues (2.13),

$$S=\sum _{k=1}^K{(e_k-e.)}^2∕k\left(e.=\sum e_k∕K\right),$$ (5.5)

looks appealing since the true eigenvalues all equal 1 when Δ = I. However (5.5) is also a monotonic function of $\widehat{\alpha }$; see Remark 6.1.

The general difficulty here is “leakage,” the fact that row-wise correlations affect the observed pattern of column-wise correlations. This becomes clearer by comparison with classical multivariate methods, where row-wise correlations are assumed away by taking = I in (4.1). Johnson and Graybill (1972) consider a two-way ANOVA problem where, after subtraction of main effects, X has the form

$$X_{ij}=a_i{\beta }_j+{∊}_{ij}\text{for}i=1,2,\dots ,m\text{and}j=1,2,\dots ,n,$$ (5.6)

a_i ~ N(0, λ) and ε_ij ~ N(0, 1), all independently, with β = (β₁, β₂, …, β_n) a fixed but unknown vector (representing “one degree of freedom for nonadditivity” in the two-way table X, Johnson and Graybill’s extension of Tukey’s procedure).

In the Kronecker notation (4.1), X ~ N_m,n(0,I ⊗ Δ) with

$$\Delta =I+\lambda \beta {\beta }^{\prime }.$$ (5.7)

Now (5.2) becomes H₀ : λ = 0. Johnson and Graybill show that, with β unknown, the likelihood ratio test rejects H₀ for large values of the eigenvalue ratio (2.13),

$$S=e_1/\sum _{k=1}^K{e}_k.$$ (5.8)

Since the m rows of X are assumed independent, they can test H₀ by comparison of S with values $S^{\star }=e_1^{\star }∕{\sum }_{k=1}^K{e}_k^{\star }$ obtained from

$${\widehat{\Delta }}^{\ast }\sim \mathrm{Wishart}(m,I)∕m,$$ (5.9)

as in (4.5).

Getting back to the correlated rows situation, Theorem 2 suggests comparing S with values S* from

$${\widehat{\Delta }}^{\ast }\sim \mathrm{Wishart}(\tilde{m},I)∕\tilde{m},$$ (5.10)

$\tilde{m}$ as in (4.8). The solid histogram in Figure 5 compares 100 S* values from (5.10), $\tilde{m}$ = 17.2 for the Cardio data, with the observed value S = .207 from the doubly standardized Cardio matrix for the healthy subjects used in Figure 3. All 100 S* values are much smaller than S, providing strong evidence against H₀ : Δ = I.

Figure 5.

Eigenratio statistic (5.8) equals .207 for 20426 × 44 Cardio matrix X; solid histogram 100 simulations S* from Wishart (5.10), $\tilde{m}=17.2$; line histogram 100 simulations from correlated-row X* matrices (5.11), α = .241, Δ = I.

The evidence looks somewhat weaker, though, if we simulate S* values with ${\widehat{\Delta }}^{\star }$ obtained from random matrices

(5.11)

doubly standardized, where has total correlation α = .241, the estimated value for X, (4.9). The line histogram in Figure 5 shows 100 such S* values, all still smaller than S, but substantially less so. (Remark 6.8 describes the construction of X*.)

Why does (5.11) produce larger “null” S* values than (5.10)? The answer is simple: even though the first and second moments of ${\widehat{\Delta }}^{\star }=X^{\star }\prime X^{\star }∕m$ match ${\widehat{\Delta }}^{\star }$ from (5.10), its eigenvalues do not. The non-zero eigenvalues of X*′X*/m equal those of . This is another example of leakage, where the fact that in (5.11) is not the identity I_m distorts the estimated eigenvalue of ${\widehat{\Delta }}^{\star }$ even if Δ = I_n.

The eigenratio statistic S = e₁/Σe_k is invariant under permutations of the columns of X, answering objection (2) to permutation testing of Section 2. Because of invariance, the eigenratio and permutation tests provide independent p-values for testing the null hypothesis of i.i.d. columns, and so can be employed together. Figure 5 is disturbing nonetheless, in suggesting that an appropriate null distribution for S depends considerably on the choice of the nuisance parameter in (5.11).

The bilinear form (4.19)-(4.20) yields another class of test statistics,

$${\widehat{\tau }}^2=w^{\prime }\widehat{\Delta }w\stackrel{.}{\sim }({\tau }^2,2{\tau }^4∕\tilde{m}),$$ (5.12)

where w is a pre-chosen n-vector and ${\tau }^2=w^{\prime }\Delta w$. Delta-method arguments give $CV\left(\widehat{\tau }\right)\stackrel{.}{=}{\left(2\tilde{m}\right)}^{-1∕2}$ for the coeffcient of variation of $\widehat{\tau }$. Defining

$$Z_i=x_i^{\prime }w\left(x_i^{\prime }\text{the}i\text{th row of}X\right),$$ (5.13)

yields the alternative form

$${\widehat{\tau }}^2=\sum _{i=1}^m{Z}_i^2∕m.$$ (5.14)

.

In a two-sample situation like that for the Cardio study, sample sizes n₁ and n₂, we can choose

$$w^{\prime }={\left(\frac{n_1{n}_2}{n_1+n_2}\right)}^{1∕2}(-1_{n_1}∕n_1,1_{n_2}∕n_2),$$ (5.15)

“1_n” indicating a vector of n 1’s. This choice makes

$$Z_i={\left(\frac{n_1{n}_2}{n_1+n_2}\right)}^{1∕2}({\stackrel{‒}{x}}_{2i}-{\stackrel{‒}{x}}_{1i}),$$ (5.16)

the multiple of the mean response difference between the two samples that has variance 1 if Δ = I. In terms of (5.12), ∥w∥² = 1 so τ² = 1.

For the Cardio study, with n₁ = 44, n₂ = 19, and $\tilde{m}$ = 17.2, we obtain $\widehat{\tau }=1.48$, coefficient of variation 0.17. This puts $\widehat{\tau }$ more than 2.8 standard errors above the null hypothesis value τ = 1, again providing evidence against column-wise independence. The Z_i values from (5.16) are nearly indistinguishable from the z_i values in Figure 1 — not surprisingly since with the rows of X standardized, Z_i is an equivalent form of the two-sample t-statistic t_i in (1.1).

Once again, however, there are difficulties with this as a test for column-wise independence. There is no question that the Z_i’s are overdispersed compared to the theoretical value τ = 1. But problems other than column dependence can cause overdispersion, in particular unobserved covariate differences between subjects in the two samples (Efron, 2004, 2008).

The statistic $S=w\prime \widehat{\Delta }w$ in (5.15) does not depend upon the order of the columns of X within each of the two samples, answering objection (2) against permutation tests, but it is the only such choice for a two-sample situation. Other w’s might yield interesting results. The version of (5.15) comparing the first 22 healthy Cardio subjects with the second 22 provided the spectacular value $\widehat{\tau }=1.87$, and here the “unobserved covariate” objection has less force.

Now, however, the test statistic depends on the order of the columns within the healthy subjects’ matrix, reviving objection (2). Again we might want to check a catalog of possible w vectors w₁, w₂, …, w_H, leading back to test statistic

$$S_B=\sum _h{w}_h^{\prime }\widehat{\Delta }{w}_h=\mathrm{tr}\left(\widehat{\Delta }\mathrm{B}\right)\left(\mathrm{B}=\sum _{\mathrm{h}}{\mathrm{w}}_{\mathrm{h}}{\mathrm{w}}_{\mathrm{h}}^{\prime }\right)$$ (5.17)

as in (2.10), the only difference being that the null distribution of $\widehat{\Delta }$ now involves normal theory rather than permutations. Remark 6.9 shows that the null first and second moments of S_B are similar to (5.12),

$$S_B\underset{H_0}{\sim }\left(\mathrm{tr}\left(B\right),\frac{2}{\tilde{\mathrm{m}}}\mathrm{tr}\left(B^2\right)\right).$$ (5.18)

.

In summary, normal-theory methods are interesting and promising, but are not yet proven competitors for the permutation tests of Section 2.

6 Remarks

This section presents some brief remarks and details supplementing the previous material.

Remark 6.1. The constant c₂ The variance constant c₂ in Theorem 1 (3.3) can be expressed as

$$c_2=\frac{K}{{\left(mn\right)}^2}\left[e^{-2}+\sum _{k=1}^K{(e_k-\stackrel{‒}{e})}^2\right]\left(\stackrel{‒}{e}\equiv \sum _1^K{e}_k∕K\right),$$ (6.1)

. so that c₂ ≥ K(ē/mn)², with equality only if the eigenvalues e_k are equal. In the doubly standardized case ē = mn/K, giving

$$c_2\ge 1∕K,$$ (6.2)

where K is the rank of X.

Remark 6.2. Permutation invariance If the columns of X are i.i.d. observations from a distribution on R^m, then the distribution of X is invariant under permutations: Xπ ~ X for any n × n permutation matrix π. Now suppose $\tilde{X}=L\left(X\right)$ where L performs the same operation on each column of X, for example replacing each column by its normal scores vector. Then

$$\tilde{X}\pi =L\left(X\right)\pi =L\left(X\pi \right)\sim L\left(X\right)=\tilde{X},$$ (6.3)

showing that $\tilde{X}$ is permutation invariant.

Similarly, suppose $\tilde{X}=R\left(X\right)$, performing the same operation ${\tilde{X}}_i=r\left(X_i\right)$ on each row of X, where now we require r(x)π = r(xπ) for all n-vectors x. The same argument as (6.3) demonstrates that $\tilde{X}$ is still permutation invariant. Iterating row and column standardizations as in Table 1 then shows that if the original data matrix X is permutation invariant, so is its doubly standardized version.

Table 1.

Successive row and column standardizations of the 20426 × 44 matrix of healthy Cardio subjects. “Col” empirical standard deviation of ${\widehat{Cor}}_{jj\prime },j<j\prime$ ; “Eig” $\widehat{\alpha }$ from (3.6); “Row” from 1% sample of ${\widehat{cor}}_{ii\prime }$ values, adjusted for overdispersion (6.6), sampling standard error :0034.

	Col	Row	Eig		Col	Row	Eig
demeaned	0.252	0.286	0.000	demeaned	0.252	0.286	0.000
col	0.252	0.249	0.251	row	0.241	0.283	0.279
row	0.242	0.255	0.246	col	0.241	0.251	0.240
col	0.242	0.241	0.242	row	0.240	0.247	0.241
row	0.241	0.246	0.235	col	0.240	0.247	0.240
col	0.241	0.244	0.241	row	0.241	0.240	0.235
row	0.241	0.245	0.234	col	0.241	0.237	0.240
col	0.241	0.238	0.241	row	0.241	0.233	0.233

Remark 6.3. Covariances after demeaning Suppose that X is normally distributed, with covariances ⊗ Δ (4.2), all columns having the same expectation vector μ. Let $\tilde{X}$ be the demeaned matrix obtained by subtracting all the row and column means of X. Then

(6.4)

where

$${\tilde{\Delta }}_{j{j}^{\prime }}={\Delta }_{j{j}^{\prime }}-\Delta {.}_{j^{\prime }}-{\Delta }_j.+\Delta ..,$$ (6.5)

dots indicating averaging over the missing subscripts, and similarly for . This shows that de-meaning tends to reduce covariances by recentering them around 0.

Remark 6.4. Standardization A matrix X is “column standardized” by individually subtracting the mean and dividing by the standard deviation of each column, and similarly for row standardization. Table 1 shows the effect of successive row and column standardizations on the 20426 × 44 demeaned matrix of healthy Cardio subjects. Here “Col” is the empirical standard deviation of the 946 column-wise correlations ${\widehat{\mathit{Cor}}}_{jj\prime },j<j\prime$; “Eig” is $\widehat{\alpha }$ in (3.6); and “Row” is the empirical standard deviation $"\widehat{\beta }"$ of a 1% sample of the row correlations ${\widehat{\mathit{cor}}}_{ii\prime }$, but adjusted for overdispersion,

$${\text{Row}}^2=\frac{n}{n-1}\left({\widehat{\beta }}^2-\frac{1}{n-1}\right).$$ (6.6)

Sampling error of the Row entries is about ±.0034.

The doubly standardized matrix X used for Figure 3 was obtained after five successive column-row standardizations. This was excessive; the Figure looked almost the same after two iterations. Other microarray examples converged equally rapidly, though small counterexamples can be constructed where double standardization isn’t possible.

Microarray analyses usually begin with some form of column-wise standardization (Bolstad et al., 2003; Qiu, Klebanov and Yakovlev, 2005b), designed to negate “brightness” differences between the n microarrays. In the same spirit, row standardization helps prevent incidental gene differences (for example, very great or very small expression level variabilities) from obscuring the actual effects of interest. Standardization tends to reduce the apparent correlations as in Remark 6.3. Without standardization, the scatterplot in Figure 1 stretches out along the main diagonal, correlation .917, driven by genes with unusually large or small inherent expression levels.

Remark 6.5. Corrected estimates of the total correlation Suppose that the true row correlations cor_ii′ have mean 0 and variance α², as in (3.8) with $\stackrel{‒}{\mathit{cor}}=0$, and that given cor_ii’ , the usual estimate ${\widehat{\mathit{cor}}}_{ii\prime },ii\prime$ has mean and variance

$${\widehat{\mathit{cor}}}_{i,i^{\prime }}\stackrel{.}{=}\left[{\mathit{cor}}_{i{i}^{\prime }},{(1-{\mathit{cor}}_{i{i}^{\prime }}^2)}^2∕(n-3)\right],$$ (6.7)

(6.7) being a good normal-theory approximation (Johnson and Kotz, 1970, Chap. 32). Letting ${\stackrel{‒}{\alpha }}^2$ be the empirical variance of the ${\widehat{\mathit{cor}}}_{ii\prime }$ values, a standard empirical Bayes derivation yields

$${\widehat{\alpha }}^2=A^2-\frac{3}{n-5}{A}^4\left[A^2=\frac{(n-3){\stackrel{‒}{\alpha }}^2-1}{n-5}\right]$$ (6.8)

as an approximately unbiased estimate of α². (If $\stackrel{‒}{\mathit{cor}}$ is not assumed to equal 0, a slightly more complicated formula applies.) Of course ${\widehat{\alpha }}^2=0$ if the right side of (6.8) is negative.

Theorem 1 implies that ${\stackrel{‒}{\alpha }}^2$ = 0 nearly equals c₂, (3.3), in the doubly standardized situation. Formula (3.6), with say

$${\tilde{\alpha }}^2=\frac{n}{n-1}\left({\stackrel{‒}{\alpha }}^2-\frac{1}{n-1}\right)$$ (6.9)

is not identical to (6.8), but provides an excellent approximation for values of $\stackrel{‒}{\alpha }\le 0.5$: with n = 44 and $\stackrel{‒}{\alpha }=.283$ as in (3.6), $\widehat{\alpha }=.2415$ while $\tilde{\alpha }=.2412$.

Remark 6.6. Column and row centerings The column correlation mean $\widehat{\mu }=-1∕(n-1)$ in (3.6) is forced by the row-wise demeaning Σ_j x_ij = 0, (2.1), centering the solid histogram in the right panel of Figure 3 at -.023. With m = 20426, the corresponding center for the line histogram is nearly 0, and the difference in the two centerings is noticeable. The dashed density curve in Figure 4, and the corresponding p-values for the FDR analysis, were shifted .023 units leftwards.

Remark 6.7. The total correlation α The total correlation α, which plays a key role in Theorem 2, (4.9), also is the central parameter of the theory developed in Efron (2007a). Equations (3.15)- (3.16) there are equivalent to (5.12) here. In both papers, α has the very convenient feature of summarizing the effects of an enormous m × m correlation matrix in a single number.

Remark 6.8. for simulation (5.11) The X* simulation used in Figure 5 began with m × n matrix Y = (y_ij),

$$y_{ij}=c_{Ij}+e_{ij}\left\{\begin{array}{cc}{e}_{ij}\hfill & \sim \mathcal{N}(0,1)\hfill \\ c_{Ij}\hfill & \sim \mathcal{N}(0,{\gamma }^2)\hfill \end{array}\phantom{\}}\right(\text{all independent}),$$ (6.10)

where I = 1, 2, 3, 4, 5 as i is in the first, second, …, last fifth of 1 through m; Y was then column standardized to give X*, so that had a block form, with large positive correlations (about 0.61) in the (m/5) × (m/5) diagonal blocks. The choice λ = 1.23 was required to yield α = .241.

Remark 6.9. Bilinear statistics Since $\tilde{\Delta }~(\Delta ,{\Delta }^{\left(2\right)}∕\tilde{m})$ (4.8), it is clear that $E\left\{{\tilde{\tau }}^2\right\}={\tau }^2$ in Corollary (4.20). The variance calculation proceeds as in Theorem 2:

$$\begin{array}{cc}\hfill \mathrm{var}\left\{{\tilde{\tau }}^2\right\}& =\sum _{jk}\sum _{lh}{\Delta }_{jk,lh}^{\left(2\right)}{w}_j{w}_k{w}_l{w}_h∕\tilde{m}\hfill \\ \hfill & =\sum _{jk}\sum _{lh}[{\Delta }_{jl}{\Delta }_{kh}+{\Delta }_{jh}{\Delta }_{kl}]w_j{w}_k{w}_l{w}_h∕\tilde{m}\hfill \\ \hfill & =\left[\sum _{jl}\sum _{kh}\left({\Delta }_{jl}{w}_j{w}_l\right)\left({\Delta }_{kh}{w}_k{w}_h\right)+\sum _{jh}\sum _{kl}\left({\Delta }_{jh}{w}_j{w}_h\right)\left({\Delta }_{kl}{w}_k{w}_l\right)\right]/\tilde{m}\hfill \\ \hfill & =2{\left(\sum _{jk}{\Delta }_{jk}{w}_j{w}_l\right)}^2/\tilde{m}=2{\tau }^4∕\tilde{m}.\hfill \end{array}$$ (6.11)

The verification of (5.18) is the same, except with element b_jk of B replacing w_jw_k above, b_lh replacing w_lw_h, etc.