Test-statistic correlation and data-row correlation

Abstract

When a statistical test is repeatedly applied to rows of a data matrix, correlations among data rows will give rise to correlations among corresponding test statistics. We investigate the relationship between test-statistic correlation and data-row correlation and discuss its implications.

1. Introduction

Many scientific data sets are organized in matrix forms and statistical inferences— such as hypothesis tests and regression analysis—are often repeatedly applied to individual rows of the data matrix. For example, in gene expression analysis, normalized expression values are often organized in a matrix with rows corresponding to genes and columns corresponding to biological samples (experimental units). In a two-group comparison experiment, a two-sample test will be applied to each row of the data matrix in order to assess differential expression (DE). For more complex experimental designs, regression analysis can be used.

Correlations may exist among the data rows: For example, between-gene correlations are commonly observed in gene expression data [1, 2, 3, 4, 5]. Data-row correlations can give rise to correlations among the test statistic values calculated from the data rows [6, 7, 8]. The dependence among test statistic values has brought methodological challenges to statistical procedures aiming to summarize the collection of test results. For example, some multiple hypothesis testing procedures determine a p-value cutoff by controlling the false discovery rate (FDR) [9] or the q-value [5]. Many FDR-control procedures are valid only when the test statistics satisfy certain independence or positive-dependence conditions [9, 10]. Furthermore, Efron [7] showed in a simulation study that for a nominal FDR of 0.1, the actual false discovery proportions (FDP) in individual experiments can easily vary by a factor of 10 when there are correlations among test statistics.

In a gene-set analysis, one tests for over-abundance of DE genes in a specified gene set (e.g., a molecular pathway or a gene ontology category) [11]. The correlations among DE test statistics, if not addressed appropriately, will undermine the validity of many gene-set tests [2, 8, 12]. A better understanding of the test-statistic correlations is thus of fundamental importance and is a first towards developing statistical methods that correctly account for correlations.

Without replicating the experiment, we cannot directly estimate the correlation between a pair of test statistic values, because there is only one observed test statistic value for each data row. For this reason, the correlation between the corresponding data rows (after treatment effects accounted for) is sometimes used as a surrogate—explicitly or implicitly—when one actually needs the test-statistic correlation. It is yet unclear when and to what extent the test-statistic correlation (e.g., as measured by the Pearson correlation coefficient) can be approximated by the corresponding data-row correlation, though some simulation results suggest connections between the two quantities. Efron [7] concluded through simulation that the distribution of z-value (the test statistic considered in that paper) correlation can be nearly represented by the distribution of sample correlation from the data rows. Barry et al. [6] showed by Monte Carlo simulation of gene expression data that a nearly linear relationship holds between test-statistic correlations and data-row correlations for several forms of test statistics they examined. These Monte Carlo simulation results were cited by Wu and Smyth [8] as a justification for estimating a variance inflation factor from data-row correlations in order to correct for test-statistic correlations.

In this paper, we derive an analytical formula for the test-statistic correlation as a function of the data-row correlation for a general class of test statistics— including the familiar two-sample t-test as a special case. We use simulation results to confirm our analytical findings. We show that 1) the test-statistic correlation is equal to data-row correlation when the test statistic is a linear combination of the observed data, but 2) in general, the test-statistic correlation is weaker than and not well approximated by the corresponding data-row correlation. In particular, our analytical formula reveal that 3) the test-statistic correlation depends on whether the test statistic has an expectation of 0 (which often corresponds to whether the null hypothesis is true). These findings urge us to give more thoughts about correlations when trying to summarize the collection of the test results.

2. Methods and Results

Suppose we have a data matrix and have applied a statistical test to individual rows of the data matrix. We will consider pairwise correlations and focus on two rows of the data matrix: X = (X₁, …, X_n)^T and Y = (Y₁,…,Y_n)^T with mean vectors μ_X and μ_Y. We will assume that the columns of the data matrix are independent so that (X_j, Y_j), j = 1, … ,n, are independent bivariate random variables: this assumption is usually reasonable in a designed experiment for two-group comparison. The mean of (X_j, Y_j) may vary across experimental units j = 1, … ,n, but we assume that the population variance-covariance structure remains the same across experimental units, that is,

$$\text{Cov}\left(\begin{array}{l}{X}_j\hfill \\ Y_j\hfill \end{array}\right)=\left(\begin{array}{cc}{\sigma }_X^2& \rho {\sigma }_X{\sigma }_Y\\ \rho {\sigma }_X{\sigma }_Y& {\sigma }_Y^2\end{array}\right)$$ (1)

for all j = 1, … ,n. We consider a general class of test statistic of the form

$$T_X=\frac{{\mathit{a}}^T\mathit{X}}{c_X{S}_X},T_Y=\frac{{\mathit{a}}^T\mathit{Y}}{c_Y{S}_Y},$$ (2)

where a is a non-zero n-vector, (c_X, c_Y) are non-random constants, and (a^TX, a^TY) and (S_X, S_Y) are independent. In particular, the familiar two-sample t-test is of this form with S_X and S_Y estimating σ_X and σ_Y respectively. So is the t-test for a regression coefficient in a linear regression model.

We want to investigate the connections between the test-statistic correlation ρ_T = Cor(T_X, T_Y) and the data-row correlation ρ = Cor(X_j, Y_j) (common to all units j). First, we present an analytical formula that relates ρ_T to ρ.

Theorem 1.

For the test statistics T_X, T_Y in (2):

$${\rho }_T=\frac{\rho {\sigma }_X{\sigma }_Y\text{\hspace{0.17em}E}\left(S_X^{-1}{S}_Y^{-1}\right)+c^{-1}{d}_X{d}_Y\text{\hspace{0.17em}Cov}\left(S_X^{-1},S_Y^{-1}\right)}{\sqrt{\left[{\sigma }_X^2\text{E}\left(S_X^{-2}\right)+c^{-1}{d}_X^2\text{\hspace{0.17em}Var}\left(S_X^{-1}\right)\right]\left[{\sigma }_Y^2\text{E}\left(S_Y^{-2}\right)+c^{-1}{d}_Y^2\text{\hspace{0.17em}Var}\left(S_Y^{-1}\right)\right]}}$$ (3)

where d_X = a^Tμ_X, d_Y = a^Tμ_Y and c = a^T a.

Proof. (c_X, c_Y) do not affect correction and can be ignored. For any (U_X, U_Y) that are independent of (S_X, S_Y), direct calculation shows that

$$\text{Cov}\left(\frac{U_X}{S_X},\frac{U_Y}{S_Y}\right)=\text{Cov}\left(U_X,U_Y\right)\text{E}\left(\frac{1}{S_X{S}_Y}\right)+\text{E}\left(U_X\right)\text{E}\left(U_Y\right)\text{Cov}\left(\frac{1}{S_X},\frac{1}{S_Y}\right),$$

and

$$\text{Var}\left(\frac{U_i}{S_i}\right)=\text{Var}\left(U_i\right)\text{E}\left(\frac{1}{S_i^2}\right)+{\left(E\left(U_i\right)\right)}^2\text{\hspace{0.17em}Var}\left(\frac{1}{S_i}\right),\text{\hspace{0.17em}for\hspace{0.17em}}i=X,Y.$$

For this theorem, we let U_X = a^TX, U_Y = a^TY, then E(U_X) = a^Tμ_X, E(U_Y) = a^Tμ_Y, $\text{Var}\left(U_X\right)={\sigma }_X^2{\mathit{a}}^T\mathit{a}$, $\text{Var}\left(U_Y\right)={\sigma }_Y^2{\mathit{a}}^T\mathit{a}$ and Cov(U_X, U_Y) = ρσ_Xσ_Ya^T a since the columns of the data matrix are assumed independent. □

To apply equation (3) in practice, we need to compute the involved moments of $\left(S_X^{-1},S_Y^{-1}\right)$, but equation (3) offers some insights without explicit calculation of those quantities.

Corollary 1. ρ_T = ρ if S_X and S_Y are constants (i.e., not random).

Proof. When S_X and S_Y are constants, $\text{Cov}\left(S_X^{-1},S_Y^{-1}\right)$, $\text{Var}\left(S_X^{-1}\right)$ and $\text{Var}\left(S_Y^{-1}\right)$ are all 0, and $\text{E}\left(S_X^{-1}{S}_Y^{-1}\right)=S_X^{-1}{S}_Y^{-1}=\sqrt{E\left(S_X^{-2}\right)E\left(S_Y^{-2}\right)}$. □

This corollary says that for z-tests, the test-statistic correlation is the same as the corresponding data-row correlation ([6] also pointed out this). This confirms the simulation results in [7]. Another intuition offered by equation (3) is that the relation between ρ_T and ρ depends on whether one or both of a^Tμ_X and a^Tμ_Y are 0—which often corresponds to whether the corresponding null hypotheses are true. When both a^Tμ_X and a^Tμ_Y are 0, equation (3) will have a simpler form

$${\rho }_T=\frac{\rho \text{E}\left(S_X^{-1}{S}_Y^{-1}\right)}{\sqrt{\text{E}\left(S_X^{-2}\right)\text{E}\left(S_Y^{-2}\right)}}.$$

Intuitively, in such cases, we can expect ρ_T ≈ ρ in large samples if S_X and S_Y are “good” estimators of σ_X and ${\sigma }_Y:\text{E}\left(S_X^{-1}{S}_Y^{-1}\right),\sqrt{E\left(S_X^{-2}\right)E\left(S_Y^{-2}\right)}$ will then both tend to ${\sigma }_X^{-1}{\sigma }_Y^{-1}$.

More generally, though, the test-statistic correlation ρ_T is not the same as the data-row correlation ρ. Next, using the important special case of two-sample t-test, we will further demonstrate that, in general, ρ_T is not well approximated by ρ, even in large samples.

For equal-variance two-sample t-test, we let $\mathit{a}={(\underset{n_1}{\underbrace{-\frac{1}{n_1},\dots ,-\frac{1}{n_1}}},\underset{n_2}{\underbrace{\frac{1}{n_2},\dots ,\frac{1}{n_2}}})}^T$, $c_X=c_Y=\sqrt{1/n_1+1/n_2}$, and $S_X^2$, $S_Y^2$ be the pooled sample variances,

$$S_i^2=\frac{\left(n_1-1\right)S_{i,1}^2+\left(n_2-1\right)S_{i,2}^2}{n_1+n_2-2},\text{\hspace{0.17em}\hspace{0.17em}for\hspace{0.17em}}i=X,Y,$$

in (2), where $S_{i,1}^2$ and $S_{i,2}^2$ are the sample variances for sample 1 and sample 2 respectively in data row i. From Basu’s lemma, (a^TX, a^TY) are independent of (S_X, S_Y). Typically, the null hypotheses to test are d_X = a^Tμ_X = 0 and d_Y = a^Tμ_Y = 0.

Theorem 2.

For the equal-variance two-sample t-test, when n = n₁ + n₂ → ∞ and n₁/n → r for some r, 0 < r < 1,

$${\rho }_T\to \frac{\rho \left(1+\beta {\delta }_X{\delta }_Y\rho \right)}{\sqrt{\left(1+\beta {\delta }_X^2\right)\left(1+\beta {\delta }_Y^2\right)}},$$ (4)

where δ_X = d_X/σ_X, δ_Y = d_Y/σ_Y , and β = r(1 − r)/2.

Proof. As n = n₁ + n ₂ → ∞ and n₁/n → r, in equation (3),

$$nc=n{\mathit{a}}^T\mathit{a}=n\left(\frac{1}{n_1}+\frac{1}{n_2}\right)\to \frac{1}{r}+\frac{1}{1-r}\text{\hspace{0.17em}and\hspace{0.17em}}{(nc)}^{-1}\to r(1-r)=2\beta .$$

The key of the proof is to determine the limits of the moments $\text{E}\left(S_X^{-1}{S}_Y^{-1}\right)$, $\text{E}\left(S_X^{-2}\right)$, $\text{E}\left(S_Y^{-2}\right)$, $\text{Cov}\left(S_X^{-1},S_Y^{-1}\right)$, $\text{Var}\left(S_X^{-1}\right)$, and $\text{Var}\left(S_Y^{-1}\right)$. By the consistency of $\left(S_X^2,S_Y^2\right)$ and the continuous mapping theorem,

$$S_i^{-1}\stackrel{p}{\to }{\sigma }_i^{-1},S_i^{-2}\stackrel{p}{\to }{\sigma }_i^{-2},\text{\hspace{0.17em}\hspace{0.17em}for\hspace{0.17em}}i=X,Y,\text{\hspace{0.17em}and\hspace{0.17em}}{S}_X^{-1}{S}_Y^{-1}\stackrel{p}{\to }{\sigma }_X^{-1}{\sigma }_Y^{-1}.$$

For large $v(=n-2),\text{E}\left(S_i^{-4}\right)={\sigma }_i^{-4}{v}^2/(v-2)(v-4)<2{\sigma }_i^{-4},\text{\hspace{0.17em}\hspace{0.17em}for\hspace{0.17em}}i=X,Y$. This implies the $S_X^{-1}$, $S_Y^{-1}$, $S_X^{-2}$, $S_Y^{-2}$ and $S_X^{-1}{S}_Y^{-1}$ are all uniformly integrable (note that $\text{E}\left(S_X^{-2}{S}_Y^{-2}\right)\le \sqrt{\text{E}\left(S_X^{-4}\right)\text{E}\left(S_Y^{-4}\right)}$, and thus

$$\text{E}\left(S_i^{-1}\right)\to {\sigma }_i^{-1},\text{\hspace{0.17em}E}\left(S_i^{-2}\right)\to {\sigma }_i^{-2},\text{for\hspace{0.17em}}i=X,Y,\text{\hspace{0.17em}E}\left(S_X^{-1}{S}_Y^{-1}\right)\to {\sigma }_X^{-1}{\sigma }_Y^{-1}.$$

In Lemma 1 in the Appendix, we show that

$$\sqrt{v}\left[\left(\begin{array}{c}{S}_X^{-1}\\ S_Y^{-1}\end{array}\right)-\left(\begin{array}{c}{\sigma }_X^{-1}\\ {\sigma }_Y^{-1}\end{array}\right)\right]\stackrel{d}{\to }N\left[\left(\begin{array}{c}0\\ 0\end{array}\right),\frac{1}{2}\left(\begin{array}{cc}{\sigma }_X^{-2}& {\rho }^2{\sigma }_X^{-1}{\sigma }_Y^{-1}\\ {\rho }^2{\sigma }_X^{-1}{\sigma }_Y^{-1}& {\sigma }_Y^{-2}\end{array}\right)\right].$$ (5)

This together with the continuous mapping theorem suggests that

$$\text{Var}\left(\sqrt{v}{S}_i^{-1}\right)=\text{E}\left[v{\left(S_i^{-1}-{\sigma }_X^{-1}\right)}^2\right]\to \frac{1}{2}{\sigma }_i^{-2},\text{for\hspace{0.17em}}i=X,Y,$$

$$\text{Cov}\left(\sqrt{v}{S}_X^{-1},\sqrt{v}{S}_Y^{-1}\right)=\text{E}\left[v\left(S_X^{-1}-{\sigma }_X^{-1}\right)\left(S_Y^{-1}-{\sigma }_Y^{-1}\right)\right]\to \frac{1}{2}{\rho }^2{\sigma }_X^{-1}{\sigma }_Y^{-1}.$$

For these moments limits to hold, we need to show that the involved moments are uniformly integrable (see, e.g., Theorem 6.2 of [13]). It is sufficient to show that $\text{E}\left[{\left(\sqrt{v}\cdot \left(S_X^{-1}-{\sigma }_X^{-1}\right)\right)}^4\right]$ is bounded for large v : in Lemma 2 in the Appendix, we show that

$$\text{E}\left[{\left(\sqrt{v}\cdot \left(S_X^{-1}-{\sigma }_X^{-1}\right)\right)}^4\right]=\frac{3}{4}{\sigma }_X^{-4}+O\left(v^{-1}\right).$$

Plugging the limiting values of $\text{E}\left(S_X^{-1}{S}_Y^{-1}\right)$, $\text{E}\left(S_X^{-2}\right)$, $\text{E}\left(S_Y^{-2}\right)$, $\text{Cov}\left(S_X^{-1},S_Y^{-1}\right)$, $\text{Var}\left(S_X^{-1}\right)$, and $\text{Var}\left(S_Y^{-1}\right)$ into equation (3) gives equation (4). □

In the Appendix, we will also explain how to compute ρ_T in finite samples for two-sample t-test. It is mainly $\text{E}\left(S_X^{-1}{S}_Y^{-1}\right)$ that is difficult to compute.

Theorem 2 (and equation (10) in the Appendix) reaffirms that the relation between ρ_T and ρ depends on (δ_X,δ_Y) = (d_X/σ_X, d_Y/σY). Figure 1 shows the contour plot of the limiting value of ρ_T when n₁ = n₂ → ∞ (r = 1/2,β = 1/8) as a function of (δ_X, δ_Y), for ρ = −0.7, −0.1, 0.1, 0.7. Note that ρ_T → ρ if d_X = d_Y = 0: typically, this means both null hypotheses are true. One can show that $|{\text{lim}}_{n\to \infty }{\rho }_T|\le \left|\rho \right|$. That is to say, in general, the test-statistic correlation is weaker than the corresponding data-row correlation.

Figure 1:

Contour plot of the limiting values of ρ_T as n₁ = n₂ → ∞. For each ρ, the asymptotic value of ρ_T is plotted as a function of (δ_X, δ_Y).

In Figure 2, we plotted ρ_T as a function of ρ when n₁ = n₂ = 3, 10 or ∞ for a few selected values of (δ_X,δ_Y). We also added simulated values of ρ_T (for n₁ = n₂ = 3, 10) to confirm our analytical findings: For each (δ_X,δ_Y) value, we let ρ vary form −1 to 1 by a step size of 0.01. For each ρ, we simulated a pair of data rows X, Y according to independent bivariate normal distributions:

$$\left(\begin{array}{l}{X}_j\hfill \\ Y_j\hfill \end{array}\right)~N\left[\left(\begin{array}{l}0\hfill \\ 0\hfill \end{array}\right),\left(\begin{array}{ll}1\hfill & \rho \hfill \\ \rho \hfill & 1\hfill \end{array}\right)\right],j=1,\dots ,n_1,$$

$$\left(\begin{array}{l}{X}_j\hfill \\ Y_j\hfill \end{array}\right)~N\left[\left(\begin{array}{l}{\delta }_X\hfill \\ {\delta }_Y\hfill \end{array}\right),\left(\begin{array}{ll}1\hfill & \rho \hfill \\ \rho \hfill & 1\hfill \end{array}\right)\right],j=n_1+1,\dots ,n_1+n_2$$

and computed the two-sample t-test statistics T_X and T_Y. H = 5000 pairs of (T_X, T_Y) were simulated and their sample correlation ${\widehat{\rho }}_T$ were shown in Figure 2.

Figure 2:

Test-statistic correlation ρ_T versus data-row correlation ρ at different (δ_X, δ_Y) values, when n₁ = n₂ = 3, 10, or ∞. The simulated values of ρ_T are also shown for n₁ = n₂ = 3 and 10. The solid (smooth) lines represent theoretical value, and dashed (jagged) lines represent simulated values.

Let ${\rho }_T^{\infty }=\text{lim}{\rho }_T$ as n₁ = n₂ → ∞. We see from Figure 2 that when δ_X = δ_Y = 0, ${\rho }_T^{\infty }=\rho$; when δ_X = 0, ${\rho }_T^{\infty }$ is a linear function of ρ; and when δ_X and δ_Y are both non-zero, ${\rho }_T^{\infty }$ is a quadratic function of ρ. These features are predictable from the analytical formula (4) in Theorem 2 and they hold approximately in finite samples if n is large. In fact, we see that when n₁ = n₂ = 10, the ρ_T values are already remarkably close to ${\rho }_T^{\infty }$.

In small samples (e.g., n₁ = n₂ = 3), there is more difference between ρ_T and ${\rho }_T^{\infty }$: ρ_T is often weaker than ${\rho }_T^{\infty }$ (i.e., $|{\rho }_T|<|{\rho }_T^{\infty }|$) with a couple of exceptions (e.g., when δ_X = ± 5, δ_Y = 5), which is reasonable since $\left(S_X^2,S_Y^2\right)$ are “noisier” in small samples and noise in general reduces correlation. When both δ_X and δ_Y are non-zero (this typically means both null hypotheses are false),ρ does not approximate ρ_T well no matter what the sample size is: |ρ| can significantly overestimate |ρ_T |. In extreme cases when δ_X δ_Y is big, ρ_T and ρ can have opposite signs.

3. Conclusion and discussion

This article discusses the relation between test-statistic correlation ρ_T and the corresponding data-row correlation ρ. Our results indicate that only in limited settings, ρ_T can be well approximated by ρ: for example, ρ_T = ρ for z-test and ρ_T ≈ ρ in large samples if both null hypotheses are true. For two-sample t-test, the relation between ρ_T and ρ will depend on (δ_X, δ_Y), the expected mean differences divided by the respective standard deviations of the data rows. When δ_X and δ_Y are both non-zero, ρ_T is a quadratic function of ρ, ρ_T can be much weaker than ρ (|ρ_T | < |ρ|), and ρ_T and ρ can sometimes have opposite signs.

Our findings have practical implications in statistical inferences aiming summarize the collection of test results. For example, our results indicate that it is not reliable to approximate the distribution of test-statistic correlations by the distribution of data-row correlations if we expect the null hypotheses to be false for a significant proportion of the rows—which is often the case in gene expression analysis. If one wants to assess the null distribution of the test-statistic p-values by permuting the columns of the data matrix, then one has to realize the permutation will also change the correlations among the test-statistic values (since (δ_X, δ_Y) values will change after each permutation). In separate ongoing work, we are delving into these and related issues to better understand the impact of test-statistic correlation on gene set enrichment analysis, where one wants to test for overabundance of DE genes in a pre-specified set ([14] is one such attempt).

Wu and Smyth [8] discussed a variance inflation factor (VIF) which is useful when estimating the variance of the sum or average of m test statistics t₁, t₂, … ,t_m when the corresponding genes (data rows) are correlated. In that paper, VIF is defined as $1+(m-1){\overline{\rho }}_T$, where ${\overline{\rho }}_T$ is the average of test-statistic correlations (i.e., ρ_T ‘s) over all pairs of data rows in the set. (If all t_i’s have the same variance τ², then $\text{Var}(\overline{t})=\text{VIF}\cdot {\tau }^2/m$). It was mentioned that ${\overline{\rho }}_T$ can be estimated by the average of data-row correlations. Our results indicate that replacing test-statistic correlations by data-row correlations will not be accurate if there are mean differences between the two groups among the data rows. For example, if we consider the two-sample t-test performed on m = 21 data rows in a matrix with correlated data rows (ρ = 0.1 for all pairs, variance σ² = 1 for all rows) and mean differences ranging from −3 to 3 (uniformly spaced, i.e., δ = −3, −2.4, −1.8, …,3 for the 21 rows) between two groups (n₁ = n₂ = 30), then the true VIF value computed using test-statistic correlations (which we can compute using asymptotic formula (4) in Theorem 2) should be 2.48; the VIF computed using the data-row correlations is 3.00, which overestimates the true VIF by 21%. In practice, we can estimate ρ_T for each pair of data rows by plugging the corresponding estimated values of ρ, d, σ values into equation (4). In the Appendix, we use a simulation to show that estimating VIF using estimated ρ_T values will outperform approximating ρ_T by the sample data-row correlations.

One reviewer asked whether our results apply to the moderated t-test where the variance estimation is based on a shrinkage method. The short answer is “no”. It is difficult to derive an analytical formula for the correlation between a pair of moderated t-test statistic values where a shrinkage method is used for estimating the variances of the test-statistic values, since information from all data rows are used for estimating the variances. Through a simple simulation where we applied moderated t-test in the limma package ([15]), we observed that the test-statistic correlations among moderated t-test statistic values depend on δ_X and δ_Y, but the relationship between test-statistic correlations and data-row correlations do not follow the analytical formula that we derived in Theorem 2. In particular, we observed that for moderated t-test, the test-statistic correlations tend to be greater than the data-row correlations in some cases where δ_X = δ_Y ≠ 0. For the usual two-sample t-test statistic, we have shown earlier that the magnitude of test-statistic correlation tends to be less than that of the data-row correlation. We included the details on the simulation settings and results of this simulation on moderated t-test in the Appendix.

In this paper, we assumed that the columns of the data matrix are independent and the explicit formula mainly focused on the two-sample t-test. We believe these are good starting points for discussing this complex issue of test-statistic correlations resulting from data-row correlations. In the future, we plan to extend our investigation into more general settings: for example, the test for regression coefficients in a generalized linear model.

The R codes for reproducing the results in this paper are available at Github: https://github.com/zhuob/CorrelatedTest.