Integrative correlation: Properties and relation to canonical correlations

Abstract

The integrative correlation coefficient was developed to facilitate the validation of expression microarray results in public datasets, by identifying genes that are reproducibly measured across studies and even across microarray platforms. In the current study, we develop a number of interesting and important mathematical and statistical properties of the integrative correlation coefficient, including a unique permutation-based null distribution with the unusual property that the variance does not shrink as the sample size increases, discussing how these findings impact its use and interpretation, and what they have to say about any method for identifying reproducible genes in a meta-analysis.

1. Introduction

The integrative correlation coefficient (ICC) was developed by Parmigiani, Garret–Mayer, Anbazhagan and Gabrielson [22] to facilitate the validation of expression microarray results in public datasets, by identifying genes that seem to be reproducibly measured across studies and even across microarray platforms. It was discovered independently by JK Lee and colleagues [14] who used it to measure cross-study reproducibility in microarrays, but did not apply it at the level of individual genes at the time, and is very similar to another approach taken by HK Lee et al. [17] Although the ICC was developed for gene expression and other genomic data, it could be used to evaluate reproducibility in any high dimensional data with sufficient dependence between variables.

It is a bit difficult to crisply define reproducibility in the multi-study context, but the general idea is that if we put the same samples on two different microarray platforms, say, we are interested in those genes whose expression values are well-correlated across platforms. It being impossible to directly assess correlation when two independent sample sets are compared, the integrative correlation solution was to map out the dependence relationships between genes, within each study, and select as reproducible those genes for which the local dependence network is the same in both studies. Thus the very simple algorithm for the integrative correlation of gene x is as follows:

1. within each study, calculate the correlation between genes x and y for every y ≠ x

2. calculate the cross-study correlation of correlations over all y; a high value indicates that inter-gene relationships are the same within study

3. compare to a null distribution to select reproducible genes.

An important consideration was that the method use only gene expression data and no additional clinical data of the kind that typically drives supervised analyses in microarray studies. In this way, we avoid compromising higher level analyses that might be carried out later on reproducible genes.

Since its introduction, the method has been applied in a variety of situations to compare platforms [14], evaluate methodology [24], aid in meta-analysis of gene expression sets [10,22,30,28], and even for comparisons across species [29]. It is also a key component of the recently developed Co-expression extrapolation (COXEN) [15,16,21,25,26] approach to the development of biomarkers for drug response starting with in-vitro response data in cell lines.

In the current study, we describe a number of important and interesting mathematical and statistical properties of the integrative correlation coefficient, including a unique permutation-based null distribution with the unusual property that the variance does not shrink as the sample size increases, discussing how these findings impact its use and interpretation, and what they have to say about any method for identifying reproducible genes in a meta-analysis.

A number of practical issues that can arise when matching similar but not identical variables across studies are not explicitly dealt with here, such as how to handle multiple probes per gene and how strictly the studies should be cross-annotated. For further discussion of these issues we refer the interested reader to these publications [14,22,5,7,30]. In the current study, it is enough to assume that we have answered these questions one way or another, and have a complete cross-study annotation.

2. Illustrative example

To demonstrate the method, we present a semi-simulated example. To create a pair of studies with a variety of ICCs we started with a single, large, breast cancer study [27] hybridized to the Affymetrix hgu133a expression array, splitting it into two independent sets of 100 samples each, measured on 3000 selected genes. The distribution of those genes is the key to the simulation. One third of the genes used for the simulation have a single probe on the array, which is used to represent the gene in both simulated studies. These are expected to show a high level of reproducibility. One half of the genes are represented by 2 different probes, with the data split so that the two simulated studies use different probes. The remaining one sixth of the genes are simulated as annotation errors, by selecting probes for different and unrelated genes in each simulated study. The null and observed distributions of integrative correlations are shown in Fig. 1 and Table 1. To define reproducible genes in Table 1, we use the 99th percentile of null integrative correlations as a cutoff.

Fig. 1.

The distribution of null integrative correlations are plotted in blue, with the 99th percentile marked as a vertical line. The observed integrative correlations are plotted in red by simulation group, and show the expected decline as the probes used in each study become more independent.

Table 1.

Each row, corresponding to one of 3 levels of reproducibility built into the simulation scheme, shows the proportion of genes that are deemed reproducible in a comparison to the null distribution.

	Non-reproducible genes	Reproducible genes
Mis-annotated genes	82.2%	17.8%
Double-probed genes	48.6%	51.4%
Single-probed genes	17.9%	82.1%

It is encouraging that the reproducibility rates are highest for the genes for which the same probe is used in both studies are found to be reproducible. Although it is not possible to determine what this number should be if all is well, we cannot expect 100% reproducibility. For example, the expression levels of some genes will not exhibit meaningful biological variation between samples, and so should not show significant correlation to other genes, in this respect 82% seems high. The genes simulated to represent annotation errors appropriately have the lowest rates of reproducibility, but again the rate is notably high with 18% of those genes found to be reproducible using the 99th percentile of null ICCs as a threshold. Again, there is no theoretical quantity to compare this to, but we can speculate as to some of the possible reasons. There is a high degree of connectivity across genes, the integrative correlation in fact deliberately exploits this feature of the genome by comparing gene interaction networks across the studies, so it is not surprising that the distribution of correlations between randomly selected pairs of genes should exceed a well-defined null distribution. What is likely a more significant cause is discussed in greater detail in Section 3.3—the method can be susceptible to batch effects, and similar artifacts which can make unexpressed genes appear to be correlated. And in fact when we apply a correction that is introduced in that section, all 3 rates drop significantly to 42.7% for reproducible genes, 20.9% for the intermediate group, and 4.8% for non-reproducible genes.

To illustrate the benefits of using integrative correlation coefficients to select reproducible genes, we extended the simulation by including 2 real, phenotypic groups in each of the simulated studies, and searched for differentially expressed genes, comparing results across study by integrative correlation level. Two important classes of breast cancer are determined by the expression level of the estrogen receptor gene. Those cancers that express the gene, here called ER+ tumors, tend to be less aggressive than the ER− tumors, and the two types are sufficiently different at the molecular level that we can be confident that the 3000 gene simulation will include a number that are actually differentially expressed in this phenotype. Accordingly, we used t-statistics to identify differentially expressed genes in each study. Fig. 2 shows the cross-study correlation between t-statistics, overall, and after stratifying genes by integrative correlation. Although the phenotype was not used in calculating the ICC, the correlation between t-statistics improves significantly when non-reproducible genes are filtered out. In this example, as above, the 99th percentile of null integrative correlations was used as the threshold for calling genes reproducible.

Fig. 2.

T-statistics comparing expression samples for ER− breast tumors and ER+ breast tumors were calculated for each probe in each study Probes are grouped according to ICC. Good probes have an ICC greater than the 99th percentile of the null distribution, while bad probes have ICC values below that threshold.

2.1. A reformulation

It is very easy to understand the ICC as a correlation of correlations, but reformulating it in matrix notation brings a number of important statistical and algorithmic features of the method into the spotlight.

To get notation and underlying assumptions out of the way first, suppose that S_a and S_b are two microarray studies, with sample sizes of n_a and n_b respectively, and a total of m common genes. The within-study correlation of two genes can be written as the inner-product of appropriately standardized variables, so we will go ahead and assume that the data is already standardized, so that in study S_a for example, each gene is assumed to have a mean expression value of 0, and a variance of 1/n_a. For a particular gene x, we will use x_a to denote the standardized expression values of the gene in study S_a and let A describe the m − 1 × n_a matrix of standardized expression values for all other genes. Notation for study S_b is, of course, identical. It would perhaps make more sense to use A_x and B_x since these matrices depend on the choice of x, but we will sacrifice the subscripts for cleaner notation (and in anticipation of soon eliminating the dependence on x). We have chosen to represent x_a and x_b as row vectors, to conform to standard representation in gene expression studies rather than as the columns that would be more typical in matrix analysis.

Finally, we note that mean-centering a vector is an idempotent linear operation, and so can be easily transcribed into matrix notation. Thus, according to common usage I_m will denote the m × m identity matrix and cE_m the m × m matrix with every element equal to c. In this notation, if υ and w are two random vectors of length m then [I_m − 1/mE_m]υ = υ − ῡ, and the quadratic form is the same, υ^t [I_m − 1/mE_m]w = cov(υ, w). For notational simplicity, we will let ℓ stand as shorthand for row centering operator, [I_m − 1/E_m].

Thus, we can write the integrative correlation coefficient for gene x in studies S_a and S_b as

$$\frac{x_a{A}^t\ell B{x}_b^t}{\sqrt{x_a{A}^t\ell A{x}_a^t}\sqrt{x_b{B}^t\ell B{x}_b^t}}$$ (1)

and define integrative covariance and integrative variance as $x_a{A}^t\ell B{x}_b^t/(m-1)$ and $x_a{A}^t\ell A{x}_a^t/(m-1)$ respectively, where ℓ is an m − 1 × m − 1 matrix, A is m − 1 × n_a and Bm − 1 × n_b. The two vectors x_a and x_b have lengths n_a and n_b respectively.

Although formally the data matrices A and B do not include gene x and so depend on the choice of gene, it makes little difference in practice whether x is deleted. The downside to leaving x in is that the correlation of x with itself is of course 1 in each dataset, so the second correlation of correlations will be slightly inflated, but unless the number of common genes, m, is very small, the effect is negligible. In contrast, the positive consequences can be substantial, and include, perhaps most dramatically, a notable reduction in computational complexity.

Here the key is that the central term in the integrative variance of x_a is the n_a × n_a matrix of covariances between samples A^tℓA, while the integrative covariance is centered around the analogous n_a × n_b matrix A^tℓB in which the i, j-th entry is the covariance between sample i of study S_a and sample j of study S_b. When integrative correlation is calculated with each gene x deleted in turn, it is necessary to produce m very slightly different versions of these matrices, and the savings obtained by doing it once and for all are remarkable.

2.2. Interpretations

So, once again what does it mean for a gene to be reproducible across studies? Suppose that in study S_a, sample i expresses gene x most highly and imagine that we have some plausible rule (not overly dependent on x itself) for identifying those study S_b samples that are most like i in terms of gene expression pattern. Then it should be a very good sign if x is also very highly expressed in those samples. Thus, the notion of reproducibility captured in this example is based on the simple idea that similar samples should have a similar expression for a reproducible gene. This is exactly what the integrative correlation measures. The cross-study matrix of covariances between samples A^tℓB describes the similarity between samples, and pre-and post-multiplication by x_a and $x_b^t$ scores the extent to which similar samples have a similar expression for gene x.

Put another way, the integrative covariance is a weighted, cross-study covariance for gene x, in which each sample in study S_a is paired in turn with each sample in S_b, but the pairings are weighted by the sample similarity measures so that pairings between similar samples contribute more to the final sum. This idea is represented graphically in Fig. 3 in which two genes with very different integrative correlations are shown. In each of these plots, n_a × n_b points are laid out in a grid, according to the expression values of the gene in each study. The plotting points vary in size, in proportion to the square of the covariance of each pair of samples. We prefer the square in this situation because it makes the points with the lowest covariances disappear almost completely, showing the patterns of co-expression for the remaining sample pairs to better effect.

Fig. 3.

In each of these plots, n_a × n_b points are laid out in a grid, according to the expression values of the gene in each study. The plotting points vary in size, in proportion to the square of the covariance of each pair of samples, but the same distribution of sizes is seen in each figure. We chose to use the square because it makes the points with the lowest covariances disappear completely, showing the patterns of co-expression for the remaining genes to better effect. The gene shown on the left has an integrative correlation near 0.5, and the largest points tend to fall along the main diagonal. The gene plotted on the right has a correlation near zero. The amount of white space around the margins is the most prominent feature of this plot; this is due both to a lack of points in those regions, and because the samples having the most extreme expression values for this gene in each study are very poorly correlated, so the few points that do fall around the margins are extremely small.

This view of the ICC has a few important implications for the usage and interpretation of the measure. Suppose that the expression level of gene x is by itself completely deterministic of some phenotype P, and that gene x is measured perfectly in both studies, and closely linked to P as expected. By any reasonable definition, the effect of the expression in x should be called reproducible, but the integrative correlation coefficient will not be significant for this gene unless the phenotype is for some reason broadly associated with the expression of many other genes as well. This is an unfortunate, but necessary characteristic of a method that does not take phenotype into account. This is a highly artificial example, but illustrates the point that genes are found to be reproducible by the ICC to the extent that their expression patterns recapitulate broad transcriptional patterns in the data. Below, we propose a variation on ICC that takes advantage of this characteristic of the method by basing the calculation of sample covariances on genes in a pathway, to specify the concept of reproducibility to a specific pathway rather than to global patterns of gene expression. We have seen a related problem arise in the presence of batch effects, which may dominate the co-expression patterns of large numbers of genes that otherwise show very little variation, enormously inflating their integrative correlations. Below, in Section 3.3 we discuss a modification that makes the method robust to this situation, illustrating the problem and solution with an example of methylation data compromised by batch effects.

3. Statistical properties of the integrative correlation coefficient

In this section, we describe several telling characteristics of the integrative correlation coefficient, indicating along the way where open questions remain.

3.1. Upper bounds on integrative correlation coefficients

A slightly different view of the ICC puts it in context in relationship to canonical correlation theory and multiple linear regression. The terms x_aA^t and $B{x}_b^t$ represent linear combinations of the columns of A and B respectively with coefficients defined by the expression values of gene x, and the integrative correlation for x is simply the Pearson correlation of these two linear combinations. Accordingly we can look to canonical correlation theory, which is concerned with characterizing maximally correlated, linear combinations of sets of random variables, for the largest integrative correlation that can be obtained between two studies. Similarly, multiple linear regression can be used to calculate particular, gene-specific upper bounds.

Canonical correlation is typically applied where the same samples have been measured on two different but related sets of variables, with the goal of finding the combinations of the variables in each set that maximize the correlation of the two. The solution is obtained via an eigenvalue decomposition of appropriately selected matrices, where the largest eigenvalue is the square of the maximum correlation that can be obtained, and a pair of corresponding eigenvectors describe the linear combinations that achieve that maximum. Subsequent eigenvalues and eigenvectors describe the maximal correlations that can be obtained over nested, orthogonal subspaces. To apply it here, it is necessary to reverse the traditional roles of the samples and variables since we are correlating linear combinations of samples over genes.

Theorem 1

Integrative correlation coefficients are bounded from above by the largest canonical correlation, calculated across studies, over common genes.

Proof

Having cast the ICC as a correlation between linear combinations of two sets of random vectors, the fundamental result is immediately established, in that the largest canonical correlation is the maximum possible correlation between linear combinations of samples from each set.

There is, however, one issue that needs to be addressed. Since x_a and x_b are centered in advance, the coefficients for ICC-defined linear combinations necessarily sum to 0, while the calculated canonical correlation is not subject to the same restriction. Thus it is possible that the upper bound calculated in this way is unachievable.

As it turns out, however, the canonical correlation does provide a tight upper bound, it does not matter whether x − a and x_b are centered since each row of A and B already is. Indeed, suppose that x_a is not centered, having mean=μ ≠ 0, and define $x_a^{\prime }=x_a-({\mu }_a\dots {\mu }_a)$. Then

$$x_a{A}^t=(x\prime +({\mu }_a\dots {\mu }_a))A^t$$ (2)

$$=x\prime A^t+({\mu }_a\dots {\mu }_a)A^t$$ (3)

$$=x\prime A^t+{\mu }_{a}0$$ (4)

$$=x\prime A^t,$$ (5)

since each row of A has mean 0.

In the same way, multiple linear regression offers gene-specific upper bounds on the integrative correlations, though these are necessarily conditional on one set of study-specific expression values.

Theorem 2

Define y = x_aA^t ℓ and find β̂_x as the least squares solution to the linear equation

$$y=\ell B{\beta }_x^t+\epsilon .$$

Then the standardized vector (β̂_x − mean(β̂_x))/var(β̂_x) is the linear combination of study 𝒮_b samples that would maximize the ICC, and the square root of the multiple R² is an upper bound on the conditional integrative correlation coefficient for the gene.

Proof

In 1966, Chow [4] showed that the least squares solution to a multiple linear regression problem maximizes the square of the correlation between the response variable Y and linear combinations of the predictors X. As in Theorem 3, the limitation in ICC to centered random variables is not a restriction.

3.2. The null distribution

When the integrative correlation coefficient was originally described [22], a null distribution was generated by calculating a full set of ICCs after individually permuting each row of each data matrix. This might be done several times and the results concatenated, depending on how thoroughly one wishes to characterize the distribution. After such a permutation, however, all samples are independent and so the elements of the cross-study sample-wise covariance matrix A^tℓB all estimate zero, while the within-study covariance matrices A^tℓA and B^tℓB are diagonal. The resulting null distribution shows too little variation, estimating ICCs for genes measured in unrelated studies, rather than approximating what we would see if a single unreproducible gene were evaluated in the current studies.

The solution we adopted was to replace x and y in Eq. (1) with permuted versions of the same while leaving the covariance matrices untouched, thus modeling the effects of comparing unrelated expression values within the actual context of the two studies under consideration. Alternatively, x and y can be drawn by sampling independently from the standard normal, or any other distribution, simulations, not included here, show that results are very similar no matter how the random variates are drawn.

3.2.1. Asymptotic null distribution

The large sample distributional properties of the null distribution are one of the most interesting aspects of the ICC method. It turns out that with a few reasonable assumptions on the distributions of the sample expression profiles, calculating the Pearson correlation between random linear combinations of the samples from each study is asymptotically equivalent to calculating the Pearson correlation between randomly selected, individual samples from each study.

The very simple form of the resulting asymptotic distribution is derived in Theorem 3 below, and as we go on to demonstrate in real expression data in Fig. 4, can provide a reasonable approximation to the formal permutation, null even with moderate sample sizes.

Fig. 4.

Quantile–quantile plots compare the observed null distribution, on the x-axis, to the asymptotic Gaussian null, on the y-axis of each plot in the figure. The number of genes common to the two studies increases by row, from 300 to 3000, while the number of samples in each study increases by column, from 5 to 100.

Theorem 3

Assume that

1. samples in each study are i.i.d. from possibly study-specific multivariate distributions with finite 4th moments

2. these distributions are such that after each study is standardized by gene to have mean = 0 and variance = 1, cross-study correlations of samples have a mean of zero, and a variance of σ²

3. to generate the null distribution, independent gene expression values are drawn as i.i.d. from a N (0, 1) distribution

then as n_a, n_b → ∞ the null distribution converges to N(0, σ²).

Notes: It is not necessary that all samples in both studies to be drawn from the same multivariate distribution. This would be an unrealistic requirement for a microarray study involving multiple platforms.

Proof

We will use classical characteristic function arguments to show convergence in distribution to the described normal distribution. Reflecting the principle that the null distribution models the effects of comparing unrelated expression values within the actual context of the two studies under consideration, the data matrices A and B are treated as constants for much of argument, eventually invoking the strong law of large numbers to pass to the limit and complete the proof. The characteristic function of the integrative covariance is written as

$$f(t)=E[\text{exp}(\mathit{\text{it}}{x}_a{A}^{t}B{x}_b^t)],$$

but a few notational changes will facilitate the arguments that follow. To avoid confusion between similarly named variables, we will call the random genes x_a and x_b by the names Z and W respectively. Genes are indexed by j ∈ 1 : G and rows of the gene data matrices using a superscripted index, so that A^(j) refers to the row vector for the jth gene in study A. Samples are indexed as u and υ in studies A and B respectively, and sample sizes denoted as N_a, N_b. With those changes,

$$f(t)=E\left[\text{exp}\right(\mathit{\text{it}}\frac{1}{G}\sum _{j=1}^G\sum _{u,\upsilon }\frac{Z_u{A}_u^{(j)}}{\sqrt{N_a}}\frac{W_{\upsilon }{B}_{\upsilon }^{(j)}}{\sqrt{N_b}}\left)\right]$$

which can be rewritten as a double expectation,

$$E\left[E\right[\text{exp}\left(\mathit{\text{it}}\frac{1}{G}\sum _{j=1}^G\sum _{u,\upsilon }\frac{Z_u{A}_u^{(j)}}{\sqrt{N_a}}\frac{W_{\upsilon }{B}_{\upsilon }^{(j)}}{\sqrt{N_b}}\right)\left|W\right]].$$

Pulling out the sum over u, this becomes

$$E\left[E\right[\text{exp}\left(\sum _u\mathit{\text{it}}{Z}_u\right(\frac{1}{G}\sum _{j=1}^G\sum _{u,\upsilon }\frac{A_u^{(j)}}{\sqrt{N_a}}\frac{W_{\upsilon }{B}_{\upsilon }^{(j)}}{\sqrt{N_b}}\left)\right)\left|W\right]].$$

Since Z_u is a standard normal random variable, the normal characteristic function can be substituted for the inner, conditional expectation, where $t={\sum }_{j=1}^G{\sum }_{\upsilon }{A}_u^{(j)}w{B}_{\upsilon }^{(j)}$ giving

$$f(t)=E\left[\text{exp}\right(-\sum _u\frac{t^2}{2}{\left(\frac{1}{G}\frac{1}{\sqrt{N_a}}\frac{1}{\sqrt{N_b}}\sum _{j=1}^G\sum _{\upsilon }{A}_u^{(j)}{W}_{\upsilon }{B}_{\upsilon }^{(j)}\right)}^2\left)\right].$$

Distinguishing indices and expanding the square further gives,

$$E\left[\text{exp}\right(-\frac{t^2}{2}\frac{1}{N_a}\frac{1}{N_b}\frac{1}{G^2}\sum _u\left(\sum _{\upsilon }\sum _j{W}_{\upsilon }{A}_u^{(j)}{B}_{\upsilon }^{(j)}\right)\left(\sum _{\upsilon \prime }\sum _{j\prime }{W}_{\upsilon \prime }{A}_u^{(j\prime )}{B}_{\upsilon \prime }^{(j\prime )}\right)\left)\right],$$

and replacing the sum over j with an inner product yields

$$E\left[\text{exp}\right(-\frac{t^2}{2}\frac{1}{N_a}\frac{1}{N_b}\frac{1}{G^2}\sum _u\left(\sum _{\upsilon }{W}_{\upsilon }{A}_u^t{B}_{\upsilon }\right)\left(\sum _{\upsilon \prime }{W}_{\upsilon \prime }{A}_u^t{B}_{\upsilon \prime }\right)\left)\right],=E\left[\text{exp}\right(-\frac{t^2}{2}\frac{1}{N_a}\frac{1}{N_b}\frac{1}{G^2}\sum _{\upsilon }\left(\sum _u{W}_{\upsilon }{A}_u^t{B}_{\upsilon }\right(\sum _{\upsilon \prime }{W}_{\upsilon \prime }{A}_u^t{B}_{\upsilon \prime }\left)\right)\left)\right]=E\left[\text{exp}\right(-\frac{t^2}{2}\frac{1}{N_b}\frac{1}{G^2}\sum _{\upsilon }\left(W_{\upsilon }{A}_{.}^t{B}_{\upsilon }\right(\sum _{\upsilon \prime }{W}_{\upsilon \prime }{A}_{.}^t{B}_{\upsilon \prime }\left)\right)\left)\right].$$

But the sum over υ and υ′ can be decomposed into two components

$$\sum _{\upsilon }\left(W_{\upsilon }{A}_{.}^t{B}_{\upsilon }\right(\sum _{\upsilon \prime }{W}_{\upsilon \prime }{A}_{.}^t{B}_{\upsilon \prime }\left)\right)=\sum _{\upsilon }\left(W_{\upsilon }{A}_{.}^t{B}_{\upsilon }\right(\sum _{\upsilon \prime =\upsilon }{W}_{\upsilon \prime }{A}_{.}^t{B}_{\upsilon \prime }\left)\right)+\sum _{\upsilon }\left(W_{\upsilon }{A}_{.}^t{B}_{\upsilon }\right(\sum _{\upsilon \prime \ne \upsilon }{W}_{\upsilon \prime }{A}_{.}^t{B}_{\upsilon \prime }\left)\right)$$

with W_υ, W_υ′, B_υ, B_υ′ and A_. all independent when υ ≠ υ′. So by conditioning on A_., it is seen that ${\sum }_{\upsilon }(W_{\upsilon }{A}_{.}^t{B}_{\upsilon }({\sum }_{\upsilon \prime \ne \upsilon }{W}_{\upsilon \prime }{A}_{.}^t{B}_{\upsilon \prime }))$ estimates 0. The other term, corresponding to υ = υ′, simplifies to ${\sum }_{\upsilon }{W}_{\upsilon }^2{G}^2{\hat{\sigma }}_{\upsilon }^2$, where ${\hat{\sigma }}_{\upsilon }^2$ is the empirical variance of the correlation between sample υ in study B and an arbitrary sample in study A. To simplify notation further, we will denote the portion of the sum that estimates 0 as ε_υ,υ′, so that f (t) can be rewritten as

$$E\left[\text{exp}\right(-\frac{t^2}{2}\frac{1}{N_b}\sum _{\upsilon }{W}_{\upsilon }^2{\hat{\sigma }}_{\upsilon }^2+{\epsilon }_{\upsilon ,\upsilon \prime }\left)\right]=E\left[\text{exp}\right(-\frac{t^2}{2}\frac{1}{N_b}\sum _{\upsilon }{W}_{\upsilon }^2{\hat{\sigma }}_{\upsilon }^2)\text{exp}({\epsilon }_{\upsilon ,\upsilon \prime })].$$

Since ${\sum }_{\upsilon }{W}_{\upsilon }^2$ is a ${\chi }_{\sqrt{N_a}/2}^2$ rv, we substitute in the ${\chi }_{\sqrt{N_a}/2}^2\mathrm{c}\mathrm{h}.\mathrm{f}.{(1-2t{\sigma }^2)}^{-{\sqrt{N_a}}^2/2}$ to give

$$\prod _{\upsilon }{(1+\frac{t^2{\hat{\sigma }}_{\upsilon }^2}{N_b})}^{-1/2}$$

which converges to exp(−t²σ²/2) implying a $\sqrt{N_a}(0,{\sigma }^2)$ limiting distribution.

We can extend the illustrative example to demonstrate rates of convergence to the asymptotic null distribution. Recollect that we used a publicly available dataset to simulate two studies, each with 100 samples, and 3000 common genes. The quantile–quantile plots in Fig. 4 show how well the asymptotic null distribution approximates the observed null distribution for 5, 25, or 100 samples per study, and 300, 1000 or 3000 genes.

What is perhaps most interesting here that the variance of the null distribution does not shrink as the sample size increases. In most testing situations, it is possible to reach statistical significance by beating a null distribution in a large-sample experiment without achieving the practical significance of a meaningfully large effect size, but this setting is very different. With appropriately chosen values for the cutoff, the measure of reproducibility suggested here can be used to identify genes that are both statistically significant and practically significant.

3.3. A variation on integrative correlation

As it is described in Section 2.2, “the cross-study matrix of covariances between samples A^tℓB describes the similarity between samples, and pre- and post-multiplication by x_a and $x_b^t$ scores the extent to which similar samples have similar expression for gene x”. This points the way to a class of extensions of standard ICC in which a different measure of sample similarity is used in place of the between-sample covariance matrix. This could be as simple as restricting the set of variables used in calculating the between-sample covariance matrix or could utilize external variables to establish sample similarity, perhaps non-molecular clinical characteristics of each patient, for example. Depending on how the similarity matrix is calculated, the connections to canonical correlation theory and regression analysis, as well as the asymptotic, null distribution theory may not be valid, though the suggested null distribution can still easily be determined in all cases.

We see this as a viable solution to some of the limitations described in Section 2.2. The case of a single gene determining a phenotype on its own may be a lost cause, but in a less extreme case, one might base the sample-wise covariance matrix on genes in a particular pathway, to focus on the co-expression patterns most relevant to the pathway function, and elevate related genes, whose expression patterns may be very rare within the whole genome, to more prominent positions in the integrative correlation analysis.

This approach is demonstrably effective when the integrative correlation coefficient is applied in the presence of batch effects, which, though they may be of small absolute magnitude, can create the illusion of widespread coordination among large numbers of otherwise unrelated, even unexpressed, genes. This seems to be a particular problem in copy number or promoter methylation array data, where the vast majority of genes can be expected to have 2 copies in every sample (or to be unmethylated in every sample). A simple but crude fix would be to base the between-sample covariance matrix on the subset of the genes with the highest sample to sample variation, excluding those in which small batch effects represent the only variation from the sample similarity calculation, although integrative correlations are still calculated for all genes. Slightly more elegant is to reduce the influence of these genes by centering, but not scaling each gene before calculating the between-sample covariance matrices, so each gene makes a contribution proportional to its variance. This is what we do below.

We will illustrate the method using data from the Cancer Genome Atlas (TCGA) data, where samples are processed in several batches, with well documented effects [18]. Fig. 5 illustrates the problem, and our solution, in TCGA lung cancer promoter methylation data. In this data, methylation takes values in the interval [0, 1], interpreted as the proportion of tumor cells in which the gene is methylated, and ICC is calculated between two tumor types, adenocarcinomas and squamous cell carcinomas. Fewer than 25% of the genes show any appreciable variation in this index and the vast majority are in fact completely unmethylated in all normal and tumor samples in either of the tumor types. The batch effects show up very clearly as patterns common to all of these genes, completely determining the between-sample covariance matrix. The standard integrative correlation exhibits a truly embarrassing preference for low variance genes, assigning high coefficients to nearly all genes with standard deviation below 0.03, a value that can be achieved if 1 sample in 100 is 30% methylated, the rest completely unmethylated.

Fig. 5.

Each heat map shows TCGA lung cancer methylation data. Squamous cell carcinomas are on the left, adenocarcinomas on the right. In each, genes are represented as rows while samples are represented as columns. Methylation level is represented by color intensity on a yellow–blue scale, where bright yellow spots have very low methylation levels (≤20%), while bright blue spots have very high levels of methylation (≥70%). The top row of figures illustrates the difficulty that can arise when standard integrative correlations are used in this data. The genes with the highest integrative correlations, shown here, include many that are completely unmethylated in all samples, in which batch effects too small to be seen on this 4 color scale dominate the correlation structure. The bottom row includes a similar number of genes, again those having the highest integrative correlations, where this time the sample similarity matrix is calculated without scaling genes so that gene contributions are proportional to their variance. The scatterplots in the rightmost column show the relationship between the s.d. of each probe, shown on the horizontal axis, and the integrative correlations on the vertical axis. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

By weighting genes in proportion to their variance, however, we are able to redefine sample similarity completely, so that genes with substantial sample to sample variation, and potentially interesting methylation patterns common to both studies achieve the highest integrative correlations. Fig. 6 shows the differences more clearly. ICC drops dramatically for nearly all genes with standard deviations below 0.03 and increases slightly for the vast majority of genes with standard deviations above 0.10. This elegant solution establishes a clear proof of principle for a valuable class of extensions to the standard integrative correlation coefficient.

Fig. 6.

The difference between filtered ICC and ordinary ICC for each gene is plotted on the vertical axis, the standard deviation on the horizontal axis.

4. Discussion

The meta-analysis of genetic data, and more generally, the integration of data from multiple sources, is a rapidly developing research problem in bioinformatics, with a number of recent publications discussing principles, risks and benefits, or proposing or evaluating methods [11,19,3,9,24,30,12,13]. These publications reflect a variety of goals: not only do we wish to increase sample sizes by combining similar studies, but also want to translate findings from cell lines to primary tumors, integrate expression data with copy number, protein or methylation and even from animal models to humans. At the same time, the very concept of the gene itself is becoming more complex [2,23,20] with recent studies suggesting that genes can be spread of very large genomic regions, with substantial overlap of other genes and regulation by transcription factors binding to sometimes quite distant sites [8]. Other studies report that a great deal of non-coding RNA is expressed and apparently plays regulatory roles [6], and identifying possible roles for pseudogenes [1]. It is more difficult and more important than we ever understood before, to be sure that the ‘gene’ we identified in one study is the same gene we are now looking at in another.

In this paper we offer an important reformulation of the ICC that brings the concept of reproducibility that is operative here into sharp focus, and makes clear the circumstances under which such genes can be accurately identified. Ultimately, the idea is a very general one, a gene is reproducible if otherwise similar samples have similar gene expression values. The same fundamental concept is operative if one calculates a within-study t-statistics with respect to some binary phenotype, and call a gene reproducible if it has similar t-statistics in the two studies; though the definition of sample similarity, and precise method of scoring reproducibility are different. As we demonstrate, by varying the sample similarity matrix, the ICC can be tuned to specific phenotypes or pathways, or to minimize the influence of batch effects.

We present a unique permutation null distribution that precisely captures the key source of variation, and prove a very interesting asymptotic result for it, showing that the null distribution is asymptotically Normally distributed, but that the variance does not necessarily shrink as the sample size increases. In consequence, the classical disconnect between statistical and practical significance, whereby vanishingly small effects become statistically significant when the sample size is large enough, is substantially reduced here. Together the null distribution and the upper-bound described in Theorem 1 provide stable anchors for interpretation of results, we have often found it useful to rescale ICC values by the canonical upper-bound in order to use the entire [−1, 1] range within which we are accustomed to consider correlations. We have also considered various ways of measuring the overall similarity between studies, using average ICC, for example. Our preferred approach, however uses metrics of the cross-study, between-sample correlation matrix in this role. The L^∞ measure, in this case as the largest absolute, cross-study correlation, is a reasonable choice and provides a more natural definition of highly reproducible than thresholds based strictly on the null distribution.

The integrative correlation coefficient reduces the problem of identifying reproducible genes to bare essentials. The concept of reproducibility operative here is very simply that a gene is reproducibly expressed across studies if similar samples in each study have similar gene expression. The method is non-parametric, and easily adapted to use any sample similarity matrix. The method does depend on correlation, or more broadly on inner products, to determine when similar samples have similar gene expression, but as this addresses a practical need to minimize cross-study differences in the location and scale of gene expression measures, we do not think it very restrictive. The default version is based on Pearson correlation, but by calculating rank-based correlations of samples, and ranking expression within gene before standardizing, a robust Spearman version is easily implemented.

Although developed for microarray analysis, the method might be applied to any high dimensional data in which the variables can be expected to have complex dependence structure. Examples would include extensive nutrition, health or behavioral surveys, where it might be expected that cultural differences, for example, might make it difficult to compare results for some questions across populations. Nor is it necessary that the variables be continuous as long as their levels are ordered; the interpretation of the correlation coefficient as a measure of similarity is maintained in this situation.