Adjusting for Principal Components of Molecular Phenotypes Induces Replicating False Positives

Biological, technical, and environmental confounders are ubiquitous in the high-dimensional, high-throughput functional genomic measurements being used to understand cellular biology and disease processes, and many approaches have been developed to estimate and correct for unmeasured confounders...

Abstract

High-throughput measurements of molecular phenotypes provide an unprecedented opportunity to model cellular processes and their impact on disease. These highly structured datasets are usually strongly confounded, creating false positives and reducing power. This has motivated many approaches based on principal components analysis (PCA) to estimate and correct for confounders, which have become indispensable elements of association tests between molecular phenotypes and both genetic and nongenetic factors. Here, we show that these correction approaches induce a bias, and that it persists for large sample sizes and replicates out-of-sample. We prove this theoretically for PCA by deriving an analytic, deterministic, and intuitive bias approximation. We assess other methods with realistic simulations, which show that perturbing any of several basic parameters can cause false positive rate (FPR) inflation. Our experiments show the bias depends on covariate and confounder sparsity, effect sizes, and their correlation. Surprisingly, when the covariate and confounder have ${\rho }^2\approx 10\%$, standard two-step methods all have $>10$-fold FPR inflation. Our analysis informs best practices for confounder correction in genomic studies, and suggests many false discoveries have been made and replicated in some differential expression analyses.

ASSOCIATION studies of molecular phenotypes have helped characterize basic biological processes, including transcription, methylation, chromatin accessibility, translation, ribosomal occupancy, and expression response to stimuli. These tests can be performed on cis and trans genetic variants to search for functional quantitative trait loci [*QTL: eQTL (Montgomery et al. 2010; Pickrell et al. 2010), mQTL (Rakyan et al. 2011), caQTL (Degner et al. 2012), pQTL (Albert et al. 2014), rQTL (Battle et al. 2015), sQTL (Rivas et al. 2015; Li et al. 2016), reQTL (Fairfax et al. 2014; Lee et al. 2014), and iQTL (Barry et al. 2017)]. Functional measurements can also be tested against nongenetic covariates with broad genomic effects, including cell type composition (Houseman et al. 2012; Jaffe and Irizarry 2014; Rahmani et al. 2017; Yao et al. 2017), disease status [e.g., cancer (van’t Veer et al. 2002), autism (Parikshak et al. 2016), and obesity (Horvath et al. 2014)], fetal developmental stage (Colantuoni et al. 2011), and ancestry (Galanter et al. 2017). We mostly refer to gene expression for simplicity, but our arguments apply to any highly structured, high-dimensional measurements.

Unfortunately, unmeasured and unknown factors are common and often have large effects in functional genomic data, reducing power and skewing null transcriptome-wide P-values (Leek and Storey 2007; Gibson 2008). Conditioning on known confounders—like technical batch—is invaluable but incomplete. Because genetic effects are typically small, even modest confounders can induce spurious genetic associations that dwarf real signal (Leek and Storey 2007; Kang et al. 2008).

Fortunately, strong confounders induce large, low-dimensional structure in the transcriptome, which is exactly what principal components (PCs) aim to capture (as do their variants, which we collectively call CCs, for confounding components). This blessing of dimensionality motivates a two-step approach where CCs are first estimated and then conditioned on downstream as surrogates for the confounders (Alter et al. 2000; Leek and Storey 2008). Domain-specific CC methods, like surrogate variable analysis (SVA) (Leek and Storey 2007) and PEER (Stegle et al. 2010), make different assumptions about the structure of the confounders, and often outperform PCA. Two-step CC correction is an essential element of thousands of functional genomics analysis pipelines (Leek et al. 2010; Rakyan et al. 2011; Stegle et al. 2012, 2015; Albert and Kruglyak 2015).

Acknowledging the substantial benefits of CCs in many settings, in this work, we explore their adverse impact on the false positive rate (FPR). Theoretically, we derive an unappreciated source of bias created by conditioning on two types of PCs. Our results suggest the two step approach is biased whenever CCs imperfectly partition the phenotype-covariate correlation. We formalize this in a unifying, unidentified likelihood (1) that the CC methods each resolve with distinct assumptions.

We also study the bias with a range of CCs and simulations using real expression data from the GEUVADIS consortium (Lappalainen et al. 2013). First, we find no nontrivial method that avoids bias even in the simple scenario where the covariate has a small effect on a single gene and is added to white noise. In more complex data, this bias can be negligible compared to confounder-induced miscalibration. Next, we perform a series of simulations varying the number and strength of covariate effects and see substantial inflation in all CC methods when their assumptions fail. Finally, we allow confounders to be correlated with the covariate—the ordinary meaning of a confounder (Leek et al. 2010)—and find that all CCs can be severely miscalibrated; further, we show these false positives replicate out-of-sample.

Confounder Estimation and Correction

We write P molecular phenotypes measured on N samples as $Y\in {\mathrm{ℝ}}^{N\times P},$ and let $y_p={Y,}_p$ be the p-th phenotype. The primary covariate of interest is $x\in {\mathrm{ℝ}}^{N\times 1}$. We assume x and the $y_p$ are standardized to mean 0 and variance 1.

We stylize the standard two-step confounder correction as:

1. Estimate a rank-K confounder U by solving

   $$\widehat{U}:=\underset{U}{\text{arg}\hspace{0.17em}\text{min}\hspace{0.17em}}\underset{\alpha ,V}{\hspace{0.17em}\text{min}}\hspace{0.17em}{\left|\left|Y-x{\alpha }^T-U{V}^T\right|\right|}_F^2+\mathcal{P}\left(\alpha ,U\right)$$ (1)

where ${\Vert \cdot \Vert }_F$ is the Frobenius norm and $\mathcal{P}$ is some penalty representing (potentially implicit) priors on the causal and confounding patterns. Here, α and V are dummy variables.

• 2. Estimate the ${\alpha }_p$ by regressing each gene $y_p$ on x given $\widehat{U}$:

  $${\widehat{\alpha }}_p:=\mathrm{OLS}\left(y_p\sim x|\widehat{U}\right)$$ (2)

where OLS indicates the regression coefficient on x from ordinary linear regression of $y_p$ on x, $\widehat{U}$, and an intercept.

We call approaches “unsupervised” when $\mathcal{P}$ constrains $\alpha =0$ and “supervised” otherwise.

Solving (1) with $\mathcal{P}(\cdot )=0$ amounts to maximum likelihood (ML) estimation under an i.i.d. Gaussian noise model for errors in Y (Leek and Storey 2007, 2008; Stegle et al. 2010). Standard ML inference for $\widehat{\alpha }$ would then (pseudo)invert the information matrix for asymptotic standard errors that account for uncertainty in U and V. This exposes one problem with two-step confounder correction: step two conditions on a fixed $\widehat{U}$ as if it were known without error. Theoretically, this difficulty can be resolved by appealing to assumptions that ensure $\widehat{U}$ perfectly estimates U so that there is no uncertainty to propagate (Leek and Storey 2008; Wang et al. 2017).

Another difficulty for ML inference in (1) is that its solution is not unique (even when requiring, e.g., $V^{T}V=I$), as

$$x{\alpha }^T+U{V}^T=x{\left(\underset{\alpha \prime }{\underbrace{\alpha +V\mathrm{\Delta }}}\right)}^T+\left(\underset{U\prime }{\underbrace{U-x^T\mathrm{\Delta }}}\right)V^T\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\forall \mathrm{\Delta }\in {\mathrm{ℝ}}^{K\times 1}$$

Because $U\prime$ satisfies the rank-K constraint, $\alpha \prime$ can obtain the same likelihood as α: adding any vector in span(V) to any α admits an equivalent solution. [We ignore the nonidentifiability of $\left(U,V\right)$ from the product $U{V}^T$ because only span$\left(U\right)$ is used in (2).]

This nonidentifiability means all (well-defined) CCs must use nontrivial penalty functions $\mathcal{P}$; below, we describe the choices of $\mathcal{P}$ roughly made by several popular CC methods. Moreover, this nonidentification means CCs depend heavily on their chosen $\mathcal{P}$ capturing the true, unknown parameter structure. In particular, we can easily design simulations where any particular CC behaves badly. This means that choice of CC method is important in practice and should be dataset-specific.

We note that we do not claim to take significant steps toward solving this identification problem, which has been analyzed from various theoretical perspectives elsewhere, e.g. West 2003; Leek and Storey 2008; Gagnon-Bartsch and Speed 2012; Sun et al. 2012; Gerard and Stephens 2017; Wang et al. 2017.

Studied Confounder Estimation Methods

Unsupervised PCA takes $\widehat{U}$ as the top eigenvectors of $Y{Y}^T$ or, equivalently, the top left singular vectors of Y. By definition, PCA solves (1) if $\mathcal{P}$ solely constrains $\alpha =0$. The key problem is that the effect of x on Y leads PCs to partially capture x, analogous to unshielded colliders in a directed graphical model (Figure 1). That is, conditioning on genomic PCs can cause, rather than remove, bias. Related concerns arise when conditioning on heritable covariates in other contexts (Aschard et al. 2015, 2017; Day et al. 2016). This bias creates test misspecification even marginally, for each gene; in contrast, previous theory assumed marginal tests were valid and focused on correlations between tests of different genes (Leek and Storey 2008).

Figure 1.

Graphical model suggesting CC conditioning causes bias. $u_1$ is the top PC, which is indirectly captures x. Although x affects only $y_1$, conditioning on the collider $u_1$ induces spurious correlation with all other $y_p$.

We approximate the PC conditioning bias for small causal effects with textbook eigenvector perturbation theory. Conditioning on one phenotypic PC, our bias approximation for gene p coincides with the bias that naively derived from Figure 1

$$\mathrm{Bia}{\mathrm{s}}_p\approx -a{V}_{p1}{V}_{q1}$$

where a is the causal effect, q is the causally affected gene, and V are the right singular vectors of Y.

A similar result can be derived for conditioning on genotypic PCs in genome-wide association studies. In Figure 1, this stylistically corresponds to replacing the phenotypes $\left(y_p\right)$ with SNPs, the covariate (x) with the tested phenotype, and reversing the arrow from x to $y_1$. However, genotype matrices typically contain three orders of magnitude more variables than expression matrices, concomitantly reducing the bias [entries of V are $O\left(P^{-1/2}\right)$]. Intuitively, causal SNPs have much lower leverage on genetic PCs than causal genes have on expression PCs.

We also test an approach we call supervised PCA, which aims to protect $\widehat{U}$ from x by first residualizing Y on x. This solves (1) when $\mathcal{P}$ constrains α to its unconditional estimate, $x^{T}Y$. We show this method simply amplifies biases in the unconditional estimate: unsupervised PCs are too correlated with x, but supervised PCs are too uncorrelated.

We study several other approaches through simulation. SVA penalizes (1) by assuming α is sparse, which is often plausible. We used the “two-step” and “irw” algorithms to implement unsupervised and supervised SVA, respectively (“irw” requires supervision, but “two-step” is infrequently used). The “irw” version learns which genes are determined by the signal α vs. the confounder V by testing with q-values. These association strengths are used in turn to weight the relative importance of each gene inside a singular value decomposition, effectively weighting PCA toward more-confounded genes to better estimate confounders. We also tested a recent reimplementation, SmartSVA (Chen et al. 2017); we found that it performs very similar to SVA with $B=50$ iterations, but much faster.

PEER explicitly penalizes U and α through priors on their respective sizes. PEER uses automatic relevance determining priors for the factors U and V and fits parameters with variational Bayes. In simulations, its default hyperparameters perform well when x explains $\approx 1\%$ of transcriptome-wide variation but less well for larger α. PANAMA is a closely related approach that greedily adds the most relevant SNPs while learning latent factors. While PANAMA can improve performance over PEER for SNPs with large effects, it is more computationally expensive and is rarely used in practice for human datasets.

RUV and related methods estimate confounders by using only a submatrix of Y that is known, a priori, to be unaffected by the primary signal (Lucas et al. 2006; Gagnon-Bartsch and Speed 2012; Gerard and Stephens 2017; Wang et al. 2017). This prior information breaks the identifiability problem by constraining ${\alpha }_S=0$ for some subset of genes S (corresponding to a barrier penalty function for $\mathcal{P}$). Latent factors can be safely identified by restricting to the negative control data, and then their effects on other genes can be extrapolated. We assume control genes or samples are unavailable, which is common, and do not study these approaches further.

We also assess LEAPP (Sun et al. 2012), a recent method that uses sparsity assumptions to disentangle α and V. Unlike other methods we study, LEAPP provably obtains oracle performance, asymptotically and assuming that confounders are strong, signals are sparse, and noise is independent across genes modulo confounders (Wang et al. 2017).

Finally, we also test the linear mixed model-based method ICE, which uses a random effect with covariance kernel $\frac{1}{P}Y{Y}^T$ to capture confounding, and tests the fixed effects of x against each gene individually (Kang et al. 2008). Conceptually, ICE seeks a few genes that are highly correlated with x compared to typical genes. Like in RUV methods, control genes can be used to improve power, which we did not study (Joo et al. 2014).

The bias for one unsupervised PC

In this section we take Y as a *QTL plus some deterministic $Y^0$:

$$Y=x\alpha +Y^0$$ (3)

We assume $x\stackrel{\mathrm{iid}}{\sim }\mathcal{N}\left(0,1\right)$. We allow $Y^0$ to be fully general to capture all noise and confounding. This is closely related to the spiked covariance model (Johnstone 2001), though we use a general $Y^0$ in place of i.i.d. Gaussian noise. We assume ${\alpha }_q=a$ and ${\alpha }_p=0$ for $p\ne q$, and we call x a local covariate for gene q.

Ideally, the OLS step (2) would condition exactly on the true, unknown confounders. We evaluate conditioning, instead, on the top PCs of Y $\left(Y^0\right)$, which we call U $\left(U^0\right)$ and define as the top left singular vectors of Y $\left(Y^0\right)$. We compare conditioning on U to $U^0$ rather than the true confounders for two reasons. First, $Y^0$ is assumed independent of x, hence conditioning on $U^0$ does not cause bias. Second, this allows us not to assume any particular form for confounding, or even its existence.

We aim to quantify the error at gene $p\ne q$ from conditioning on the top feasible PC, $u_1$, instead of the top oracle PC, $u_1^0$:

$$\mathrm{Erro}{\mathrm{r}}_p:={\widehat{\alpha }}_p-{\widehat{\alpha }}_p^0$$

where $\widehat{\alpha }$ $\left({\widehat{\alpha }}^0\right)$ solves (2) given $u_1$ $\left(u_1^0\right)$. Since $y_p=y_p^0$ for $p\ne q$, any error can only be caused by the effect of x on U.

The bias is the expected error over x, the only randomness:

$$\mathrm{Bia}{\mathrm{s}}_p:=\mathcal{E}\left(\mathrm{Erro}{\mathrm{r}}_p\right)=\mathcal{E}\left({\widehat{\alpha }}_p-{\widehat{\alpha }}_p^0\right)=\mathcal{E}\left({\widehat{\alpha }}_p\right)-{\alpha }_p$$

We study $\mathrm{Erro}{\mathrm{r}}_p$ rather than $\mathrm{Bia}{\mathrm{s}}_p$ to focus away from the ordinary regression error due to noise and the onto the error caused by x’s perturbation of U. This enables stronger results—particularly, that $\mathrm{Erro}{\mathrm{r}}_p\approx \mathrm{Bia}{\mathrm{s}}_p$ deterministically. Because of this, we often refer to this perturbed PC conditioning error as the bias, i.e., the randomness in $\mathrm{Erro}{\mathrm{r}}_p$ is negligible.

We assume α is small so that we can use a standard approximation to the perturbed eigenvector [e.g., (Allez and Bouchaud 2012), Sec. II]: for any small E, the first eigenvector of $1/P,Y^{T}Y+E$ is approximately

$$u_1=u_1^0+{\displaystyle \sum _{j=2}^N\frac{u_1^{0^T}E{u}_j^0}{{\lambda }_1^0-{\lambda }_j^0}{u}_j^0}+O\left({\left|\left|E\right|\right|}^2\right)$$ (4)

where ${\lambda }_j^0$ is the j-th eigenvalue of $\frac{1}{P}{Y}^0{Y}^{0^T}$.

Under our assumption on α, $E=a{y}_q{x}^T+ax{y}_q^T+a^{2}x{x}^T$, giving the perturbation approximation

$$u_1=u_1^0+a{\displaystyle \sum _{j>1}\frac{{\stackrel{\sim }{y}}_{1q}{\stackrel{\sim }{x}}_j+{\stackrel{\sim }{y}}_{jq}{\stackrel{\sim }{x}}_1}{{\lambda }_1^0-{\lambda }_j^0}}{u}_j^0+O\left(a^2\right)$$ (5)

This uses simplifying definitions based on rotating with U:

$$\stackrel{\sim }{Y}:=U^{T}Y; \stackrel{\sim }{x}:=U^{T}x$$

Note that $\stackrel{\sim }{x}$ is still a spherical Gaussian random variable.

We show in Supplemental Material, Section S1.1 that this approximation can be combined with the standard two-step least squares expression for $\widehat{\alpha }$ to give

$$\mathrm{Erro}{\mathrm{r}}_p\approx -2a\overline{c}{\displaystyle \sum _{j=1}^N{\stackrel{\sim }{Y}}_{jp}{\stackrel{\sim }{Y}}_{jq}{w}_j^{\stackrel{\sim }{x}}}$$ (6)

$\overline{c}$ is a condition number for $Y^0$ and $w^{\stackrel{\sim }{x}}$ are random weights:

$$\begin{array}{l}\overline{c}:=\frac{1}{N-1}{\displaystyle \sum _{j>1}\frac{1}{{\lambda }_1^0-{\lambda }_j^0}}\\ w_j^{\stackrel{\sim }{x}}:=\frac{1}{2\left(N-1\right)\overline{c}}\frac{{\stackrel{\sim }{x}}_1^2}{{\lambda }_1^0-{\lambda }_j^0}\left(\text{for}\hspace{0.17em}j\ne 1\right)\\ w_i^{\stackrel{\sim }{x}}:=\frac{1}{2\left(N-1\right)\overline{c}}{\displaystyle \sum _{k>1}\frac{{\stackrel{\sim }{x}}_k^2}{{\lambda }_1^0-{\lambda }_k^0}}\end{array}$$

The $w^{\stackrel{\sim }{x}}$ partition the perturbation among PCs and are proportional to the (random) squared correlations between x and the PCs (i.e., ${\stackrel{\sim }{x}}^2$). The $w^{\stackrel{\sim }{x}}$ are nonnegative and sum to one in expectation. $\overline{c}$ is deterministic—depending only on the spectrum of $Y^0$—and quantifies the susceptibility of the first PC to perturbation.

These properties of $w^{\stackrel{\sim }{x}}$ mean the error in (6) is a (randomly) weighted correlation between the projections of genes p and q—the tested and the causal genes—onto the eigen-axes, i.e.

$$\mathrm{Erro}{\mathrm{r}}_p\approx -2a\overline{c}{\rho }^{w^{\stackrel{\sim }{x}}}\left({\stackrel{\sim }{y}}_p,{\stackrel{\sim }{y}}_q\right)$$ (7)

where ${\rho }^{\pi }$ is the correlation weighted by some π. In particular, if $1_N$ is a vector of 1s, ${\rho }^{1_N}\left({\stackrel{\sim }{y}}_p,{\stackrel{\sim }{y}}_q\right)=\rho \left(y_p,y_q\right)$ is the ordinary correlation between the two genes. In contrast, ${\rho }^{w^{\stackrel{\sim }{x}}}$ randomly weights the eigen-axes, but with far greatest weight on axis 1 and successively less expected weight on subsequent axes.

${\rho }^{w^{\stackrel{\sim }{x}}}$ is the only remaining randomness, so the error depends on x only through this (random) notion of correlation. And even this randomness is often negligible. First, $w_1^{\stackrel{\sim }{x}}\gg w_j^{\stackrel{\sim }{x}}$ for $j>1$ (the former is the sum over the $N-1$ latter), and this gap grows for increasingly confounded data (Figure S1). Second, $w_1^{\stackrel{\sim }{x}}$ should be very well approximated by its expectation because it is an average over $N-1$ variables.

Together, this suggests the approximations $w_1^{\stackrel{\sim }{x}}\approx \frac{1}{2}$ and $w_j^{\stackrel{\sim }{x}}\approx 0$ for $j>1$, giving a deterministic approximation to the random error. Because the error is approximately deterministic, it can immediately be recognized a bias approximation, as well:

$$\mathrm{Bia}{\mathrm{s}}_p\approx \mathrm{Erro}{\mathrm{r}}_p\approx -a{V}_{p1}{V}_{q1}$$ (8)

This uses the approximation $\overline{c}\approx 1/{\lambda }_1^0$ (Equation S3). We find (8) is accurate in a realistic simulation (Figure S2).

While ${\widehat{\alpha }}_p$ is biased conditional on q and p, this conditional bias itself has mean zero on average over q or p ($V_{,1}$ is mean zero). Nonetheless, our conditional definition of bias conveys the fact that p and q are biologically meaningful and replicable indices. Moreover, even random biases with mean zero introduce overdispersion that still causes false positive inflation.

We have not generalized these calculation to $K>1$ PCs, though we suspect an analogous result will hold after appropriately modifying w. If correct, ${\rho }^w$ will move toward ${\rho }^{1_N}$ as K grows, suggesting the correlation between causal and tested traits is a good intuitive proxy for the bias.

The bias for supervised PCs

An apparent solution is to project out x before computing PCs, which we call supervised PCA. We show this is deeply flawed, even when x has no causal effect, supporting existing simulation results (Leek and Storey 2007).

First, after residualizing x from Y, x is the bottom supervised PC, hence orthogonal to the others. Thus the unconditional OLS estimates for ${\alpha }_p$ are unchanged by conditioning on supervised PCs. Similarly, the ratio of the conditional and unconditional SEs is just the ratio of the overall regression error estimates, i.e., ${\widehat{\sigma }}_{\mathrm{cond}}^2/{\widehat{\sigma }}_{\mathrm{uncond}}^2$. Together, the ratio of t-statistics testing ${\alpha }_p=0$ is

$$\frac{t_{\mathrm{cond}}}{t_{\mathrm{uncond}}}=\frac{{\widehat{\alpha }}_{\mathrm{cond}}/{\widehat{\sigma }}_{\mathrm{cond}}}{{\widehat{\alpha }}_{\mathrm{uncond}}/{\widehat{\sigma }}_{\mathrm{uncond}}}=\frac{{\widehat{\sigma }}_{\mathrm{uncond}}}{{\widehat{\sigma }}_{\mathrm{cond}}}$$ (9)

By definition, U explains large amounts of variance in Y, making ${\widehat{\sigma }}_{\text{cond}}$ smaller than ${\widehat{\sigma }}_{\text{uncond}}$. Formally, the ratio (9) is inflated on average (over genes and x) and, in practice, is usually inflated (Figure S3 and Section S1.4).

Local covariate simulations with white noise

We now demonstrate the CC conditioning bias in a simplistic simulation using (3): the background expression $Y^0$ is drawn i.i.d. standard normal; the $x_i$ are (independently) i.i.d. standard normal; and $\alpha =a{e}_q$, so that x is a local *QTL for gene q; finally, q is drawn, independently of x and Y, uniformly from $\left\{1,\dots ,P\right\}$, and its effect a is varied over $\left\{0,.3,.6,.9\right\}$. We chose $\left(N,P\right)=\left(375,13120\right)$ to match the GEUVADIS data (see below for details).

After simulating Y and x, we test for ${\alpha }_p=0$ using either PCA, SVA, PEER, or their supervised versions to estimate $\widehat{U}$. We also test the mixed model implemented in ICE (which failed to converge in a few simulated datasets).

For 1000 independently simulated datasets, we perform one-sided Kolmogorov-Smirnov (KS) tests for deflation in the regression P-values at noncausally affected genes. A two-sided test should be used in the first step when testing for general miscalibration (Leek and Storey 2007, 2008); we, however, are testing for estimator bias, which decreases P-values, and discriminates inflation from deflation.

Figure 2 presents the QQ plots for the resulting KS P-values. Unsurprisingly, excluding all CCs (None) delivers well-calibrated P-values because we did not simulate confounders. Confirming our theory, unsupervised PCs cause noticeable bias for large a; however, no bias is detected for small a, emphasizing that the bias can be negligible for local covariates. Unsupervised PEER results are similar for small a, but for large a PEER becomes conservative (such observations require one-sided KS tests for the regression P-values). ICE is mostly conservative, especially when using REML and $a>0$. Unsupervised SVA correctly declares $K=0$ [with the permutation test from (Buja and Eyuboglu 1992)] and is thus equivalent to “None”; although this is ideal behavior, any CC method could use this (or other) tests to choose K, and analysts in practice often turn to a different method in this situation (e.g., Pierce et al. 2014).

Figure 2.

Confounder correction causes P-value miscalibration in simulations with a local effect and white noise. QQ plots show one-sided KS test P-values for the nominally null regression P-values in 5000 simulations. 2-sided KS tests of the KS P-values are in the legends. Variation explained in the causal gene is 0% (black), 30% (red), 60% (green), or 90% (blue).

The three supervised methods qualitatively share a different type of bias, growing with K and depending little on a. We theoretically characterized this for supervised PCA, but PEER and, especially, SVA seem less biased.

Overall, Figure 2 shows that all tested (nontrivial) CC methods create P-value miscalibration even in the complete absence of confounding, though the problem is small for small a.

Data availability

We used a high-quality RNA-sequencing dataset from the GEUVADIS consortium (Lappalainen et al. 2013) as a realistic simulation baseline. We aligned the raw transcript reads from the European individuals to the reference hg19 transcriptome using RSEM (Li and Dewey 2011). We removed perfectly correlated genes and quantile-normalized the rest to standard normal. The final matrix has $N=375$ samples (rows) and $P=13,120$ genes (columns) and column-means and -variances equal to 0 and 1. Its spectrum is shown in Figure S10. Supplemental material available at Figshare: https://doi.org/10.25386/genetics.7040186.

Global Covariate Simulations with Real Traits

We now simulate a global effect, meaning α is much denser and larger. We set 90% of its entries to 0 and the others to i.i.d. Gaussian with mean zero and variance such that x explains 1% of transcriptome-wide variation. Nongenetic x can easily have these sorts of effects, e.g., even early studies with small sample sizes found broad expression profiles that still inform breast cancer treatment (Sparano et al. 2015; Cardoso et al. 2016). If x were genetic, it would be an extremely strong trans-*QTL.

We now use the GEUVADIS expression for the noise $Y^0$ to make the simulation more realistic, and let $x_i\stackrel{\mathrm{iid}}{\sim }$ Binomial(2,20%). Finally, as we do not aim to match the perturbation theory here, we adopt standard practice and normalize x and columns of Y to mean 0, variance 1.

We assess empirical FPR and true positive rate (TPR) at the nominal $p=.01$ level and average over 250 independently simulated datasets (averaging before log-transforming, Figure 3, a and d). All unsupervised methods and supervised PCA are badly miscalibrated for $K\ge 10$, while other supervised CC methods, ICE, and LEAPP were calibrated. These calibrated methods had power similar to Oracle, which uses PCs of the pure noise term $Y^0$, except ICE.

Figure 3 .

Mean FPR (a–c) and TPR (d–f) (on log scale) for a simulated global α added to GEUVADIS expression using three settings. The oracle is always calibrated and often covered by other lines. Two-step CC methods are at top-right; others are bottom-right. ICE has essentially 0 FPR and is omitted from the top plots.

We then decrease the sparsity of α from 90% zeros to 5% zeros, violating the sparsity assumptions of supervised (Smart)SVA and LEAPP. This leads (Smart)SVA to roughly 10-fold FPR inflation at $K=10$ (Figure 3b) and LEAPP to lose essentially all power (Figure 3e). Supervised PEER, however, retains near-oracle power and calibration.

Next, we return to 90% sparsity in α but increase its variance explained from 1 to 25%. This apparently violates PEER’s assumptions as it is $\approx$five-fold inflated at $K=10$ (Figure 3c).

These conclusions qualitatively remain when using a $q=.01$ threshold (Storey 2003) or $p=.001$ (Figures S4 and S5).

Confounders Correlated with a Global Covariate

The previous simulations only added a causal x effect to some independent $Y^0$. We now add confounders correlated with x, which we feel is common in practice for nongenetic x (genetic x can only be confounded by population structure). For example, even within-tissue, PEER factors had ${\rho }^2$ ranging from 10% to 60% with known covariates (GTEx Consortium et al. 2015). Further, experimental procedures are often correlated with biological factors (Gilad and Mizrahi-Man 2015). And correlation between technical confounders and primary biological signal can be pernicious: tissue dissociation in quiescent muscle stem cells can resemble cellular activation from muscle injury (van den Brink et al. 2017).

We now add a confounder, u, that has correlation ρ with x:

$$Y=x\alpha +u\beta +Y^0$$

We draw each $\left(x_i,u_i\right)$ pair i.i.d. from a bivariate Gaussian with mean zero, variances equal to 1 and correlation ρ. We repeated our above pipeline, simulating 250 independent datasets and plotting the FPR in Figure 4. Here, we always take $K=12$ CCs, because $K=10$ was used originally for $Y^0$ (Lappalainen et al. 2013) and we have added two rank-one effects.

Figure 4 .

FPR and TPR at a nominal $p=.01$ level for testing a strong covariate x correlated with a confounder u. x and u have squared correlation ${\rho }^2$, and their respective variances explained are ${\sigma }_x^2=1\%$ and ${\sigma }_u^2=1\%$ (a and d), 3% (b and e), or 10% (c and f). Two-step CC methods are in the top legend; others are in the bottom legend.

We draw α as in Figure 3a: 10% of its entries are nonzero, drawn i.i.d.Gaussian with mean zero and variance such that ${\sigma }_x^2=1\%$ is the transcriptome-wide fraction of variance explained. Because u represents a confounder, we draw β i.i.d. Gaussian with variance such that ${\sigma }_u^2=10\%$. This is roughly in line with GTEx (Aguet et al. 2017), where the top 15–35 PEER factors collectively explained 59−78% of transcriptome-wide variation. We vary ${\rho }^2$ from 0 to 1 (ρ and $-\rho$ are equivalent).

The results in Figure 4 show that supervised PCA badly inflates FPR, even at $\rho =0$. The other supervised methods are nearly as inflated for $\rho >0$, except PEER for larger ${\sigma }_u^2$ and modest ρ. Unsupervised CCs also inflate FPR for ${\sigma }_u^2=1\%$, though this diminishes as ${\sigma }_u^2$ grows and u becomes near-perfectly captured. In particular, the apparently naive approach of simply ignoring confounding (None) can be less inflated than all CC methods, particularly for small ${\rho }^2$ or ${\sigma }_u^2$. This shows that, even when confounding exists, CC adjustment can cause more harm than good. Qualitatively similar patterns hold using $q=.01$ or $p=.001$ thresholds (Figures S6 and S7).

We found that ICE was always calibrated and LEAPP was always close to calibrated, with a maximum of roughly threefold inflation. The LEAPP inflation occurred for smaller ${\sigma }_u^2$ and larger ${\rho }^2$, scenarios excluded by assumptions in Wang et al. (2017). LEAPP was often more powerful than ICE, but these were typically settings where CC methods also performed well. For example, LEAPP was similar to supervised PEER/(Smart)SVA when ${\rho }^2=0$, and LEAPP was similar to unsupervised CCs and supervised PEER when ${\sigma }_u^2=10\%$ (though better calibrated). In the intermediate range (e.g., ${\sigma }_u^2=3\%$ and $.1<{\rho }^2<.5$), however, LEAPP outperformed all competitors. LEAPP was far slower than the other methods, taking $\approx$2 hr on average over simulated datasets in Figure 4, compared to $\approx$1–5 min for two-step methods and $\approx$10 min for ICE (Figure S8) nonetheless, recent re-implementations of LEAPP are faster (Wang et al. 2017).

Similar simulations can be found in Figure S2 of Leek and Storey (2008), but they reached the opposite conclusion, i.e. that SVA is calibrated even when ${\rho }^2>0$. To test if this discrepancy is due to their smaller tested data dimensions, $\left(N,P\right)=\left(20,1000\right)$, we repeated our simulations in Figure 4 after downsampling to 20 samples and 1000 genes (uniformly without replacement, and independently downsampling for each simulation). This reduced FPR for SVA, though SVA can still be inflated (e.g., $\approx$10-fold for ${\rho }^2\approx 50\%$ and ${\sigma }_u^2=10\%$, Figure S9). This remaining discrepancy is likely partially because there is, in fact, inflation in Figure S2 of Leek and Storey (2008): e.g., row 2, column 4 visually seems miscalibrated.

False positive replication

Assuming a and q are biologically determined and universal, the PCA bias we derived depends only on the top PCs of Y. As N grows large, the bias remains, precision grows, and the null is rejected for every gene if x affects even one.

Analogously, biases will be similar between datasets with similar top PCs, which occurs in the presence of strong and similar confounders, e.g. batch effects (Leek et al. 2010), population structure (Listgarten et al. 2010), cell type composition (Jaffe and Irizarry 2014), or the signal x itself if it is strong.

We empirically assess replication rates by simulating as in Figure 4 with ${\sigma }_x^2=1\%$, ${\rho }^2=.25$, and 10$\text{\%}$ of α’s entries drawn i.i.d. Gaussian. We then split the data (x and Y) into halves, test each separately, compute the replication rate as the fraction of false positive discoveries from the first half that are deemed positive in the second half, and repeat after transposing the splits. We use a significance level of $p=.01$ in each split. We independently repeat the process 250 times, ignoring initial splits without false discoveries. Splitting an existing dataset simulates worst-case confounder sharing between discovery and replication cohorts.

The left panel of Figure 5 shows the average (positive) false replication rate. Unsupervised methods perform reasonably well and supervised SmartSVA and PEER are calibrated. Either performing no confounder correction or using supervised SVA or PCA creates severe spurious replication, with nearly all false positives replicating when ${\sigma }_u^2$ is large (e.g., $>20\%$). In this section, we do not assess ICE as it has low total positive rates.

Figure 5.

False positive replication rate at a nominal $p=.01$ level. ${\sigma }_u^2$ is the confounder strength. The signal strength ${\sigma }_x^2$ is either 1% (baseline) or 10% (others). The squared signal-confounder correlation ${\rho }^2$ is 25%, or 0 for the third panel.

We then increased the signal to ${\sigma }_x^2=10\%$. Unsupervised methods performed worse, as they more readily capture the larger x effect, and supervised PEER became miscalibrated, especially for larger ${\sigma }_u^2$. Next, we reduced the confounder correlation to 0: unsupervised methods performed slightly better, while supervised methods (except PCA) became roughly calibrated. Finally, we increased the density of α to 90%, causing supervised SmartSVA to suffer similarly to PEER.

Discussion

We have evaluated unappreciated sources of bias induced by conditioning on estimated confounders in functional genomic association tests. We used a combination of theory and simulation to cover different cases of interest. Overall, no two-step CC method we evaluated generally had calibrated FPR; moreover, most studies use PCA, one of the worst-performing methods in our simulations. Although all methods behave well when their assumptions hold, and these assumptions are often reasonable in practice, confounders even modestly correlated with the primary signal can cause substantial bias. We also showed these false positives can replicate at a high rate.

*QTL studies are often performed only within local genomic windows, which are called cis-*QTL studies. In this context, genetic effects are small and restricted to nearby genes, and our results suggest that the bias induced from CC conditioning is minimal. However, unlike in our Figure 2 simulations, cis-windows may contain many highly correlated genes, and cis-*QTL often causally affect genes other than the nearest one (Zhu et al. 2016), both of which serve to inflate FPR.

*QTL studies can also be performed genome-wide, which are called trans-*QTL studies. While such *QTL are biologically central, they are difficult to reliably uncover because the signals tend to be dispersed across the transcriptome and genome. Unlike cis-*QTL, trans-*QTL can have much larger effects on CCs, which has led modern studies to diametrically opposed methodology for testing trans-*QTL. For example, Brynedal et al. (2017) do not use CCs, despite the fact that “all [significant] gene sets were significantly correlated to ... the top 20 PEER factors.” On the other hand, Yao et al. (2017) adjust for 20 PEER factors and cell type composition, and many of the resulting “hotspot” signals are in fact loci known to affect cell type composition¹. Similar to (Yao et al. 2017), (Aguet et al. 2017) adjust for PEER factors computed per tissue. We have proposed GBAT to address these and other limitations by testing gene-level trans associations (Liu et al. 2018). GBAT uses supervised SmartSVA, which performs best in our simulations without correlated confounders. This is feasible because we test only thousands of genes rather than millions of SNPs, though SNPs could be prescreened with unsupervised CCs. More generally, we recommend ICE or LEAPP when feasible: ICE was more reliably calibrated but had low power than LEAPP, while two-step CCs could perform very poorly when their assumptions are violated.

Our global covariate simulations are relevant for differential expression studies performed with linear regression and/or CC correction. Such tests have been broadly applied, including to differences between tissues, sexes, or ages. These factors can easily correlate with latent confounders when experiments are not randomized and can substantially affect the expression of many genes. This is the setting in our simulations underlying Figure 4, where no approach (except ICE) generally gives calibrated P-values and CC correction can be worse than a completely uncorrected analysis.

A key limitation of two-step approaches is that step 1 uncertainty is not propagated to the test in step 2. To address this, a multiple imputation-style approach can be used, performing step 2 on several draws from the step 1 CC posterior. We have not evaluated this concept as it is currently developed only within the RUV framework (Gerard and Stephens 2017). Related, the two steps can be integrated, which analytically conveys first-step uncertainty, though this has much greater computational cost when run genome-wide (Stegle et al. 2010; Fusi et al. 2012; Sun et al. 2012; Wang et al. 2017).

In the future, it may be useful to pursue other assumptions on $Y^0$ in the bias calculation we derived for unsupervised PCA. For example, we could use a spiked covariance model for $Y^0$, using results from Nadler (2008) to approximate the perturbed eigenvector (e.g., replacing our Equation 5 with their Equation 2.15). An advantage of our approach, however, is that we allow general correlations between traits.

In more complex scenarios, the appropriate covariate and confounding model can be unclear. For example, coexpression studies learn complex and subtle graphical models from (partial) covariance (Horvath 2011; Shin et al. 2014). But uncorrected confounders (or biased corrections) will yield statistically significant, biologically meaningless networks. Latent variable graphical models may suit this problem (Chandrasekaran et al. 2012), and a related two-step approximation was recently proposed for genomics (Parsana et al. 2017). Finally, mixed models that learn specifically-genetic graphical models may be adaptable to adjust for low-dimensional confounding (Dahl et al. 2013).