Estimating and accounting for unobserved covariates in high-dimensional correlated data

Abstract

Many high dimensional and high-throughput biological datasets have complex sample correlation structures, which include longitudinal and multiple tissue data, as well as data with multiple treatment conditions or related individuals. These data, as well as nearly all high-throughput ‘omic’ data, are influenced by technical and biological factors unknown to the researcher, which, if unaccounted for, can severely obfuscate estimation of and inference on the effects of interest. We therefore developed CBCV and CorrConf: provably accurate and computationally efficient methods to choose the number of and estimate latent confounding factors present in high dimensional data with correlated or nonexchangeable residuals. We demonstrate each method’s superior performance compared to other state of the art methods by analyzing simulated multi-tissue gene expression data and identifying sex-associated DNA methylation sites in a real, longitudinal twin study.

1. Introduction

The development of high-throughput technologies has provided biologists with a cornucopia of genetic, proteomic and metabolomic data that can help elucidate the genetic components of phenotypes and the mediation of environmental exposures. Many of these ‘omic’ data have complex sample-correlation structure, which includes longitudinal data (Baumgart et al., 2016; McKennan et al., 2018), multi-tissue data (GTEx Consortium, 2017), data with multiple treatment conditions (Knowles et al., 2018) and data with related individuals (Martino et al., 2013; Tung et al., 2015). Longitudinal studies have even been cited as a critical area of future DNA methylation research to assess the stability of methylation marks over time (Breton et al., 2017; Martin and Fry, 2018), so it is crucial that suitable methods exist to analyze these data. An important feature of these high-throughput data is the presence of unmeasured factors that influence the measured response, which include technical factors like batch variables and biological factors like cell composition (Leek et al., 2010; Jaffe and Irizarry, 2014). When unaccounted for, these factors can bias test statistics, reduce power and lead to irreproducible results (Yao et al., 2012; Peixoto et al., 2015).

There have been a number of methods developed by the statistical community to estimate and correct for latent factors in high throughput biological data (Buja and Eyuboglu, 1992; Leek and Storey, 2008; Gagnon-Bartsch and Speed, 2012; Sun et al., 2012; Gagnon-Bartsch et al., 2013; Houseman et al., 2014; Owen and Wang, 2016; Fan and Han, 2017; Lee et al., 2017; Wang et al., 2017; McKennan and Nicolae, 2019). However, these methods make the critical assumption that conditional on both the observed and unobserved covariates, samples are independent with homogeneous residual variances, and tend to perform poorly when these assumptions are violated. The goal of this paper is to provide a provably accurate method to both choose the number of and estimate latent factors from the measured correlated data so that downstream inference on the effects of interest is as accurate as when all latent factors are observed. To the best of our knowledge, this is the first method to estimate and correct for latent covariates in high throughput biological data with correlated samples.

We use DNA methylation quantified in p ≈ 8 × 10⁵ methylation sites (CpGs) in n / 2 = 183 unrelated individuals at birth and age 7 from McKennan et al. (2018) as a motivating data example, although we analyze other methylation data with a more complex covariance structure in Section 6. The aim of that study was to jointly model methylation at birth and age 7 to determine if the effects due to ancestry on methylation levels changed or remained constant over time. If we ignore observed nuisance variables like the intercept, a reasonable model relating ancestry $\mathit{A}\in {ℝ}^{n/2}$ to methylation levels at CpG g at birth, ${\mathit{y}}_{g,0}\in {ℝ}^{n/2}$, and age 7, ${\mathit{y}}_{g,7}\in {ℝ}^{n/2}$, in the presence of unobserved covariates $\mathit{C}\in {ℝ}^{n\times K}$ is

$$\begin{gathered} {\mathit{y}}_g={\left({\mathit{y}}_{g,0}^T{\mathit{y}}_{g,7}^T\right)}^T=\mathit{X}{\mathit{\beta }}_g+{{\mathit{C}}^{\mathcal{l}}}_g+{\mathit{e}}_g,{\mathit{e}}_g~N_n\left(0,{\mathit{V}}_g\right)(g=1,\dots ,p) \\ {\mathit{V}}_g=v_{g,1}{\mathit{B}}_1+v_{g,2}{\mathit{B}}_2+v_{g,3}{\mathit{B}}_3(g=1,\dots ,p), \end{gathered}$$ (1)

where X = A ⊗ A, β_g = (β_g,0 β_g,7)^T and β_g,0, β_g,7 are the effects of interest. ${\mathit{B}}_1\in {ℝ}^{n\times n}$ is a partition matrix that groups samples by individuals and captures the within-individual variability. ${\mathit{B}}_2\in {ℝ}^{n\times n}$ and ${\mathit{B}}_3\in {ℝ}^{n\times n}$ capture the potential difference in residual variance at birth and age 7 and are diagonal matrices with ones in the first n / 2 and second n / 2 diagonal entries, respectively. In order to avoid overestimating the residual variance or biasing our estimates for β_g, we must first estimate C and its latent dimension K. One approach would be to use methods from the economics literature that allow for dependent residuals in their theoretical arguments (Bai, 2009; Li et al., 2016; Lu and Su, 2016; Su and Ju, 2018). However, their assumptions on the homogeneity of the effects β₁, …, β_p are too restrictive to have any application in genetic data. Another strategy would be to use existing methods designed for genetic data to estimate the factors at each age separately. However, this reduces the sample size by 50% and risks underestimating K, and subsequently biasing estimates for β_g. To avoid these biases, a second option would be to estimate K and C using all n samples. However, naively estimating K with methods commonly applied to genetic data with independent and identically distributed residuals, like parallel analysis (Buja and Eyuboglu, 1992) and bi-cross validation (Owen and Wang, 2016), will typically overestimate K to be on the order of the sample size, as they will be unable to distinguish the low dimensional factors C from the high dimensional random effect. Even if K were known, the correlation among the residuals obfuscates existing estimates for C.

Our proposed method uses all of the available data to estimate K and C, and subsequently the effects of interest β₁, …, β_p, in data whose genomic unit-specific covariance can be written as a linear combination of known matrices. This covariance structure is quite general and includes longitudinal, multi-tissue, multi-treatment and twin studies, as well as studies with individuals related through a kinship matrix.

The remainder of the paper is organized as followed. We describe the data and set up the problem in Section 2 and present a detailed description of our method in Section 3. We prove its efficacy in Section 4 by proving its estimate for K is consistent, its estimate for the column space of C is nearly as accurate as when samples are independent (i.e. V_g in (1) is a multiple of the identity) and that inference on the effects of interest is asymptotically equivalent to that when C is known. In Section 5 we analyze simulated multi-tissue gene expression data with a complex, gene-dependent correlation structure that demonstrates our method’s superior performance in both choosing K and estimating C. We lastly apply our method to a longitudinal DNA methylation dataset with measurements made on pairs of twins to identify CpGs with sex-dependent methylation levels in Section 6. An R package called CorrConf that implements our method to estimate K and C is freely available on GitHub. The proofs of all theorems are in the Supplement.

2. Notation and problem set-up

2.1. Notation

We define ${\mathbf{1}}_n\in {ℝ}^n$ to be the vector of all ones, $I_n\in {ℝ}^{n\times n}$ to be the identity matrix, [n] = {1, …, n} and [x]_i to be the ith element of $x\in {ℝ}^n$. For $M\in {ℝ}^{n\times m}$, we let [M]_ij be the (i, j) element of M, and define P_M and $P_M^{\perp }$ to be the orthogonal projections matrices onto $\mathrm{Im}(M)=\left\{Mv:v\in {ℝ}^m\right\}$ and $M^{\perp }=\left\{v\in {ℝ}^n:M^{T}v=\mathbf{0}\right\}$. If d = dim (M^⊥) > 0, we define $Q_M\in {ℝ}^{n\times d}$ to be a matrix whose columns form an orthonormal basis for M^⊥, i.e. $P_M^{\perp }=Q_M{Q}_M^T$. Lastly, we use the relation $X\stackrel{d}{=}\mathit{y}$ if the two random variables, vectors or matrices X and Y have the same distribution.

2.2. A description of and generative probability model for the data

Let ${\mathit{y}}_g\in {ℝ}^n$ be the measured expression or methylation of genomic unit g ∈ [p] in n potentially correlated samples, and let $\mathit{X}\in {ℝ}^{n\times d}$ be the covariates of interest. When latent factors $\mathit{C}\in {ℝ}^{n\times K}$ influence the data y_g, we assume y_g is generated as

$${\mathit{y}}_g=\mathit{X}{\mathit{\beta }}_g+{\mathit{C}}_g^{\mathcal{l}}+{\mathit{e}}_g,{\mathit{e}}_g~N_n\left(\mathbf{0},{\mathit{V}}_g\right)(g=1,\dots ,p)$$ (2a)

$${\mathit{V}}_g=v_{g,1}{\mathit{B}}_1+\cdots +v_{g,b}{\mathit{B}}_b(g=1,\dots ,p)$$ (2b)

$${\mathit{y}}_{p\times n}={\left[{\mathit{y}}_1\cdots {\mathit{y}}_p\right]}^T={\mathit{\beta }}_{p\times d}{\mathit{X}}_{n\times d}^T+{\mathit{L}}_{p\times K}{\mathit{C}}_{n\times K}^T+{\mathit{E}}_{p\times n}.$$ (2c)

The gth rows of β, L and E are β_g, ℓ_g and e_g, respectively, e₁,…, e_p are independent, and our goal is to estimate and perform inference on β₁, …, β_p. We assume C is a random matrix that is independent of E, and whose mean may be a function of X in our theoretical results in Section 4. Therefore, accounting for C can be interpreted as a way to alleviate both the dependence across genomic units and potential biases when estimating and performing inference on β₁, …, β_p. We postpone discussing the distributional assumptions on C until Section 4 because, other than C being independent of E, they play no part in the methodology presented in Section 3.

The observed matrices ${\mathit{B}}_1,\dots ,{\mathit{B}}_b\in {ℝ}^{n\times n}$ describe how the n samples are related, and Model (2b) is a typical model for the variance in mixed effect models in high throughput biological data (Martino et al., 2013; Tung et al., 2015; Chen et al., 2017; Knowles et al., 2018; McKennan et al., 2018). When b = 1 and B₁ = I_n, Model (2) reduces to the model typically used to estimate β₁, …, β_p in high throughput data with independent samples (Leek and Storey, 2008; Sun et al., 2012; Lee et al., 2017; Wang et al., 2017; Gerard and Stephens, 2018; McKennan and Nicolae, 2019). We assume the unknown variance multipliers v_g = (v_g,1 … v_g,b)^T lie in the convex polytope

$$\mathrm{\Theta }=\left\{\mathit{x}\in {ℝ}^{\mathit{b}}:{\mathit{A}}_{\mathcal{E}}\mathit{x}=\mathbf{0},A_{\mathcal{I}}\mathit{x}\ge \mathit{0}\right\},$$ (3)

where the matrices $A_{\mathcal{E}}\in {ℝ}^{N_{\epsilon }\times b}$ and $A_I\in {ℝ}^{N\tau \times b}$ are the equality and inequality constraints on v_g. For example, we may know that certain multipliers must be larger than others, or that the sum of two sets of multipliers must be equal to ensure the marginal variances are the same.

In many applications, there are other observed covariates $\mathit{Z}\in {ℝ}^{n\times r}$ that may influence the response y_g but whose effects are not of interest. In that case, the model for y_g would be

$${\mathit{y}}_g=\mathit{X}{\mathit{\beta }}_g+\mathit{Z}{\mathit{\omega }}_g+{\mathit{C}}^{\mathcal{l}}{}_g+{\mathit{se}}_g,$$ (4)

and we can get back to Model (2) by multiplying y_g by ${\mathit{Q}}_z^T$, provided r does not grow with n:

$$\begin{gathered} {\mathit{Q}}_Z^T{y}_g=\left({\mathit{Q}}_Z^T\mathit{X}\right){\mathit{\beta }}_g+{\left({\mathit{Q}}_Z^{T}C\right)}^{\mathcal{l}}{}_g+{\overline{\mathit{e}}}_g,{\overline{\mathit{e}}}_g\sim N_{n-r}\left(\mathit{0},{\overline{\mathit{V}}}_g\right) \\ {\overline{\mathit{V}}}_g=v_{g,1}\left({\mathit{Q}}_z^T{\mathit{B}}_1{\mathit{Q}}_z\right)+\cdots +v_{g,b}\left({\mathit{Q}}_z^T{\mathit{B}}_b{\mathit{Q}}_z\right). \end{gathered}$$

Therefore, we assume that the only observed covariates are contained in X.

Evidently, C in Model (2) is not identifiable. However, we show in Section 4 that conditional on C, Im (C) is identifiable when β satisfies a modest sparsity assumption. Therefore, our primary goal is to develop a procedure to estimate and perform inference on β₁, …, β_p that only relies on an accurate estimate for Im (C). Our secondary goal is to simply estimate Im (C) when there are no covariates of interest, i.e. d = 0. Since accomplishing the former implies we can achieve the latter, we only consider the case when d ≥ 1.

We estimate Im (C) by separately computing, and then combining, estimates for the parts of Im (C) in Im (X) and X^⊥. Let $\mathit{G}\in {ℝ}^{n\times n}$ be a positive definite matrix. We estimate the part of Im (C) in Im (X) using the variation in y₁, …, y_p explainable by X:

$${\mathit{y}}_{g_1}={\left({\mathit{X}}^T{\mathit{G}}^{-1}\mathit{X}\right)}^{-1}{\mathit{X}}^T{\mathit{G}}^{-1}{\mathit{y}}_g={\mathit{\beta }}_g+{\mathbf{\Omega }}_g^{\mathcal{l}}+{\mathit{e}}_{g_1}\in {ℝ}^d(g=1,\dots ,p)$$ (5a)

$$\mathbf{\Omega }={\left({\mathit{X}}^T{\mathit{G}}^{-1}\mathit{X}\right)}^{-1}{\mathit{X}}^T{\mathit{G}}^{-1}\mathit{C}\in {ℝ}^{d\times K}$$ (5b)

$${\mathit{Y}}_1={\left[\begin{array}{lll}{\mathit{y}}_{1_1}\hfill & \cdots \hfill & {\mathit{y}}_{p_1}\hfill \end{array}\right]}^T=\mathit{\beta }+\mathit{L}{\mathbf{\Omega }}^T+{\mathit{E}}_1\in {ℝ}_{p\times d}.$$ (5c)

Here ${\mathit{y}}_{g_1}$ is the naive weighted least squares estimator for β_g that ignores C and Ω is the weighted least squares regression coefficient for the regression of C onto X. We show how to choose the appropriate G in Section 3.4. Next, we use the variation in y₁, …, y_p explainable by X^⊥ to estimate the part of Im (C) in X^⊥ :

$${\mathit{y}}_{g_2}={\mathit{Q}}_{\mathit{X}}^T{\mathit{y}}_g={\mathit{C}}_{\perp }{\mathcal{l}}_g+{\mathit{e}}_{g_2}\in {ℝ}^{n-d},{\mathit{e}}_{g_2}~N_{n-d}\left(\mathit{0},{\mathit{Q}}_{\mathit{X}}^T{V}_g{\mathit{Q}}_{\mathit{X}}\right)(g=1,\dots ,p)$$ (6a)

$${\mathit{C}}_{\perp }={\mathit{Q}}_{\mathit{X}}^T\mathit{C}\in {ℝ}^{(n-d)\times K}$$ (6b)

$${\mathit{Y}}_2={\left[\begin{array}{lll}{\mathit{y}}_{1_2}\hfill & \cdots \hfill & {\mathit{y}}_{p_2}\hfill \end{array}\right]}^T=\mathit{L}{\mathit{C}}_{\perp }^T+E_2\in {ℝ}^{p\times (n-d)}.$$ (6c)

Here C_⊥ and ${\mathit{y}}_{g_2}$ lie in the space orthogonal to X, where the latter no longer depends on β_g. Algorithm 1 below, which we call CBCV-CorrConf, provides a cursory description of how we use the objects defined in (5) and (6) to estimate Im (C) and subsequently β₁,…,β_p.

Algorithm 1 (CBCV-CorrConf).

1. For $k\in \left\{0,1,\dots ,K_{\max }\right\}$, use Y₂ to obtain estimates $\mathit{L}\in {ℝ}^{p\times k}$ and ${\mathit{C}}_{\perp }\in {ℝ}^{(n-d)\times k}$ using a factor analysis procedure we call ICaSE (Iterative Correlation and Subspace Estimation) (Section 3.2).

2. Use Y₂ and a cross-validation procedure we call CBCV (Correlated Bi-Cross Validation) to define $\widehat{K}$, the estimate for dim {Im (C)} (Section 3.3).

3. Define G and estimate Ω using Y₁ and the estimators $\mathit{L}\in {ℝ}^{p\times \widehat{K}}$, ${\mathit{C}}_{\perp }\in {ℝ}^{(n-d)\times \widehat{K}}$ from steps 1 and 2. Let the estimate for C be $\mathit{C}=\mathit{X}\mathbf{\Omega }+\mathit{G}{\mathit{Q}}_X{\left({\mathit{Q}}_X^T\mathit{G}{\mathit{Q}}_X\right)}^{-1}{\mathit{C}}_{\perp }\in {ℝ}^{n\times \widehat{K}}$. For $\widehat{K}$ given, we call this method of estimating C CorrConf (Section 3.4).

4. Use the design matrix [X C] to estimate β₁ , …, β_p with generalized least sqaures.

The estimates for Im (C_⊥) and Im (C) are then Im (C_⊥) and Im (C), where the estimate for C in step 3 follows from the fact that $\mathit{C}=\mathit{X}\mathbf{\Omega }+\mathit{G}{\mathit{Q}}_{\mathit{X}}{\left({\mathit{Q}}_{\mathit{X}}^T\mathit{G}{\mathit{Q}}_{\mathit{X}}\right)}^{-1}{\mathit{C}}_{\perp }$ for any positive definite G. We show in Section 4 that our estimator in Step 4 only depends on Im (C). If b = 1, B₁ = I_n and K were known, steps 1 and 3 with G = I_n are similar to methods used by Sun et al. (2012); Wang et al. (2017); Lee et al. (2017); McKennan and Nicolae (2019).

3. Estimating the factor dimension K and Im (C)

3.1. A computationally tractable model for the data

The generative model assumed in (2) is not conducive to estimating Im (C), since it requires jointly estimating Im (C) and all p covariance matrices V₁,…,V_p. Instead, we use a simpler, but incorrect, model that assumes ${\mathit{V}}_1=\cdots ={\mathit{V}}_p={\delta }^2\mathit{V}$ to estimate K, L and Im (C):

$$\mathit{Y}=\mathit{\beta }{\mathit{X}}^T+\mathit{L}{\mathit{C}}^T+\mathit{E},\mathit{E}~M{N}_{p\times n}\left(\mathit{0},{\delta }^2{I}_p,\mathit{V}\right),\det \left({\mathit{Q}}_{\mathit{X}}^T\mathit{V}{\mathit{Q}}_{\mathit{X}}\right)=1,$$ (7)

where we introduce δ² so that V is scale-invariant. We define δ² in terms of the determinant for reasons discussed in Section 3.3.

Let ${\delta }_{*}^2$ and V_* be the values of δ² and V that, conditional on C, minimize the KL-divergence between the data generating model in (2) and the incorrect model in (7). Then

$${\delta }_{*}^2{\mathit{V}}_{*}=p^{-1}\left({\mathit{V}}_1+\cdots +{\mathit{V}}_p\right),\det \left({\mathit{Q}}_{\mathit{X}}^T{\mathit{V}}_{*}{\mathit{Q}}_{\mathit{X}}\right)=1.$$ (8)

In Sections 3.2, 3.3 and 3.4 we show that, quite remarkably, we only need to estimate V_* and not V₁, …,V_p to get accurate estimates for Im (C).

Nevertheless, without any assumptions on V_*, this is a challenging, if not an intractable, problem. However, since ${\mathit{V}}_g=v_{g,1}{\mathit{B}}_1+\cdots +v_{g,b}{\mathit{B}}_b$ for all g = 1,…, p and B₁,…, B_b are known, then ${\mathit{V}}_{*}={\tau }_{{*}_1}{\mathit{B}}_1+\cdots +{\tau }_{{*}_b}{\mathit{B}}_b$ by (8), where

$${\mathit{\tau }}_{*}={\left({\tau }_{{*}_1}^{\cdots }{\tau }_{{*}_b}\right)}^T={\left({\delta }_{*}^{2}p\right)}^{-1}\left({\mathit{v}}_1+\cdots +{\mathit{v}}_p\right),{\delta }_{*}^2=\det {\left\{p^{-1}{\mathit{Q}}_{\mathit{X}}^T\left({\mathit{V}}_1+\cdots +{\mathit{V}}_p\right){\mathit{Q}}_X\right\}}^{1\text{/(}n\text{−}d\text{)}}.$$

Therefore, we need only estimate τ_* to estimate V_*, and subsequently Im (C).

3.2. ICaSE: an iterative factor analysis to estimate Im (C_⊥)

Here we present the algorithm ICaSE, a method to estimate Im (C_⊥), ${\delta }_{*}^2$ and τ_* using Y₂. Recall from (6) that the mean of Y₂ is $\mathit{L}{\mathit{C}}_{\perp }^T$ and is not dependent on X. Let ${\mathit{W}}_{*}={\mathit{Q}}_X^T{\mathit{V}}_{*}{\mathit{Q}}_X$, which implies ${\mathit{W}}_{*}={\tau }_{{*}_1}{\mathit{Q}}_X^T{\mathit{B}}_1{\mathit{Q}}_X+\cdots +{\tau }_{{*}_b}{\mathit{Q}}_X^T{\mathit{B}}_b{\mathit{Q}}_X$ and ${\delta }_{*}^2{\mathit{W}}_{*}=p^{-1}{\mathit{Q}}_X^T\left({\mathit{V}}_1+\cdots +{\mathit{V}}_p\right)\mathit{Q}$. Therefore,

$${\mathit{S}}_2=p^{-1}{\mathit{Y}}_2^T{\mathit{Y}}_2\approx {\mathit{C}}_{\perp }\left(p^{-1}{\mathit{L}}^T\mathit{L}\right){\mathit{C}}_{\perp }^T+p^{-1}{\mathit{E}}_2^T{\mathit{E}}_2\approx {\mathit{C}}_{\perp }\left(p^{-1}{\mathit{L}}^T\mathit{L}\right){\mathit{C}}_{\perp }^T+{\delta }_{*}^2{\mathit{W}}_{*}.$$

Since W_* is not a multiple of the identity, the span of the first K eigenvectors of S₂ will not be an accurate estimate for Im (C_⊥), in general. If W_* were known, we could instead first estimate $\mathrm{Im}\left({\mathit{W}}_{*}^{-1/2}{\mathit{C}}_{\perp }\right)$ as the span of the first K eigenvectors of ${\mathit{W}}_{*}^{-1/2}{\mathit{S}}_2{\mathit{W}}_{*}^{-1/2}$, and then rescale the estimate by ${\mathit{W}}_{*}^{-1/2}$ to estimate Im (C_⊥). On the other hand, if Im (C_⊥) were known, one could easily estimate ${\delta }_{*}^2$ and τ_* using restricted maximum likelihood. ICaSE iterates between these two steps to estimate Im (C_⊥), ${\delta }_{*}^2$ and τ_*.

It remains to show how to find a starting point for ${\delta }_{*}^2$ and τ_* that avoids incorporating signal from the random effect into the estimate for Im (C_⊥), as imprudent starting points often beget biased estimates for K and Im (C_⊥). We Y₂ due to C_⊥ from the random effect by employing a separate the variation in “warm start” technique often used to solve penalized regression problems. First, we initialize our estimates for ${\delta }_{*}^2$ and τ_* assuming K = 0 (i.e. $\mathit{L}{\mathit{C}}_{\perp }^T=\mathit{0}$ ). We then use the estimate for τ_* when we assume dim {Im (C_⊥)} = k − 1 as the starting point when we assume dim {Im (C_⊥)} = k. This ensures that we attribute as much variability as possible to the random effect and only ascribe variability to the latent covariates if the observed signal is not amenable to the model for the variance. Algorithm 2 enumerates these steps in ICaSE.

Algorithm 2 (ICaSE).

Let $\mathit{W}(\mathit{x})=\sum _{j=1}^b{[\mathit{x}]}_j{\mathit{Q}}_X^T{\mathit{B}}_j{\mathit{Q}}_X$ for $\mathit{x}\in {ℝ}^b$.

1. For k = 0, estimate ${\delta }_{*}^2$ and τ_* by maximum likelihood using the model ${\mathit{Y}}_2~M{N}_{p\times (n-d)}\left\{\mathit{0},{\delta }^2{\mathit{I}}_p,\mathit{W}(\mathit{\tau })\right\}$ under the restriction that det {W (τ)} = 1 and δ² τ ∈ Θ.

2. Let W = W (τ). For k ≥ 1 and given estimates ${\widehat{\delta }}^2$ and τ for ${\delta }_{*}^2$ and τ_*, estimate W^−1/2 C_⊥ as the first k right singular vectors of Y₂ W^−1/2. Re-scale the estimate by (n − d)^1/2 W^1/2 to get an estimate for ${\mathit{C}}_{\perp },{\mathit{C}}_{\perp }\in {ℝ}^{(n-d)\times k}$.

3. Given C_⊥, obtain ${\widehat{\delta }}^2$ and τ by restricted maximum likelihood using the model

   $${\mathit{Y}}_2~M{N}_{p\times (n-d)}\left\{\mathit{L}{\mathit{C}}_{\perp }^T,{\delta }^2{I}_p,\mathit{W}(\mathit{\tau })\right\}{\text{where}}^{\det \{\mathit{W}(\mathit{\tau })\}=1}\text{and}{\delta }^2\mathit{\tau }\in \mathrm{\Theta }.$$

4. Iterate between steps 1 and 2 and stop on step 1 of the second iteration. Repeat this for k = 1, …, K_max, using τ, ${\widehat{\delta }}^2$ obtained when dim {Im (C_⊥)} = k − 1 as the starting point for step 1 when dim {Im (C_⊥)} = k.

Remark 1.

If b = 1 and ${\mathit{B}}_1=I_n,{\mathit{C}}_{\perp }\in {ℝ}^{(n-d)\times k}$ is a scalar multiple of the first k right singular vectors of Y₂ for each k ∈ [K_max].

Evidently, iterating between steps 1 and 2 is not explicitly maximizing an objective function. However, Theorem 1 in Section 4 proves that ICaSE’s estimates for Im (C_⊥) are as accurate as those obtained from principal component analysis when samples are independent.

3.3. Correlated Bi-Cross Validation to estimate K

Here we use Y₂ and a cross-validation paradigm developed in Owen and Wang (2016) to estimate K = dim {Im (C)}. Besides having substantially shorter computation time and providing a less variable estimate for K, what differentiates our method from the one developed in Owen and Wang (2016) is we provide an approach amenable to dependent data, while their method is only valid for independent data.

Our primary concern is ensuring our estimates for K are not biased by the correlations in the data (Hastie et al., 2009). An equivalent concern is we want to avoid including factors arising from the random effect in our estimate for K. To describe our procedure, we assume for notational convenience that ${\mathit{Y}}_{p\times n}={\mathit{L}}_{p\times K}{\mathit{C}}_{n\times K}^T+{\mathit{E}}_{p\times n}$ where the rows e₁, …, e_p of E are independent and e_g ~ N_n (0,V_g) for all g ∈ [p]. Algorithm 3 provides an outline of the algorithm we use to estimate K, which we call Correlated Bi-Cross Validation (CBCV).

Algorithm 3 (CBCV).

Randomly partition the rows of Y (i.e. genes) into F = O (1) folds and let f ∈ [F]. Without loss of generality, assume

$$\mathit{Y}=\left[\begin{array}{c}{\mathit{Y}}_{(-f)}\\ {\mathit{Y}}_f\end{array}\right]=\left[\begin{array}{c}{\mathit{L}}_{(-f)}{\mathit{C}}^T\\ {\mathit{L}}_f{\mathit{C}}^T\end{array}\right]+\left[\begin{array}{c}{\mathit{E}}_{(-f)}\\ {\mathit{E}}_f\end{array}\right]$$

where ${\mathit{Y}}_{(-f)}\in {ℝ}^{p_{(-f)}\times n}$ and ${\mathit{Y}}_f\in {ℝ}^{p_f\times n}$ are the training and test sets.

1. Let $k\in \left\{0,1,\dots ,K_{\max }\right\}$. Obtain $\mathit{C}\in {ℝ}^{n\times k}$ and V_{(− f)} from Y_{(− f)} using Algorithm 2.

2. Let ${\overline{\mathit{Y}}}_f={\mathit{Y}}_f{\mathit{V}}_{(-f)}^{-1/2}$ and $\overline{\mathit{C}}={\mathit{V}}_{(-f)}^{-1/2}\mathit{C}$. Define the loss for this fold, dimension pair as the leave-one-out cross validation loss:

   $${\mathcal{L}}_f(k)=\sum _{i=1}^n{\Vert {\overline{\mathit{y}}}_{f,i}-{\mathit{L}}_{f,(-i)}{\widehat{\overline{\mathit{c}}}}_i\Vert }_2^2,$$ (9)

   where ${\overline{\mathit{y}}}_{f,i}\in {ℝ}^{p_f}$ and ${\widehat{\overline{\mathit{c}}}}_i\in {ℝ}^k$ are the ith columns of ${\overline{\mathit{Y}}}_f$ and ${\overline{\mathit{C}}}^T$, respectively. L_{f,(− i)} is the ordinary least squares regression coefficient from the regression of ${\overline{\mathit{Y}}}_{f,(-i)}$ onto ${\overline{\mathit{C}}}_{(-1)}$, where ${\overline{\mathit{Y}}}_{f,(-i)}$ and ${\overline{\mathit{C}}}_{(-i)}^T$ are submatrices of ${\overline{\mathit{Y}}}_f$ and ${\overline{\mathit{C}}}^T$ with the ith columns removed.

3. Repeat this for folds f = 1,…, F and k = 0,1,…, K max and set $\widehat{K}$ to be

$$\widehat{K}=\underset{\text{k}\in \left\{0,1,\dots ,{\text{K}}_{\max }\right\}}{\arg \text{m}\mathrm{in}}\left\{\sum _{\text{f}=1}^{\text{F}}{\mathcal{L}}_{\text{f}}(\text{k})\right\}.$$ (10)

Since (9) is the leave one out cross validation squared loss using the design matrix $\overline{\mathit{C}}$ and the scaled data ${\overline{\mathit{Y}}}_f$, it only depends on Im (C) and V_{(− f)}.

Scaling by ${\mathit{V}}_{(-f)}^{-1/2}$ in (9) is sensible because it places more importance on correctly estimating the portion of C not captured by the model for the residual variance. However, unless proper care is taken, this scaled loss function would underestimate K simply because the estimated residual variance is larger for underspecified models. The restriction that $\det \left\{{\mathit{V}}_{(-f)}\right\}=1$ alleviates this issue by making the loss function scale-invariant.

To understand why CBCV gives accurate estimates for K, consider the expected leave-one-out squared error for a particular fold, dimension pair, where ${\overline{c}}_i$ is the ith row of ${\mathit{V}}_{(-f)}^{-1/2}\mathit{C}$ :

$$\begin{gathered} \mathbb{E}\left\{{\mathcal{L}}_f(k)\mid {\mathit{Y}}_{(-f)}\right\}=\underset{I}{\underbrace{\sum _{i=1}^n\mathbb{E}\left\{{\Vert {\mathit{L}}_f{\overline{\mathit{c}}}_i-{\mathit{L}}_{f,(-i)}{\widehat{\overline{\mathit{c}}}}_i\Vert }_2^2\mid {\mathit{Y}}_{(-f)}\right\}}}+\underset{II}{\underbrace{p_f{\delta }_{f^{*}}^2\text{Tr}\left\{{\mathit{V}}_{(-f)}^{-1}{\mathit{V}}_{f^{*}}\right\}}}+\underset{III}{\underbrace{\mathrm{cross}-\text{term}}} \\ {\delta }_{f^{*}}{\mathit{V}}_{f^{*}}=p_f^{-1}\sum _{g\in \text{fold}f}{\mathit{V}}_g,\det \left({\mathit{V}}_{f^{*}}\right)=1. \end{gathered}$$

In standard cross validation, V_{(− f)} = I_n, meaning the residual variance term (II) would be constant for all k = 0,1,…, K_max and the cross-term (III) would be 0, since L_{f,(− i)} would be independent of the ith column of Y_f. The minimizer of the squared bias term (I) would then also minimize the expected cross validated error. However, since we must account for the correlation between samples, we now need to ensure that I + II is minimized when $\widehat{K}=K$ and that the cross-term does not contribute to the loss. The restriction that det {V_{(− f)} } = 1 helps ensure this is the case, since ${\left(n{p}_f\right)}^{-1}II\ge {\delta }_{f^{*}}^2$, where the inequality holds with equality if and only if V_{(− f)} = V_{f *} by Jensen’s Inequality. Since an accurate estimate of Im (C) begets an accurate estimate of V_{(− f)*} ≈_{f *}, the minimizer of I + II, and also I + II + III if III ≈ 0, should be close to K. We make this rigorous and prove the consistency of $\widehat{K}$ in Theorem 2 in Section 4.

3.4. De-biasing estimates for the main effect in correlated data

Recall from (5b) that for some positive definite $\mathit{G}\in {ℝ}^{n\times n}$, $\mathbf{\Omega }={\left({\mathit{X}}^T{\mathit{G}}^{-1}\mathit{X}\right)}^{-1}{\mathit{X}}^T{\mathit{G}}^{-1}\mathit{C}$, which helps quantify the proportion of the variability of C explainable by X. While C_⊥ enables one to estimate ℓ_g and V_g, Ω allows us to estimate the effect of X on y_g while controlling for the latent factors C. This helps make results reproducible.

To estimate Ω, we must first specify a suitable weighting matrix G. Section 3.1 suggests that a reasonable choice would be $\mathit{G}=\sum _{j=1}^b{[\mathit{\tau }]}_j{\mathit{B}}_j=\mathit{V}$, where τ is computed using Algorithm 2. From here on out, we assume K is known and define ${\mathit{Y}}_{g_1}$, Ω and Y₁ in (5) by setting G = V.

Our strategy for estimating Ω is to regress Y₁ onto the estimate for L obtainable after completing Algorithm 2, where for $\mathit{W}={\mathit{Q}}_X^T\mathit{V}{\mathit{Q}}_X,\mathit{L}={\mathit{Y}}_2{\mathit{W}}^{-1}{\mathit{C}}_{\perp }{\left({\mathit{C}}_{\perp }^T{\mathit{W}}^{-1}{\mathit{C}}_{\perp }\right)}^{-1}$. One possible estimator for Ω is the ordinary least squares estimate ${\mathbf{\Omega }}^{\text{naive}}={\mathit{Y}}_1^T\mathit{L}{\left({\mathit{L}}^T\mathit{L}\right)}^{-1}$. When samples are independent, this strategy is similar to those used in Sun et al. (2012); Houseman et al. (2014); Lee et al. (2017); Wang et al. (2017); Fan and Han (2017), although the exact estimators differ slightly. However, McKennan and Nicolae (2019) showed that Ω^naive tends to underestimate Ω when samples are independent because L is a noisy estimate for the design matrix L. Therefore, we use the following de-biased estimate of Ω, which is analogous to the estimator used in McKennan and Nicolae (2019) when samples are assumed independent:

$$\mathbf{\Omega }={\mathit{Y}}_1^T\mathit{L}{\left\{{\mathit{L}}^T\mathit{L}-p{\widehat{\delta }}^2\left({\mathit{C}}_{\perp }^T{\mathit{W}}^{-1}\mathit{C}+1\right)\right\}}^{-1}\}.$$ (11)

The $p{\widehat{\delta }}^2{\left({\mathit{C}}_{\perp }^T{\mathit{W}}^{-1}{\mathit{C}}_{\perp }\right)}^{-1}$ term in (11) removes the bias in L^T L and reduces to the bias correction used in McKennan and Nicolae (2019) when b = 1 and B₁ = I_n.

Lastly, we can express C as $\mathit{C}=\mathit{X}\mathbf{\Omega }+\mathit{V}{\mathit{Q}}_X{\mathit{W}}^{-1}{\mathit{C}}_{\perp }$. Therefore, our estimate for C is

$$\mathit{C}=\mathit{X}\mathbf{\Omega }+\mathit{V}{\mathit{Q}}_X{\mathit{W}}^{-1}{\mathit{C}}_{\perp }.$$ (12)

We subsequently estimate V_g and β_g for all g ∈ [p] with restricted maximum likelihood followed by generalized least squares using the design matrix [X C].

4. Theoretical justification of CBCV-CorrConf

For all theoretical results, we assume d, b, K_max = O (1), and y₁,…, y_p are generated by (2), where e₁,…, e_p are independent. The latter assumption is based off of the repeated observation in experimental genetic data that, for the purposes of marginal testing, it is sufficient to assume the dependence between genomic units is driven by a relatively small number of latent variables (Leek and Storey, 2007; Maksimovic et al., 2015; Gerard and Stephens, 2018). It is also a common assumption among the literature that motivates their models with high throughput biological data (Leek and Storey, 2008; Sun et al., 2012; Lee et al., 2017; Wang et al., 2017; Dobriban and Owen, 2019). We also provide simulation evidence in Section S1.3 of the Supplement that suggests our method is robust to dependence among e₁,…, e_p.

Assumption 1.

$\mathit{X}\in {ℝ}^{n\times d}$ and ${\mathit{B}}_1,\dots ,{\mathit{B}}_b\in {ℝ}^{n\times n}$ are observed, non-random matrices. Let c₁ > 1 be a constant. Define $M\in {ℝ}^{b\times b}$ to be [M]_ij = n⁻¹Tr (B_i B_j) for all i, j ∈ [b]. For any matrix D, let D_{(− i)} be the sub-matrix of D with the ith row removed. Assume:

1. The random matrix C is independent of E, where $\mathbb{E}\left\{{\left({[\mathit{C}]}_{ik}\right)}^2\right\}<\infty$ for all i ∈ [n] and k ∈ [K]. Further, $\underset{n\to \infty }{\lim }{n}^{-1}{\mathit{X}}^T\mathit{X}={\mathit{\Sigma }}_X\succ \mathit{0}$, $M\succcurlyeq {\mathit{c}}_1^{-1}{I}_b$, ${\mathit{B}}_j={\mathit{B}}_j^T$, ${\Vert {\mathit{B}}_j\Vert }_2\le {\mathit{c}}_1$ for all j ∈ [b] and $\mathit{L}\in {\mathbf{\Theta }}_{\mathit{L}}=\left\{\mathit{L}\in ℝp\times K:{\mathit{L}}_{(-g)}containstwodistinct\mathrm{rank}Ksubmatricesforallg\in [p]\right\}.$

2. Let ${\mathit{W}}_{*}={\mathit{Q}}_X^T{\mathit{V}}_{*}{\mathit{Q}}_X$ and $\mathbf{\Psi }=\mathbb{E}\left(n^{-1}{\mathit{C}}_{\perp }^T{\mathit{W}}_{*}^{-1}{\mathit{C}}_{\perp }\right)$. Then ${}_g^{\mathcal{l}T}{\mathbf{\Psi }}_g^{\mathcal{l}}\le c_1$ for all g ∈ [p], and the first K eigenvalues λ₁ > … >λ_K > λ_K+1 = 0 of np⁻¹ L Ψ L^T satisfy ${\lambda }_1,\dots ,{\lambda }_K\in \left[c_1^{-1},n{c}_1\right]$.

3. ${\mathit{V}}_g\succcurlyeq c_1^{-1}{I}_n$, $\left|v_{g,j}\right|\le c_1$ for all g ∈ [p] and j ∈ [b]. Further, ${\Vert n^{-1}{\mathit{C}}_{\perp }^T{\mathit{W}}_{*}^{-1}{\mathit{C}}_{\perp }-\mathbf{\Psi }\Vert }_2=o_P(1)$ as n, p → ∞ and $\Vert \mathbf{\Psi }{\Vert }_2,{\Vert {\mathbf{\Psi }}^{-1}\Vert }_2\le c_1$.

4. $1-{\lambda }_{k+1}/{\lambda }_k\ge c_1^{-1}$ for all k ∈ [K], λ₁ / λ_K ≤ c₁ and one of the following holds:

   $${\lambda }_K\ge c_1^{-1}nOR{\lambda }_K\to \infty \text{but}{\lambda }_K/n\to 0\text{as}n,p\to \infty OR{\lambda }_K\le c_1.$$

5. p is a non-decreasing function of n such that n / p, n^3/2 / (p λ_K) → 0 as n → ∞.

Remark 2.

We prove V₁,…,V_p ≻ 0 and L Ψ L^T are identifiable under Assumption 11 in Proposition S1 in Section S3.1 in the Supplement. The set Θ_L is a classic way to identify parameters in factor models when ${\mathit{V}}_g={\sigma }_g^2{I}_n$ for all g ∈ [p] (Anderson and Rubin, 1956; Wang et al., 2017).

Remark 3.

The assumptions on C imply $\mathbb{E}\left({\mathit{y}}_g\right)=\mathit{X}{\mathit{\beta }}_g+\mathbb{E}{(\mathit{C})}_g^{\mathcal{l}}$ and $\mathrm{Cov}\left({\left[{\mathit{y}}_g\right]}_i,{\left[{\mathit{y}}_h\right]}_j\right){=}_g^{\mathcal{l}T}{\mathbf{\Psi }}_{ij}^{\mathcal{l}}+{\left[{\mathit{V}}_g\right]}_{ij}I(g=h)$, where Ψ_ij is the covariance between the ith and jth rows of C.

The condition on M ensures v₁,…, v_p are identifiable, and one can easily verify the condition on ║B_j║₂ in data from McKennan et al. (2018) and Martino et al. (2013) discussed in Sections 1 and 6, as well as in the simulated multi-tissue data from Section 5. We provide additional data examples to illustrate the ubiquity of the assumption on ║B_j║₂ in Section S4.1 of the Supplement. The eigenvalues λ₁,…, λ_K quantify the average variation in y₁,…, y_p due to C that can be distinguished from that due to X, and we give general sufficient conditions under which ${\Vert n^{-1}{\mathit{C}}_{\perp }^T{\mathit{W}}_{*}^{-1}{\mathit{C}}_{\perp }-\mathbf{\Psi }\Vert }_2=o_P(1)$ in Lemma S1 in the Supplement. Item 5 is standard when ${\mathit{V}}_g={\sigma }_g^2{I}_n$ for all g ∈ [p] (Wang et al., 2017; McKennan and Nicolae, 2019).

Assumption 2.

Let c² > 1 be a constant not dependent on n or p. The estimates ${\widehat{\delta }}^2$ and τ from step 2 in Algorithm 2 are such that ${\widehat{\delta }}^2\mathit{\tau }$ lies in the convex set

$${\mathrm{\Theta }}_{*}=\mathrm{\Theta }\cap \left\{\mathit{x}\in {\mathit{R}}^b:\Vert \mathit{x}{\Vert }_2\le b{c}_2{c}_1,\sum _{j=1}^b{\left[\mathit{x}\right]}_j{\mathit{B}}_j-{\left(c_2{c}_1\right)}^{-1}{I}_n\succ \mathit{0}\right\},$$

where c₁ was defined in Assumption 1.

This makes the residual variance parameter space compact and is analogous to Assumption D in Bai and Li (2012) and Assumption 2 in Wang et al. (2017). It is also a standard assumption in likelihood theory (Wald, 1949; Ferguson, 1996; Douc et al., 2004). We now state Theorem 1.

Theorem 1 (Accuracy of ICaSE).

Suppose Assumptions 1 and 2 hold and we apply Algorithm 2 for k = 1,…, K_max , where K ≤ K max. Then the estimates for ${\delta }_{*}^2$, τ_* and C_⊥ satisfy

$$\left|{\widehat{\delta }}^2-{\delta }_{*}^2\right|,{\Vert \mathit{\tau }-{\mathit{\tau }}_{*}\Vert }_2=O_P\left(n^{-1}\right),k\ge K$$

$${\Vert P_{{\mathit{C}}_{\perp }}-P_{{\mathit{C}}_{\perp }}\Vert }_F^2=O_P\left\{n/\left({\lambda }_{K}p\right)+{\left(p{\lambda }_K\right)}^{-1/2}+{\left(n{\lambda }_K\right)}^{-1}\right\},k=K\ge 1$$

$${\Vert P_{{\mathit{C}}_{\perp }}-P_{{\mathit{C}}_{\perp }}{P}_{{\mathit{C}}_{\perp }}\Vert }_F^2=O_P\left\{n/\left({\lambda }_{K}p\right)+{\left(p{\lambda }_K\right)}^{-1/2}+{\left(n{\lambda }_K\right)}^{-1}\right\},k\ge K\ge 1$$

where ${\widehat{\delta }}^2$ and τ depend on k. Further, ${\Vert P_{{\mathit{C}}_{\perp }}-P_{{\mathit{C}}_{\perp }}\Vert }_F^2,{\Vert P_{{\mathit{C}}_{\perp }}-P_{{\mathit{C}}_{\perp }}{P}_{{\mathit{C}}_{\perp }}\Vert }_F^2$ and the estimates ${\widehat{\delta }}^2$, τ are invariant of the choice of Q_X.

Theorem 1 implies that Im (C_⊥) is estimated well when k = K and its column space is approximately a subspace of Im (C_⊥) when we overestimate K. This result is quite remarkable because besides the additional factor ${\left(n{\lambda }_K\right)}^{-1}+{\left(p{\lambda }_K\right)}^{-1/2}<<n^{-1/2}$, this is the same rate obtained from principal components analysis when the samples are independent (McKennan and Nicolae, 2019). Theorem 1 makes no assumption on the starting points for ${\widehat{\delta }}^2$ and τ (other than the REML estimates are estimated in Θ_*), which proves the efficacy of our warm-start technique.

Theorem 2 (Consistency of CBCV).

Let $\widehat{K}$ be as defined in (10). Suppose the assumptions of Theorem 1 hold, ${\lambda }_K\ge {\delta }_{*}^2+ϵ$ for some constant ϵ > 0 and we sample Q_X uniformly over the space of all orthonormal bases for X^⊥. Then $\underset{n,p\to \infty }{\lim }\text{P}(\widehat{K}=K)=1$.

Remark 4.

Let ${\mathit{Q}}_{*}\in {ℝ}^{n\times (n-d)}$ be a non-random matrix such that $P_{\mathit{X}}^{-}={\mathit{Q}}_{*}{\mathit{Q}}_{*}^T$. Then Q_X = Q_* M, where $M\in {ℝ}^{(n-d)\times (n-d)}$ is independent of C and E, and is sampled uniformly over the space of unitary matrices. This guarantees the maximum leverage score of ${\mathit{W}}_{*}^{-1/2}{\mathit{C}}_{\perp }$ is o_P (1) as n → ∞, and is a common technique to uniformize the leverage scores of a matrix (Mahoney, 2011).

This theorem shows that recovering K is easier when the sample size n is larger, since λ₁,…, λ_K tend to grow with n. This is why it is advisable to use all of the available samples to estimate K, rather than partitioning the data to alleviate correlation, as discussed in Section 1.

The requirement that ${\lambda }_K\ge {\delta }_{*}^2+ϵ$ is tight, as Owen and Wang (2016) demonstrate that ${\delta }_{*}^2$ is the lower limit of detection when b = 1 and B₁ = I_n. This lower limit of detection is smaller than parallel analysis’, which often fails to recover moderate to small factors (Owen and Wang, 2016). We discuss this in more detail after the statement of Theorem S2 in Section S4 of the Supplement. Theorem 2 is also, to the best of our knowledge, the first result proving that bi-cross validation-like methods are consistent. The authors of Owen and Perry (2009); Owen and Wang (2016) only showed their estimates for K minimize the expected loss when samples are independent.

Assumption 3.

$s=p^{-1}\sum _{g=1}^{p}I\left({\mathit{\beta }}_g\ne 0\right)=o\left(n^{-3/2}{\lambda }_K\right)$, ${\max }_{g\in [p]}{\Vert {\mathit{\beta }}_g\Vert }_2\le c_3$ for some constant c₃ > 0 , and for $\mathit{S}={\left({\mathit{X}}^T{\mathit{V}}_{*}^{-1}\mathit{X}\right)}^{-1}{\mathit{X}}^T{\mathit{V}}_{*}^{-1}\mathit{C}$, ${\Vert \mathit{S}{\mathbf{\Psi }}^{-1}{\mathit{S}}^T\Vert }_2=O_p(1)$ as n, p→ ∞.

Remark 5.

We prove β is identifiable under Assumptions 11–2, 15 and the sparsity assumption on s in Proposition S2 in the Supplement. Other than the conditions on $n^{-1}{\mathit{C}}_{\perp }{\mathit{W}}_{*}^{-1}{\mathit{C}}_{\perp }$ and S in Assumptions 1 and 3, we make no assumptions about the relationship between C and X.

This provides an explicit relationship between the maximum allowable sparsity on the main effect and the informativeness of the data Y for estimating C: the more signal in $\mathit{L}{\mathit{C}}_{\perp }^T$, the part of LC^T unequivocally distinguishable from βX^T, the less stringent Assumption 3 becomes. This sparsity condition is consistent with many modern genetic studies (Liu et al., 2016; Morales et al., 2016; Yang et al., 2017; Zhang et al., 2018), and is the same sparsity assumed in existing theoretical work when b = 1 and B₁ = I_n (Wang et al., 2017; McKennan and Nicolae, 2019). We show through simulation in Section 5 that we can accurately recover C even when this assumption is violated.

Theorem 3 (Inference on β)

Let g ∈ [p]. Suppose Assumptions 1, 2 and 3 hold and we estimate C according to (12). Let ${\widehat{\mathit{v}}}_g\in {\mathrm{\Theta }}_{*}$ and β_g be the restricted maximum likelihood (REML) and generalized least squares (GLS) estimates for v_g and β_g using the design matrix [X C]. Then

$${\Vert {\mathit{V}}_g-{\mathit{V}}_g\Vert }_2=o_P(1),{\mathit{V}}_g=\sum _{j=1}^b{\left[{\widehat{\mathit{v}}}_g\right]}_j{\mathit{B}}_j$$

$$M_g^{-1/2}{\left({\mathit{\beta }}_g-{\mathit{\beta }}_g\right)}^d=\mathit{Z}+o_P(1),M_g={\left({\mathit{X}}^T{\mathit{V}}_g^{-1}\mathit{X}\right)}^{-1}+{\mathbf{\Omega }}_g{\left\{{\mathit{C}}_{\perp }^T{\left({\mathit{Q}}_X^T{\mathit{V}}_g{\mathit{Q}}_X\right)}^{-1}{\mathit{C}}_{\perp }\right\}}^{-1}{\mathbf{\Omega }}_g^T$$

as n → ∞ , where Z ~ N_d (0, I_d) and ${\mathbf{\Omega }}_g={\left({\mathit{X}}^T{\mathit{V}}_g^{-1}\mathit{X}\right)}^{-1}{\mathit{X}}^T{\mathit{V}}_g^{-1}\mathit{C}$. Further, ${\Vert M_g{M}_g^{-1}-I_d\Vert }_2=o_P(1)$ as n → ∞, where M_g is the variance of the GLS estimate for β_g when C and V_g are known.

Remark 6.

The REML and GLS estimates V g, β_g and M_g only depend on C through Im (C). We prove that conditional on C, Im (C) and M_g are identifiable for all g ∈ [p] in Proposition S3 in Section S5.2 in the Supplement.

5. Simulated multi-tissue gene expression data analysis

5.1. Simulation setup and parameters

We simulated the expression of p = 15, 000 genes from 50 individuals across three tissues with a complicated tissue-by-tissue correlation structure to compare our method against other state of the art methods designed to estimate K and β₁,…, β_p. We include comparisons of our method to estimate C_⊥ in Section S1.1 in the Supplement. We first randomly chose 25 individuals to be in the treatment group and set X ∈ {0,1}ⁿ to be the treatment status for the n = 150 samples and Z = 1₅₀ ⊗ I₃ to be the tissue-specific intercept. For each g ∈ [p], we let V_g = I₅₀ ⊗ M_g be the gene-specific covariance matrix for gene g and simulated $M_g\in {ℝ}^{3\times 3}$ in each dataset so the average correlation matrix across all p genes was as given in Table 1.

Table 1.

The correlation matrix corresponding to the covariance matrix $\underset{t\to \infty }{\lim }{t}^{-1}\sum _{g=1}^t{M}_g$.

	Tissue 1	Tissue 2	Tissue 3
Tissue 1	1	0.72	0.58
Tissue 2	0.72	1	0.80
Tissue 3	0.58	0.80	1

Specifically, we assumed tissues two and three were more similar to one another than to the first and for each g ∈ [p], set M_g to be

$$M_g=\text{Cov}\left\{{\left(\begin{array}{lll}ϵ\hfill & ϵ\hfill & ϵ\hfill \\ g1\hfill & g2\hfill & g3\hfill \end{array}\right)}^T\right\}$$

$${ϵ}_{g1}={\alpha }_{g1}+{\xi }_{g1},{ϵ}_{g2}={\varphi }_{g2}{\alpha }_{g1}+{\alpha }_{g2}+{\xi }_{g2},{ϵ}_{g3}={\varphi }_{g3}{\alpha }_{g1}+{\rho }_{g3}{\alpha }_{g2}+{\xi }_{g3}$$

$${\alpha }_{g1}~\left(0,v_{g1}^2\right),{\alpha }_{g2}~\left(0,v_{g2}^2\right),{\xi }_{gj}~\left(0,{\sigma }_{gj}^2\right)(j=1,2,3).$$

The constants $v_{g1}^2,{\varphi }_{g2}$, $v_{g2}^2,{\varphi }_{g3},{\rho }_{g3}$ and ${\sigma }_{gj}^2$ were simulated from Gamma distributions with means 0.8, 1.25, 0.4, 0.75, 1 and 0.2, respectively, each with coefficient of variation equal to 0.2. We subsequently re-scaled these parameters so that ${\delta }_{*}^2=1$. This complex tissue-by-tissue covariance is amenable to the variance model assumed in (2b), since

$${\mathit{V}}_g=\sum _{r=1}^3\sum _{s=r}^3{v}_{grs}\left\{I_{50}\otimes \left({\mathit{a}}_{rs}{\mathit{a}}_{rs}^T\right)\right\}=\sum _{r=1}^3\sum _{s=r}^3{v}_{grs}{\mathit{B}}_{rs}$$ (13)

where ${\mathit{a}}_{rs}\in {ℝ}^3$ has a 1 in the rth and sth coordinates and 0 everywhere else and

$$v_{g12},v_{g13},v_{g23}\ge 0,v_{g11}+v_{g12}+v_{g13},v_{g22}+v_{g12}+v_{g23},v_{g33}+v_{g13}+v_{g23}\ge 0.$$

We next simulated data with K = 10 latent factors. The parameters X, Z, K and α, defined below, were fixed across all simulations. For ${\mathit{W}}_{*}={\mathit{Q}}_{[XZ]}^T{\mathit{V}}_{*}{\mathit{Q}}_{[XZ]}$, we simulated ${\mathit{y}}_g\in {ℝ}^n$, the expression of gene g in the n (individual, tissue) pairs, as follows:

$$\begin{gathered} {\mathit{y}}_g=\mathit{X}{\mathit{\beta }}_g+{\mathit{C}}_g^{\mathcal{l}}+{(3/4)}^{1/2}{\mathit{V}}_g^{1/2}{\tilde{\mathit{e}}}_g,{\left[{\tilde{\mathit{e}}}_g\right]}_i~T_s(g=1,\dots ,p;i=1,\dots ,n) \\ {\mathit{\beta }}_g~0.8{\delta }_0+0.2N_1\left(0,{0.4}^2\right)(g=1,\dots ,p) \\ \mathit{C}=\alpha \mathit{X}{\mathit{1}}_K^T+\Xi {\left(n^{-1}{\Xi }^T{\mathit{Q}}_{[\mathit{X}Z]}{\mathit{W}}_{*}^{-1}{\mathit{Q}}_{[\mathit{X}Z]}^T\Xi \right)}^{-1/2},\Xi ~M{N}_{n\times K}\left(0,I_n,I_K\right) \\ {\left[\begin{array}{l}\mathcal{l}\hfill \\ g\hfill \end{array}\right]}_k~{\pi }_k{\delta }_0+\left(1-{\pi }_k\right)N_1\left(0,{\eta }_k^2\right)(g=1,\dots ,p;k=1,\dots ,K). \end{gathered}$$ (14)

Here δ₀ is the point mass at 0 and T₈ is the t-distribution with eight degrees of freedom, and was chosen to emulate data with heavy tails. We chose to simulate a non-sparse main effect β to show that we can violate Assumption 3 and still do inference that is just as accurate as when C is known. The re-scaling of Ξ was simply to make the diagonal elements of n p⁻¹ L^T L approximately equal to the eigenvalues λ1,…, λ10 defined in Assumption 1. This re-scaling also caused λ1,…, λ₁₀ to shrink by a factor of 1.5, on average, thus making it harder to recover K and C. The constant α was chosen so that $P_{{\mathit{V}}_{*}^{{}^{-1/2}}z}^{\perp }{\mathit{V}}_{*}^{-1/2}\mathit{C}$ explained approximately 30% of the variability in $P_{{\mathit{V}}_{*}^{{}^{-1/2}}z}^{\perp }{\mathit{V}}_{*}^{-1/2}\mathit{X}$, on average. We defined α by scaling X, C and Z by ${\mathit{V}}_{*}^{-1/2}$ because this is what one would do in generalized least squares when the covariance of the observations is V_*. The values for π_k, η_k and the resulting λ₁,…,λ₁₀ are provided in Table 2.

Table 2.

The π_k and η_k values used to simulated L and the resulting average λ_k (^{k = 1, …, 10}).

Factor number (k)	1	2	3	4	5	6	7	8	9	10
π k	0	0.45	0.60	0.71	0.79	0.85	0.90	0.92	0.94	0.96
η k	1	0.4	0.4	0.4	0.4	0.4	0.4	0.4	0.4	0.5
λ k	143	12.7	9.5	6.6	4.8	3.3	2.5	1.8	1.4	1.4

For each simulated dataset, we let X be the covariate of interest and the tissue specific intercepts, Z, be nuisance covariates and estimated K using CBCV with F = 3 folds, C_⊥ and V_* using ICaSE and C using (12). We then estimated V_g and β_g for each g ∈ [p] using restricted maximum likelihood and generalized least squares, respectively. We include additional results for data simulated such that conditional on C, expression across genes was correlated in Section S1.3 of the Supplement.

5.2. Comparison of estimators for K and β

Since, as far as we are aware, our methods for estimating K and C are the first methods that account for correlation among samples in high throughput biological data, we compared our estimates for K and β with state of the art methods designed for high throughput biological data with independent samples. We first assessed our estimates for K in 100 simulated datasets and compared them with three widely used methods: the method proposed in Bai and Ng (2002), parallel analysis (Buja and Eyuboglu, 1992) and bi-cross validation (Owen and Wang, 2016). The results are given in Figure 1. We could not compare our method to the deterministic version of parallel analysis designed for data with independent samples proposed in Dobriban and Owen (2019), since the authors did not provide R code to implement their method. However, their simulations show that their method tends to select at least as many factors as parallel analysis.

Fig. 1.

Estimates for K = 10 in 100 simulated datasets using our proposed method (CBCV), the method proposed in Bai and Ng (2002) (Bai & Ng), parallel analysis (PA) and bi-cross validation (BCV).

The fact that CBCV recovers K in all simulated datasets is quite remarkable given that we simulate data with heavy tails and ${\lambda }_K\approx {\delta }_{*}^2$ (see Theorem 2). We discuss the behavior of CBCV-CorrConf when λ_K is smaller than ${\delta }_{*}^2$ in Sections S1.3 and S4.2 of the Supplement. The method proposed in Bai and Ng (2002) severely underestimates K because it is only able to recover latent factors with overtly large effects (Owen and Wang, 2016). On the other hand, bi-cross validation and parallel analysis overestimate K because both methods are treating the high dimensional random effect as part of the low dimensional effect $\mathit{L}{\mathit{C}}_{\perp }^T$. We discuss possible adjustments to ameliorate bi-cross validation’s and parallel analysis’ estimates for K in these simulated data in Section S1.2 of the Supplement. Suffice it to say that these adjustments either did not change the estimates, or caused them to consistently underestimate K.

We lastly estimated β via generalized least squares with the design matrix M = [X Z C], where C was estimated with CBCV followed by CorrConf (our estimator specified in (12)), BCconf (McKennan and Nicolae, 2019), Cate-RR (Wang et al., 2017), dSVA (Lee et al., 2017), IRW-SVA (Leek and Storey, 2008), RUV-2 (Gagnon-Bartsch and Speed, 2012), RUV-4 (Gagnon-Bartsch et al., 2013) and when C was known. BCconf, Cate-RR, dSVA, IRW-SVA, RUV-2 and RUV-4 were all applied assuming K = 10 was known. In order to make the estimation of V₁,…,V_p computationally tractable, we modeled the simulated residuals e₁,…, e_p as

$${\mathit{e}}_g~N_n\left(\mathbf{0},v_g\mathit{V}\right)(g=1,\dots ,p),\mathit{V}=\sum _{r=1}^3\sum _{s=r}^3{\mathit{\tau }}_{rs}{\mathit{B}}_{rs},$$

where B_rs was defined in (13), and estimated v₁,…, v_p and T_rs with restricted maximum likelihood for each of the eight methods using the estimated design matrix M. We then computed the P value for the null hypothesis β_g = 0 for all g ∈ [p] by comparing the t-statistics to a t-distribution with n – 4 − K = n − 14 degrees of freedom, used these P values as input into q-value (Storey, 2001) to control the false discovery rate and deemed a gene as being differentially expressed across the two treatment conditions if its q-value was no greater than 0.2. We chose q-value to control the false discovery rate because this is the software most popular among biologists. Figure 2 plots the true false discovery proportion in 100 simulated datasets among genes with a q-value less than or equal to 0.2 for each of the eight methods. The false discovery proportion when we completely ignored C was uniformly greater than that of IRW-SVA’s.

Fig. 2.

The false discovery proportion (FDP) among genes with a q-value no greater than 0.2 in 100 simulated datasets. We randomly chose 90 genes with β_g = 0 as control genes in RUV-2 and RUV-4. The FDP when we ignored C was uniformly greater than IRW-SVA’s.

The performance of our method, CorrConf, as the statement of Theorem 3 suggests, is nearly indistinguishable from the generalized least squares estimator when C is known, with nearly identical power. On the other hand, the other six methods tend to introduce more type I errors because their estimates for C do not account for the dependence between the residuals and therefore fail to recover Ω. When K is underestimated, these six methods perform even worse than when K = 10 is known, and we show in Section S1.3 in the Supplement that these methods still fail to control the false discovery rate and, due to the reduction in residual degrees of freedom, exhibit a decrease in power when K is overestimated to the extent suggested by bi-cross validation and parallel analysis in Figure 1.

6. Sex-specific DNA methylation in a longitudinal twin study

We next applied our method to identify sex-specific DNA methylation patterns from a longitudinal twin study using data previously published in Martino et al. (2013). The authors measured the DNA methylation of 10 monozygotic (MZ) and 5 dizygotic (DZ) Australian twin pairs (all DZ twins were both male or both female) at birth and 18 months on the Infinium HumanMethylation450 BeadChip platform in buccal epithelium, a relatively homogeneous tissue. After probe and sample quality filtering and data-normalization, the authors were left with p = 330, 168 methylation sites (CpGs) whose methylation was quantified in 29 male and 24 female (n = 53) samples as the difference between log-methylated and log-unmethylated probe intensity (see Martino et al. (2013) for all pre-processing steps). We then used our proposed method (CBCV-CorrConf), BCconf, Cate-RR, dSVA and IRW-SVA to identify CpGs whose methylation differs in males and females, which we refer to as sex-associated CpGs. We subsequently validated each method’s findings using sex-associated CpGs identified at birth in previous studies with substantially larger sample sizes. We did not compare our method with RUV-2 or RUV-4, since we did not have access to control CpGs.

We first show that we can write the covariance of the 53 observations at each CpG as a linear combination of six matrices. Let y_{m, t,a} be the measured DNA methylation at an arbitrary CpG for twin t ∈{1, 2}, from mother m ∈ [15] at age a ∈{0,18}, where samples with different mothers were assumed to be independent and twin 1 and twin 2 from the same mother were assumed to be exchangeable. A preliminary analysis showed that the correlation between MZ and DZ twins’ methylomes was approximately the same at both ages, which is consistent with the observation that methylation patterns are in large part determined by environmental exposures (Galanter et al., 2017; Martin and Fry, 2018). Therefore, the 4 × 4 covariance matrix for y_m = (y_m,1,0 y_m,1,18 y_m,2,0 y_m,2,18)^T completely determined the n × n covariance matrix for each CpG. We show in Section S2 of the Supplement that by only making assumptions on the pairwise covariances, one can write Cov (y_m) as

$$\begin{gathered} \mathrm{Cov}\left({\mathit{y}}_m\right)=v_{\alpha }{\mathit{1}}_4{\mathit{1}}_4^T+v_{\eta }\left({\mathit{1}}_2{\mathit{1}}_2^T\oplus {\mathit{1}}_2{\mathit{1}}_2^T\right)+v_{\varphi ,0}{\mathit{a}}_0{\mathit{a}}_0^T+v_{\varphi ,18}{\mathit{a}}_{18}{\mathit{a}}_{18}^T+v_0\mathrm{diag}\left({\mathit{a}}_0\right) \\ +v_{18}\text{diag}\left({\mathit{a}}_{18}\right) \end{gathered}$$ (15)

where ${\mathit{a}}_0^T=(1,0,1,0)$, ${\mathit{a}}_{18}^T=(0,1,0,1)$. The variance multipliers are such that for t₁ ≠ t₂,

$$v_{\alpha }=\mathrm{Cov}\left(y_{m,t_1,0},y_{m,t_2,18}\right)$$

$$v_{\alpha }+v_{\eta }=\mathrm{Cov}\left(y_{m,t_1,0},y_{m,t_1,18}\right),v_{\alpha }+v_{\varphi ,a}=\mathrm{Cov}\left(y_{m,t_1,a},y_{m,t_2,a}\right)(a=0,18)$$

$$v_{\alpha }+v_{\eta }+v_{\varphi ,a}+v_a=\mathrm{Var}\left(y_{m,t_1,a}\right)(a=0,18)$$

and lie in a convex polytope defined in (3). We derive these variance multiplier relationships in Section S2 of the Supplement.

Since there was no evidence that the difference in methylation between males and females changed from birth to 18 months, we assumed the methylation at each CpG was a linear combination of the subject’s age (birth or 18 months), sex and other unobserved factors to be estimated, where age was a nuisance covariate and sex was the phenotype of interest. We used CBCV with F = 5 folds and estimated K to be 2, and subsequently estimated V_* and C with ICaSE and CorrConf. Our estimates for the six average variance multipliers were all greater than 0, the average residual variance at 18 months was 25% larger than that at birth and the correlation between methylation for twins at 18 months was nearly 20% larger than that at birth. This indicated this set of twin’ s methylomes tended to converge over the first 18 months of life, which is consistent with previous observations that one’s methylome reflects one’s environmental exposure history (Galanter et al., 2017; Martin and Fry, 2018).

We next computed each of the other four method’s estimates for C by first using each method’s default software to choose K: bi-cross validation, the default for Cate-RR and BCconf, or parallel analysis, the default for dSVA and IRW-SVA. These methods estimated K to be 4 and 15, respectively. The fact that both estimated the latent factor dimension to be greater than CBCV’s estimate of 2 is not surprising, as these methods tend to overestimate K when samples are correlated. We subsequently estimated C with each of the four methods using their software’s default settings.

We lastly estimated the effect due to sex on methylation and corresponding q-values to control the false discovery rate, exactly as we did for the simulated data in Section 5. We deemed a CpG a sex-associated CpG if its q-value in that method was no greater than 0.2. Since we did not know the ground truth, we used sex-associated CpGs identified at birth in Yousefi et al. (2015) and Maschietto et al. (2017) as a validation set to help judge the veracity of each method’s findings. Yousefi et al. (2015) and Maschietto et al. (2017) measured DNA methylation in umbilical cord blood on the Infinium HumanMethylation450 BeadChip platform in children born to 111 unrelated Brazilian and 71 unrelated Mexican American mothers, respectively. The authors of both studies measured and corrected for cord blood cellular composition and identified 2,355 and 1,928 sex-associated CpGs that were also among the 330,168 CpGs studied in Martino et al. (2013). Table 3 gives the fraction of sex-associated CpGs identified using C estimated with our method (CBCV-CorrConf), along with the other four methods, that are also among the 3,532 sex-associated CpGs identified in Maschietto et al. (2017) or Yousefi et al. (2015). Since the other four methods are not designed for dependent data, we discuss two adjustments designed to alleviate potential biases in their estimates for both K and C in Section S2 of the Supplement. The overlaps with the modified methods were at least as poor as those presented in Table 3.

Table 3.

The fraction of sex-associated CpGs identified using data from Martino et al. (2013) that are also one of the 3,532 sex-associated CpGs confidently identified in Yousefi et al. (2015) or Maschietto et al. (2017).

CBCV-CorrConf (K = 2)	BCconf (K = 4)	Cate-RR (K = 4)	dSVA (K = 15)	IRW-SVA (K = 15)
38% (278/729)	23% (424/1839)	20% (474/2404)	19% (487/2517)	28% (341/1240)

While it may be the case that most of BCconf, Cate-RR, dSVA and IRW-SVA are actual sex-associated CpGs, the results in Table 3 mirror the trends observed in Figure 2. That is, while these four methods nominally identify more sex-associated CpGs, we are less confident in their results because their estimates for C reduce the residual variance but likely do not suitably account for the variation in sex explainable by C, which makes their results less reproducible.

These results also highlight the importance of the choice of K. Estimating K with CBCV, and cross-validation in general, tends to yield more reproducible results because we only include a latent factor if prediction performs suitably well on new, held-out data. When we applied all five methods with K = 2, BCconf, Cate-RR and dSVA performed similarly with overlaps no greater than 30%. However, 272 out of IRW-SVA’s 662 sex-associated CpGs (41%) were in the validation set, which is nearly identical to CorrConf’s results in Table 3. Similarly, when we set K = 4 for all methods, dSVA performed nearly identically to Cate-RR, whereas CorrConf and IRW-SVA had overlaps of 27% and 26%, respectively, and both ostensibly identified approximately 1,500 sex-associated CpGs. This similarity arises because unlike BCconf, Cate-RR and dSVA, IRW-SVA circumvents estimating Ω with a noisy estimate for L and estimates C by performing factor analysis directly on Y, restricted to the genomic units that show little marginal correlation with X. In fact, we use the proof of Theorem 1 in Sections S1.4 and S2 in the Supplement to show that under certain conditions on C, V_* and λ_K, which appear to be satisfied in these data, IRW-SVA can accurately recover C even when residuals are correlated. We believe K = 2 is the most appropriate choice of K for this dataset because CorrConf’s estimate for C appears to explain enough of the variance in methylation to achieve reasonable power, while also accurately recovering Ω to control for false discoveries and ensuring that the results are reproducible.

7. Discussion

To the best of our knowledge, we have developed the first method to account for latent factors in high throughput biological data with correlated samples. We proved that our estimate for K is consistent and that our estimate for Im (C) is accurate enough so that inference on the main effects β₁,…, β_p is just as accurate as when C is known. We also demonstrated our method’s finite sample properties by analyzing complex, multi-tissue simulated gene expression data, and used a real longitudinal DNA methylation data from a twin study to show our method tends to give more reproducible results compared to existing methods.

Our proposed procedure is certainly not a panacea for data with arbitrary correlation structure, and relies on the residual variance V_g being a linear combination of known matrices. Data with more complex, non-linear sample correlation structure may not be amenable to (2b), since a linear combination of p non-linear functions will not necessarily have an apriori known functional form. This could be an interesting area of future research.