Large-Scale Simultaneous Testing of Cross-Covariance Matrices with Applications to PheWAS

Abstract

Motivated by applications in phenome-wide association studies (PheWAS), we consider in this paper simultaneous testing of columns of high-dimensional cross-covariance matrices and develop a multiple testing procedure with theoretical guarantees. It is shown that the proposed testing procedure maintains a desired false discovery rate (FDR) and false discovery proportion (FDP) under mild regularity conditions. We also provide results on the magnitudes of the signals that can be detected with high power. Simulation studies demonstrate that the proposed procedure can be substantially more powerful than existing FDR controlling procedures in the presence of correlation of unknown structure. The proposed multiple testing procedure is applied to a PheWAS of two auto-immune genetic markers using a rheumatoid arthritis patient cohort constructed from the electronic medical records of Partners Healthcare System.

1. Introduction

Simultaneously assessing the associations among a large number of variables is an important problem in statistics with a wide range of applications. For example, large-scale testing tools are frequently needed for biomedical studies such as genome-wide association studies (GWAS) (Bush and Moore (2012)); high throughput gene expression profiling studies of microRNAs and mRNAs (Katagiri and Glazebrook (2009); Nelson et al. (2004)); and phenome-wide association studies (PheWAS) which examine the relationships between a large number of disease phenotypes and some candidate genomic markers (Denny et al. (2010)).

A critical step in performing a large-scale multiple testing is to control the false discovery rate (FDR). Standard FDR control procedures, such as the Benjamini-Hochberg (BH) procedure (Benjamini and Hochberg (1995)), typically built under the independence assumption, would fail to provide desired error controls in the presence of strong correlation. Alternative methods that allow for the general dependency, including the Benjamini-Yekutieli (BY) procedure (Benjamini and Yekutieli (2001)) and those proposed in Romano and Shaikh (2006a,b) and Guo and Rao (2008), tend to be overly conservative (see simulation results in Section 4). Furthermore, existing methods largely focus on the association between a single outcome variable with a large number of candidate covariates ((Efron, 2004, 2007; Owen, 2005; Fan et al., 2012; Fan and Han, 2017, e.g.)). However, multivariate outcomes are often of interest in many applications such as the PheWAS. These aforementioned methods may not be valid or powerful for such PheWAS settings due to the more complex correlation structure of the test statistic and the exact null distribution being unknown.

In this paper, we formulate the problem of assessing the association between a large number of variables and a vector of outcomes as the statistical problem of simultaneous testing of columns of high-dimensional cross-covariance matrices; and develop a general framework for such a testing problem without requiring strong assumptions on the correlation structures.

1.1. The Problem

We consider a problem of multiple testing for columns of high-dimensional cross-covariance matrices. Let $\mathcal{D}=\left\{{\left({\mathit{Y}}_k^{\prime },{\mathit{X}}_k^{\prime }\right)}^{\prime },1\le k\le n\right\}$, where Y_k = (Y_k1, …, Y_kd)^′ and X_k = (X_k1, …, X_kp)^′, be a random sample consisting of n independent and identically distributed copies of (Y, X). In the PheWAS setting, Y may be a vector of genomic markers and X represents all phenotypic disease conditions of interest. For 1 ≤ i ≤ p, define the cross-covariance vector between Y and X_i by ${\sigma }_i={\left({\sigma }_1^{(i)},\dots ,{\sigma }_d^{(i)}\right)}^{\prime }$, where ${\sigma }_j^{(i)}=Cov\left(Y_j,X_i\right)$. Thus, σ_i is the ith column of the cross-covariance matrix

$${\Sigma }_{YX}\equiv \text{Cov}(\mathit{Y},\mathit{X})=\left[{\sigma }_1,\dots ,{\sigma }_p\right]$$

between Y and X. We wish to simultaneously test the collection of p hypotheses

$$H_{0i}:{\mathit{\sigma }}_i=0\text{\hspace{1em} versus \hspace{1em}}{H}_{1i}:{\mathit{\sigma }}_i\ne 0,1\le i\le p$$ (1.1)

based on the random sample $\mathcal{D}$, while controlling the overall false discovery rate (FDR) and false discovery proportion (FDP). We are interested in multiple testing of the columns, not individual entries, of the cross-covariance matrix Σ_YX. Here the ith hypothesis examines whether the marker vector Y is associated with the ith disease condition. We are particularly interested in the setting where the number of the true alternative hypotheses is relatively small as in the case of the applications in PheWAS. Here, we consider linear associations between Y and X but note that transformations can be performed on the original data. For example we typically transform the ICD9 code X_j as log(1 + X_j) to increase the power of analysis since the count of ICD9 codes tend to be highly skewed.

Large-scale multiple testing of the columns of a high-dimensional cross-covariance matrix is technically difficult due to the complex entrywise dependence structures. No existing multiple testing procedures can be applied to this problem to provide accurate FDR or FDP control. Available methods and theoretical results largely focus on z- or t-tests for univariate Y and commonly require the knowledge of the null distribution. For example, Fan et al. (2012) and Fan and Han (2017) considered multiple testing for normal means and required the covariance matrix to be known or well estimated. Efron (2004, 2007) developed FDR controlling procedures for multiple t-tests. The cross-covariance testing considered here is more complicated due to the involvement of multiple outcomes and the unknown and complicated dependence structure of the entries of the cross-covariance matrix. The current setting necessitates the use of test statistics with only approximate null distributions. No existing theoretical results can ensure accurate control of the FDR and FDP under such complex dependence structure without assuming that the null distribution of the test statistics is known.

The choice of the null distribution may substantially affect the simultaneous inference procedure (Efron (2004); Liu and Shao (2014)). In fact, Liu and Shao (2014) showed that in multiple t-tests with the dimension much larger than the sample size, if the p-values are calculated from the asymptotic distribution, such as normal distribution or t-distribution, then the FDR and FDP of the BH method can converge to one. It is thus critical to justify the use of asymptotic null distribution for proper FDR control. If the number of true signals p₁ ≥ cp for some c > 0, the method in Storey et al. (2004) incorporates the estimation of the proportion of the true nulls into the BH method. In the PheWAS setting, the signal is sparse and p₁ is of order o(p), which leads to the equivalence of the Storey et al. (2004) and BH procedures. Due to the sparsity, the strong and complex dependence between the test statistics as well as the need to estimate the null distribution, neither of the Storey et al. (2004) and BH procedures is able to control FDR or FDP in our setting, as confirmed via our simulation studies.

We first discuss the motivating application of PheWAS with multiple outcomes and then discuss the challenges in large-scale multiple testing of columns of high-dimensional cross-covariance matrices and summarize the main contributions of the paper.

1.2. PheWAS of a Set of Genomic Markers

Complementary to GWAS, PheWAS investigates the association between a set of candidate genomic markers and a diverse range of phenotypes. PheWAS enables the discovery of genetic markers with pleiotropic effects, and thus may provide a broader view of the relationship between genetic variation and networks of phenotypes (Pendergrass et al. (2013); Hall et al. (2014)). This is highly desirable since recent genetic studies have suggested that many genetic loci appear to harbor variants associated with multiple traits (Solovieff et al. (2013)).

PheWAS has only recently become feasible due to the wide availability of the electronic medical record (EMR) systems linked with biorepositories. The EMR system provides detailed patient level phenotypic data including ICD9 codes for a wide range of disease conditions. To enable efficient genetic research, research institutions such as Partners Healthcare System (PHS) also link the EMR to bio-specimen repositories that collect blood samples. Anonymized patient level phenotype data linked with genotype data can be extracted and stored for clinical research (Liao et al. (2015)). Through such a process, a cohort of 1837 rheumatoid arthritis (RA) patients with their genomic and phenotypic information available has been established at PHS for discovery research (Liao et al. (2010); Kurreeman et al. (2011); Liao et al. (2013)).

Such linked EMR data enables us to conduct PheWAS to rigorously study the association between a large number of disease phenotypes X and any given set of candidate genomic markers Y. In contrast to GWAS, the large number of phenotypic variables in PheWAS are often substantially correlated. The complex yet unknown correlation along with the multi-dimensionality of Y contribute significantly to the difficulty of the multiple testing problem. We translate the PheWAS problem into the problem of testing high dimensional cross-covariance matrices and our proposed procedures can overcome such difficulties to draw valid conclusions from the PheWAS.

1.3. Our Contributions

There are two main challenges for simultaneous testing of columns of high-dimensional cross-covariance matrices: (i) the construction of suitable test statistics for individual hypotheses and providing proper estimates for the null distribution of the test statistics; (ii) the construction of a good procedure to account for the multiplicity of the tests so that the overall FDR and FDP are controlled in the presence of dependency and additional approximation error due to the use of the estimated null distribution. We summarize the main contributions of this paper as follows.

1. We propose a test statistic T_i that mimics the Hotelling′s T² statistic for testing an individual hypothesis H_0i. It is shown that in the presence of correlation induced by the observed data, the null distribution of this test statistic can be well approximated by a χ² distribution within an appropriate range.

2. We develop a large-scale multiple testing procedure for {H_0i, i = 1, …, p} by thresholding the test statistics{T_i, i = 1, …, p}with theoretical guarantees for FDR and FDP control under flexible correlation structures. Both theoretical and numerical properties are investigated. It is proven that, under mild regularity conditions, the proposed multiple testing procedure based on the asymptotic null distribution of T_i controls both the FDR and FDP. In addition, we study the power property of the procedure and provide results on the magnitudes of the signals that can be detected with high power. Simulation studies demonstrate that the proposed methods can be substantially more powerful than existing FDR controlling procedures in the presence of correlation.

3. Our methods also contribute to the practice of PheWAS using EMR data. Standard PheWAS defines each disease phenotype X_j based on whether a patient has at least 1 or 2 ICD9 billing codes and then tests for the association between the binary phenotype and a single marker Y (Denny et al. (2010); Liao et al. (2013)). Our proposed procedures allow clinical investigators to make use of the counts of ICD9 codes, which could be more powerful and more robust since it is difficult to choose the appropriate threshold for each disease as the ICD9 codes have varying degree of accuracy (Liao et al. (2010)).

Our methods allow one to identify the subset of phenotypes that are associated with multiple, say d, related markers as a group. An alternative strategy is to perform FDR-controlled marginal testing of dp hypotheses to identify the set of phenotype and marker pairs that are associated. However, this approach is generally less powerful than the proposed approach based on Hotelling T statistics. Similar patterns have been previously reported when comparing entry-wise testing versus group level testing ((Xia et al., 2017, e.g.).)

4. The proposed procedure is applied to a PheWAS of two important auto-immune genetic markers using the EMR RA cohort described above. The goal is to comprehensively evaluate how these genetic variations may contribute to comorbidities of RA. The results show that these risk alleles are potentially associated with RA severity, chronic fatigue syndrome, back pain and anemia.

1.4. Structure of the Paper

The rest of the paper is organized as follows. The proposed methodology detailing the test statistics for individual hypotheses as well as the simultaneous testing procedure is introduced in Section 2. A theoretical analysis of the multiple testing procedure, presented in Section 3 and proved in the supplementary materials, shows that the proposed method controls the FDR and the false discovery proportion (FDP) at a desired nominal level asymptotically. We discuss how the proposed procedure relates to and differs from existing FDR controlling methods in Section 3.3. Results from simulation studies are given in Section 4, along with those of the application of the proposed procedures to the aforementioned RA cohort. Section 5 discusses a few related issues. Proofs for the theoretical results are given in the Supplementary Materials.

2. Methodology

We detail the proposed multiple testing procedure in this section. We first introduce the test statistics for testing the individual hypothesis H_0i : σ_i = 0. Let ${\widehat{\mathit{Z}}}_{ki}=\left({\mathit{Y}}_k-\overline{\mathit{Y}}\right)\left(X_{ki}-{\overline{X}}_i\right)$ where $\overline{\mathit{Y}}=n^{-1}{\sum }_{k=1}^n{\mathit{Y}}_k$ and ${\overline{X}}_i=n^{-1}{\sum }_{k=1}^n{X}_{ki}$ for 1 ≤ i ≤ p and 1 ≤ k ≤ n. Then

$${\widehat{\mathbf{\sigma }}}_i=n^{-1}\sum _{k=1}^n{\widehat{\mathit{Z}}}_{ki}$$

is a consistent and asymptotically unbiased estimator of σ_i, the covariance vector between Y and X_i. A test for H_0i may be constructed based on the observed $n^{1/2}{\widehat{\mathbf{\sigma }}}_i$ along with its covariance matrix that can be approximated using the sample covariance matrix of $\left\{{\widehat{\mathit{Z}}}_{ki},k=1,\dots ,n\right\}$,

$${\widehat{\Sigma }}_{Zi}=n^{-1}\sum _{k=1}^n\left({\widehat{\mathit{Z}}}_{ki}-{\widehat{\mathbf{\sigma }}}_i\right){\left({\widehat{\mathit{Z}}}_{ki}-{\widehat{\mathbf{\sigma }}}_i\right)}^{\prime }.$$

Inspired by the Hotelling′s T² statistic for testing a multivariate normal mean vector, we propose to test H_0i using the test statistic

$$T_i=n{\left({\widehat{\mathbf{\sigma }}}_i\right)}^{\prime }{\left({\widehat{\mathrm{\Sigma }}}_{Zi}\right)}^{-1}{\widehat{\mathbf{\sigma }}}_i.$$ (2.2)

Let ${\mathcal{H}}_0=\left\{i:{\mathbf{\sigma }}_i=0\right\}$ be the index set of all null hypotheses, ${\mathcal{H}}_1=\left\{i:{\mathbf{\sigma }}_i\ne 0\right\}$, and $p_0=Card\left({\mathcal{H}}_0\right)$. As indicated in Lemma ?? in the Supplementary Materials, under the conditions of Theorem 1,

$$\underset{i\in {\mathcal{H}}_0}{\text{max}}\left|\frac{\mathbb{P}\left(T_i\ge t\right)}{G(t)}-1\right|\to 0$$

uniformly in t ∈ [0, a_p], where $G(t)=\mathbb{P}\left({\chi }_d^2\ge t\right)$ and

$$a_p=2\text{\hspace{0.17em}log\hspace{0.17em}}p+(d-1)\text{\hspace{0.17em}log\hspace{0.17em}log\hspace{0.17em}}p.$$ (2.3)

Hence, when n is large, the chi-squared distribution provides an accurate approximation to the null distribution of T_i in the range [0, a_p].

We next propose an FDR-controlled multiple testing procedure by thresholding the test statistics {T_i, i = 1, …, p}. Specifically, let t > 0 be a rejection threshold so that H_0i is rejected if and only if T_i ≥ t. For any given threshold t > 0, the false discovery proportion (FDP) based on the random sample $\mathcal{D}$ is

$$\text{FDP}(t)=\frac{{\sum }_{i\in {\mathcal{H}}_0}I\left(T_i\ge t\right)}{\text{max}\left\{{\sum }_{i=1}^{p}I\left(T_i\ge t\right),1\right\}}.$$ (2.4)

To maximize the power of the test or equivalently the rejection rate among ${\mathcal{H}}_1$ while maintaining an FDP level of α, the optimal threshold t is then

$${\widehat{t}}_0=\text{inf}\{t:\text{FDP}(\text{t})\le \alpha \}.$$

Since the denominator of FDP(t) in (2.4) is observable, the key to empirically control the FDP is to find a good estimate of the numerator ${\sum }_{i\in {\mathcal{H}}_0}I\left(T_i\ge t\right)$. We will show that

$$\underset{0\le t\le b_p}{\text{sup}}\left|\frac{{\sum }_{i\in {\mathcal{H}}_0}I\left(T_i\ge t\right)}{p_{0}G(t)}-1\right|\to 0\text{\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}in\hspace{0.17em}probability},$$

where

$$b_p=2\text{\hspace{0.17em}log\hspace{0.17em}}p+(d-3)\text{\hspace{0.17em}log\hspace{0.17em}}(\text{\hspace{0.17em}log\hspace{0.17em}}p).$$

The range [0, b_p] is nearly optimal. When t ≥ b_p + log(log p), G(t) may not consistently estimate $p_0^{-1}{\sum }_{i\in {\mathcal{H}}_0}I\left(T_i\ge t\right)$. Here p₀ can be further estimated by p due to the sparsity in the number of alternative hypotheses in many data applications. Based on this analysis, we propose a multiple testing procedure for simultaneously testing the hypotheses in (1.1):

FDR control procedure.

Calculate T_i in (2.2) and, for any given nominal FDR level α ∈ (0, 1), reject H_0i whenever $T_i\ge \widehat{t}$, where

$$\widehat{t}=\text{inf}\left\{0\le t\le b_p:G(t)\le \frac{a\text{max}\left\{{\sum }_{1\le i\le p}I\left(T_i\ge t\right),1\right\}}{p}\right\}$$ (2.5)

if the righthand side of (2.5) exists; and $\widehat{t}=a_p$ otherwise.

When $\widehat{t}$ in (2.5) does not exist, the FDR control procedure simply thresholds the test statistics at the value a_p given in (2.3). Unlike the conventional BH procedure, the proposed thresholding rule enables the corresponding multiple testing procedure to control both the FDR and the FDP. As shown numerically in Section 4, if the number of alternatives p₁ is small, then $\widehat{t}$ in (2.5) may not exist and the true FDP of the BH method can be much higher than α. In this case, the proposed method uses a_p as the thresholding level adaptively and is able to control the FDP efficiently. See more discussion in Section 5.

3. Theoretical results

We investigate in this section the theoretical properties of the proposed multiple testing procedure and discuss its relation to existing methods. Proof of these results can be found in the Supplementary Materials. Numerical performance of the procedure will be studied in Section 4. We first state some conditions on the correlation structure. Let Z_i = (Y − μ_Y)(X_i − μ_i), Σ_Zi = Cov(Z_i) and ${\xi }_i={\Sigma }_{Zi}^{-1/2}\left(Z_i-{\mathbf{\sigma }}_i\right)$. As in canonical correlation analysis, define the maximum correlation coefficients ${\rho }_{ij}^{*}={\text{max}}_{\Vert a\Vert =1,\Vert b\Vert =1}\left|corr\left({\mathit{a}}^{\prime }{\xi }_i,{\mathit{b}}^{\prime }{\xi }_j\right)\right|$, where ∥·∥ denotes the Euclidean norm. ${\rho }_{ij}^{*}$ characters the dependence between T_i and T_j.

Define

$$\mathcal{B}(\delta )=\left\{(i,j):i\in {\mathcal{H}}_0,j\in {\mathcal{H}}_0,{\rho }_{ij}^{*}\ge \delta ,i\ne j\right\}\text{\hspace{1em} with }\delta \in (0,1).$$

The set $\mathcal{A}(\epsilon )=\mathcal{B}\left\{{(\text{log\hspace{0.17em}}p)}^{-2-\epsilon }\right\}$ includes the pairs ξ_i and ξ_j that are strongly correlated for $i,j\in {\mathcal{H}}_0$. The first condition requires the number of strongly correlated pairs to be not too large.

(C1). There exist some ε > 0 and some δ > 0 such that

$$\sum _{(i,j)\in \mathcal{A}(\epsilon )}{p}^{\frac{2{\rho }_{ij}^{*}}{1+{\rho }_{ij}^{*}}+\delta }=O\left\{p^2{(\text{log\hspace{0.17em}}p)}^{-2}\right\}.$$ (3.6)

Remark 1. When (Y′, X′) has an elliptically contoured distribution, it is easy to see that ${\rho }_{ij}^{*}\le \left|{\rho }_{ij}\right|$ for $i,j\in {\mathcal{H}}_0$, where (ρ_ij)_1≤i,j≤p is the correlation matrix of X. In this case, (3.6) is reduced to

$$\sum _{(i,j)\in {\mathcal{A}}_1(\epsilon )}{p}^{\frac{2\left|{\rho }_{ij}\right|}{1+|p_{ij}]}+\delta }=O\left\{p^2{(\text{log\hspace{0.17em}}p)}^{-2}\right\},$$ (3.7)

where ${\mathcal{A}}_1(\epsilon )$ is defined as $\mathcal{A}(\epsilon )$ with ${\rho }_{ij}^{*}$ replaced by |ρ_ij|. Condition (3.6) holds if $\text{Card}\{\mathcal{B}(\delta )\}=O\left(p^{\rho }\right)$ for any 0 < δ < 1 and some ρ < 2/(1 + δ), and $\text{Card}\{\mathcal{A}(\epsilon )\}=O\left(p^{\rho }\right)$ for some ρ < 2 and ε > 0. The correlation condition (3.6) can be further weakened if the number of signals becomes larger and can be easily satisfied in many applications. For example, in the scale-free network, only a few variables are associated with many variables and most of variables are only associated with a few others. In PheWAS settings, most diseases only have a few co-morbidities and hence these assumptions for $\text{Card}\{\mathcal{B}(\delta )\}$ and $\text{Card}\{\mathcal{A}(\epsilon )\}$ are reasonable.

Remark 2. When $i\in {\mathcal{H}}_0$ and $j\in {\mathcal{H}}_0$, we have Y uncorrelated with (X_i, X_j). If we assume that Y is independent from (X_i, X_j) for $i\in {\mathcal{H}}_0$ and $j\in {\mathcal{H}}_0$, then ${\rho }_{ij}^{*}=\left|{\rho }_{ij}\right|$. In this case, (C1) is reduced to a correlation condition on X which is quite natural.

3.1. FDR and FDP Control

For the multiple testing procedure defined in Section 2, the FDP and FDR are given by

$$\text{FDP}=\frac{{\sum }_{i\in {\mathcal{H}}_0}I\left(T_i\ge \widehat{t}\right)}{\text{max}\left\{{\sum }_{1\le i\le p}I\left(T_i\ge \widehat{t}\right),1\right\}}\text{\hspace{1em} and \hspace{1em}FDR}=\mathbb{E}(\text{FDP}).$$

The next two theorems show that the proposed multiple testing procedure controls FDR and FDP asymptotically. Furthermore, the actual FDR and FDP converge to αp₀/p asymptotically when the number of non-trivial signals,

$$m_1(c)=\text{Card}\left\{i:1\le i\le p,{\mathbf{\sigma }}_i^{\prime }{\Sigma }_{Zi}^{-1}{\mathbf{\sigma }}_i\ge c(\text{log\hspace{0.17em}}p)/n\right\},$$

is not too small for some c. Let λ_i1 and λ_id be the largest and smallest eigenvalues of Σ_Zi, respectively.

Theorem 1.

Suppose that p ≤ n^β for some β > 0, $\mathbb{E}{\Vert Y-{\mu }_Y\Vert }^{8\beta +4+ϵ}\le K$, ${\text{max}}_{1\le i\le p}\mathbb{E}|X_i-{{\mu }_i|}^{8\beta +4+ϵ}\le K$ and c₁ ≤ λ_id ≤ λ_i1 ≤ c₂ for all 1 ≤ i ≤ p and some ϵ > 0, K > 0, c₁ > 0 and c₂ > 0. Under (C1), we have, for any ε > 0,

$$\underset{(n,p)\to \infty }{\text{lim}}\mathbb{P}(\text{FDP}\le \alpha +\epsilon )=1and\underset{(n,p)\to \infty }{\text{lim\hspace{0.17em}sup}}\text{FDR}\le \alpha .$$

Theorem 2.

Suppose the conditions in Theorem 1 hold. If

$$m_1(c)\ge \text{log\hspace{0.17em}}pforsomec>2,$$ (3.8)

then

$$\underset{(n,p)\to \infty }{\text{lim}}\frac{\text{FDR}}{\alpha p_0/p}\to 1and\frac{\text{FDP}}{\alpha p_0/p}\to 1inprobabilityas(n,p)\to \infty .$$ (3.9)

When additional assumptions are imposed on the sparsity and strength of the signals, the condition on the correlation matrix R can be further relaxed. For example, under

$$m_1(c)\ge p^{\theta }\text{\hspace{1em} for some }0<\theta <1\text{ and }c>2(1-\theta ),$$ (3.10)

(C1) can be weakened as follows.

$$\left(\text{C}{1}^{*}\right).\sum _{(i,j)\in \mathcal{A}(\epsilon )}{p}^{\frac{2(1-\theta ){\rho }_{ij}^{*}}{1+{\rho }_{ij}^{*}}+\delta }=O\left\{p^2/{(\text{log\hspace{0.17em}}p)}^2\right\},\text{\hspace{1em} for some }\epsilon >0\text{ and }\delta >0.$$

Under these alternative conditions, we have similar results on the FDR and FDP control.

Theorem 3.

Suppose that p ≤ n^β for some β > 0, $\mathbb{E}{\Vert \mathit{Y}-{\mathit{\mu }}_Y\Vert }^{8\beta +4+ϵ}\le K$, ${\text{max}}_{1\le i\le p}\mathbb{E}|X_i-{{\mu }_i|}^{8\beta +4+ϵ}\le K$ and c₁ ≤ λ_id ≤ λ_i1 ≤ c₂ for all 1 ≤ i ≤ p and some ϵ > 0, K > 0, c₁ > 0 and c₂ > 0. Under (C1*) and (3.10), we have (3.9) holds.

When the number of non-trivial signals increases to the magnitude of m₁(c) = p/(log p)^λ for some λ > 0 and c > 0, we only require $\text{Card}\{\mathcal{A}(\epsilon )\}\le p^{2-\delta }$ for some δ > 0. The number of pairs (ξ_i, ξ_j) with non-trivial correlations can be as large as p^2−δ.

When d = 1, Owen (2005) considered multiple tests assessing the association between Y and X_i for i = 1, …, p based on the sample correlation coefficients. He developed the variance of the number of falsely rejected hypotheses ${\sum }_{i\in {\mathcal{H}}_0}I\left\{\left|{\widehat{\rho }}_i\right|\ge t\right\}$ for a fixed t > 0. He showed that the variance can be affected significantly by the correlations between X_i, 1 ≤ i ≤ p. No FDR controlling procedures were provided. When considering FDR control, our results obtained under different sets of conditions suggest that the effect of the correlations from X is related to the number of signals. As θ in (3.10) increases, the total number of signals increases and the condition on the correlation becomes weaker.

3.2. Power Properties

We now consider the power of the procedure. Define the power by

$$\widehat{PO}=\frac{{\sum }_{i\in {\mathcal{H}}_1}I\left(T_i\ge \widehat{t}\right)}{\text{Card}\left({\mathcal{H}}_1\right)}.$$

Suppose (3.10) holds and the magnitude of all signals in ${\mathcal{H}}_1$ satisfies

$${\mathbf{\sigma }}_i^{\prime }{\Sigma }_{Zi}^{-1}{\mathbf{\sigma }}_i\ge \frac{c\text{\hspace{0.17em}log\hspace{0.17em}}p}{n}\text{\hspace{1em} for }i\in {\mathcal{H}}_1.$$ (3.11)

Theorem 4.

Suppose that p ≤ n^β for some β > 0, $\mathbb{E}{\Vert Y-{\mu }_Y\Vert }^{8\beta +4+ϵ}\le K$, ${\text{max}}_{1\le i\le p}\mathbb{E}|X_i-{{\mu }_i|}^{8\beta +4+ϵ}\le K$ and c₁ ≤ λ_id ≤ λ_i1 ≤ c₂ for all 1 ≤ i ≤ p and some ϵ > 0, K > 0, c₁ > 0 and c₂ > 0. If (3.10) and (3.11) hold, then

$$\widehat{PO}\to 1inprobability.$$

Theorem 4 shows that the proposed multiple testing procedure has overwhelming power in detecting the signals whose magnitudes satisfy (3.11). The constant factor c in (3.11) cannot be replaced by o(1). Otherwise, it is not even possible to detect the global signals (σ_i ≠ 0 for some i); see Donoho and Jin (2004) in the setting of signal detection under a sparse normal mixture model.

3.3. Relation with the existing FDR control methods

Under the PheWAS setting with multi-dimensional Y, controlling the FDR or FDP is more complicated due to the non-trivial dependence among the test statistics, the need to estimate the null distribution of the test statistics or p-values and the sparsity of the signals. Most existing FDR control procedures require the exact p-values. When the number of the true signals is fixed as the dimension p → ∞, Proposition 2.1 in Liu and Shao (2014) shows that the BH procedure is unable to control the FDP. The PheWAS setting is even more challenging with the additional complication of strong and complex dependence among the test statistics.

Compared with the BH procedure, the proposed method differs in the additional thresholding step, which is critical in controlling the FDR and FDP. To see the differences in the properties of the two procedures, we investigate the theoretical performance of the BH procedure. Let the p-values p_i = G(T_i). The BH procedure rejects H_0i if ${\text{p}}_i\le {\text{p}}_{(\widehat{k})}$, where the p-values{p_i, 1 ≤ i ≤ p} are sorted as p₍₁₎ ≤ ⋯ ≤ p_(p) and $\widehat{k}$ satisfies

$$\widehat{k}=\text{max}\left\{k:{\text{p}}_{(k)}\le \alpha k/p\right\}.$$

The BH procedure is equivalent to rejecting H_0i if $T_i\ge {\widehat{t}}_{BH}$, where $\widehat{t}$ is defined by

$${\widehat{t}}_{BH}=\text{inf}\left\{t\ge 0:G(t)\le \frac{\alpha \text{\hspace{0.17em}max}\left\{{\sum }_{1\le i\le p}I\left(T_i\ge t\right),1\right\}}{p}\right\}.$$ (3.12)

The FDR and FDP of the BH procedure are given by

$${\text{FDP}}_{BH}=\frac{{\sum }_{i\in {\mathcal{H}}_0}I\left\{T_i\ge {\widehat{t}}_{BH}\right\}}{\text{max}\left(1,{\sum }_{1\le i\le p}I\left\{T_i\ge {\widehat{t}}_{BH}\right\}\right)}\text{\hspace{1em} and \hspace{1em}}{\text{FDR}}_{BH}=\mathbb{E}\left[{\text{FDP}}_{BH}\right].$$

We show that the BH method cannot control the FDP if the number of true alternatives $\left|{\mathcal{H}}_1\right|$ is fixed as p → ∞. To simplify the proof, we illustrate the point in the case that Σ = Cov(X) is diagonal.

Proposition 1. Suppose the conditions in Theorem 3.1 hold and Σ is diagonal. If $\left|{\mathcal{H}}_1\right|$ is fixed as p → ∞, then for any 0 < ξ < 1, we have

$$\underset{(n,p)\to \infty }{\text{lim\hspace{0.17em}inf\hspace{0.17em}}}\mathbb{P}\left({\text{FDP}}_{BH}\ge \xi \right)\ge \eta$$ (3.13)

for some η > 0 which may depend on ξ and ${\mathcal{H}}_1$.

This proposition indicates that in the extremely sparse case, the BH method is not suitable for the control of FDP. In contrast, Theorem 3.1 shows that our procedure can still control the FDP when $\left|{\mathcal{H}}_1\right|$ is fixed.

When $\left|{\mathcal{H}}_1\right|$ goes to infinity at certain rates, we will have $\widehat{t}\le b_p$ with probability tending to one. In this case, $\widehat{t}$ in (2.5) exists and our procedure is equivalent to the BH method. So the BH method can control the FDP/FDR asymptotically.

Proposition 2. Suppose the conditions in Theorem 3.1 hold. If (3.8) holds, we have

$$\underset{(n,p)\to \infty }{\text{lim}}\frac{{\text{FDR}}_{BH}}{\alpha p_0/p}=1and\frac{{\text{FDP}}_{BH}}{\alpha p_0/p}\to 1inprobabilityas(n,p)\to \infty .$$

When ${\widehat{t}}_{BH}\le b_p$, our procedure coincides with the BH procedure; if ${\widehat{t}}_{BH}\le b_p$, then our procedure rejects hypotheses with test statistics exceeding ap. This threshold is necessary since the asymptotic null distribution $G(t)=\mathbb{P}\left({\chi }_d^2\ge t\right)$ may not approximate $p_0^{-1}{\sum }_{i\in {\mathcal{H}}_0}I\left(T_i\ge t\right)$ sufficiently well for large t in that $p_0^{-1}{\sum }_{i\in {\mathcal{H}}_0}I\left(T_i\ge t\right)/G(t)\cancel{\to }1$ when t ≥ b_p + log(log p). As shown in our simulation studies, the probability of ${\widehat{t}}_{BH}>b_p$ approaches 1 under extreme sparsity, and is non-trivial under moderate sparsity. This also sheds light on why the proposed procedure controls the FDR and FDP while the BH procedure fails under the PheWAS setting. As shown in Theorem 3.1, our thresholding rule based procedure adaptively controls the FDR and FDP without prior knowledge of the degree of sparsity.

4. Numerical Results

We investigated the numerical performance of the proposed multiple testing procedure through simulation studies. Numerical comparison with alternative methods is given. The proposed procedure was applied to an RA cohort to comprehensively evaluate how two important genetic markers for auto-immune diseases can contribute to comorbidities of RA.

4.1. Simulation Studies

We performed extensive simulations to examine the performance of the procedure in finite sample with practical sample sizes and p. We let d = 4 for the outcome Y, considered p = 500, 1000 and 2000 for X, and the sample size n = 100 and 150. Under each configuration, the results are summarized based on 500 simulated datasets for FDR estimates and 100 simulated datasets for power calculations. For each dataset, we performed testing based on our methods and several existing methods In addition to to the BH, BY methods, we compared to the Storey et al. (2004) (JS) procedure, the Romano and Shaikh (2006b) step-up procedure controlling FDP at 50% with probability at least 1 − α (RSuFDP), the Romano and Shaikh (2006a) step-down procedure controlling FDP at 50% with probability at least 1 − α (RSdFDP) and controlling FDR at α (RSdFDR), as well as the Guo and Rao (2008) procedure (GR).

We generated the entire data vector W := (Y₁, … , Y₄, X₁, … , X_p)^′ from W = Σ^1/2ε, with two settings of Σ, as described below, were chosen to reflect different correlation structures, and ε = (ε₁, … , ε_p+4)^′ are independent and identically distributed. The distribution of ε_i is taken to be (i) the standard normal N(0, 1); (ii) the exponential with mean 1; and (iii) a mixture of N(−1, 0.5²) with probability 0.1 and N(0, 0.5²) with probability 0.9. As our test statistics are invariant to the variances, we take the diagonal entries of Σ to be 1. Let ${\Sigma }_1=\left({\mathbf{\sigma }}_{ij1}\right)\in {ℝ}^{\left(p_1+4\right)\times \left(p_1+4\right)}$ and ${\Sigma }_2=\left({\mathbf{\sigma }}_{ij2}\right)\in {ℝ}^{\left(p-p_1\right)\times \left(p-p_1\right)}$. In all models, we let Σ = diag(Σ₁, Σ₂) and ${\mathbf{\sigma }}_{ij1}={\{(2+\delta )\text{log\hspace{0.17em}}p/n\}}^{I(i\ne j)/2}$.

• Model 1: ${\sigma }_{ij2}={\wp }^{|j-i|}$. Here $\wp =0.5$ in Model (1A) and $\wp =0.8$ in Model (1B).

• Model 2: Σ₂ = diag(D₁, … , D_m, I), where m = [(p − p₁)/10], $D_k\in {ℝ}^{10\times 10}$, 1 ≤ k ≤ m. I is the identity matrix. All off-diagonal entries of D_k are taken to be $\wp$. We let $\wp =0.8$ in Model (2A) and $\wp =0.5$ in Model (2B).

Under these models, only X₁, X₂, …, X_p1 are correlated with Y.

We examined the probability that $\widehat{t}$ in (2.5) exists under different settings, which reflects the degree to which our procedure differs from the BH procedure. Figure 1 summarizes the frequency that $\widehat{t}$ in (2.5) exists when δ = −0.5, α = 0.05 and 0.10, n = 100 and p = 1000. The probability that $\widehat{t}$ does not exist approaches 1 when the signals are extremely sparse, is non-trivial under moderate sparsity and gradually decreases to near 0 as p₁ further increases. These results demonstrate that the proposed procedure differs substantially from the BH procedure under sparse settings.

Figure 1:

The frequency that $\widehat{t}$ exists when n = 100, p = 1000 with normal error.

We next investigated the performance of various procedure in controlling for the FDR in finite samples. Figures 2 and 3 present the empirical FDR for α = 0.05 and 0.1 when δ = −0.5 with various choices of p, p₁ and n. The results show that the proposed procedures maintain the desired FDR levels well when p₁ = 30. For p₁ = 10, our method also generally maintains the FDR level but with slight inflation for a few scenarios when n = 100. On the other hand, the BH procedure tends to result in inflated FDR under a variety of settings and the inflation can be quite substantial for some scenarios. The JS procedure, requiring an estimate of the null proportion, tends to give substantially inflated FDR, due to the inaccurate estimate of the null proportion. All other procedures, including the BY and FDP controlling procedures are able to control the FDR but tend to be conservative with empirical FDR substantially lower than the target levels. For example, when p = 2000, p₁ = 10, n = 100, and α = 0.1, the empirical FDR was 0.110, 0.111 and 0.068 for the proposed procedure, and 0.144, 0.151 and 0.097 for the BH procedure, under Model (1A) with mixture, normal and exponential error distributions, respectively. To better preserve the FDR, our procedure also demonstrates superior performance in maintaining the FDP when compared to the BH and the JS procedure, as illustrated in Figure 4.

Figure 2:

Empirical FDR of various procedures, including the proposed, BH, BY, Storey et al. (2004) (JS), Guo and Rao (2008) (GR), the Romano and Shaikh (2006b) step-up procedure controlling FDP at 50% with probability at least 1 - α (RSuFDP), the Romano and Shaikh (2006a) step-down procedure controlling FDP at 50% with probability at least 1−α (RSdFDP) as well as the Romano and Shaikh (2006a) step-down FDR controlling procedure (RSdFDR), under the very sparse setting with p₁ = 10.

Figure 3:

Empirical FDR of various procedures, including the proposed, BH, BY, Storey et al. (2004) (JS), Guo and Rao (2008) (GR), the Romano and Shaikh (2006b) step-up procedure controlling FDP at 50% with probability at least 1 - α (RSuFDP), the Romano and Shaikh (2006a) step-down procedure controlling FDP at 50% with probability at least 1− α (RSdFDP) as well as the Romano and Shaikh (2006a) step-down FDR controlling procedure (RSdFDR), under the moderately sparse setting with p₁ = 30.

Figure 4:

Empirical survival distribution of the FDP under different models with p₁ = 30 or 10 when n = 100, p = 1000, α = 0.1 and the error is normal.

We now turn to the comparison of powers between our procedure and existing procedures that are robust to the dependence structure. We varied the signal strength by letting δ in σ_ij1 vary from −0.5 to 1. Figure 5 summarizes the empirical power of various procedures over different values of δ when n = 100, p = 2000 and α = 0.1. Across all settings, the power of our procedure is substantially higher than that of competing methods, with the advantage even more significant under the very sparse setting of p₁ = 10.

Figure 5:

Comparing power of different procedures under different models with p₁ = 30 or 10 when n = 100, p = 2000, α = 0.1 and the error is normal.

These numerical results are consistent with the theoretical analysis discussed earlier. Both the BH and the Storey procedures fail to control for the FDR or FDP due to the complex correlation structure and the estimated null distribution under the sparse setting. On the other hand, the BY procedure along with other competing FDR and FDP controlling procedures are overly conservative with substantially lower power compared to our method.

4.2. Application to PheWAS of Autoimmune Risk Alleles with EMR

We applied our procedure to an EMR cohort of 1837 RA subjects (Liao et al. (2010)). Blood samples were collected for these RA cases using the BWH Specimen Bank from 2009–2010 and a range of single nucleotide polymorphisms (SNPs) associated with various auto-immune diseases were genotyped (Kurreeman et al. (2011)). We limited our study to 1237 individuals of European ancestry, this was the majority (80%) of our cohort. For this analysis, we focused on the two major genetic risk factors for RA, the human leukocyte antigen shared epitope (HLA-SE) region (tag SNP rs6910071) and a loci in the PTPN22 gene (rs2476601). We were interested in conducting a PheWAS using these two SNPs to determine whether carriage of these risk alleles are associated with RA comorbidities. To perform the PheWAS, the ICD9 codes were grouped into clinically relevant diseases, termed as PheWAS code, as suggested in Denny et al. (2010). For each of the PheWAS codes, the value of the variable represents total count of ICD9 codes a patient received across all hospital encounters. This analysis included 352 PheWAS codes that have a prevalence of ≥ 2%, where the prevalence is calculated as the fraction of patients with non-zero counts. Since these counts are highly skewed, a log(x + 1) transformation was applied to each of the code count to obtain the p = 352 dimensional vector X for the association analysis. The 2-dimensional vector Y consists of the allele counts of the HLA-SE and PTPN22 SNPs, taking values 0, 1 or 2.

At an FDR level of 0.05, our testing procedures identified 4 phenotypes as significantly associated with the HLA-SE and PTPN22: RA (p-value = 3.8E-7), chronic fatigue syndrome (p-value = 1.1E-4), back pain (p-value = 1.2E-4) and anemias (p-value = 7.0E-4). It is interesting here that all 4 phenotypes are related to severity of the disease. Patients with more severe RA tend to have a higher number of RA ICD9 codes, suggesting more visits related to RA. Anemia of chronic disease is also frequently associated in RA patients with high disease activity and higher levels of inflammation (Masson (2011)). Our findings are consistent with previous studies demonstrating an association between carriage of the HLA-SE and RA disease severity (Weyand et al. (1992); Jaraquemada et al. (1986)) and hence is likely to be associated with the phenotypes associated with RA severity. Chronic fatigue syndrome (CFS), a multifactorial condition, is highly related to autoimmune diseases. High levels of inflammation result in symptoms of profound fatigue and may explain the association with chronic fatigue syndrome (CFS). In addition, recent gene expression studies have confirmed that several immune function related genes are candidate markers of CFS (Fang et al. (2006)). Since both the HLA-SE and PTPN22 genes are important in immune function and both pre-dispose to other autoimmune diseases (Criswell et al. (2005); Davidson and Diamond (2001)), it is interesting to see their association with CFS. When we applied the BY method to this example, only RA was deemed as significant at FDR level of 0.05. This again demonstrates that our proposed method is more powerful than the BY method.

5. Discussion

We introduced in this paper a multiple testing procedure for simultaneously testing columns of high-dimensional cross-covariance matrices. There is an important difference between our FDR control procedure given in (2.5) and the well-known BH procedure. When the test statistics T_i are used and the p-values are calculated from $\mathbb{P}\left({\chi }_d^2\ge t\right)$, the BH procedure is equivalent to rejecting H_0i whenever $T_i\ge {\widehat{t}}_{BH}$. For our procedure, if t_BH > b_p we do not use ${\widehat{t}}_{BH}$ but instead threshold at a_p = 2 log p+(d–1) log(log p) to control the FDP and FDR. Thus, for t > b_p, we do not use pG(t) to estimate ${\widehat{R}}_0(t)\equiv {\sum }_{i\in {\mathcal{H}}_0}I\left\{T_i\ge t\right\}$ while the BH method does. When pG(t) is bounded or converges to infinity slowly, it is not a good estimator for ${\widehat{R}}_0(t)$ and can even be inconsistent. For example, if we treat T_i as i.i.d. random variables with the cumulative distribution function 1 – G(t), then ${\widehat{R}}_0(t)$ is a binomial random variable with success probability G(t). So if pG(t) is bounded, then ${\widehat{R}}_0(t)$ approximately follows a Poisson distribution with rate p₀G(t) and pG(t) is no longer a consistent estimator for ${\widehat{R}}_0(t)$. Thus thresholding test statistics at ${\widehat{t}}_{BH}$ would lead to unstable behavior of the FDP and ultimately fail to control the FDR when the signals are very sparse.

We next argue that, if $\widehat{t}$ in (2.5) does not exist, then thresholding the test statistics at a_p = 2 log p + (d – 1) log(log p) is a reasonable way to control FDP. To explain, we assume the number of true alternative p₁ = 10. Let ${\widehat{m}}_0$ be the number of wrong rejections by any multiple tests procedure. Then FDP ≥ ${\widehat{m}}_0/\left(10+{\widehat{m}}_0\right)$. So, if we want to control FDP ≤ 0.05, for example, then it is necessary to make sure ${\widehat{m}}_0=0$. In this case, control FDP is equivalent to controlling FWER. For general fixed p₁, controlling FDP at level 1/(p₁ + 1) is essentially equivalent to controlling FWER. When p₁ is fixed as p → ∞, $\widehat{t}$ in (2.5) does not exist with probability tending to one and our procedure would simply threshold the test statistics at a_p to control FDP. In fact, when p₁ is fixed, Liu and Shao (2014) showed that the BH method is unable to control FDP at any level 0 < α < 1. Specifically, consider p tests with independent p-values ${\mathcal{P}}_1,\dots ,{\mathcal{P}}_p$. If ${\text{min}}_{i\in {\mathcal{H}}_1}{\mathcal{P}}_i=o_P\left(p^{-1}\right)$, then for any 0 < α < 1, there exists c₀ > 0 such that $\text{lim\hspace{0.17em}}{\text{inf}}_{p\to \infty }\mathbb{P}(\text{FDP}\ge \alpha )\ge c_0$.

Owen (2005) investigated the variance of the number of falsely rejected hypotheses under the assumption that all ρ_j = 0, 1 ≤ j ≤ p, where ρ_j is the correlation coeffcient between the univariate response Y and the covariates X_i. This work is related to the control of false discovery number (FDN) which is different from the control of FDR. In addition, Owen (2005) used the sample correlation coefficients as the test statistics so that the dependence structure between the test statistics can be calculated explicitly. Our test statistics are more complicated. It is difficult to calculate the correlation between T_i and T_j. Hence, the results in Owen (2005) are not applicable in our setting.

Our procedure tests for correlatedness between Y and X_i′s, which is an easier task than testing for independence, especially in the setting of high dimension and low sample size. In the Gaussian case, uncorrelatedness is equivalent to independence. When data are not Gaussian, we can take transformations of the data prior to testing such that the transformed data are approximately normal. The covariance testing is also valid for detecting dependency under other models. For example, if

$$h\left(X_i\right)={\beta }_i^{\text{T}}\mathit{Y}+{ϵ}_i,{ϵ}_i\perp \mathit{Y},h(\cdot )\text{\hspace{0.17em}\hspace{0.17em}strictly increasing}$$

then it is not difficult to show that

$${\beta }_i^{\text{T}}\mathbf{\sigma }=cov\left({\beta }_i^{\text{T}}\mathit{Y},X_i\right)=cov\left(h\left(X_i\right),X_i\right)=E\left[\left\{h\left(X_i\right)-h\left({\mu }_i\right)\right\}\left(X_i-{\mu }_i\right)\right]>0$$

under mild conditions on the distribution of X_i. Thus, our test based on σ_i can in fact detect non-linear relationships although proper transformation may increase power. Future research is warranted to test for more complex non-linear associations in the high dimensional setting.