A Fast and Accurate Algorithm to Test for Binary Phenotypes and Its Application to PheWAS

Abstract

The availability of electronic health record (EHR)-based phenotypes allows for genome-wide association analyses in thousands of traits and has great potential to enable identification of genetic variants associated with clinical phenotypes. We can interpret the phenome-wide association study (PheWAS) result for a single genetic variant by observing its association across a landscape of phenotypes. Because a PheWAS can test thousands of binary phenotypes, and most of them have unbalanced or often extremely unbalanced case-control ratios (1:10 or 1:600, respectively), existing methods cannot provide an accurate and scalable way to test for associations. Here, we propose a computationally fast score-test-based method that estimates the distribution of the test statistic by using the saddlepoint approximation. Our method is much (∼100 times) faster than the state-of-the-art Firth’s test. It can also adjust for covariates and control type I error rates even when the case-control ratio is extremely unbalanced. Through application to PheWAS data from the Michigan Genomics Initiative, we show that the proposed method can control type I error rates while replicating previously known association signals even for traits with a very small number of cases and a large number of controls.

Introduction

Over the last decade, genome-wide association studies (GWASs) have proved instrumental to unravelling the genetic complexities of hundreds of diseases and traits and their associations with common genomic variations. To date, thousands of GWASs have identified more than 4,000 significant loci to be associated with human diseases and traits.¹ However, because most GWASs investigate a single disease or trait, they cannot exploit the cross-phenotype associations or pleiotropy,² where a single genetic variant can be associated with multiple phenotypes. The phenome-wide association study (PheWAS) has been proposed as an alternative approach to take advantage of the pleiotropy phenomenon by studying the impact of genetic variations across a broad spectrum of human phenotypes or “phenome.” It is a complementary approach to the GWAS in the sense that whereas a GWAS attempts to identify phenotype-to-genotype associations, a PheWAS uses a genotype-to-phenotype approach. The first PheWAS³ was published as a proof-of-principle study that demonstrated that the PheWAS strategy could be applied to successfully identify the expected gene-disease associations. Additional studies^{4, 5, 6, 7, 8} have shown that the PheWAS approach can further identify previously unreported disease-SNP associations.⁹

The PheWAS approach depends on the availability of detailed phenotypic information. Currently, most PheWASs are applied to clinical cohorts linked to electronic health records (EHRs) and utilize the International Classification of Disease (ICD) billing codes to define clinical phenotypes. The ICD codes provide intuitive phenotype ordering based on clinical disease and trait classifications. Given that the current genotyping and imputation technologies allow for genotyping of tens of millions of variants at a very low cost,¹⁰ an extensive PheWAS can attempt to investigate the genotype-phenotype associations by performing genome-wide association analyses in thousands of traits. We can interpret the PheWAS result of a single genetic variant by observing its associations across the phenome. Such a PheWAS is exhaustive in nature and has great potential to identify variants associated with clinical diseases.

One of the main challenges of the PheWAS approach is that most of the phenotypes are binary phenotypes with unbalanced (1:5) or often extremely unbalanced (1:600) case-control ratios (see Figure S1), given that the data are collected in cohorts. Although standard asymptotic tests (such as the Wald, score, and likelihood-ratio tests) are relatively well calibrated and asymptotically equivalent¹¹ for common (minor allele frequency [MAF] > 0.05) variants in balanced case-control studies, they can inflate type I error for low-frequency (0.01 < MAF ≤ 0.05) and rare (MAF ≤ 0.01) variants in unbalanced case-control studies.¹² Moreover, because the Wald and likelihood-ratio tests need to calculate the likelihood or the maximum-likelihood estimator under the full model, which is computationally expensive, they are not scalable for the amount of tests that PheWASs attempt. On the other hand, the score test is computationally efficient because it does not need to calculate the maximum likelihood under the full model. However, as mentioned before, it suffers from having highly inflated type I error rates in unbalanced studies. Ma et al. proposed Firth’s penalized likelihood-ratio test¹³ as a solution to control the type I error rates in such situations. Firth’s test, despite being well calibrated and robust for testing low-frequency and rare variants in unbalanced studies, lacks computational efficiency because it also involves calculating the maximum likelihood under the full model. For instance, the projected computation time for testing 1,500 phenotypes across 10 million SNPs is ∼117 CPU years (2,000 cases and 18,000 controls). Thus, it is impractical to apply Firth’s test for analyzing large PheWAS datasets.

In this paper, we propose a score-based single-variant test for binary phenotypes that is well calibrated for controlling type I error and can adjust for covariates even in extremely unbalanced case-control studies. Moreover, our test is computationally efficient and scalable to testing thousands of phenotypes across millions of SNPs in large PheWAS datasets. Our proposed test (SPA) is based on score statistics and estimates the null distribution by using the saddlepoint approximation^{14, 15, 16} instead of the normal approximation¹⁷ traditionally used in score tests. We further develop an improvement of our test (fastSPA) that renders the most computationally challenging steps dependent only on the number of carriers (subjects with at least one minor allele) rather than the sample size. This improved test can substantially reduce the computation time, especially for low-frequency and rare variants, where the number of carriers is much lower than the sample size. Our method’s projected computation time for testing 1,500 phenotypes across 10 million SNPs is ∼400 CPU days (2,000 cases and 18,000 controls), which is more than a 100 times better than that of Firth’s test. In addition, through extensive simulation studies and analysis of the Michigan Genomics Initiative (MGI) data, we demonstrate that the proposed approach can control type I error and is powerful enough to replicate known association signals.

Material and Methods

Logistic Regression Model and Saddlepoint Approximation Method

We consider a case-control study with sample size n. For the i^th subject, let Y_i = 1 or 0 denote the case-control status, X_i denote the k × 1 vector of non-genetic covariates (including the intercept), and G_i denote the number of minor alleles (G_i = 0, 1, 2) of the variant to be tested. To relate genotypes to phenotypes, we use the following logistic regression model:

$$\text{logit}\left[\mathrm{Pr}\left(Y_i=1|X_i,G_i\right)\right]=X_i^T\beta +G_i\gamma \text{for}i=\mathrm{1,2},\dots ,n,$$ (Equation 1)

where β is a k × 1 vector of coefficients of the covariates, and γ is the genotype log odds ratio. Under this model, we are interested in testing for the genetic association by testing the null hypothesis H₀: γ = 0. Let ${\hat{\mu }}_i$ be the estimate of μ_i = Pr(Y_i = 1|X_i), which is the probability of being a case under H₀. A score statistic for γ from the model (Equation 1) is given by $S={\sum }_{i=1}^n{G}_i\left(Y_i-{\hat{\mu }}_i\right)$. Suppose $X=\left(X_1^T,\dots ,X_n^T\right)$ is the n × k matrix of covariates, $G={\left(G_1,\dots ,G_n\right)}^T$ is the genotype vector, W is a diagonal matrix with ${\hat{\mu }}_i\left(1-{\hat{\mu }}_i\right)$ as the i^th diagonal element, and $\tilde{G}=G-X{\left(X^{T}WX\right)}^{-1}{X}^{T}WG$ is a covariate-adjusted genotype vector in which covariate effects are projected out from the genotypes (details are given in Appendix A). Then, S can be written as

$$S=\sum _{i=1}^n{\tilde{G}}_i\left(Y_i-{\hat{\mu }}_i\right),$$ (Equation 2)

and the mean and variance of S under H₀ are $E_{H_0}\left(S\right)=0$ and $V_{H_0}\left(S\right)={\sum }_{i=1}^n{\tilde{G}}_i^2{\hat{\mu }}_i\left(1-{\hat{\mu }}_i\right)$, respectively, where ${\tilde{G}}_i$ is the i^th element of $\tilde{G}$.

The traditional score test approximates the null distribution by using a normal distribution, which depends only on the mean and the variance of the score statistic. We can obtain the p value by comparing the observed test statistic (s) and $N\left(0,V_{H_0}\left(S\right)\right)$. Normal approximation works well near the mean of the distribution but performs very poorly at the tails. The performance is especially poor when the underlying distribution is highly skewed, such as in unbalanced case-control outcomes,¹² because normal approximation cannot incorporate higher moments such as skewness. In addition, the convergence rate of normal approximation^{18, 19, 20} is O(n^−1/2), which is not fast enough for rare variants.

Saddlepoint approximation was introduced by Daniels¹⁴ as an improvement over the normal approximation. Contrary to normal approximation, where only the first two cumulants (mean and variance) are used for approximating the underlying distribution, saddlepoint approximation uses the entire cumulant-generating function (CGF). Jensen²¹ further showed that saddlepoint approximation has a relative error bound of $O\left(n^{-3/2}\right)$, making it a considerable improvement over the normal approximation.

To use the saddlepoint approximation, we first derive the CGF of S from the fact that Y_i ∼ Bernoulli (μ_i) under H₀. Let $\hat{\mu }$ be an n × 1 vector with ${\hat{\mu }}_i$ as the i^th element. From Equation 2, the estimate of the CGF of the score statistic S is

$$K\left(t\right)=\log \left(E_{H_0}\left(e^{tS}\right)\right)=\sum _{i=1}^n\log \left(1-{\hat{\mu }}_i+{\hat{\mu }}_i{e}^{{\tilde{G}}_{i}t}\right)-t\sum _{i=1}^n{\tilde{G}}_i{\hat{\mu }}_i,$$

and the estimates of the first- and second-order derivatives of K are

$$K^{\prime }\left(t\right)=\sum _{i=1}^n\frac{{\hat{\mu }}_i{\tilde{G}}_i}{\left(1-{\hat{\mu }}_i\right)e^{-{\tilde{G}}_{i}t}+{\hat{\mu }}_i}-\sum _{i=1}^n{\tilde{G}}_i{\hat{\mu }}_i$$

and

$$K^{\prime \prime }\left(t\right)=\sum _{i=1}^n\frac{\left(1-{\hat{\mu }}_i\right){\hat{\mu }}_i{\tilde{G}}_i^2{e}^{-{\tilde{G}}_{i}t}}{{\left[\left(1-{\hat{\mu }}_i\right)e^{-{\tilde{G}}_{i}t}+{\hat{\mu }}_i\right]}^2},$$

respectively. We note that K, K′, and K″ are plug-in estimates in which we plug in ${\hat{\mu }}_i$ instead of μ_i. Then, according to the saddlepoint method (Barndorff-Nielson^{15, 16}), the distribution of S at s can be approximated by

$$\mathrm{Pr}\left(S<s\right)\approx \tilde{F}\left(s\right)=\mathrm{\Phi }\left\{w+\frac{1}{w}\log \left(\frac{v}{w}\right)\right\},$$

where $w=\mathrm{sgn}\left(\hat{t}\right)\sqrt{2\left(\hat{t}s-K\left(\hat{t}\right)\right)},v=\hat{t}\sqrt{K^{\prime \prime }\left(\hat{t}\right)}$, $\hat{t}$ is the solution to the equation $K^{\prime }\left(\hat{t}\right)=s$, and Φ is the distribution function of a standard normal distribution.

Implementation Details and Approaches to Reducing Computation Time

The saddlepoint approximation method involves finding the root of the saddlepoint equation $K^{\prime }\left(t\right)=s$. It is easy to verify that K′ strictly increases as K″(t) > 0 for all −∞ < t < ∞, and $s={\sum }_{i=1}^n{\tilde{G}}_i\left(Y_i-{\hat{\mu }}_i\right)$ lies between $\underset{t\to \infty }{\lim }{K}^{\prime }\left(t\right)={\sum }_{i:{\tilde{G}}_i>0}{\tilde{G}}_i-{\sum }_{i=1}^n{\tilde{G}}_i{\hat{\mu }}_i$ and $\underset{t\to -\infty }{\lim }{K}^{\prime }\left(t\right)={\sum }_{i:{\tilde{G}}_i<0}{\tilde{G}}_i-{\sum }_{i=1}^n{\tilde{G}}_i{\hat{\mu }}_i$. Therefore, a unique root exists, and we can use popular root-finding algorithms (Newton-Raphson,^{22, 23} bisection,²³ secant,²³ and Brent’s method²⁴) to efficiently solve this equation. For our simulation studies and real-data applications, we applied a combination of the Newton-Raphson and bisection method to solve the saddlepoint equations.

The most computationally demanding step in this saddlepoint approximation method is calculating the CGF and its derivatives. Here, we propose several approaches to reducing the computational complexities associated with these calculations.

Faster Calculation of the CGF by a Partially Normal Approximation Approach

The most computationally intensive step in the saddlepoint method is the calculation of the CGF K and its derivatives. In each step of the root-finding algorithm, we need to calculate K, K′, and K″, each of which needs O(n) computations. Using the fact that many elements of G are zeroes (i.e., homozygous major genotypes), we propose a fast computation method that speeds up the computation to O(m), where m is the number of non-zero elements in G. Without loss of generality, we assume that the first m subjects have at least one minor allele each and the rest have homozygous major genotypes. We can then express S as S = S₁ + S₂, where $S_1={\sum }_{i=1}^m{\tilde{G}}_i\left(Y_i-{\hat{\mu }}_i\right)$ and $S_2={\sum }_{i=m+1}^n{\tilde{G}}_i\left(Y_i-{\hat{\mu }}_i\right)$. Let $Z={\left(X^{T}WX\right)}^{-1}{X}^{T}WG$, and let Z_l be the l^th element of Z. Then, we can further express S₂ as

$$\begin{array}{c}{S}_2=\sum _{i=m+1}^n{\tilde{G}}_i\left(Y_i-{\hat{\mu }}_i\right)=\sum _{i=m+1}^n\left(0-X_{i}Z\right)\left(Y_i-{\hat{\mu }}_i\right)\\ =-\sum _{i=m+1}^n\sum _{l=1}^k{X}_{il}{Z}_l\left(Y_i-{\hat{\mu }}_i\right)=-\sum _{l=1}^k{Z}_l\sum _{i=m+1}^n{X}_{il}\left(Y_i-{\hat{\mu }}_i\right)\\ =-\sum _{l=1}^k{Z}_l{S}_{2l},\end{array}$$

where $S_{2l}={\sum }_{i=m+1}^n{X}_{il}\left(Y_i-{\hat{\mu }}_i\right)$. Now, if we assume that the non-genetic covariates are relatively balanced in the sample, then the normal distribution should be a good approximation of the null distribution of each S_2l. Because S₂ is a weighted sum of the S_2l variables, we can also approximate the null distribution of S₂ by using a normal distribution where the mean and variance under H₀ are given by $E_{H_0}\left(S_2\right)=0$ and $V_{H_0}\left(S_2\right)={\sum }_{i=m+1}^n{\tilde{G}}_i^2{\hat{\mu }}_i\left(1-{\hat{\mu }}_i\right)$, respectively. Then, the CGF of $S_2$ can be approximated by

$$K_2\left(t\right)=\frac{1}{2}{t}^2{V}_{H_0}\left(S_2\right),$$

and the CGF of S = S₁ + S₂ can be approximated by

$$K\left(t\right)=\sum _{i=1}^m\log \left(1-{\hat{\mu }}_i+{\hat{\mu }}_i{e}^{{\tilde{G}}_{i}t}\right)-t\sum _{i=1}^m{\tilde{G}}_i{\hat{\mu }}_i+\frac{1}{2}{t}^2{V}_{H_0}\left(S\right).$$ (Equation 3)

In order to calculate the first two terms on the right side of Equation 3, we will need ${\tilde{G}}_i$ values for i = l, …, m, which can be calculated in O(m) computations given that G has only m non-zero elements and the quantity $X{\left(X^{T}WX\right)}^{-1}{X}^{T}W$ can be pre-calculated. Then, the first two terms will require only O(m) computations because both of them sum over m elements. Next, the variance $V_{H_0}\left(S\right)$ can be further broken down into

$$\begin{array}{c}{V}_{H_0}\left(S\right)=\sum _{i=m+1}^n{\tilde{G}}_i^2{\hat{\mu }}_i\left(1-{\hat{\mu }}_i\right)=\sum _{i=m+1}^n{\left(X_{i}Z\right)}^2{\hat{\mu }}_i\left(1-{\hat{\mu }}_i\right)\\ =\sum _{i=1}^n{\left(X_{i}Z\right)}^2{\hat{\mu }}_i\left(1-{\hat{\mu }}_i\right)-\sum _{i=1}^m{\left(X_{i}Z\right)}^2{\hat{\mu }}_i\left(1-{\hat{\mu }}_i\right)\\ =Z^T\left(X^{T}WX\right)Z-\sum _{i=1}^m{\left(X_{i}Z\right)}^2{\hat{\mu }}_i\left(1-{\hat{\mu }}_i\right).\end{array}$$

Because X^TWX can be pre-calculated and Z is a k × 1 vector, the first term requires O(k) computations, and the second term requires O(m) computations, which implies that the calculation of $V_{H_0}\left(S_2\right)$ requires O(m) calculations under the assumption that k < m, i.e., the number of non-genetic covariates is smaller than the number of subjects with at least one minor allele each. Hence, the CGF K(t) can be calculated in O(m) computations. Using similar arguments, we can further show that the derivatives K′(t) and K″(t) can also be calculated in O(m) computations. Therefore, this partially normal approximation reduces the computational complexity of our test from O(n) to O(m), which is especially useful for rare variants, where m is much smaller than n.

Using Normal Approximation near the Mean for Faster Computation

Because the normal approximation behaves well near the mean of the distribution, we can use it to obtain the p value when the observed score statistic (s) lies close to the mean (0). Moreover, saddlepoint approximation can be numerically unstable very close to the mean of the distribution. We can also avoid such situations by using normal approximation near the mean. One possible approach is to use a fixed threshold in which we apply normal approximation to obtain the p value if the absolute value of the observed score statistic, |s| < rσ, where $\sigma =\sqrt{V_{H_0}\left(S\right)}$ and r is a pre-specified value. For example, we used r = 2 in our simulation studies and real-data analyses. For a given level α, this approach does not inflate type I error rates if r < Φ⁻¹(1 − α/2), where Φ⁻¹ is the inverse function of the standard normal distribution function, Φ(x).

Alternatively, we can adaptively select the threshold by using the error bound of the normal approximation given by the Berry-Esseen theorem. Suppose we are interested in controlling the type I error rate at level α. Let F_n(x) be the true distribution function of the standardized score test statistic $S/\sqrt{V_{H_0}\left(S\right)}$. Then, according to Berry-Esseen theorem,^{18, 19, 20} the maximum error bound in approximating F_n(x) by Φ(x) is

$$\underset{x\in \mathbb{R}}{\sup }\left|F_n\left(x\right)-\Phi \left(x\right)\right|\le B_n=C{\left({\sigma }^2\right)}^{-3/2}\left(\sum _{i=1}^n{\rho }_i\right),$$ (Equation 4)

where ${\rho }_i=E_{H_0}\left[{\left|{\tilde{G}}_i\left(Y_i-{\hat{\mu }}_i\right)\right|}^3\right]={\tilde{G}}_i^3{\hat{\mu }}_i\left(1-{\hat{\mu }}_i\right)\left[{\hat{\mu }}_i^2+\left(1-{\hat{\mu }}_i^2\right)\right]$ and C is a constant. As of now, the best-known estimate for C is 0.56, given by Shevtsova.²⁵ Suppose p_F and p_N are F_n(x)- and Φ(x)-based p values, respectively. From the Berry-Esseen theorem, we can show p_N ≤ p_F + B_n. Suppose q = B_n + α/2 and r_α = Φ⁻¹(1 − q). Then, p_N ≥ q indicates p_F ≥ α/2. Therefore, we use $r_{\alpha }\sigma$ as a threshold at level α in which we will apply normal approximation if |s| < r_ασ.

Numerical Simulations

To evaluate the computation times, type I error rates, and power of the proposed method, we carried out extensive simulation studies. We considered three different case-control ratios: balanced with 10,000 cases and 10,000 controls, moderately unbalanced with 2,000 cases and 18,000 controls, and extremely unbalanced with 40 cases and 19,960 controls. For each choice of case-control ratio, the phenotypes were simulated on the basis of the following logistic model:

$$\text{logit}\left[\mathrm{Pr}\left(Y_i=1\right)\right]={\beta }_0+X_{1i}+X_{2i}+\gamma G_i,$$

where the two non-genetic covariates X_1i and X_2i were simulated from X_1i ∼ Bernoulli (0.5) and X_2i ∼ N(0, 1), respectively. The intercept β₀ was chosen to correspond to a prevalence of 0.01. The genotype G_i values were generated from a binomial(2, p) distribution where p was the MAF. The parameter γ represents the genotype log odds ratio.

To estimate computation times and type I error rates in realistic scenarios, we randomly sampled the MAF (p) from the MAF distribution in the MGI data. To compare computation times, we simulated 10⁴ variants with γ = 0. To compare type I error rates, we simulated 10⁹ variants with γ = 0 and recorded the number of rejections at α = 5 × 10⁻⁵ and 5 × 10⁻⁸. We also used fixed MAFs to evaluate the effect of MAF on computation time and type I error rates. For the power calculations, we considered two different choices for MAF (p = 0.01 and 0.05) and wide ranges of γ (Figure 4). For each choice of p and γ, we generated 5,000 variants.

Figure 4.

Empirical Power Curves for Score, fastSPA-2, and Firth’s Test

The top and bottom panels consider MAF = 0.05 and 0.01, respectively. From left to right, the plots consider case-control ratios 10,000:10,000 (balanced), 2,000:18,000 (moderately unbalanced), and 40:19,960 (extremely unbalanced). In each plot, the x axis represents genotype odds ratios, and the y axis represents the empirical power. Empirical power was estimated from 5,000 simulated datasets at the test-specific α levels where their empirical type I errors were equal to 5 × 10⁻⁸.

We compared the computation times of seven different tests: a traditional score test using normal approximation (Score), the saddlepoint-approximation-based test with a standard-deviation threshold at 0.1 and 2 (SPA-0.1 and SPA-2, respectively), the fast saddlepoint-approximation-based test with the partially normal approximation improvement and a standard-deviation threshold at 0.1 and 2 (fastSPA-0.1 and fastSPA-2, respectively), the fastSPA test with the Berry-Esseen bound threshold at α = 5 × 10⁻⁸ (fastSPA-BE), and Firth’s penalized likelihood-ratio test. Next, we compared the empirical type I errors and power curves for fastSPA-2, Score, and Firth’s test at 5 × 10⁻⁸. Because performing Firth’s test 10⁹ times, which is required for estimating type I error rates at 5 × 10⁻⁸, is practically impossible given the heavy computational burden, we performed a hybrid approach in which we used Firth’s test only when the fastSPA-2 p values were smaller than 5 × 10⁻³. For the power comparison, because Score has extremely inflated type I errors in the unbalanced and extremely unbalanced case-control scenarios (as shown in the Results), it might not be appropriate to directly compare the power of Score with that of the other two tests at the same nominal α level. In order to provide a more meaningful comparison, we compared their powers at the empirical α levels where their empirical type I errors became 5 × 10⁻⁸. The empirical α levels were selected on the basis of the type I error simulations, whereby variants were simulated with MAF randomly sampled from the MAF distribution of the MGI data. This approach is similar to performing resampling (e.g., permutation) to control family-wise error rates. We also estimated the powers at the nominal fixed α = 5 × 10⁻⁸. In order to compare the p values resulting from different tests, we also simulated 5 × 10⁻⁶ variants with MAFs randomly sampled from the MAF distribution of the MGI data. We further compared the inflation factors of the genomic controls at different p value quantiles for fastSPA-2, fastSPA-BE, and fastSPA-0.1 in order to explore the effect of the standard-deviation threshold on the inflation factor.

Application to MGI Data

To illustrate the performance of the proposed methods in real-data application, we analyzed four selected phenotypes in the MGI data. The main goal of MGI is to create an institutional repository of genetic data together with rich clinical phenotypes for a broad portfolio of future medical research. DNA from blood samples of >20,000 individuals who underwent surgical procedures at the University of Michigan Health System was genotyped (with their informed consent) on the Illumina HumanCoreExome v.12.1 array, which is a combined GWAS plus exome array composed of >500,000 SNPs. Genotypes of the Haplotype Reference Consortium²⁶ (chromosomes 1–22: HRC release 1; chromosome X: HRC release 1.1) were imputed into the phased MGI genotypes (SHAPEIT2²⁷ on autosomal chromosomes and Eagle2²⁸ on chromosome X) with Minimac3.²⁹ Excluding variants with low imputation quality (R² < 0.3) resulted in dense mapping at over 39 million quality-imputed genetic markers.

Phenotypes derived from 8,940 ICD-9 billing codes were classified into 1,815 PheWAS disease states of shared disease etiology, of which 1,448 had at least 20 cases. Standard code translations were used for converting the taxonomy of diagnostic ICD-9 codes into PheWAS code groups (PheWAS code translation table v.1.2³⁰). Cases were derived from EHRs of individuals with at least two encounters with an ICD-9 billing code. This is a typical example of many recent large-scale PheWASs. To compare our proposed fastSPA-2 with Score and the current gold-standard Firth’s test in analyzing such PheWAS data, we performed genome-wide association analyses for four selected traits—skin cancer (PheWAS code: 172), type 2 diabetes (PheWAS code: 250.2; MIM: 125853), primary hypercoagulable state (PheWAS code: 286.81; MIM: 188055), and cystic fibrosis (PheWAS code: 499; MIM: 219700)—in 18,267 unrelated individuals of European ancestry while adjusting for age, sex, and four principal components. Genotyped samples with any missing covariate information were excluded from the analysis. Given that imputation quality is low for very rare variants,²⁶ we excluded the imputed variants with MAF < 0.001 in our main analysis, which resulted in 13 million variants. For Firth’s test, we used the hybrid approach used in the type I error simulation, where Firth’s test was performed only when the fastSPA-2 p value was smaller than 5 × 10⁻³.

Results

Numerical Simulations

We examine the computation time, type I error control, and power of the proposed fastSPA and two existing approaches, Score and Firth’s test, across ranges of case-control imbalance and MAFs.

Comparison of Computation Times

The projected computation times for testing 1,500 phenotypes across 10 million variants by different testing methods are presented in Figure 1. To obtain computation time under realistic scenarios of the MAF distribution, we randomly sampled the MAFs of the simulated SNPs from the MAF spectrum of the MGI data (Figure S2). fastSPA-2 performs 100–300 times faster than Firth’s test. In the unbalanced case-control setup of 2,000 cases and 18,000 controls, for example, Firth’s test takes 117 CPU years to analyze 10 million SNPs across 1,500 phenotypes, whereas fastSPA-2 takes only 1.09 CPU years. This indicates that on a cluster with 100 CPU cores, the proposed test would require 4 days (without data reading), but Firth’s test would need more than a year. When we compare fastSPA and SPA, fastSPA-0.1 performs 4–6 times faster than SPA-0.1 (e.g., 2.90 versus 12.32 CPU years when the case-control ratio = 2,000:18,000), and fastSPA-2 performs 1.5–2 times faster than SPA-2 (e.g., 1.09 versus 1.62 CPU years when the case-control ratio = 2,000:18,000). Expectedly, the computation time for fastSPA-BE is in between the computation times for fastSPA-2 and fastSPA-0.1. fastSPA-BE performs 1.3–1.8 times faster than fastSPA-0.1 and 1.6–2.8 times slower than fastSPA-2 (e.g., 1.09, 1.86, and 2.9 CPU years for fastSPA-2, fastSPA-BE, and fastSPA-0.1, respectively, when the case-control ratio = 2,000:18,000).

Figure 1.

The Projected Computation Times for Testing 10 Million Variants across 1,500 Phenotypes by Various Tests with MAFs Sampled from the MAF Distribution of the MGI Data

The computation times are based on testing 10,000 simulated variants on an Intel i7 2.70GHz processor and then projecting them onto a PheWAS with 10 million variants and 1,500 phenotypes.

We also recorded the computation times for variants with three different fixed MAFs (0.1, 0.01, and 0.001) in order to assess the effect of MAF on the performance of the tests. Similar to Figure 1, Table 1 shows the superior performance of fastSPA-2 over all other tests. Moreover, whereas the computation time of SPA increases with decreasing MAFs, which could be due to the slow convergence caused by the discrete nature of the underlying distribution, fastSPA requires less computation time for rarer variants (smaller MAFs) than for more common variants (larger MAFs). This demonstrates the potential of the partially normal approximation improvement in terms of faster computation of the p values, especially for low-frequency and rare variants.

Table 1.

Computation Times for Various Tests of 10,000 Simulated Variants with Different MAFs

Case-Control Ratio	MAF	Score	SPA-0.1	fastSPA-0.1	fastSPA-BE	SPA-2	fastSPA-2	Firth’s Test
10,000:10,000	0.1	20	214	75	37	28	23	7,251
	0.01	19	225	38	35	27	20	6,918
	0.001	19	242	33	36	30	20	5,304
2,000:18,000	0.1	21	256	84	37	36	24	3,940
	0.01	20	284	39	36	35	21	4,312
	0.001	19	326	34	41	40	20	3,804
40:19,960	0.1	21	376	98	70	38	24	3,615
	0.01	20	477	42	58	44	21	3,598
	0.001	20	647	38	51	79	21	3,525

All computation times are in CPU seconds on an Intel i7 2.70GHz processor.

Type I Error Comparison

The type I error rates from 10⁹ simulated datasets are presented in Figure 2. Because of the heavy computation burden for testing these extremely large numbers of datasets, in this comparison, we considered only Score, fastSPA-2, and the hybrid version of Firth’s test, in which we used Firth’s test only when the fastSPA-2 p values were smaller than 5 × 10⁻³. We note that both fastSPA-2 and Firth’s test had well-calibrated quantile-quantile (Q-Q) plots up to 10⁻⁶ p values (Figure 5), and whenever fastSPA-2 p values were greater than 5 × 10⁻³, Firth’s test p values were greater than 4.8 × 10⁻⁴ (see p Value and Inflation Factor Comparison), indicating that the hybrid approach can provide very accurate estimation of the type I error rates of Firth’s test at very stringent α levels.

Figure 2.

Type I Error Comparison between Score, fastSPA-2, and Firth’s Test for Variants Simulated with MAFs Sampled from the MAF Distribution of the MGI Data

Type I error rates were estimated on the basis of 10⁹ simulated datasets. From left to right on the x axis, the plots consider case-control ratios 10,000:10,000 (balanced), 2,000:18,000 (moderately unbalanced), and 40:19,960 (extremely unbalanced). The top and bottom panels show empirical type I error rates at α = 5 × 10⁻⁵ and 5 × 10⁻⁸, respectively.

Figure 5.

Q-Q Plots for Score, fastSPA-2, SPA-2, and Firth’s Test on 5 × 10⁶ Simulated Variants with MAF Randomly Sampled from the MAF Distribution of the MGI Data

The top, middle, and bottom panels show Q-Q plots in the balanced (case-control ratio = 10,000:10,000), moderately unbalanced (case-control ratio = 2,000:18,000), and extremely unbalanced (case-control ratio = 40:19,960) case-control scenarios, respectively. In each plot, the x axis represents –log₁₀ expected p values, and the y axis represents –log₁₀ observed p values.

Score had greatly inflated type I error rates for moderately unbalanced and extremely unbalanced case-control ratios, whereas fastSPA-2 could control the type I error in such situations. At the genome-wide significance level of α = 5 × 10⁻⁸, for example, the empirical type I error rates of Score were 32 (1.63 × 10⁻⁶ when the case-control ratio = 2,000:18,000) and 26,600 (1.33 × 10⁻³ when the case-control ratio = 40:19,960) times higher than the nominal α = 5 × 10⁻⁸. In contrast, fastSPA-2 had empirical type I error rates nearly identical to (4.9 × 10⁻⁸ when the case-control ratio = 2,000:18,000) or slightly lower than (3.5 × 10⁻⁸ when the case-control ratio = 40:19,960) the nominal α = 5 × 10⁻⁸. Firth’s test also had well-controlled type I error rates in the balanced and moderately unbalanced case-control scenarios (4.7 × 10⁻⁸ and 4.9 × 10⁻⁸, respectively, at α = 5 × 10⁻⁸). Interestingly, it showed slight inflation (7.8 × 10⁻⁸ at α = 5 × 10⁻⁸) in the extremely unbalanced scenario. We also estimated empirical type I error rates at six different MAFs (Figure 3). Score had deflated type I error rates for low-frequency and rare variants for the balanced case-control ratio and inflated and extremely inflated type I error rates for moderately and severely unbalanced case-control ratios. fastSPA-2 had overall well-controlled type I error rates regardless of MAF and case-control ratio. Firth’s test had either well-controlled or slightly conservative type I error rates when the case-control ratio was balanced or moderately unbalanced. However, when the case-control ratio was extremely unbalanced, Firth’s test had inflated type I error rates, especially when the minor allele count was small (e.g., 1.33 × 10⁻⁷ and 1.47 × 10⁻⁷ for MAF = 0.0005 and 0.001, respectively, at α = 5 × 10⁻⁸ when the case-control ratio = 40:19,960).

Figure 3.

Type I Error Comparison at Different MAFs between Score, fastSPA-2, and Firth’s Test

The top and bottom panels show empirical type I error rates at α = 5 × 10⁻⁵ and 5 × 10⁻⁸, respectively. From left to right, the plots consider case-control ratios 10,000:10,000 (balanced), 2,000:18,000 (moderately unbalanced), and 40:19,960 (extremely unbalanced). In each plot, the x axis represents MAF with the expected minor allele count (MAC) in parentheses, and the y axis represents empirical type I error rates. Empirical type I error rates were estimated on the basis of 10⁹ simulated datasets. 95% confidence intervals at different MAFs are also presented.

Power Comparison

Next, we compared the power curves of fastSPA-2, Score, and Firth’s test. Note that Firth’s test is a current gold-standard method.¹³ Because Score had greatly inflated type I error rates, we compared the empirical powers of different tests at their test-specific empirical α levels. Figure 4 shows power by odds ratios when the MAF of the variant was 0.05 (top panel) and 0.01 (bottom panel). As expected, the power was higher when the case-control ratio was balanced. The empirical powers of fastSPA-2 and Firth’s test were nearly identical for all case-control ratios and MAFs, which suggests that our proposed test does not suffer from any loss in power in comparison with Firth’s test. The empirical powers of Score were almost identical to those of fastSPA-2 and Firth’s test for the balanced case-control ratio. However, Score showed substantially lower power than the other two tests for the unbalanced case-control ratios as a result of the very small empirical α levels, and the power gap was especially large when the case-control ratio was extremely unbalanced. The simulation results clearly show that the proposed approach improves power over Score when type I error rates are properly controlled. When we used the nominal α = 5 × 10⁻⁸ level instead of the empirical α levels, Score had higher power than the other two approaches (Figure S3) as expected, given that its type I error rates were not controlled.

p Value and Inflation Factor Comparison

To compare p value distributions of various tests, we generated Q-Q plots and calculated the inflation factor (λ) of the genomic control. Figure 5 suggests strong deflation (smaller than expected) in the p values from Score in the moderately unbalanced and extremely unbalanced case-control setups, whereas fastSPA-2, SPA-2, and Firth’s test resulted in well-calibrated Q-Q plots, which suggests that these methods can control for type I errors. Moreover, the minimum Firth’s test p value was 4.8 × 10⁻⁴ for the variants with a fastSPA-2 p value > 5 × 10⁻³ among all case-control setups, which justifies our hybrid approach of performing Firth’s test only when the fastSPA-2 p value is less than 5 × 10⁻³ in the type I error simulation studies.

None of fastSPA-2, fastSPA-BE, and fastSPA-0.1 showed any inflation or deflation in genomic controls (λ) in the balanced and moderately unbalanced case-control setups (Table S1). In the extremely unbalanced case-control setup, fastSPA-2 resulted in a greatly deflated λ (0.48) at the median p value (q = 0.5). Interestingly, fastSPA-BE and fastSPA-0.1 resulted in an inflated λ (both 1.83) at q = 0.5, which could be due to the discrete nature of p values. However, when λ was measured at p value quantiles q = 0.01 and 0.001, all three tests provided λ very close to unity.

Analysis of MGI Data

We applied Score, Firth’s test, and fastSPA-2 to the MGI data with four phenotypes: skin cancer, type 2 diabetes, primary hypercoagulable state, and cystic fibrosis, which were selected on the basis of case-control ratios. Skin cancer (2,359 cases and 15,265 controls) and type 2 diabetes (1,987 cases and 14,906 controls) were moderately unbalanced, whereas primary hypercoagulable state (168 cases and 16,401 controls) and cystic fibrosis (28 cases and 18,212 controls) were extremely unbalanced phenotypes.

The Manhattan plots (Figure 6) show that Score produced a large number of potentially spurious associations for all of these phenotypes, whereas all of the significant variants from our proposed test at the genome-wide significance level of α = 5 × 10⁻⁸ can be verified as truly associated with the phenotypes on the basis of previous findings (Table 2). In the analysis of skin cancer, variants in or near IRF4 (MIM: 601900), MC1R (MIM: 155555), RALY (MIM: 614663), and SLC45A2 (MIM: 606202) were significant at α = 5 × 10⁻⁸, and all four of these genes were previously identified as associated with pigmentation traits and skin cancers.^{31, 32, 33, 34, 35, 36} In the other traits, variants in TCF7L2 (MIM: 602228), F5 (MIM: 612309), and CFTR (MIM: 602421) were significantly associated with type 2 diabetes,³⁷ primary hypercoagulable state,³⁸ and cystic fibrosis,³⁹ respectively, and all of these genes are well known to be associated with the risk of their respective diseases. The Q-Q plots (Figure 7) also suggest that the p values based on Score are much smaller than expected, especially for low-frequency and rare variants, whereas the p values based on fastSPA-2 closely follow the uniform distribution. We also observed the Manhattan plots (Figure S4) including the imputed variants with MAF < 0.001 in the analysis. The inclusion of rarer variants resulted in extreme inflation in the number of potentially spurious associations for Score. However, our proposed test still produced none to very few new associations. The Manhattan plots and Q-Q plots for Firth’s test were almost identical to those of our proposed test.

Figure 6.

Manhattan Plots for Four Different Phenotypes from MGI Data

All imputed variants with MAF > 0.001 and all directly genotyped variants were included in this analysis. From left to right, the three panels show associations based on fastSPA-2, Firth’s test, and Score. The red line represents the genome-wide significance level α = 5 × 10⁻⁸.

Table 2.

Significant SNP-Phenotype Associations Based on fastSPA-2 on MGI Data and Previous Findings Confirming Such Associations

Phenotype	Location	dbSNP ID	Nearest Gene	Alleles	MAF	p Value	Previous Findings
Skin cancer	6:396321	rs12203592	IRF4	C>T	0.16	6.71 × 10−18	Zhang et al.31 Sulem et al.,32 Jacobs et al.,33 and Liu et al.34
	16:89986117	rs1805007	MC1R	C>T	0.077	1.86 × 10−14	Zhang et al.31 Sulem et al.,32 Jacobs et al.,33 and Liu et al.34
	20:32538391	rs62211989	RALY	G>C	0.075	5.59 × 10−13	Zhang et al.31 Sulem et al.,32 Jacobs et al.,33 and Liu et al.34
	5:33951693	rs16891982	SLC45A2	C>G	0.038	7 × 10−9	Liu et al.,34 Barrett et al.,35 and Nan et al.36
Type 2 diabetes	10:114754071	rs34872471	TCF7L2	T>C	0.29	3.4 × 10−11	Scott et al.37
Primary hypercoagulable state	1:169519049	rs6025	F5	T>C	0.029	4.9 × 10−39	Bertina et al.38
Cystic fibrosis	7:117299434	rs113827944	CFTR	G>A	0.018	3.11 × 10−15	Kerem et al.39

Figure 7.

Q-Q Plots for Four Different Phenotypes from MGI Data

From left to right, the three panels show the Q-Q plots based on fastSPA-2, Firth’s test, and Score. The plots are color coded according to different MAF categories. 95% confidence bands are presented in gray to signify the deviance from the uniform distribution.

Further, on the basis of the p values from our proposed test, we obtained the inflation factor (λ) of the genomic control at different p value quantiles (q) and different MAF cutoffs (Table S2). Only the imputed variants were removed when we used different MAF cutoffs. The SNPs present on the Illumina HumanCoreExome v.12.1 array were preserved. To evaluate whether using a smaller standard-deviation threshold (r) improves the estimation of λ, we also applied fastSPA with r = 0.1 (fastSPA-0.1) and fastSPA with the Berry-Esseen bound threshold at α = 5 × 10⁻⁸ (fastSPA-BE) on these four phenotypes. When all variants were included in the analysis, there was slight inflation (λ = 1.11, type 2 diabetes) or great deflation (λ = 0.12, cystic fibrosis) at the median level for fastSPA-2. However, the genomic controls were very close to unity at q = 0.01 and 0.001. When we considered only the variants with MAF > 0.001, fastSPA-2 did not show any significant inflation in λ at the median for skin cancer, type 2 diabetes, or primary hypercoagulable state. However, it showed deflated genomic control for cystic fibrosis (λ = 0.63) as a result of the discrete nature of the underlying distribution. However, when we excluded the rare variants and considered only the variants with MAF > 0.01, all four of the phenotypes showed λ very close to unity. Neither fastSPA-0.1 nor fastSPA-BE showed a significant inflation or deflation in λ at any quantiles or MAF cutoffs, except for cystic fibrosis (both with λ = 1.27) when all variants were considered and genomic control was measured at the median level.

Discussion

In this paper, we propose a fast and scalable test for analyzing large PheWAS datasets that is well calibrated even in extremely unbalanced case-control settings. The method uses computationally efficient saddlepoint approximation to accurately calculate p values of score test statistics. We further propose an improved version of our test that substantially reduces the computation time, especially for low-frequency and rare variants. Our proposed test can also adjust for additional covariates. Through extensive numerical studies, we have demonstrated that our test can perform 100–300 times faster than the currently used Firth’s test while retaining similar power and well-controlled type I error rates. Analysis of MGI data illustrates that by applying the proposed method to PheWAS datasets, we can identify true association signals while controlling for type I error, even for traits with a very small number of cases and a large number of controls.

Our test calculates p values on the basis of Score if the score statistics lie sufficiently close to the mean. Even though normal approximation is accurate near the mean, those p values might not be well calibrated. In such cases, because the median p values might come from Score, we can encounter a slightly inflated or deflated inflation factor at the median. When the case-control ratio is extremely unbalanced, this phenomenon is more pronounced. One way to circumvent this issue is to measure the inflation factor at more extreme quantiles (0.01, 0.001, etc.) or to exclude rare variants when estimating the inflation factor. Another approach is to decrease the standard-deviation threshold so that the median p values come from the saddlepoint approximation. In the analysis of MGI data, fastSPA-0.1 produced substantially improved inflation-factor estimates than fastSPA-2. However, the use of threshold 0.1 instead of 2 would increase the computation time from ∼3 to 4 times. The Berry-Esseen threshold can be viewed as a compromise between these two thresholds. If there is no restriction in computational resource, we recommend using fastSPA-0.1 so that most of the p values are calculated by the saddlepoint approximation. If computational resource is limited, or researchers want to obtain results quickly, either a larger threshold (i.e., fastSPA-2) or Berry-Esseen bound can be a better choice.

As sequencing costs continue to drop, whole-exome or whole-genome sequencing will be used for PheWASs to identify rare variants associated with clinical phenotypes.⁴⁰ In rare-variant association analysis, gene- or region-based multiple-variant tests are commonly used to improve power.⁴¹ When case-control ratios are unbalanced, popular rare-variant tests, including burden tests, SKAT, and SKAT-O, can also have substantially inflated type I error rates. Although resampling-based approaches have been developed to address this problem,⁴² the existing methods are not fast enough to be used in PheWASs. One possible approach is to first adjust single-variant score statistics by SPA and then use the adjusted score statistics to control for the type I error. We have left this for future research.

In summary, we have proposed an accurate and scalable method for PheWAS data analysis. With the growing effort to build large research cohorts for precision medicine,⁴⁰ future PheWASs will have hundreds of thousands of samples and hundreds of millions of variants. Our method will provide a scalable solution for this large-scale problem and contribute to finding genetic components of complex traits. All of our tests are implemented in the R package SPAtest.

Published: June 8, 2017

Appendix A: Explanation behind Using $\tilde{\mathit{G}}$ instead of G

We first note that $S={\tilde{G}}^T\left(Y-\hat{\mu }\right)=G^T\left(Y-\hat{\mu }\right)$ given that $\hat{\mu }$ is the maximum-likelihood estimator of μ under the null model and $X^T\left(Y-\hat{\mu }\right)=0$. Now, the score function and the observed information matrix under the null model are given by

$$U_0=\left[\begin{array}{c}{X}^T\left(Y-\hat{\mu }\right)\\ G^T\left(Y-\hat{\mu }\right)\end{array}\right]=\left[\begin{array}{c}0\\ S\end{array}\right]$$

and

$$I_0=\left[\begin{array}{cc}{X}^{T}WX& X^{T}WG\\ G^{T}WX& G^{T}WG\end{array}\right],$$

respectively.

Therefore, the variance of S under H₀ is given by

$$V_{H_0}\left(S\right)=G^{T}WG-G^{T}WX{\left(X^{T}WX\right)}^{-1}{X}^{T}WG=G^{T}W\tilde{G}={\tilde{G}}^{T}W\tilde{G}.$$

So, even though the two expressions of S are algebraically the same, the variance can be expressed as a weighted sum of ${\hat{\mu }}_i\left(1-{\hat{\mu }}_i\right)$ values, where the weights are given by ${\tilde{G}}_i$ values. Therefore, we used $\tilde{G}$ instead of G to express the score statistic.

Web Resources

• Michigan Genomics Initiative, https://www.michigangenomics.org/

• OMIM, http://www.omim.org

• SPAtest R-package, https://sites.google.com/a/umich.edu/leeshawn/software