A flexible and nearly optimal sequential testing approach to randomized testing: QUICK-STOP

Abstract

In the analysis of current life science datasets, we often encounter scenarios in which the application of asymptotic theory to hypothesis testing can be problematic. Besides improved asymptotic results, permutation/simulation-based tests are a general approach to address this issue. However, these randomized tests can impose a massive computational burden, for example, in scenarios in which large numbers of statistical tests are computed, and the specified significance level is very small. Stopping rules aim to assess significance with the smallest possible number of draws while controlling the probabilities of errors due to statistical uncertainty. In this communication, we derive a general stopping rule, QUICK-STOP, based on the sequential testing theory that is easy to implement, controls the error probabilities rigorously, and is nearly optimal in terms of expected draws. In a simulation study, we show that our approach outperforms current stopping approaches for general randomized tests by factor 10 and does not impose an additional computational burden. We illustrate our approach by applying our stopping rule to a single variant analysis of a whole-genome sequencing study for lung function.

Introduction

The analysis in genetic studies often involves association testing of single variants, genetic regions, gene-gene-interaction, or gene expression data. Due to decreasing sequencing costs, study sample sizes and the number of available variants have increased by several magnitudes over the last few years. For many studies, this has created a large computational burden and a severe problem of multiple testing. Given the number of tests computed, very small alpha levels must be achieved to establish that test results are considered significant.

Furthermore, the evaluation of p-values itself can be problematic for many study designs and test statistics. Rare genetic variants, imbalanced case control ratios, phenotypic outliers (Stranger et al., 2005), or non-normally distributed phenotypes can lead to scenarios in which standard asymptotic theory does not provide reliable results. For some test statistics of interest, for example in gene-based/region-based analysis (Chen, Hsu, Gamazon, Cox, & Nicolae, 2012; Lee, Abecasis, Boehnke, & Lin, 2014; Liu et al., 2010; Mishra & Macgregor, 2015), the asymptotic distribution of the test statistic is even not tractable and permutation/simulation procedures have to be applied which can be computationally challenging (Malik et al., 2018; Sugasawa, Noma, Otani, Nishino, & Matsui, 2017).

Researchers have proposed various approaches and strategies to overcome the described issues regarding the assessment of statistical significance. These approaches include the transformation of the data to normality so that standard methodology can be applied, more specialized asymptotic results/approximations for particular test statistics, the application of permutation/simulation-based tests, approximations of the permutation-based p-values, and efficient resampling algorithms. In the setting where non-normally distributed traits are tested for association, for example, lung volume and blood markers, or in the presence of phenotypic outliers, a proposed strategy is to apply inverse normal transformations to constrain the phenotype information to a normal distribution form (Beasley, Erickson, & Allison, 2009). Then, the standard methodology can be applied to test the phenotype data for association. This procedure was applied and investigated in several recent publications (Boueiz et al., 2016; Vavoulis, Taylor, Schuh, & Bar-Joseph, 2017; Zhang, Xie, Liang, & Xiong, 2016). Although these transformations can solve the problem of unreliable asymptotic p-values for some scenarios, they can also lead to a loss of power (Beasley et al., 2009). In Supplementary Material A, we included simulations that indicate a loss of power in regression analysis when rank transformations are applied.

While the corresponding theory for approaches that approximate the permutation/resampling-based p-values or derive more specific asymptotic results is much more involved than the straightforward permutation/simulation-based procedure, it is also restricted to specific scenarios/test statistics. The approaches are not universally applicable, and many epidemiological studies and association test statistics nevertheless require randomized testing approaches based on permutations or simulations (e.g., Chen et al., 2012; Malik et al., 2018; Zhu, Zhang, & Sha, 2018).

As explained above, if randomized testing is applied in a large-scale setting, the computational burden is enormous. To reduce this computational burden, some simple stopping rules were proposed (Chang et al., 2015; Che, Jack, Motsinger-Reif, & Brown, 2014; Hasegawa et al., 2016). Instead of estimating the empirical p-value based on a pre-specified number of permutations/simulations per test, these approaches aim to stop randomization testing early if the corresponding p-value is clearly nonsignificant and therefore reduce the number of randomized draws and the computational time.

Closely related is the methodology of the so-called sequential Monte Carlo testing (Besag & Clifford, 1991; Gandy, 2009). The approach by Gandy (2009), called SIMCTEST, describes a stopping rule that guarantees to control the probability of a wrong decision, but the implementation requires recursions and additional tuning parameters.

In this communication, we develop a general stopping rule for randomized tests, QUICK-STOP (QS), that is based on sequential testing theory, and decides as fast as possible if the unknown p-value is below a specific significance level of interest. In contrast to the sequential Monte Carlo testing literature (Gandy, 2009), we introduce an arbitrary small indifference region between both hypotheses in which both decisions are acceptable. This leads to much simpler derivations and an intuitive stopping rule. For applications, the indifference region can be selected as arbitrarily small so that it is not of practical relevance but guarantees a finite runtime. The approach reduces the computational burden substantially compared to the confidence interval-based (CI) stopping rule in the popular genetic analysis tool PLINK1.9 (Chang et al., 2015).

Furthermore, we can utilize the existing theoretical sequential testing results to show that our procedure is nearly optimal in terms of the number of expected replicates, minimizing the computation burden. It is important to note that our methodology can be applied to any randomized test, is not restricted to a specific scenario, and does not impose a significant computational burden. Our approach combines computational efficiency and generality with an intuitive approach.

Methods

We consider a genetic epidemiological study where many statistical tests are performed, and significance of findings is declared based on an appropriate significance level $p_{significance}$, that is corrected for multiple testing. For example, $p_{significance}=5*{10}^{-8}$ in a GWAS. We assume that each statistical test is based on a suitable association test statistic T that can be computed based on observed data, but significance cannot be evaluated by asymptotic theory. We assume that one can draw random permutations (e.g. in a regression model) or simulate the corresponding null distribution (e.g. a suitable normal distribution as in gene-based tests (Liu et al., 2010; Mishra & Macgregor, 2015)) such that we can interpret the procedure of comparison between the randomized test statistic and the observed statistic as a sequence $x_i\in \left\{0,1\right\}$ of independent Bernoulli draws with parameter p that corresponds to the true, unknown p-value. After n permutations/simulations, the empirical p-value is usually estimated by ${\widehat{p}}_{MLE}=\frac{1}{n}{\sum }_{r=1}^n{x}_r$. Given unrestricted computational resources and time, all p-values could be estimated with high accuracy based on a very large number of permutations/simulations, and significant results identified. To avoid this infeasible computational burden, a general stopping rule aims to decide between the hypotheses

$$H_1:p\le p_{significance}\text{and}{H}_2:p>p_{significance}$$

based on as few permutations/simulations as possible. Since p is estimated based on permutations/simulations, a rigorous stopping rule needs to control the probability of a wrong decision due to statistical uncertainty in the estimation. In the following, we will refer to a type I decision error if the hypothesis H₂ is true and the stopping rule chooses H₁ A type II decision error occurs if the hypothesis H₁ is true and the stopping rule chooses H_2.

An important consequence is that, if the probability of a type I or type II decision error by the stopping rule is negligibly small, randomized testing based on the statistic T in combination with the stopping rule basically maintains the corresponding type 1 error and power of T but reduces the computational burden.

QUICK-STOP

In this section, we will introduce the setting and objects of our stopping rule QUICK-STOP and describe the theoretical properties. QUICK-STOP is based on the so-called Adaptive Sequential Likelihood Ratio Test (ASLRT) derived by Pavlov (Pavlov, 1991) and Tartakovsky (Tartakovsky, 2014).

The two hypotheses that we want to distinguish between are:

$$H_1:p\le p_1\text{and}{H}_2:p\ge p_2=p_1+d=p_{significance}$$

where the indifference parameter $d>0$ is a technical parameter that is chosen very small and separates the two hypotheses. Within the indifference region $\left(p_1,p_2\right)$, we assume that both hypotheses are acceptable. For example, $d={10}^{-8}$ would correspond to the resolution level of 10⁸ permutations/simulations but can be chosen arbitrary small. We will elaborate on the theoretical aspects of the indifference region/parameter d in more detail below.

Besides the parameters p₁ and d, our stopping rule requires the specification of the parameters ${\alpha }_1$ and ${\alpha }_2$ As we will show, these two parameters control the type I and the type II decision error probabilities introduced above. Since we usually we would like to avoid such wrong decisions in practice, we suggest very small values, for example ${\alpha }_1={\alpha }_2={10}^{-10}$. We introduce the objects

$${\tau }_1\left({\alpha }_1\right):=min\left\{n:\frac{{\pi }_n}{\underset{p\in D_1}{\sup }{b}_n\left(p,x^n\right)}\ge {\alpha }_1^{-1}\right\}$$

and

$${\tau }_2\left({\alpha }_2\right):=min\left\{n:\frac{{\pi }_n}{\underset{p\in D_2}{\sup }{b}_n\left(p,x^n\right)}\ge {\alpha }_2^{-1}\right\},$$

where $x^n=\left(x_1,\dots ,x_n\right)$, ${\pi }_n:={\prod }_{r=1}^{n}b\left({\widehat{p}}_{r-1},x_r\right),b_n\left(p,x^n\right)={\prod }_{r=1}^{n}b\left(p,x_r\right)$ and $D_1=\left[0,p_1\right]$, $D_2=\left[p_2,1\right]$. Here, $b\left(p,x\right)$ is the probability distribution of a Bernoulli random variable with success probability p,$x\in \left\{0,1\right\}$. The estimate ${\widehat{p}}_{r-1}$ is defined by ${\widehat{p}}_{r-1}=\frac{{\sum }_{k=1}^{r-1}{x}_k+\frac{1}{2}}{r}$, a slightly modified version of the maximum likelihood estimator for the success probability pthat is still asymptotically consistent. Our stopping rule is denoted by $\left(N,\delta \right)$, where $\delta =1$ or $\delta =2$ defines the selected hypothesis, and N is the number of permutations/simulations computed until this decision is made. If ${\tau }_1\left({\alpha }_1\right)\le {\tau }_2\left({\alpha }_2\right)$, we set $\delta =2$ and $N={\tau }_1\left({\alpha }_1\right)$. If ${\tau }_2\left({\alpha }_2\right)<{\tau }_1\left({\alpha }_1\right)$, we set $\delta =1$ and $N={\tau }_2\left({\alpha }_2\right)$.

An illustrative example

The intuition behind our stopping rule QUICK-STOP is that the objects $\frac{{\pi }_n}{{\sup }_{p\in D_1}{b}_n\left(p,x^n\right)}$ and $\frac{{\pi }_n}{{\sup }_{p\in D_2}{b}_n\left(p,x^n\right)}$ are similar to likelihood ratios between the currently estimated p-value and the best explaining parameter in the respective hypothesis bin.

To illustrate our stopping rule and the corresponding objects, we demonstrate an application with the following parameters ${\alpha }_1={\alpha }_2={10}^{-10}$, $p_1=4.99*{10}^{-8}$ and $p_2=5*{10}^{-8}$, implying $d={10}^{-10}$. First, we consider the case $p=0.1$, a clearly non-significant p-value. As described above, the sequence $x_1,x_2,\dots$ can be thought to be generated by independent Bernoulli(p) draws. A simulated example gives us the observed sequence $x_1=0,x_2=0,x_3=1,x_4=0,x_5=0,x_6=0,x_7=1,\dots$. According to the definitions above, for n = 6 this results in ${\pi }_6=\left(1-\frac{1}{2}\right)*\left(1-\frac{1}{4}\right)*\frac{1}{6}*\left(1-\frac{3}{8}\right)*\left(1-\frac{3}{10}\right)*\left(1-\frac{3}{12}\right)\approx 0.02051$. At this point, we obtain the values $\frac{{\pi }_6}{{\sup }_{p\in D_1}{b}_6\left(p,x^6\right)}=410978.3$ and $\frac{{\pi }_6}{{\sup }_{p\in D_2}{b}_6\left(p,x^6\right)}=0.30618$ ; therefore, both objects did not reach their corresponding threshold. After n = 7 draws, we have ${\pi }_7=\left(1-\frac{1}{2}\right)*\left(1-\frac{1}{4}\right)*\frac{1}{6}*\left(1-\frac{3}{8}\right)*\left(1-\frac{3}{10}\right)*\left(1-\frac{3}{12}\right)*\frac{3}{14}\approx 0.004394531$ and we get $\frac{{\pi }_7}{{\sup }_{p\in D_1}{b}_7\left(p,x^7\right)}=1.76*{10}^{12}$ and $\frac{{\pi }_7}{{\sup }_{p\in D_2}{b}_7\left(p,x^7\right)}=0.2895$, showing that the first object reaches the threshold of $\frac{1}{{\alpha }_1}={10}^{10}$, indicating that the observed sequence is extremely unlikely under hypothesis H₁. Our stopping rule chooses $\delta =2$ and sets n = 7. Therefore, QUICK-STOP used 7 permutations/simulations to decide the unknown p-value is non-significant and chose hypothesis H₂.

For a second example, we consider $p={10}^{-9}$. Here, we expect to draw many permutations/simulations without observing a more extreme test statistic. A simulated example resulted in $x_i=0$ for $i\le 763193033,i\ne 17652977$, and $x_{17652977}=1$. A straightforward computation for $n=763193033$ gives $\frac{{\pi }_n}{{\sup }_{p\in D_1}{b}_n\left(p,x^n\right)}=2.78*{10}^{-5}$ and $\frac{{\pi }_n}{{\sup }_{p\in D_2}{b}_n\left(p,x^n\right)}\approx {10}^{10}=\frac{1}{{\alpha }_2}$, that means the point where QUICK-STOP stops and selects hypothesis $\delta =1$ with $N=763,193,033$.

Theorem 1 summarizes two important properties of our stopping rule.

Theorem 1.

• (i)$P_p\left[\delta =2\right]\le {\alpha }_1$ for $p\in D_1$, and $P_p\left[\delta =1\right]\le {\alpha }_2$ for $p\in D_2$
• (ii)Let $K\left(\alpha ,t_1,t_2\right)$ be the class of all stopping rules $\left(N^{\prime },{\delta }^{\prime }\right)$ such that property (i) is fulfilled for ${\alpha }_1:=\alpha t_1$ and ${\alpha }_2:=\alpha t_2$. Then,

$$\frac{E_p\left[N\right]}{\underset{\left(N^{\prime },{\partial }^{\prime }\right)\in K\left(\alpha ,t_1,t_2\right)}{\inf }{E}_p\left[N^{\prime }\right]}=1+o\left(1\right)as\alpha \to 0\mathit{for\; all\; p}\in [0,1].$$

Theorem 1 states that for fixed parameters p₁and p₂ (and so d), our stopping rule QUICK-STOP guarantees to control the error probabilities at the arbitrary, pre-specified rates α₁ and α₂. The key for the proof of this result is the usage of a one-step delayed estimator ${\widehat{p}}_{r-1}$ that allows utilizing Doob’s martingale inequality (Pavlov, 1991; Tartakovsky, 2014). Besides, for small rates α₁ and α₂ that are of interest in practice, the expected number of permutations/simulations N of QUICK-STOP approaches the theoretical lower bound among all comparable stopping rules that also control the error probabilities at the same levels. The theoretical lower bound depends on p, p₁, p₂ (and therefore d), α₁ and α₂. Therefore, our procedure is nearly optimal regarding the number of expected draws. The explicit terms for the lower bound are described in Theorem 1 in the Supplementary Material B. The indifference parameter d determines a tradeoff between worst-cased expected runtime and the interpretability of the results due to the indifference region. In contrast to the scenario without separated hypotheses, the introduction of the indifference region implies that the procedure stops eventually (Gandy, 2009), and the expected number of permutations/simulations is finite. It is important to note that our result includes the deterministic cases $p\in \left\{0,1\right\}$ (Supplementary Material B). After the decision, the current empirical p-value provides an accurate p-value estimate for tests with p-values close to the thresholds p₁ and p₂. If one is also interested in an accurate estimation of p-values that are clearly not significant, one can combine the stopping rule with a minimum number of permutations/simulations.

Application and numerical study

In this section, we describe the results of a simulation study and the application of our approach to a whole-genome sequencing study for lung function.

Simulation study

In this simulation study, we compare QUICK-STOP with a different, simple stopping rule that is based on the adaptive permutation procedure in the popular genetic analysis tool PLINK1.9 (Chang et al., 2015). Both stopping rules can be applied to any randomized test. We directly apply both stopping rules to a sequence $x_1,x_2,\dots$ of independent Bernoulli variables with unknown parameter p, mimicking a general scenario of randomized testing. We assessed how fast both rules could decide which hypothesis is true and how the rates of wrong decisions behave empirically. The PLINK1.9 related stopping rule is based on an asymptotic confidence interval approximation. It aims to decide between the two hypotheses

$$H_1:p\le p_{significance}and{H}_2:p>p_{significance}$$

After n permutations/simulations, the p-value is estimated by the proportion of randomized test statistics that were more extreme than the observed statistic, that means: ${\widehat{p}}_{MLE}=\frac{1}{n}{\sum }_{r=1}^n{x}_i$. Based on the corresponding estimated standard error $\sqrt{\frac{{\widehat{p}}_{MLE}\left(1-{\widehat{p}}_{MLE}\right)}{n}}$, a $1-\alpha$ confidence interval utilizing the asymptotic normal distribution is constructed. If this confidence interval does not contain the significance level $p_{significance}$, the stopping rule draws the decision based on the location of the estimated p-value below or above the significance level. We will refer to this approach by CI. We note that the estimation of the standard error requires an estimated p-value that is not 0 or 1.

For QUICK-STOP, we chose ${\alpha }_1={\alpha }_2={10}^{-10}$ and for CI we considered a $\left(1-{10}^{-10}\right)$ confidence interval (based on asymptotic normal distribution assumption).

In Tables 1 - 4, we report the average number of draws required by both approaches for multiple combinations of p, $p_{significance}$ and d. For QUICK-STOP, we considered two different indifference parameters d, a scenario with a very small value and a scenario where the indifference region might be of practical impact (see the difference of p₂ and p₁ in Tables 1–4).

Table 1.

Averaged number of draws until the decision for CI and QUICK-STOP (QS). The significance level is chosen as $p_{siginificace}=0.005$. Results are based on 10,000 replicates.

	Significance cutoff	0.5	0.1	0.05	10−2	p 10−3	10−4	10−5	10−6	10−7	10−8
CI	$p_{siginificace}=0.005$	22	379	937	16453	2666	10012	100480	976758	9.98e06	9.81e07
QS	$p_1/p_2=4.999*{10}^{-3}/5*{10}^{-3}$	13	124	372	14311	11374	6051	5617	5574	5569	5569
	$p_1/p_2=4.9*{10}^{-3}/5*{10}^{-3}$	13	123	367	13584	11374	6051	5617	5574	5569	5569

Table 4.

Averaged number of draws until the decision for CI and QUICK-STOP (QS). The significance level is chosen as $p_{siginificace}={10}^{-9}$. Results for $p\ge {10}^{-7}$ are based on 10,000 replicates, results for $p={10}^{-8}$ are based on 1,000 replicates due to computational reasons.

	Significance cutoff	0.5	0.1	0.05	10−2	p 10−3	10−4	10−5	10−6	10−7	10−8
CI	$p_{siginificace}=1*{10}^{-9}$	22	343	767	4155	41975	420480	4.20e06	4.20e07	4.30e08	5.06e09
QS	$p_1/p_2=0.999*{10}^{-9}/1*{10}^{-9}$	4	20	39	200	2690	29952	400473	5.62e06	9.35e07	2.35e09
	$p_1/p_2=0.9*{10}^{-9}/1*{10}^{-9}$	4	20	39	200	2658	29860	398335	5.53e06	9.09e07	2.21e09

Compared to CI, QUICK-STOP generally reduces the number of draws dramatically in most scenarios. In Tables 1 and 2, we see a huge reduction for very small p, since the CI approach cannot estimate the confidence interval before the estimated p-value is non-zero. The parameter d does not have a strong impact on the QUICK-STOP results, indicating that this parameter should be chosen very small in practice, reducing the indifference region. However, if p is of the same magnitude as $p_{significance}$, there are scenarios where the CI approach does require fewer draws than QUICK-STOP. The reason for this is that the probability of a wrong decision by the CI approach is much larger than the $\left(1-{10}^{-10}\right)$ confidence level suggests.

Table 2.

Averaged number of draws until the decision for CI and QUICK-STOP (QS). The significance level is chosen as $p_{siginificace}=5*{10}^{-4}$. Results are based on 10,000 replicates.

	Significance cutoff	0.5	0.1	0.05	10−2	p 10−3	10−4	10−5	10−6	10−7	10−8
CI	$p_{siginificace}=5*{10}^{-4}$	22	345	784	4594	167198	26682	101565	976847	9.98e06	9.81e07
QS	$p_1/p_2=4.999*{10}^{-4}/5*{10}^{-4}$	9	61	148	1344	149838	119452	63133	58647	58191	58158
	$p_1/p_2=4.9*{10}^{-4}/5*{10}^{-4}$	9	61	147	1330	142424	119452	63133	58647	58191	58158

We analyzed the empirical wrong decisions rates for $p_{significance}=5*{10}^{-3}$ and $p_{significance}=5*{10}^{-4}$. We considered values for p close to these significance levels; the results are reported in Table 5. As we can see, the empirical rates for wrong decisions are increased for the CI approach. Since the CI approach does not consider the ‘trajectory’ along the draws and relies on an asymptotic distribution, the confidence interval does not correspond to an error rate of 10⁻¹⁰. In comparison, we did not observe a wrong decision by QUICK-STOP. This is expected since we chose ${\alpha }_1={\alpha }_2={10}^{-10}$ and according to Theorem 1, the error rates are controlled by these parameters.

Table 5.

Empirical analysis of the probability of a wrong decision for multiple combinations of p, $p_{significance}$ and d. Results are based on 10⁷ replications (*10⁷ replications, ** 10⁴replications, *** 10³replications, due to computational restrictions).

		p
	Significance cutoff	10−2	5.1 * 10−2	10−3	5.1 * 10−3	10−4	10−5
CI	$p_{significance}=5*{10}^{-3}$	4.2e-06	0.004**	1.0092e-03	5.003e-04	1.029e-4	9.6e-06
QS	$p_1/p_2=4.999*{10}^{-3}/5*{10}^{-3}$	0	0**	0	0	0	0
	$p_1/p_2=4.9*{10}^{-3}/5*{10}^{-3}$	0	0**	0	0	0	0
CI	$p_{significance}=5*{10}^{-4}$	0	0	3e-06*	0.002***	1.021e-4	9.6e-06
QS	$p_1/p_2=4.999*{10}^{-4}/5*{10}^{-4}$	0	0	0*	0***	0	0
	$p_1/p_2=4.9*{10}^{-4}/5*{10}^{-4}$	0	0	0*	0***	0	0

The increased error rate of CI can also be computed directly. For example, for $p=5*{10}^{-8}$, CI stops if, after 149,400,000 simulations, only 1 success was observed. The corresponding estimated p-value is $6.69*{10}^{-9}<p_{significance}$. If we assume that the true p-value is $p=5.01*{10}^{-8}>p_{significance}$, the probability of observing exactly 1 success after 149,400,000 simulations is 0.0042. The strength of QUICK-STOP is that it guarantees the probability of an error, even if the true p-value is very small and close to the significance level.

Application to COPD whole-genome sequencing study

In the simulation study above, we analyzed the performance of QUICK-STOP and the CI-based approach for fixed p. To illustrate the application and advantages of our approach in a real data example where the true, unknown $p\text{'}s$ are drawn from a realistic distribution, we considered a genome-wide single variant association analysis for a whole-genome sequencing dataset from the COPDGene study (Regan et al., 2010).

The COPDGene Study consists of >10,000 current or former smokers with and without chronic obstructive pulmonary disease (COPD). Subjects were of non-Hispanic White or African-American ancestry and age between 45 and 80 years. Also, a minimum of 10 pack-years of smoking and no lung disease (other than COPD or asthma) were ascertainment criteria. The Boston Early-Onset COPD study (Silverman et al., 1998) (BEOCOPD) is an extended pedigree study with probands age below 53 and severe COPD (defined as forced expiratory volume in one second (FEV₁) < 40% predicted). As part of the National Heart, Lung, and Blood Institute Trans-Omics in Precision Medicine (TOPMed) project, 2000 severely affected cases from BEOCOPD and COPDGene, and controls with normal spirometry from COPDGene were selected (Prokopenko et al., 2018) for whole-genome sequencing. Our analysis dataset consisted of 51,715,479 genetic variants and 1,794 samples, after selecting variants with non-missingness, no lower MAF cutoff, and the quality control described in Prokopenko et al. (2018). Using covariates for pack-years and 10 eigenvectors from Jaccard population stratification analysis (Prokopenko et al., 2016), we computed the covariate-adjusted quantitative lung function trait (FEV₁ percent predicted) via linear regression. To test for association between this adjusted phenotype and genotype, we also consider a linear regression model. However, corresponding asymptotic p-values are not reliable since the phenotype distribution in the sample is highly skewed, and most variants in a WGS study are rare. Therefore, we evaluated significance by permutation of the phenotype information.

We compare our approach QUICK-STOP with the adaptive, confidence interval-based permutation stopping rule implemented in the popular genetic analysis tool PLINK1.9 (Chang et al., 2015). This approach corresponds to the stopping rule CI in the previous simulation study. For QUICK-STOP, we chose ${\alpha }_1={\alpha }_2={10}^{-10}$ and ran the analysis with $p_1=5*{10}^{-8},p_2=6*{10}^{-8}$ as the first scenario and with $p_1={10}^{-9}$,$p_2=2*{10}^{-9}$ as the second scenario.

For the analysis with PLINK1.9, we applied the aperm command, which requires the specification of six parameters (Chang et al., 2015). We specified the minimum and maximum number of permutations by 2 and 10⁹ (maximum possible value), respectively, and that variants can be pruned out after every permutation. We set the significance level to $5*{10}^{-8}$ in the first scenario, and 10⁻¹⁹ in the second. In addition, we chose $\beta =0.005$, such that PLINK1.9 computes a $\left(1-\frac{0.005}{51715479}\right)\approx \left(1-{10}^{-10}\right)$ confidence interval (based on the normal distribution approximation) and stops if this confidence interval does not contain the significance level.

In Table 6, we report the results of this analysis. For each lower cutoff for p-values, we extracted all variants for which the final p-value estimate provided by PLINK1.9 was above this threshold. We report the corresponding number of variants and the overall number of computed permutations along these single nucleotide polymorphisms (SNPs) for both methods. We observe that our approach reduces the number of required permutations by a factor around 10 along most SNPs. In addition, we observe that most of the permutations are performed on a small set of ‘interesting/promising’ SNPs. For example, in the first scenario, our approach utilizes more than 50% of the overall number of permutations for 278 variants. The corresponding fraction for the confidence interval-based approach is only approximately 23%.

Table 6:

Analysis of the COPD dataset with our sequential testing approach QUICK-STOP and PLINK1.9.

Scenario	Lower p- value cutoff	Number of variants	QUICK-STOP	PLINK1.9	Ratio PLINK1.9/ QUICK-STOP
1	0.05	49,226,335	3.96e08	3.99e09	10.08
	10−2	51,244,126	7.70e08	7.43e09	9.65
	10−3	51,676,557	1.17e09	1.27e10	10.85
	10−4	51,712,482	1.58e09	1.68e10	10.63
	10−5	51,715,193	1.98e09	1.98e10	10.00
	10−6	51,715,445	2.60e09	2.28e10	8.77
	–	51,715,471	4.37e09	2.57e10	5.88
2	0.05	49,226,335	3.81e08	3.99e09	10.47
	10−2	51,244,126	6.22e08	7.43e09	11.95
	10−3	51,676,557	9.16e08	1.27e10	13.87
	10−4	51,712,482	1.24e09	1.68e10	13.55
	10−5	51,715,193	1.50e09	1.98e10	13.2
	10−6	51,715,446	1.79e09	2.27e10	12.68
	–	51,715,479	3.06e09	2.87e10	9.38

For each p-value lower cutoff, we considered all variants where the final p-value estimate of PLINK1.9 was above this cutoff. We excluded the genetic variants where PLINK1.9 reached the maximum of 10⁹ permutations.

We excluded variants from this comparison where PLINK1.9 reached the maximum number of permutations (10⁹), since this number is truncated by the software implementation. This only affected the first scenario, where we excluded eight genetic variants. One of these genetic variants was reported to be significant by our approach with an estimated p-value of $\widehat{p}\approx 5.24*{10}^{-8}$. In the second scenario, neither PLINK1.9 nor our approach required 10⁹ permutations for a variant, and no genetic variant was reported to be genome-wide significant with respect to this significance level.

Overall, this real data example demonstrates the practical advantages of our approach; the results are in line with our simulation study above.

Discussion

The analysis of recent datasets in the field of life science often encounters scenarios in which asymptotic theory cannot be applied, and the general approach to significance testing in these scenarios is permutation or simulation.

To avoid the huge computational effort, researchers have proposed several methods to approximate permutation-based p-values for particular test statistics or designed efficient resampling procedures that utilize special properties of common application scenarios.

On the other hand, a general approach that aims to reduce the computational burden of permutation/simulation-based testing without losing flexibility and robustness is the utilization of stopping rules. The goal of such stopping rules is to decide as fast as possible if the unknown p-value is below a pre-specified level of significance.

Here, we proposed such a general and intuitive sequential testing-based stopping rule, that is easy to implement, requires almost no additional computational effort, and controls the error probabilities rigorously. Based on the theory for sequential testing, it can be shown that our stopping rule is nearly optimal in terms of the expected number of permutations/simulations. In a simulation study, we investigated the performance of our approach and showed that our stopping rule reduces the number of permutations/simulations substantially compared to the current methodology. In an application to a whole-genome sequencing study for lung function, we demonstrated the implementation and the practical value of our stopping rule. An implementation scheme and a simulation tool for QUICK-STOP can be found at https://github.com/julianhecker/QUICK-STOP.

Table 3.

Averaged number of draws until the decision for CI and QUICK-STOP (QS). The significance level is chosen as $p_{siginificace}=5*{10}^{-8}$. Results for $p\ge {10}^{-7}$ are based on 10,000 replicates, results for $p={10}^{-8}$ are based on 1,000 replicates due to computational reasons.

	Significance cutoff	0.5	0.1	0.05	10−2	p 10−3	10−4	10−5	10−6	10−7	10−8
CI	$p_{siginificace}=5*{10}^{-8}$	22	343	765	4176	41955	420576	4.29e06	4.68e07	1.67e09	2.72e08
QS	$p_1/p_2=4.999*{10}^{-8}/5*{10}^{-8}$	4	20	46	294	3547	47337	737914	1.57e07	1.73e09	1.41e09
	$p_1/p_2=4.9*{10}^{-8}/5*{10}^{-8}$	4	20	45	294	3535	47246	735007	1.56e07	1.64e09	1.41e09