MERIT: Controlling Monte-Carlo error rate in large-scale Monte-Carlo hypothesis testing

Abstract

The use of Monte-Carlo (MC) p-values when testing the significance of a large number of hypotheses is now commonplace. In large-scale hypothesis testing, we will typically encounter at least some p-values near the threshold of significance, which require a larger number of MC replicates than p-values that are far from the threshold. As a result, some incorrect conclusions can be reached due to MC error alone; for hypotheses near the threshold, even a very large number (eg, 10⁶) of MC replicates may not be enough to guarantee conclusions reached using MC p-values. Gandy and Hahn (GH)⁶⁻⁸ have developed the only method that directly addresses this problem. They defined a Monte-Carlo error rate (MCER) to be the probability that any decisions on accepting or rejecting a hypothesis based on MC p-values are different from decisions based on ideal p-values; their method then makes decisions by controlling the MCER. Unfortunately, the GH method is frequently very conservative, often making no rejections at all and leaving a large number of hypotheses “undecided”. In this article, we propose MERIT, a method for large-scale MC hypothesis testing that also controls the MCER but is more statistically efficient than the GH method. Through extensive simulation studies, we demonstrate that MERIT controls the MCER while making more decisions that agree with the ideal p-values than GH does. We also illustrate our method by an analysis of gene expression data from a prostate cancer study.

1 |. INTRODUCTION

Modern scientific studies often require testing a large number of hypotheses; in particular, modern biological and biomedical studies of -omics such as genomics, proteomics, and metabolomics typically test hundreds or thousands of hypotheses at a time. In these studies, p-values for testing individual hypotheses are first obtained and a procedure that corrects for multiplicity, such as the procedure of Benjamini and Hochberg (BH),¹ is then applied to make decisions on the hypotheses. Specifically, denote the $m$ null hypotheses by $H_{\mathrm{1,0}},H_{\mathrm{2,0}},\dots ,H_{m,0}$. Ideally, the p-values are calculated based on an (accurate) analytical method or exhaustive resampling of replicates, which we call ideal p-values and denote by $p_1^{\mathrm{*}},p_2^{\mathrm{*}},\dots ,p_m^{\mathrm{*}}$. When the BH procedure with nominal false-discovery rate (FDR) $\varphi$ is applied to the ideal p-values, they are sorted in an ascending order $p_{\left(1\right)}^{\mathrm{*}}\le p_{\left(2\right)}^{\mathrm{*}}\le \cdots \le p_{\left(m\right)}^{\mathrm{*}}$ and then the largest integer $k$ such that $p_{\left(k\right)}^{\mathrm{*}}\le \varphi \times k/m$ is found; hypotheses corresponding to $p_{\left(1\right)}^{\mathrm{*}},p_{\left(2\right)}^{\mathrm{*}},\dots ,p_{\left(k\right)}^{\mathrm{*}}$ are rejected while the remaining hypotheses are accepted. Thus, we call ${\tau }^{\mathrm{*}}\equiv \varphi \times k/m$ the $BH$ cutoff for the ideal p-values, which separates the ideal p-values for rejected and accepted hypotheses.

When the ideal p-values cannot be calculated analytically, we frequently obtain p-values by resampling methods such as permutation or bootstrap. For ease of exposition, hereafter we generically refer to any resampling-based method as a Monte-Carlo (MC) method, and refer to individual resamples as MC replicates. The recommended form of an MC p-value is ${\hat{p}}_i=\left(X_i+1\right)/(n+1)$^2–4 for the ith hypothesis, where $n$ is the number of MC replicates (ie, MC sample size) and $X_i$ is the number of “exceedances”, that is, when the statistic based on an MC replicate “exceeds” the observed test statistic. When using a finite $n$, there will be a difference between the MC p-value and the ideal p-value.

It is well known that the MC sample size required to test a hypothesis having an ideal p-value near the BH cutoff ${\tau }^{\mathrm{*}}$ is much larger than that required for a hypothesis having an ideal p-value that is far from ${\tau }^{\mathrm{*}}$. What is less appreciated is that, when testing a large number of hypotheses, it is very likely that at least some hypotheses have ideal p-values near ${\tau }^{\mathrm{*}}$. As a result, if we repeat the MC experiment with a different set of replicates (and so obtain a slightly different set of MC p-values), there can be noticeable variability in the list of hypotheses that are rejected. The source of this variability is that random fluctuations in the MC procedure for a hypothesis having ideal p-value near the BH cutoff can result in the MC p-value randomly crossing the BH cutoff. Because the MC sample size is finite, these lists can also differ from the list based on the ideal p-values. This is true whether the test procedure adopts a fixed stopping criterion, that is, mandating a fixed MC sample size for all hypotheses, or a sequential stopping criterion,⁵ that is, allowing some tests to stop early after collecting sufficient information about its decision. Note that even for test procedures (eg, the procedure of Sandve et al.⁵) that always control the FDR in any list of rejected hypotheses resulting from an arbitrary set of MC replicates, there can still be variability in these lists. This failure in reproducibility is clearly undesirable to investigators. Even a large number (eg, 10⁶) of MC replicates may not be enough to guarantee a list free of MC error.

To date, only Gandy and Hahn (GH)^6–8 have directly addressed this problem. They developed a method that can be coupled with any MC test method that has a sequential stopping criterion. Their method makes decisions on multiple hypotheses by controlling the family-wise Monte-Carlo error rate (MCER), which is the probability that any of their decisions on accepting or rejecting a hypothesis are different from what would have been obtained from the ideal p-values. It works in a sequential manner, giving a decision among “rejected”, “accepted”, and “undecided” to each test after each iteration and continuing sampling only for hypotheses that are “undecided”.

Denote the decisions (between “rejected” and “accepted”) made by the BH procedure that is applied to the ideal p-values by $D_1^{\mathrm{*}},D_2^{\mathrm{*}},\dots ,D_m^{\mathrm{*}}$. Denote the decisions (among “rejected”, “accepted”, and “undecided”) made by an MC test procedure by ${\widehat{D}}_1,{\widehat{D}}_2,\dots ,{\widehat{D}}_m$. Following GH, we define the total number of MC errors to be the sum of all disagreements between the $D_i^{\mathrm{*}}\mathrm{s}$ and ${\widehat{D}}_i\mathrm{s}$ among hypotheses that are determined as “rejected” or “accepted” by the MC test procedure:

$$V=\sum _{i=1}^m\mathbb{I}\left({\widehat{D}}_i=\mathrm{rejected},D_i^{*}=\mathrm{accepted}\right)+\sum _{i=1}^m\mathbb{I}\left({\widehat{D}}_i=\mathrm{accepted},D_i^{*}=\mathrm{rejected}\right),$$ (1)

where $\mathbb{I}(\cdot ,\cdot )$ is an indicator function. Controlling the family-wise MCER at level $\alpha$, that is, $\mathrm{P}\mathrm{r}(V\ge 1)\le \alpha$, assures that, among those tests that the MC test procedure actually decides, the probability that we reach a conclusion that is different from the decision we would make if we had the ideal p-values is less than $\alpha$. Note that there is no error control on “undecided” hypotheses ${(\widehat{D}}_i=\mathrm{u}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{d})$, which would require more MC replicates to reach a decision.

At each iteration of the MC test procedure, the GH method forms a two-sided Robbins-Lai confidence interval (CI)^9,10 for each ideal p-value that maintains a joint coverage over all iterations (ie, accounting for the possible differences in MC sample size for each test); the level of each CI is determined by using the Bonferroni correction. Then, the GH method applies the BH procedure to the collection of upper CI limits and, separately, to the collection of lower CI limits. Those tests for which both lower and upper limits are rejected are determined as “rejected,” those for which both limits are accepted are determined as “accepted”; the remaining tests are determined as “undecided.” The GH method controls MCER at every iteration. However, it is extremely conservative due to the use of the Bonferroni correction, and thus tends to make no rejections at all when the MC sample size is not very large.

While the sequential stopping criterion has advantages (eg, it can greatly reduce the MC sample size for tests that are either highly insignificant or highly significant), use of a fixed, pre-determined MC sample size is more common. Compared to the sequential stopping criterion, use of a fixed MC sample size reduces the chance of making different decisions on a hypothesis when the MC experiment is repeated. Further, in many instances there may be little or no computational advantage to stopping some tests early, for example, when the cost of evaluating the stopping criterion exceeds the cost of calculating the test statistics. For these reasons, we now restrict our attention to MC procedures with fixed MC sample sizes.

In this article, we propose a method, called MERIT (Monte-Carlo Error Rate control In large-scale MC hypothesis Testing), that makes decisions on the hypotheses by controlling the same family-wise MCER as the GH method, but it works for MC test procedures with a fixed stopping criterion. We aim to maximize detection efficiency by minimizing the number of “undecided” hypotheses at a given MC sample size or by making “rejected” or “accepted” decisions for all hypotheses with fewer MC replicates. We present our method in Section 2, examine its properties through simulation studies in Section 3, and apply it to a study of prostrate cancer with gene expression data in Section 4. Because one immediate advantage of using a fixed MC sample size is that the Robbins-Lai CI required for the sequential GH approach is much wider than any single-stage CI,¹¹ in Section 2 we also develop a fixed-MC-sample-size version of GH that uses a single-stage CI, to allow for a more fair comparison in Sections 3 and 4. Finally, we conclude with a discussion in Section 5.

2 |. METHODS

Suppose that a fixed number, $n$, of MC replicates have been collected for testing all $m$ hypotheses. For testing the ith hypothesis $H_{i,0}$, let $s_i$ be the observed test statistic and $S_{ij}$ be the test statistic calculated from the jth MC replicate. The input of our method is the matrix $I={\left\{I_{ij}\right\}}_{m\times n}$, where $I_{ij}=\mathbb{I}\left(\left|S_{ij}\right|\ge \left|s_i\right|\right)$ is the exceedance indicator corresponding to a symmetric, two-sided statistic (note that $I_{ij}$ can be redefined for one-sided tests or tests with asymmetric null distributions). We frequently use the total number of exceedances for each test, $X_i={\sum }_{j=1}^n{I}_{ij}$. Because the replicates are sampled independently from the null distribution, we have $X_i~\mathrm{B}\mathrm{i}\mathrm{n}\left(n,p_i^{\mathrm{*}}\right)$, which is a binomial variable with $n$ trials and “success” rate being the ideal p-value $p_i^{\mathrm{*}}$. This is the key assumption we use to develop our method, same as in the development of the GH method.

We partition the total MC errors in (1) into the type-I and type-II MC errors, which correspond to the first and second terms in (1) and are denoted by $V^{\mathrm{I}}$ and $V^{\mathrm{I}\mathrm{I}}$, respectively. In Sections 2.1,2.2, and 2.3, we present a procedure that makes decisions on the hypotheses between “rejected” and “non-rejected” while controlling the family-wise type-I MCER $\mathrm{P}\mathrm{r}\left(V^{\mathrm{I}}\ge 1\right)$ at ${\alpha }_1$. In Section 2.4, we present another procedure, which is obtained by modifying the first procedure and makes decisions on the hypotheses between “accepted” and “non-accepted” while controlling the family-wise type-II MCER $\mathrm{P}\mathrm{r}\left(V^{\mathrm{I}\mathrm{I}}\ge 1\right)$ at ${\alpha }_2$. Either of these procedures can be used separately, if control of only one type of MCER is desired. In Section 2.5, we show how to combine the two procedures to determine each hypothesis as “rejected”, “accepted”, or possibly “undecided”, while controlling the overall family-wise MCER at ${\alpha }_1+{\alpha }_2$, because $\mathrm{P}\mathrm{r}(V\ge 1)\le \mathrm{P}\mathrm{r}\left(V^{\mathrm{I}}\ge 1\right)+\mathrm{P}\mathrm{r}\left(V^{\mathrm{I}\mathrm{I}}\ge \right.1)$. In what follows, we omit the term “family-wise” for simplicity.

2.1 |. A procedure that rejects original hypotheses while controlling the type-I MCER

Recall that ${\tau }^{\mathrm{*}}$ is the BH cutoff for the ideal p-values $p_1^{\mathrm{*}},p_2^{\mathrm{*}},\dots ,p_m^{\mathrm{*}}$. If ${\tau }^{\mathrm{*}}$ is known a priori, the result of the BH procedure can be viewed as the truth of the following hypotheses:

$$H_{i,0}^{\mathrm{*}}:p_i^{\mathrm{*}}>{\tau }^{\mathrm{*}}\text{vs.}{H}_{i,1}^{\mathrm{*}}:p_i^{\mathrm{*}}\le {\tau }^{\mathrm{*}},i=1,2,\dots ,m.$$ (2)

Recall that a type-I MC error occurs when a hypothesis $H_{i,0}^{\mathrm{*}}$ is true (ie, not rejected using the ideal p-value) but rejected using the MC p-value; this would happen when the MC p-value is smaller than its corresponding ideal p-value. Then, a test procedure based on MC p-values that controls the family-wise error rate (FWER) for testing $H_{i,0}^{\mathrm{*}}$ at ${\alpha }_1$ would control the type-I MCER at ${\alpha }_1$.

In reality, ${\tau }^{\mathrm{*}}$ is unknown since the ideal p-values are unknown. A reasonable strategy to control type-I MCER would be to choose some threshold $\tau$ satisfying $\tau <{\tau }^{\mathrm{*}}$; this is intuitive, since, as we make $\tau$ smaller, we require smaller MC p-values to reject a hypothesis, and smaller MC p-values correspond to increased confidence in any rejections we make. Thus, we divide the problem of testing hypotheses in (2) into two sub-problems and develop a two-step procedure. In the first step, we find a lower limit ${\tau }_l$ for ${\tau }^{\mathrm{*}}$ that satisfies ${\tau }_l<{\tau }^{\mathrm{*}}$ with a probability of at least $\left(1-{\tilde{\alpha }}_1\right.$), where ${\tilde{\alpha }}_1$ is chosen such that $0<{\tilde{\alpha }}_1<{\alpha }_1$. In the second step, we consider the revised hypotheses, treating ${\tau }_{\ell }$ as a constant:

$${\tilde{H}}_{i,0}:p_i^{\mathrm{*}}>{\tau }_{\ell }\text{vs.}{\tilde{H}}_{i,1}:p_i^{\mathrm{*}}\le {\tau }_{\ell },i=1,2,\dots ,m.$$ (3)

We develop a test procedure that tests (3) while controlling the FWER at level $\left({\alpha }_1-{\tilde{\alpha }}_1\right)$. Note that the same decision on ${\tilde{H}}_{i,0}$ is transferred to $H_{i,0}^{\mathrm{*}}$ and $H_{i,0}$. Then, by the relationship

$$\begin{array}{l}\left\{V^{\mathrm{I}}\ge 1\right\}=\left\{{\tau }_{\ell }>{\tau }^{\mathrm{*}},V^{\mathrm{I}}\ge 1\right\}\cup \{{\tau }_{\ell }\le {\tau }^{\mathrm{*}},V^{\mathrm{I}}\ge 1\}\\ \subset \left\{{\tau }_{\ell }>{\tau }^{\mathrm{*}}\right\}\cup \{{\tau }_{\ell }\le {\tau }^{\mathrm{*}},\text{at}\text{least}\text{one}{H}_{i,0}^{\mathrm{*}}\text{is}\text{true}\text{but}\text{rejected}\}\\ \subset \left\{{\tau }_{\ell }>{\tau }^{\mathrm{*}}\right\}\cup \{{\tau }_{\ell }\le {\tau }^{\mathrm{*}},\text{at}\text{least}\text{one}{\tilde{H}}_{i,0}\text{is}\text{true}\text{but}\text{rejected}\},\end{array}$$

the type-I MCER $\mathrm{P}\mathrm{r}\left(V^{\mathrm{I}}\ge 1\right)$ is always bounded by the sum of $\mathrm{P}\mathrm{r}\left({\tau }_{\ell }>{\tau }^{\mathrm{*}}\right)$ and the FWER of the test procedure in testing (3), which is ${\tilde{\alpha }}_1+\left(1-{\tilde{\alpha }}_1\right)\left({\alpha }_1-{\tilde{\alpha }}_1\right)\le {\tilde{\alpha }}_1+\left({\alpha }_1-{\tilde{\alpha }}_1\right)={\alpha }_1$, since ${\tilde{\alpha }}_1<{\alpha }_1$. As the default, we choose ${\tilde{\alpha }}_1={\alpha }_1/2$, that is, partitioning the type-I MCER equally into the two sub-problems. We show in Supplementary Materials S1 that the equal partition scheme generally yields the highest power.

2.2 |. Step 1: Construct a $\left(1-{\tilde{\mathit{\alpha }}}_1\right)$-level one-sided CI $\left[{\mathit{\tau }}_{\ell },1\right]$ for ${\mathit{\tau }}^{\mathrm{*}}$

We now turn our attention to constructing a one-sided CI $\left[{\tau }_{\ell },1\right]$ for ${\tau }^{\mathrm{*}}$. Note that, for any set of p-values, there is a corresponding BH cutoff; thus, we find the lower limit ${\tau }_{\ell }$ for ${\tau }^{\mathrm{*}}$ from a set of p-values that are “lower limits” for the ideal p-values $p_i^{\mathrm{*}}$. To this aim, we consider the following sets of p-values, with each set indexed by a positive continuous value $c$:

$$p_{c,i}=\frac{X_i+1+c\sqrt{X_i+1}}{n+1+c\sqrt{X_i+1}},i=1,2,\dots ,m.$$ (4)

Recall that the MC p-value ${\hat{p}}_i=\left(X_i+1\right)/(n+1)$ provides a consistent estimate of $p_i^{\mathrm{*}}$. The term $\sqrt{X_i+1}$ in (4) is an approximation to $\sqrt{n{\hat{p}}_i\left(1-{\widehat{p}}_i\right)}$, which is the estimated standard error of $X_i+1$; the term $\left(1-{\widehat{p}}_i\right)$ is omitted because only the small p-values (in the neighborhood of ${\tau }^{\mathrm{*}}$) are of interest. As $c$ goes to 0 and $n$ goes to infinity, $p_{c,i}$ converges to $p_i^{\mathrm{*}}$. In addition, $p_{c,i}$ is asymptotically larger than $p_i^{\mathrm{*}}$ for $c>0$ because $p_{c,i}$ is always larger than ${\hat{p}}_i$.

Let $F^{\mathrm{*}}(t)$ and $F_c\left(t\right)$ be the empirical distribution functions (EDF) of $\left(p_1^{\mathrm{*}},\dots ,p_m^{\mathrm{*}}\right)$ and $\left(p_{c,1},\dots ,p_{c,m}\right)$, respectively, and ${\tau }^{\mathrm{*}}$ and ${\tau }_c$ be their BH cutoffs. Storey et al¹² gave a general result that, a BH cutoff $\tau$ can also be obtained from an EDF $F\left(t\right)$ of p-values by the formula $\tau =\mathrm{m}\mathrm{a}\mathrm{x}\{t:F(t)\ge t/\varphi ,t\in [\mathrm{0,1}\left]\right\}$, which means that $\tau$ is the intersection of $F(t)$ and the straight line $t/\varphi$, as illustrated in Figure 1. When $t/\varphi$ is constrained to [0, 1] as $F(t)$, $t$ must be constrained to $[0,\varphi ]$. Then we see from Figure 1 (right) that the intersection $\tau$ must lie in $[0,\varphi ]$ and is fully determined by the EDF of p-values in the range of $[0,\varphi ]$. If $F_c(t)$ is always no greater than $F^{\mathrm{*}}(t)$ over $[0,\varphi ]$, as shown in Figure 1 (left), we must have ${\tau }_c\le {\tau }^{\mathrm{*}}$. It follows that

$$\mathrm{Pr}\left({\tau }_c\le {\tau }^{\mathrm{*}}\right)\ge \mathrm{Pr}(F_c\prec F^{\mathrm{*}}),$$

in which we use $\prec$ to denote that one function is always no greater than another function for $t\in [0,\varphi ]$. Note that ${\tau }_c$ and $F_c$ are random because $p_{c,i}$ involves $X_i$ that is based on MC replicates; ${\tau }^{\mathrm{*}}$ and $F^{\mathrm{*}}$ are constant because we condition on the observed data. To construct a $\left(1-{\tilde{\alpha }}_1\right)$-level one-sided CI $\left[{\tau }_{\ell },1\right]$ for ${\tau }^{\mathrm{*}}$, we are interested in $c$ that satisfies

$$\mathrm{P}\mathrm{r}(F_c\prec F^{\mathrm{*}})\ge 1-{\tilde{\alpha }}_1.$$ (5)

FIGURE 1.

An example of $F^{\mathrm{*}}(t)$, $F_c(t)$, and $F_{-c}(t)$ and their corresponding BH cutoffs ${\tau }^{\mathrm{*}}$, ${\tau }_c$, and ${\tau }_{-c}$. The black straight line is $t/\varphi$, where $\varphi =10\mathrm{\%}$. The left plot is a zoom-in view of the bottom left corner of the right plot.

The probability on the left of (5) increases as $c$ increases (because the increase of $p_{c,i}$ leads to the decrease of $F_c(t)$ at a given $t$). To make the CI for ${\tau }^{\mathrm{*}}$ the tightest, we wish to find the smallest $c$ that satisfies (5), denoted by $c_{\ell }$. Finally, we set the lower limit ${\tau }_{\ell }$ to ${\tau }_{c_{\ell }}$, which is the BH cutoff corresponding to the set of p-values in (4) indexed by $c_{\ell }$.

It is difficult to find the smallest $c$ that satisfies (5) analytically because $F^{\mathrm{*}}$ is unknown. Thus, we propose to obtain $c_{\ell }$ by bootstrap resampling. In the b-th bootstrap, we sample $n$ columns from the input matrix $I$ with replacement to form the bootstrap matrix $I^{\left(b\right)}$ and sum up the rows of $I^{\left(b\right)}$ to obtain the corresponding numbers of exceedances $(X_1^{\left(b\right)},X_2^{\left(b\right)},\dots ,X_m^{\left(b\right)})$. The number of bootstrap replicates $B$ needs to be large, for example, $B={10}^4$. Define the bootstrap p-values indexed by $c$ as

$$p_{c,i}^{\left(b\right)}=\frac{X_i^{\left(b\right)}+1+c\sqrt{X_i^{\left(b\right)}+1}}{n+1+c\sqrt{X_i^{\left(b\right)}+1}},i=1,2,\dots ,m.$$

Let $\widehat{F}$ and $F_c^{\left(b\right)}$ be the EDF for $\left({\widehat{p}}_1,\dots ,{\widehat{p}}_m\right)$ and $\left(p_{c,1}^{\left(b\right)},\dots ,p_{c,m}^{\left(b\right)}\right)$, respectively. Then we wish to find $c_{\ell }$ to be the smallest $c$ that satisfies

$$B^{-1}\sum _{b=1}^B\mathbb{I}\left(F_c^{\left(b\right)}\prec \widehat{F}\right)\ge 1-{\tilde{\alpha }}_1,$$ (6)

which is an empirical version of (5). In fact, it is still difficult to directly find the smallest $c$ that satisfies (6). As an even simpler alternative, we find, for every $b$, the smallest $c$ that guarantees $F_c^{\left(b\right)}\prec \widehat{F}$, denoted by $c_{\ell }^{\left(b\right)}$, and then set $c_{\ell }$ to be the $\left(1-{\tilde{\alpha }}_1\right)$ th quantile of $\left(c_{\ell }^{\left(1\right)},c_{\ell }^{\left(2\right)},\dots ,c_{\ell }^{\left(B\right)}\right)$. This strategy ensures that $\left(1-{\tilde{\alpha }}_1\right)\times B$ indicator functions in (6) take the value of 1 when evaluated at $c=c_{\ell }$. The procedure is summarized in Step 1 of Algorithm 1.

2.3 |. Step 2: Test hypotheses in (3) given ${\mathit{\tau }}_{\ell }$ while controlling the FWER at $\left({\mathit{\alpha }}_1-{\tilde{\mathit{\alpha }}}_1\right)$

For testing multiple hypotheses while controlling the FWER, step-wise procedures (where the criterion for rejecting hypotheses becomes less stringent in subsequent steps in case some hypotheses have been rejected in an earlier step) such as Holm’s¹³ are typically more powerful than single-step procedures such as the Bonferroni correction. Thus, we develop a step-wise procedure for testing hypotheses in (3) following the general framework of Romano and Wolf.^14,15 We use $X_1,X_2,\dots ,X_m$ defined earlier as the test statistics and let $x_1,x_2,\dots ,x_m$ denote their observed values. Let $x_{\left(1\right)}\le x_{\left(2\right)}\le \cdots \le x_{\left(m\right)}$ be the ordered observed values, and ${\tilde{H}}_{\left(1\right),0},{\tilde{H}}_{\left(2\right),0},\dots ,{\tilde{H}}_{\left(m\right),0}$ be the corresponding hypotheses.

We start by testing the global null ${\tilde{H}}_{\left(1\right),0}\cap {\tilde{H}}_{\left(2\right),0}\cap \cdots \cap {\tilde{H}}_{\left(m\right),0}$, which means that all $p_i^{\mathrm{*}}$ s are greater than ${\tau }_{\ell }$. We use $x_{\left(1\right)}$ as the test statistic for this test. Romano and Wolf^14,15 proposed to calculate the p-value under the global null in this form:

$$\mathrm{Pr}\left(\underset{1\le j\le m}{\mathrm{m}\mathrm{i}\mathrm{n}}{X}_j\le x_{\left(1\right)}\right),$$ (7)

where each $X_j$ is a binomial random variable with $n$ trials and some “success” rate ${\theta }_j$ that is under the null and determined below. We use the Bonferroni inequality to obtain an upper bound for (7), namely

$$\sum _{j=1}^m{F}_{\mathrm{Bin}}\left(x_{\left(1\right)};n,{\theta }_j\right),$$ (8)

Algorithm 1.

The procedure that rejects original hypotheses while controlling type-I MCER

Input: matrix $I$, nominal FDR $\varphi$, nominal type-I MCER ${\alpha }_1$

Step 1: construct a $\left(1-{\tilde{\alpha }}_1\right)$-level one-sided CI $\left[{\tau }_{\ell },1\right]$ for ${\tau }^{\mathrm{*}}$.

Calculate $\left(X_1,X_2,\dots ,X_m\right)$, $\left({\hat{p}}_1,{\hat{p}}_2,\dots ,{\hat{p}}_m\right)$, and then $\widehat{F}$.

For each bootstrap replicate $b=1,2,\dots ,B$,

Sample $n$ columns from $I$ with replacement to form $I^{\left(b\right)}$.

Calculate $(X_1^{\left(b\right)},X_2^{\left(b\right)},\dots ,X_m^{\left(b\right)})$, $(p_{c,1}^{\left(b\right)},p_{c,2}^{\left(b\right)},\dots ,p_{c,m}^{\left(b\right)})$, and then $F_c^{\left(b\right)}$.

Find $c_l^{\left(b\right)}$ to be the smallest $c$ that guarantees $F_c^{\left(b\right)}\prec \widehat{F}$.

Find $c_{\ell }$, the smallest $c$ that satisfies (5), to be the $\left(1-{\tilde{\alpha }}_1\right)$ quantile of $(c_l^{\left(1\right)},c_l^{\left(2\right)},\dots ,c_l^{\left(B\right)})$.

Find the lower limit ${\tau }_{\ell }$ to be the BH cutoff of p-values in (4) indexed by $c_{\ell }$.

Step 2: test hypotheses in (3) given ${\tau }_{\ell }$ while controlling the FWER at $\left({\alpha }_1-{\tilde{\alpha }}_1\right)$.

Set $i=1$.

while $i\le m$,

If (9) does not hold, accept all remaining hypotheses ${\tilde{H}}_{\left(i\right),0}$ S and exit the loop.

Otherwise, reject ${\tilde{H}}_{\left(i\right),0}$, update $i=i+1$.

Whenever ${\tilde{H}}_{\left(i\right),0}$ is rejected, $H_{\left(i\right),0}$ is rejected.

Output: a decision for every hypothesis $H_{i,0}$ between “rejected” and “non-rejected”.

where $F_{\mathrm{B}\mathrm{i}\mathrm{n}}(x;n,\theta )$ denotes $\mathrm{P}\mathrm{r}(X\le x)$ for $X~\mathrm{B}\mathrm{i}\mathrm{n}(n,\theta )$. Expression (8) can be calculated analytically, although this computation efficiency comes at the cost of losing some statistical power compared to using (7) directly. Note that the upper bound in (8) is valid under any dependence structure among the $m$ tests.

Now we consider the problem of determining ${\theta }_j$ s. When all $p_i^{\mathrm{*}}\mathrm{s}$ are close to the boundary ${\tau }_{\ell }$, we should choose ${\theta }_j={\tau }_{\ell }$ for all $j$s. Hansen¹⁶ noted that it is possible to gain more power by making adjustments for null hypotheses that are “deep in the null”, that is, $p_i^{\mathrm{*}}$ s that are not near the boundary ${\tau }_{\ell }$ but instead satisfy $p_i^{\mathrm{*}}\gg {\tau }_{\ell }$. In the presence of a large number of tests, we expect many $p_i^{\mathrm{*}}\mathrm{s}$ are “deep in the null”. Intuitively, the hypotheses associated with very large $p_i^{\mathrm{*}}\mathrm{s}$ can be removed and the “effective” number of tests that we need to adjust for can be considerably reduced, leading to improved power. Ideally, we should choose ${\theta }_j=\mathrm{m}\mathrm{a}\mathrm{x}\left\{{\tau }_{\ell },p_j^{\mathrm{*}}\right\}$. However, $p_j^{\mathrm{*}}$ is unknown. Hansen¹⁶ proposed an estimator for $p_j^{\mathrm{*}}$ that leads to the choice

$${\theta }_j=\left\{\begin{array}{ll}{\tau }_{\ell },& \text{if}\sqrt{n}\left(x_j/n-{\tau }_{\ell }\right)/\sqrt{{\tau }_{\ell }\left(1-{\tau }_{\ell }\right)}\le \sqrt{2\log \log n,}\\ x_j/n,& \text{otherwise.}\end{array}\right.$$

Therefore, if the statistic $x_j$ is sufficiently large, ${\theta }_j$ is set to the sample-based estimator $x_j/n$, and otherwise it is left unchanged at ${\tau }_{\ell }$.

Using this ${\theta }_j$, if

$$\sum _{j=1}^m{F}_{\mathrm{B}\mathrm{i}\mathrm{n}}\left(x_{\left(1\right)};n,{\theta }_j\right)\le {\alpha }_1-{\tilde{\alpha }}_1,$$

we reject ${\tilde{H}}_{\left(1\right),0}$ and move to the next joint null ${\tilde{H}}_{\left(2\right),0}\cap \cdots \cap {\tilde{H}}_{\left(m\right),0}$ that excludes ${\tilde{H}}_{\left(1\right),0}$; otherwise, we stop and declare that none of the hypotheses should be rejected. In general, for testing the joint null ${\tilde{H}}_{\left(i\right),0}\cap \cdots \cap {\tilde{H}}_{\left(m\right),0}$ for any $i$, we use $x_{\left(i\right)}$ as the test statistic. If

$$\sum _{j=i}^m{F}_{\mathrm{Bin}}\left(x_{\left(i\right)};n,{\theta }_j\right)\le {\alpha }_1-{\tilde{\alpha }}_1,$$ (9)

we reject ${\tilde{H}}_{\left(i\right),0}$ and move to the next joint null that excludes ${\tilde{H}}_{\left(i\right),0}$; otherwise, we stop and accept the remaining hypotheses. This step-wise procedure asymptotically controls the FWER in testing (3) at $\left({\alpha }_1-{\tilde{\alpha }}_1\right)$.^15,16 Whenever ${\tilde{H}}_{\left(i\right),0}$ is rejected, the corresponding original hypothesis $H_{\left(i\right),0}$ is rejected. This procedure is summarized in Step 2 of Algorithm 1.

The step-wise procedure here can be understood as a combination of Holm’s test and Hansen’s adjustment. We can see that, without Hansen’s adjustment, this procedure reduces to Holm’s test. Specifically, the left hand side of (9), with ${\theta }_j$ replaced by ${\tau }_{\ell }$, becomes $(m-i+1)p_{\left(i\right)}$, where $p_{\left(i\right)}=F_{\text{Bin}}\left(x_{\left(i\right)};n,{\tau }_{\ell }\right)$ is the ith smallest p-value among all p-values for testing ${\tilde{H}}_{\left(i\right),0}$. Thus, (9) becomes $p_{\left(i\right)}\le \left({\alpha }_1-{\tilde{\alpha }}_1\right)/(m-i+1)$, which is Holm’s test. By adjusting for null hypotheses that are “deep in the null”, we have $F_{\mathrm{B}\mathrm{i}\mathrm{n}}\left(x_{\left(i\right)};n,{\theta }_j\right)\approx 0$ for very large ${\theta }_j$ s and (9) becomes approximately $p_{\left(i\right)}\le \left({\alpha }_1-{\tilde{\alpha }}_1\right)/m_0$, where $m_0$ is a number that is smaller than $m-i+1$. Therefore, our procedure gains more power than Holm’s test.

2.4 |. A procedure that accepts original hypotheses while controlling the type-II MCER

We can readily modify the procedure in Sections 2.1,2.2, and 2.3 for testing a set of hypotheses that flip the null and alternative hypotheses $H_{i,0}^{\mathrm{*}}$ and $H_{i,1}^{\mathrm{*}}$ in (2). Thus, a rejection of the new null hypothesis, which is now $H_{i,1}^{\mathrm{*}}$, leads to an acceptance of the original hypothesis $H_{i,0}$. As with the previous procedure, we divide the problem into two sub-problems, the first of which is to construct an at-least- $\left(1-{\tilde{\alpha }}_2\right)$-level one-sided CI $\left[0,{\tau }_u\right]$ for ${\tau }^{\mathrm{*}}$, where $0<{\tilde{\alpha }}_2<{\alpha }_2$, and the second of which is to test the hypotheses that flip the null and alternative hypotheses ${\tilde{H}}_{i,0}$ and ${\tilde{H}}_{i,1}$ in (3) as well as replace ${\tau }_{\ell }$ by ${\tau }_u$. The details of this procedure are deferred to Appendix and are summarized in Algorithm 2, which can also be found in that section. One important feature of this algorithm is that ${\tau }_u$ is found by using a negative value for $c$ in (4), corresponding to the curve $F_{-c}(t)$ in Figure 1, guaranteeing that ${\tau }_{\ell }<{\tau }_u$, a result we require in the next subsection.

2.5 |. Combining results of the two procedures

If $H_{i,0}$ is rejected by the first procedure and non-accepted by the second procedure, it is determined as “rejected”; if $H_{i,0}$ is accepted by the second procedure and non-rejected by the first procedure, it is determined as “accepted”; if $H_{i,0}$ is non-rejected and non-accepted, it is determined as “undecided”. Note that our method precludes the possibility that $H_{i,0}$ is both rejected and accepted, as this would correspond to simultaneously concluding that both $p_i^{\mathrm{*}}<{\tau }_{\ell }$ and ${\tau }_u<p_i^{\mathrm{*}}$ hold simultaneously, which is not possible since ${\tau }_{\ell }<{\tau }_u$. Finally, the overall MCER among the decisions that are either “rejected” or “accepted” (excluding those “undecided”) is controlled at the level that is the sum of the type-I and type-II MCER. The whole workflow of MERIT is depicted in Figure 2.

FIGURE 2.

MERIT workflow.

2.6 |. Modifying the GH method for a fixed MC sample size

Because the GH method is fully sequential while MERIT uses a pre-determined MC sample size, it is difficult to compare them directly. For this reason, we created a modified GH method with a fixed stopping rule. We also replaced each two-sided Robbins-Lai interval in GH by two one-sided Wilson intervals¹⁷ because the Robbins-Lai interval cannot be one-sided and because the Wilson interval is more efficient.¹⁸ In Supplementary Materials S2, we illustrate, using simulated data, that the two-sided Wilson interval (which is the intersection of the two one-sided Wilson intervals) is always narrower than the two-sided Robbins-Lai interval for 0 ≤ p ≤ 0.2, the interesting range of p-values. To complete our modification of GH, we applied the BH procedure to the upper limits of the Wilson intervals to obtain rejected hypotheses and to the lower limits to obtain accepted hypotheses as in the original GH procedure. Similar arguments to those found in Gandy and Hahn^6,7 can be used to prove our modified GH procedure controls the type-I and type-II MCER.

3 |. SIMULATION STUDIES

3.1 |. Setup

We conducted extensive simulation studies to evaluate the performance of our method. We evaluated the two procedures of our method (one rejecting hypotheses and one accepting hypotheses). We also compared MERIT to our modified GH method. In addition, we evaluated the naive approach that applies the BH procedure to the MC p-values, for which we evaluated the type-I and type-II MCER separately. We refer to our method, the modified GH method, and the naive method as MERIT, GH, and NAIVE, respectively.

We considered $m=100,1000$, and 5000 tests. For the bulk of our simulations, we assumed that 80% of $m$ tests are under the null and the remaining tests are under the alternative; we sampled the p-values for tests under the null independently from the uniform distribution $\mathrm{U}\left[\mathrm{0,1}\right]$, and sampled the p-values for tests under the alternative independently from the Gaussian right-tailed probability model: $p_i^{\mathrm{*}}=1-\mathrm{\Phi }\left(Z_i\right)$, where $Z_i~\mathrm{N}(\beta ,1)$ and $\mathrm{\Phi }$ is the standard normal cumulative distribution function. We set $\beta$ to 1.5, 2 and 2.5; a larger value of $\beta$ implies higher sensitivity to reject the alternative hypotheses. For each pair of values of $m$ and $\beta$, we sampled 100 different sets of ideal p-values $\left(p_1^{\mathrm{*}},p_2^{\mathrm{*}},\dots ,p_m^{\mathrm{*}}\right)$. In addition to these simulations, we also considered a scenario when 80% tests are under the alternative and a worst-case scenario when all “significant” ideal p-values (defined to be p-values that would be rejected by the BH procedure) are close to the threshold of significance. The details of these last two settings, along with their results, are provided in Supplementary Materials S3 and S4, respectively. Instead of generating actual data and calculating test statistics, we directly simulated the matrix $I$ by independently drawing Bernoulli samples with the “success” rate $p_i^{\mathrm{*}}$. The MC sample size $n$ ranged between 5,000 and 1,000,000. The nominal type-I and type-II MCER ${\alpha }_1$ and ${\alpha }_2$ were both set to 10%; the nominal FDR $\varphi$ used to define the BH threshold was set to 10%.

For evaluating the rejecting procedure in each method, we use two metrics, the empirical type-I MCER and sensitivity, the latter of which is

$$\text{Sensitivity}=\frac{\sum _{i=1}^m\mathbb{I}({\widehat{D}}_i=\text{rejected,}{D}_i^{\mathrm{*}}=\text{rejected})}{1\vee \sum _{i=1}^m\mathbb{I}(D_i^{\mathrm{*}}=\text{rejected})}.$$

For evaluating the accepting procedure in each method, we also use two metrics, the empirical type-II MCER and specificity, given by

$$\text{Specificity}=\frac{\sum _{i=1}^m\mathbb{I}({\widehat{D}}_i=\text{accepted,}{D}_i^{\mathrm{*}}=\text{accepted})}{1\vee \sum _{i=1}^m\mathbb{I}(D_i^{\mathrm{*}}=\text{accepted})}.$$

For each set of ideal p-values and MC sample size $n$, we evaluated each of these metrics using 1,000 replicates of the sampled matrix $I$.

3.2 |. Simulation results

Figure 3 shows the type-I MCER and sensitivity for the three methods, MERIT, GH, and NAIVE, for testing $m=1000$ hypotheses. Each box displays the variation of results of a method over the 100 sets of ideal p-values: the top and bottom lines represent the 75% and 25% quantiles and the middle bar indicates the median. Each point in each box represents results from 1,000 replicates. We can see that, for any combination of $n$ and $\beta$, both MERIT and GH controlled the type-I MCER below the nominal level, while NAIVE did not. In fact, most of the empirical type-I MCER from MERIT and GH are zero, which is expected because both methods guard against the worse case when all “significant” p-values are concentrated at the threshold of significance, while the p-values in these simulations were spread out. Nevertheless, MERIT always had higher sensitivity than GH, which is more pronounced when $n$ is not very large. In fact, MERIT rejected more hypotheses than GH for every set of ideal p-values (results not shown). As expected, the type-I MCER of NAIVE was highly inflated when $n$ was low; although the error rate decreased as $n$ was increased, it was still above the nominal level for many sets of ideal p-values even when $n$ was as large as one million. One reason that the sensitivity of NAIVE was so high is because it did not allow “undecided” tests, so that every test was categorized either as “rejected” or “accepted”. The results of type-II MCER and specificity are displayed in Figure 4, which shows similar patterns as Figure 3. Note that the specificity remained relatively unchanged for different values of $\beta$ because specificity pertains to null hypotheses only. The proportions of undecided tests are shown in Figure S5, which demonstrates that MERIT consistently yielded a lower proportion of undecided tests compared to GH in all scenarios.

FIGURE 3.

The empirical type-I MCER (upper panel) and sensitivity (lower panel) for $m=1000$ tests. The gray dashed lines represent the nominal level 10%. Each point represents a different set of ideal p-values; results for each set of p-values are based on 1,000 replicates.

FIGURE 4.

The empirical type-II MCER (upper panel) and specificity (lower panel) for $m=1000$ tests.

The results for $m=100$ are shown in Figures S6 and S7. With this small number of tests, NAIVE was more likely to control the type-I and type-II MCER because the sampled ideal p-values were less likely to be near the threshold of significance. For the same reason, MERIT had higher sensitivity and specificity than in the $m=1000$ case (for the same values of $n$ and $\beta$). Again, MERIT and GH both controlled the type-I and type-II MCER and MERIT always had higher sensitivity and specificity than GH. However, NAIVE still lost control of MCER for some samples of 100 p-values, even with 10⁶ MC replicates. Note that this behavior with changing $m$ is because our ideal p-values were assumed to be fairly uniformly spread. If we had assumed a worst-case scenario in which most, or all, “significant” ideal p-values were at or near the threshold, we would not expect to see this improved behavior when m is decreased.

The results for $m=5000$ are shown in Figures S8 and S9. This scale of testing is more commonly seen in modern biological and biomedical studies of omics. Indeed, the prostate cancer data that we will apply our method to has 6033 tests corresponding to 6033 genes. In this case, NAIVE had highly inflated type-I and type-II MCER even with a million MC replicates, implying that the naive results had very poor reproducibility. Again, MERIT and GH both controlled the type-I and type-II MCER and MERIT always had higher sensitivity and specificity than GH. Interestingly, while the sensitivity of GH reduced from $m=1000$ to $m=5000$, the sensitivity of MERIT stayed quite robust, which suggests an increasing advantage of MERIT over GH for very large numbers of tests.

4 |. APPLICATION TO THE PROSTATE CANCER STUDY

We applied MERIT, GH, and NAIVE to detect differentially expressed genes in a prostate cancer study.¹⁹ The data contains microarray gene expressions for 6,033 genes and 102 subjects comprised of 52 prostate cancer patients and 50 healthy controls. We calculated the t-statistic (assuming equal variance) for each gene based on the observed data and $n$ MC replicates (by permuting the case-control labels), from which we obtained the matrix $I$. As in the simulation studies, we considered a wide range of values for $n$ and we set the nominal type-I and type-II MCER to 10% and the nominal FDR to 10%. For evaluation of reproducibility, we obtained the results for 10 different runs with different seeds for generating MC replicates.

The results are shown in Figure 5. In all cases, MERIT yielded more rejected tests (when there were any rejections) and more accepted tests than GH; the advantage of MERIT is more clear when $n$ is between 50,000 and 500,000. Interestingly, the relative numbers of rejections and acceptances of the three methods seem to match well with the sensitivity and specificity in the simulation study with $m=5000$ and $\beta =1.5$. To interpret the results in more detail, we focus on the first run of $n=100,000$ MC replicates. MERIT detected 51 genes as differentially expressed and 5961 genes as non-differentially expressed, and left 21 gene “undecided”; in the same run, GH detected 0 gene as differentially expressed and 5954 as non-differentially expressed, and left 79 genes “undecided”. The chance for a single gene or more to be declared as differentially expressed by MERIT (or GH) but declared as non-differentially expressed with an infinite number of MC replicates is less than 10%; we have a similar guarantee for the genes determined as non-differentially expressed by MERIT (or GH).

FIGURE 5.

Results from analysis of the prostate cancer data. The upper panel shows the number of rejected hypotheses and the lower panel shows the last two digits (the first two digits are 59; eg, the last two digits 35 correspond to the complete number 5935) of the number of accepted hypotheses. For $n=5,000$ and 10,000, both MERIT and GH rejected no hypotheses.

Because MERIT includes a bootstrap step, MERIT will require more run time than NAIVE. For the prostate cancer data, MERIT required ~20 min (not including the generation of MC replicates and the calculation of t statistic) on a single core for one run of n = 1,000,000. Because the bootstrap procedure in Step 1 is the most computationally intensive part in our algorithm, the run time can be significantly reduced by parallelizing the computation of bootstrap replicates to multiple cores. For example, one run of MERIT for the prostate cancer data with $n=1,000,000$ required only ~2 min on 10 cores.

5 |. DISCUSSION

Monte-Carlo error is a threat to reproducibility in two related ways, when testing hypotheses using resampling techniques such as permutation or bootstrap. First, MC error may lead to false rejection of a hypothesis that we would not reject given the ideal p-value. Second, if we re-run an MC hypothesis testing procedure, we may obtain a different list of rejected or accepted hypotheses. Note that when the outcome “undecided” is available, these two types of MC error are distinct. The control of MCER guaranteed by MERIT means that, even if two runs give different lists of rejected or accepted hypotheses, we are assured that the chance of reaching the wrong conclusion about any hypothesis is bounded by level $\alpha$. Thus, while successive runs of MERIT may have different lists of rejected or accepted hypotheses, these lists will only differ by switches between “rejected” and “undecided”, or between “accepted” and “undecided”, with probabilities at most ${\alpha }_1$ and ${\alpha }_2$, respectively. The situation is very different for the NAIVE procedure where the BH threshold is applied directly to the MC p-values. Here the absence of an “undecided” category means that all errors are switches between “rejected” and “accepted”, even though the overall (experiment-level) false discovery rate may be controlled. Many of these false discoveries will be due to MC error alone.

MERIT (as well as our modification of GH) differ from the original GH proposal in that type-I and type-II MCER can be separately controlled. This may be useful for some users who care more about discoveries (ie, rejections) than they care about lists of hypotheses for which the null hypothesis is accepted. For these users, type-I MCER may be the only error worth controlling. In this situation, we may set the type-II MCER to ${\alpha }_2=0$ so that we only obtain two possible outcomes: “rejected” or “undecided”. Alternatively, a small type-II MCER ${\alpha }_2$ can be selected so that hypotheses that are very far from the threshold of significance can still be labeled as “accepted”.

MERIT achieves control of type-I MCER at level ${\alpha }_1$ by lowering the BH threshold for rejecting hypotheses. We note that we could address the chance of switches between “rejected” and “undecided” hypotheses by lowering this threshold even further. If we lowered the threshold such that the probability of observing any switches between “rejected” and “undecided” was bounded by ${\alpha }_3$, then we could control the probability of observing any irreproducible rejections by ${\alpha }_1+{\alpha }_3$. This control would, however, come at the cost that some hypotheses that can be rejected by controlling type-I MCER at level ${\alpha }_1$ will no longer be rejected by the new criterion. Still, it may be worth considering. Finally, the number of ideal p-values near the threshold of significance controls the number of “undecided” hypotheses in MERIT, and the distance between the MC p-value and this threshold presumably controls the likelihood of a switch between “rejected” and “undecided”. In our experience, when there are a large number of “undecided” hypotheses, changing the nominal FDR level $\varphi$ to a value where there are relatively few empirical p-values can sometimes reduce the number of “undecided” hypotheses.

When there are “undecided” hypotheses, it is tempting to run the MERIT algorithm several times using the same number of replicates, in the hopes of drawing some inference about which hypotheses switch between undecided and decided (eg, rejected). Unfortunately, there does not seem to be a simple way to combine the information in several MERIT runs except using MERIT on the pooled set of replicates. Combining the results in any other way is tantamount to developing a sequential version of MERIT. While developing a sequential version of MERIT seems worthwhile, it may be difficult. One approach might be to consider conditional inference, in which we condition on an “undecided” outcome at the previous step. However, our bootstrap procedure for the second step (see Equation 8) would need to be modified to account for this conditioning. Alternatively, we could consider an alpha spending approach,²⁰ which is commonly used to address multiple decision endpoints in sequential experiments. However, our two-step approach is non-standard and would require custom decision boundaries. Moreover, this approach may not be ideal for practical applications as we may find that simply using the maximum number of MC replicates we can afford gives a higher detection sensitivity and a lower number of undecided tests.

When developing MERIT, we chose to control the family-wise MCER, rather than a percentage or rate of MC errors among all decided tests, since this MC “false discovery rate” may be misleading. The chance of an MC error for hypotheses with ideal p-values well below the BH threshold is typically fairly small; the presence of these “easily rejected” hypotheses in a false-discovery-like list of MC errors can therefore lead to fairly loose control of MC error for hypotheses that are closer to the BH threshold, even though the overall error rate is controlled. In contrast, controlling the family-wise MCER bounds the probability of any MC error, which seems to be more appropriate in the current context.

Here we have considered only MC schemes that have a fixed stopping criterion. We feel this is reasonable, as our impression is that most large-scale MC hypothesis testing reported in the biological sciences used a fixed MC sample size. This also enables us to achieve higher efficiency, as it allows for a fairly sophisticated correction for multiple testing to be applied only once, after all tests have been completed. For fairness, we have also modified the original, sequential GH approach to allow for a more efficient approach when a fixed stopping criterion is used. Even with this boost, we showed that MERIT outperformed GH unless the MC sample size was 1,000,000 or greater, after which performance of the two methods was similar. In this sense, our method is the best available approach to MC hypothesis testing that features control of MCER with a fixed stopping criterion. We have implemented our method in the R package MERIT available on GitHub at https://github.com/yijuanhu/MERIT in formats appropriate for Macintosh or Windows.

APPENDIX A. A PROCEDURE THAT ACCEPTS ORIGINAL HYPOTHESES WHILE CONTROLLING THE TYPE-II MCER

We consider the following hypotheses that “flip” the hypotheses in (2):

$$H_{i,0}^{\mathrm{*}\prime }:p_i^{\mathrm{*}}\le {\tau }^{\mathrm{*}}\text{vs.}{H}_{i,1}^{{\mathrm{*}}^{\prime }}:p_i^{\mathrm{*}}>{\tau }^{\mathrm{*}},i=1,2,\dots ,m.$$

Note that a rejection of $H_{i,0}^{\mathrm{*}\prime }$ corresponds to an acceptance of the original hypothesis $H_{i,0}$. Recall that a type-II MC error occurs when $H_{i,0}^{\mathrm{*}\prime }$ is true (hence $D_i^{\mathrm{*}}=\mathrm{r}\mathrm{e}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}$) but rejected using the MC p-value (hence ${\widehat{D}}_i=\mathrm{a}\mathrm{c}\mathrm{c}\mathrm{e}\mathrm{p}\mathrm{t}\mathrm{e}\mathrm{d}$). Then, a test procedure based on MC p-values that controls the FWER for testing $H_{i,0}^{\mathrm{*}\prime }$ at ${\alpha }_2$ would control the type-II MCER at ${\alpha }_2$. Here, we develop a two-step procedure following the same idea as in Sections 2.1,2.2, and 2.3. In the first step, we construct an at least $\left(1-{\tilde{\alpha }}_2\right.$)-level one-sided CI $\left(0,{\tau }_u\right]$ for ${\tau }^{\mathrm{*}}$, where $0<{\tilde{\alpha }}_2<{\alpha }_2$. In the second step, we consider the revised hypotheses, treating ${\tau }_u$ as fixed:

$${\tilde{H}}_{i,0}^{\prime }:p_i^{\mathrm{*}}\le {\tau }_u\text{vs.}{\tilde{H}}_{i,1}^{\prime }:p_i^{\mathrm{*}}>{\tau }_u,i=1,2,\dots ,m.$$ (A1)

We develop a test procedure that tests (A1) while controlling the FWER at level $\left({\alpha }_2-{\tilde{\alpha }}_2\right)$. Then, by the relationship that

$$\begin{array}{l}\left\{V^{\mathrm{I}\mathrm{I}}\ge 1\right\}=\left\{{\tau }_u<{\tau }^{\mathrm{*}},V^{\mathrm{I}\mathrm{I}}\ge 1\right\}\cup \{{\tau }_u\ge {\tau }^{\mathrm{*}},V^{\mathrm{I}\mathrm{I}}\ge 1\}\\ \subset \left\{{\tau }_u<{\tau }^{\mathrm{*}}\right\}\cup \{{\tau }_u\ge {\tau }^{\mathrm{*}},\text{at}\text{least}\text{one}{H}_{i,0}^{{\mathrm{*}}^{\prime }}\text{is}\text{true}\text{but}\text{rejected}\}\\ \subset \{{\tau }_u<{\tau }^{\mathrm{*}}\}\cup \{{\tau }_u\ge {\tau }^{\mathrm{*}},\text{at}\text{least}\text{one}{\tilde{H}}_{i,0}^{\prime }\text{is}\text{true}\text{but}\text{rejected}\},\end{array}$$

the type-II MCER $\mathrm{P}\mathrm{r}\left(V^{\mathrm{I}\mathrm{I}}\ge 1\right)$ is always bounded by the sum of $\mathrm{P}\mathrm{r}\left({\tau }_u<{\tau }^{\mathrm{*}}\right)$ and the FWER in testing (A1), which is ${\alpha }_2$. As the default choice, we choose ${\tilde{\alpha }}_2={\alpha }_2/2$.

Step 1: construct a $\left(1-{\tilde{\alpha }}_2\right)$-level one-sided $\mathrm{C}\mathrm{I}\left[0,{\tau }_u\right]$ for ${\tau }^{\mathrm{*}}$

We find ${\tau }_u$ from a set of p-values that are “upper limits” for the ideal p-values. To this aim, we consider the following sets of p-values, with each set indexed by a positive continuous value $c$:

$$p_{-c,i}=\frac{X_i+1-c\sqrt{X_i+1}}{n+1-c\sqrt{X_i+1}}.$$ (A2)

Note that we impose the restriction that $c<\sqrt{n+1}$; otherwise $n+1-c\sqrt{X_i+1}$ may become 0 or even negative and then $p_{-c,i}$ is not well-defined. However, negative values of the numerator are allowed, as illustrated in Figure 1 where the curve $F_{-c}(t)$ has non-zero value at $t=0$. As $c$ goes to $0,p_{-c,i}$ converges to $p_i^{\mathrm{*}}$. When $c$ is sufficiently large, $p_{-c,i}$ is asymptotically smaller than $p_i^{\mathrm{*}}$, as least for small $p_i^{\mathrm{*}}\mathrm{s}$.

Let $F_{-c}(t)$ be the EDF of $\left(p_{-c,1},\dots ,p_{-c,m}\right)$ and ${\tau }_{-c}$ be the BH cutoff. If $F_{-c}(t)$ is always no less than $F^{\mathrm{*}}(t)$ over $(0,\varphi ]$, as shown in Figure 1 (left), we must have ${\tau }_{-c}\ge {\tau }^{\mathrm{*}}$. It follows that

$$\mathrm{Pr}\left({\tau }_{-c}\ge {\tau }^{\mathrm{*}}\right)\ge \mathrm{Pr}(F^{\mathrm{*}}\prec F_{-c}).$$

To construct a $\left(1-{\tilde{\alpha }}_2\right)$-level one-sided $\mathrm{C}\mathrm{I}\left(0,{\tau }_u\right]$ for ${\tau }^{\mathrm{*}}$, we are interested in $c$ that satisfies

$$\mathrm{P}\mathrm{r}(F^{\mathrm{*}}\prec F_{-c})\ge 1-{\tilde{\alpha }}_2.$$ (A3)

The probability on the left of (A3) increases as $c$ increases (because the decrease of $p_{-c,i}$ leads to the increase of $F_{-c}(t)$ at a given $t$). Thus we wish to find the smallest $c$ that satisfies (A3), denoted by $c_u$. Finally, we set the upper limit ${\tau }_u$ to ${\tau }_{c_u}$.

We obtain $c_u$ using the same bootstrap replicates as in Section 2.2. Define the bootstrap p-values indexed by $c$ as

$$p_{-c,i}^{\left(b\right)}=\frac{X_i^{\left(b\right)}+1-c\sqrt{X_i^{\left(b\right)}+1}}{n+1-c\sqrt{X_i^{\left(b\right)}+1}},i=1,2,\dots ,m.$$

Let $F_{-c}^{\left(b\right)}$ be the EDF for $(p_{-c,1}^{\left(b\right)},\dots ,p_{-c,m}^{\left(b\right)})$. We wish to find $c_u$ to be the smallest $c$ that satisfies

$$B^{-1}\sum _{b=1}^B\mathbb{I}\left(\widehat{F}\prec F_{-c}^{\left(b\right)}\right)\ge 1-{\tilde{\alpha }}_2,$$

Algorithm 2.

The procedure that accepts original hypotheses while controlling type-II MCER

Input: matrix $I$, nominal FDR $\varphi$, nominal type-II MCER ${\alpha }_2$

Step 1: construct a $\left(1-{\tilde{\alpha }}_2\right)$-level one-sided $\mathrm{C}\mathrm{I}\left[0,{\tau }_u\right]$ for ${\tau }^{\mathrm{*}}$.

Calculate $(X_1,X_2,\dots ,X_m)$, $({\hat{p}}_1,{\hat{p}}_2,\dots ,{\hat{p}}_m)$, and then $\widehat{F}$.

For each bootstrap replicate $b=1,2,\dots ,B$,

Sample $n$ columns from $I$ with replacement to form $I^{\left(b\right)}$.

Calculate $(X_1^{\left(b\right)},X_2^{\left(b\right)},\dots ,X_m^{\left(b\right)})$, $(p_{-c,1}^{\left(b\right)},p_{-c,2}^{\left(b\right)},\dots ,p_{-c,m}^{\left(b\right)})$, and then $F_{-c}^{\left(b\right)}$.

Find $c_u^{\left(b\right)}$ to be the smallest $c$ that guarantees $\widehat{F}\prec F_{-c}^{\left(b\right)}$.

Find $c_u$, the smallest $c$ that satisfies (A3), to be the $\left(1-{\tilde{\alpha }}_2\right)$ quantile of $(c_u^{\left(1\right)},c_u^{\left(2\right)},\dots ,c_u^{\left(B\right)})$.

Find the upper limit ${\tau }_u$ to be the BH cutoff of p-values in (A2) indexed by $c_u$.

Step 2: Test hypotheses in (A1) given ${\tau }_u$ while controlling the FWER at $\left({\alpha }_2-{\tilde{\alpha }}_2\right)$.

Set $i=m$.

While $i\ge 1$,

If (A4) does not hold, accept all remaining hypotheses ${\tilde{H}}_{\left(i\right),0}^{\prime }\mathrm{s}$ and exit the loop.

Otherwise, reject ${\tilde{H}}_{\left(i\right),0}^{\prime }$, update $i=i-1$.

Whenever ${\tilde{H}}_{\left(i\right),0}^{\prime }$ is rejected, $H_{\left(i\right),0}$ is accepted.

Output: a decision for every hypothesis $H_{i,0}$ between “accepted” and “non-accepted”.

which is the empirical version of (A3). To this aim, we find, for every $b=1,\dots ,B$, the smallest $c$ that guarantees $\widehat{F}\prec F_{-c}^{\left(b\right)}$, denoted by $c_u^{\left(b\right)}$, and then set $c_u$ to be the $\left(1-{\tilde{\alpha }}_2\right)$ th quantile of $(c_u^{\left(1\right)},c_u^{\left(2\right)},\dots ,c_u^{\left(B\right)})$. The procedure here is summarized in Step 1 of Algorithm 2.

Step 2: test hypotheses in (A1) given ${\tau }_u$ while controlling the FWER at (${\alpha }_2-{\tilde{\alpha }}_2$)

Let ${\tilde{H}}_{\left(1\right),0}^{\prime },{\tilde{H}}_{\left(2\right),0}^{\prime },\dots ,{\tilde{H}}_{\left(m\right),0}^{\prime }$ be the ordered hypotheses (A1) that correspond to the ordered observed test statistics $x_{\left(1\right)},x_{\left(2\right)},\dots ,x_{\left(m\right)}$. We start by testing the global null ${\tilde{H}}_{\left(1\right),0}^{\prime }\cap {\tilde{H}}_{\left(2\right),0}^{\prime }\cap \cdots \cap {\tilde{H}}_{\left(m\right),0}^{\prime }$, which means that all $p_i^{\mathrm{*}}$ are less than ${\tau }_u$. We use $x_{\left(m\right)}$ as the test statistic for this test. Following Romano and Wolf, we propose to calculate the p-value under the global null:

$$\mathrm{Pr}\left(\underset{1\le j\le m}{\mathrm{m}\mathrm{a}\mathrm{x}}{X}_j\ge x_{\left(m\right)}\right),$$

where each $X_j$ is a binomial random variable with $n$ trials and “success” rate ${\theta }_j^{\prime }$ that is under the null. Using the Bonferroni inequality, we obtain an upper bound of the p-value to be

$$\sum _{j=1}^m{F}_{\mathrm{B}\mathrm{i}\mathrm{n}}^{\prime }\left(x_{\left(m\right)};n,{\theta }_j^{\prime }\right),$$

where $F_{\text{Bin}}^{\prime }(x;n,\theta )$ denotes $\mathrm{P}\mathrm{r}(X\ge x)$ for $X~\mathrm{B}\mathrm{i}\mathrm{n}(n,\theta )$. Using Hansen’s adjustments for null hypotheses that are “deep in the null”, we choose ${\theta }_j^{\prime }$ to be

$${\theta }_j^{\prime }=\left\{\begin{array}{ll}{\tau }_u,& \text{if}\sqrt{n}\left(x_j/n-{\tau }_u\right)/\sqrt{{\tau }_u\left(1-{\tau }_u\right)}\ge -\sqrt{2\log \log n,}\\ x_j/n,& \text{otherwise.}\end{array}\right.$$

Using this ${\theta }_j^{\prime }$, if

$$\sum _{j=1}^m{F}_{\mathrm{B}\mathrm{i}\mathrm{n}}^{\prime }\left(x_{\left(m\right)};n,{\theta }_j^{\prime }\right)\le {\alpha }_2-{\tilde{\alpha }}_2,$$

we reject ${\tilde{H}}_{\left(m\right),0}^{\prime }$ and move to the next joint null ${\tilde{H}}_{\left(1\right),0}^{\prime }\cap \cdots \cap {\tilde{H}}_{(m-1),0}^{\prime }$ that excludes ${\tilde{H}}_{\left(m\right),0}^{\prime }$; otherwise, we stop and declare that none of the hypotheses should be rejected. In general, for testing the joint null ${\tilde{H}}_{\left(1\right),0}^{\prime }\cap \cdots \cap {\tilde{H}}_{\left(i\right),0}^{\prime }$ for any $i$, we use $x_{\left(i\right)}$ as the test statistic. If

$$\sum _{j=1}^i{F}_{\mathrm{Bin}}^{\text{'}}(x_{\left(i\right)};n,{\theta }_j^{\text{'}})\le {\alpha }_2-{\tilde{\alpha }}_2,$$ (A4)

we reject ${\tilde{H}}_{\left(i\right),0}^{\prime }$ and move to the next joint null that excludes ${\tilde{H}}_{\left(i\right),0}^{\prime }$; otherwise, we stop and accept the remaining hypotheses. This step-wise procedure asymptotically controls the FWER in testing (A1) at $\left({\alpha }_2-{\tilde{\alpha }}_2\right)$. Whenever ${\tilde{H}}_{\left(i\right),0}^{\prime }$ is rejected, the corresponding original hypothesis $H_{\left(i\right),0}$ is accepted. This procedure is summarized in Step 2 of Algorithm 2.

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are openly available in the R package sda in the repository CRAN.