Abstract
Multiple testing has been a prominent topic in statistical research. Despite extensive work in this area, controlling false discoveries remains a challenging task, especially when the test statistics exhibit dependence. Various methods have been proposed to estimate the false discovery proportion (FDP) under arbitrary dependencies among the test statistics. One key approach is to transform arbitrary dependence into weak dependence and subsequently establish the strong consistency of FDP and false discovery rate under weak dependence. As a result, FDPs converge to the same asymptotic limit within the framework of weak dependence. However, we have observed that the asymptotic variance of FDP can be significantly influenced by the dependence structure of the test statistics, even when they exhibit only weak dependence. Quantifying this variability is of great practical importance, as it serves as an indicator of the quality of FDP estimation from the data. To the best of our knowledge, there is limited research on this aspect in the literature. In this paper, we aim to fill in this gap by quantifying the variation of FDP, assuming that the test statistics exhibit weak dependence and follow normal distributions. We begin by deriving the asymptotic expansion of the FDP and subsequently investigate how the asymptotic variance of the FDP is influenced by different dependence structures. Based on the insights gained from this study, we recommend that in multiple testing procedures utilizing FDP, reporting both the mean and variance estimates of FDP can provide a more comprehensive assessment of the study’s outcomes.
Keywords: asymptotic expansion, asymptotic variance, multiple testing, weak dependence
1. INTRODUCTION
Multiple hypothesis testing is a widely explored subject in statistical research, with applications in diverse scientific domains such as biology, medicine, genetics, neuroscience, and finance. In the context of multiple testing, we confront the challenge of simultaneously assessing p hypotheses. Within this setting, we use the notation 
to represent the number of true null hypotheses, and 
to denote the number of false null hypotheses. The outcomes of these tests can be effectively summarized using Table 1, which provides a concise representation of the various possibilities.
TABLE 1.
Summary of potential outcomes from multiple hypothesis testing.
| Not rejected | Rejected | Total | |
|---|---|---|---|
| True null | U | V |
|
| False null | T | S |
|
| Total |
|
R | p |
Here, V represents the number of hypotheses that are incorrectly rejected, indicating the number of type I errors made, while R denotes the total number of hypotheses rejected.
Traditional approaches aim to maintain control over the family-wise error rate (FWER): 
(Bonferroni, 1936; Šidák, 1967; Holm, 1979; Simes, 1986; Holland and Copenhaver, 1987; Hochberg, 1988; Rom, 1990); alternatively, one can control the generalized family-wise error rate (gFWER): 
, where 
, which provides a broader perspective (Dudoit et al., 2004; Pollard and van der Laan, 2004; Lehmann and Romano, 2005). These conventional methods aim to guarantee the probability of making zero or a limited number of false positives; therefore, they tend to be conservative, especially when dealing with a large number of hypotheses. Testing a large number of hypotheses simultaneously is common in practice; for instance, in genome-wide association studies, researchers assess hundreds of thousands or even millions of single nucleotide polymorphisms (SNPs) for each subject in relation to their disease status. This approach results in a substantial number of hypothesis tests.
In scenarios where p becomes large, the concept of false discovery proportion (FDP) and its statistical properties have attracted significant attention. Particularly, false discovery rate (FDR) has emerged as a widely studied criterion, initially introduced by Benjamini and Hochberg (1995) and recognized for its practicality. Specifically, it is defined to be
 |
where 
if 
. Extensive research on FDR can be found in works by Benjamini and Hochberg (1995), Benjamini et al. (2001), Storey (2002), Storey and Tibshirani (2003), Storey et al. (2004), Ferreira and Zwinderman (2006), Clarke et al. (2009), and the references therein. It is worth noting that most of these methods are grounded in the assumption that the test statistics are either independent or conform to some restrictive dependence structure.
Given the inherent complexity of practical data, test statistics derived from real-world scenarios often exhibit diverse and arbitrary dependence structures. The impact of such dependence on FDR has been a subject of extensive study, explored by Benjamini et al. (2001), Finner and Roters (2002), Owen (2005), Sarkar (2006), Efron (2007), and others. More recently, novel methods have been developed to accommodate and estimate FDR in the presence of complex and general dependence structures or even to integrate these dependencies into the testing process. These innovative approaches include the local index of significance testing procedure with the dependence modeled by a hidden Markov model (Sun and Cai, 2009), pointwise and clusterwise analyses designed to address spatial dependence (Sun et al., 2015), and the principal factor approximation (PFA) method suitable for handling arbitrary dependence structures (Fan et al., 2012, 2019; Fan and Han, 2017). Notably, Fan et al. (2012) delved into the study of FDP and FDR under arbitrary dependence using a 2-step procedure. Their approach initially introduces a PFA method to transform arbitrary dependence into weak dependence. Subsequently, they establish the theoretical strong consistency of both FDP and FDR. As a result, their method remains valid for test statistics exhibiting arbitrary dependence, while under weak dependence, both FDP and FDR converge to the same asymptotic limit.
In addition to FDR, another commonly used multiple testing criterion is known as false discovery exceedance (FDX), also often referred to as the tail probability for the proportion of false positives (TPPFP) in the literature. FDX quantifies the tail probability that FDP exceeds a specified threshold 
, defined as
 |
Compared to FDR, which provides a single summary value, FDX has the potential to offer a more comprehensive characterization of the uncertainty and distribution of FDP as the threshold value q varies. However, the literature has not fully explored the impact of dependence among test statistics on FDX. Notably, some works on FDX make the assumption of independence among tests (Genovese and Wasserman, 2004; Guo and Romano, 2007; Ge and Li, 2012). Moreover, the majority of studies focus on developing procedures to control FDX as a single-value summary of FDP, rather than investigating the influence of dependence on the variability and distribution of FDP (Korn et al., 2004; van der Laan et al., 2004; Lehmann and Romano, 2005; Genovese and Wasserman, 2006; Delattre and Roquain, 2015; Hemerik et al., 2019; Döhler and Roquain, 2020; Basu et al., 2021). Consequently, these works do not provide the valuable insights into how FDP varies across different studies.
To the best of our knowledge, the majority of the aforementioned studies have concentrated on proposing estimators for FDP and its related statistical characteristics. While this focus is instrumental in ensuring the control of FDP and its associated attributes, it becomes evident that the asymptotic uncertainty of FDP can significantly hinge on the dependence structure inherent in the test statistics. Consequently, there is substantial practical value in quantifying this variability, as it can serve as a crucial indicator of the quality and reliability of FDP estimates, thereby augmenting the research efforts in the fields of FDR and FDX methods. More specifically, the availability of FDP variance estimates equips researchers with the ability to draw more confident conclusions about the effectiveness of FDP control when managing its corresponding mean at a given level. Additionally, it has the potential to contribute to the advancement of methods for estimating and controlling FDX. Despite its considerable potential impact, the exploration of FDP uncertainty remains relatively limited within the research community. Notable examples include the work by Ge and Li (2012), who derived the asymptotic distribution of FDP under independence, and the study by Delattre and Roquain (2011), which addressed Gaussian equi-correlation dependence. Furthermore, Delattre and Roquain (2016) established the asymptotic distribution of FDP under specific assumptions regarding the dependence structure of the test statistics and the equivalence of expected values in the alternatives.
In this paper, our primary objective is to conduct a rigorous theoretical investigation into the asymptotic uncertainty of FDP within the framework where the test statistics are assumed to be normally distributed and exhibit weak dependence. Our analysis encompasses deriving the theoretical expansion of FDP, which emerges as a linear combination of the test results, enabling the computation of its asymptotic variance. Moreover, we explore how this variance evolves across diverse dependence structures among the test statistics, both in theoretical and numerical contexts. An intriguing observation arises from our investigation: FDP based on weakly dependent test statistics converges to the same limit as that grounded in independent test statistics, as demonstrated by Fan et al. (2012); however, what sets them apart is that the asymptotic variance of the former almost consistently exceeds that of the latter, both theoretically and numerically. These findings motivate our recommendation that, in the context of multiple testing conducted via an FDP procedure, it would be beneficial to report both the mean and variance estimates of FDP. Such a practice can substantially enhance the comprehensiveness of study outcomes and facilitate more informed interpretations. To demonstrate practical applications, we analyze a real genome-wide association study (GWAS) dataset, showing how our insights impact real studies.
2. ASYMPTOTIC LIMIT OF FDP UNDER WEAK DEPENDENCE
Suppose our objective is to simultaneously conduct the following p hypothesis tests:
 |
(1) |
For each test j, we have obtained a corresponding test statistic 
. These test statistics collectively follow a multivariate normal distribution, as expressed by
 |
(2) |
where 
represents the variance-covariance matrix for 
.
In practical scenarios, it is common to encounter multiple testing problems within the frameworks defined by Equations 1 and 2, where the variance-covariance matrix 
is typically non-diagonal, indicating interdependence among the test statistics 
. To illustrate, let’s consider a GWAS involving n subjects, where each subject’s disease status and p SNPs are measured. This data can be represented as 
and an 
matrix 
. Here, 
represents the disease status of subject i, and the ith row of 
contains the associated SNPs. When 
values are measured on a continuous scale, we can model the association between the jth SNP and the disease status using the marginal linear regression model:
 |
(3) |
where 
and 
are regression parameters, and 
’s represent random errors. The estimates 
and their corresponding test statistics 
for the hypotheses
 |
(4) |
can be derived using a standard least squares procedure. Proposition 1 in Fan et al. (2012) has verified that the estimates 
are correlated and derived their joint distribution under appropriate regularity conditions. Consequently, the testing problems defined in Equation 4 and the corresponding test statistics 
can be effectively integrated into the multiple testing framework defined by Equations 1 and 2.
Naturally, we can categorize the testing problems in Equation 1 into 
and 
, respectively, being the sets of indices for the true nulls and the true alternatives, and let 
and 
denote their cardinalities. For a given threshold 
, the jth test is given by 
. In this context, 
signifies the rejection of the null hypothesis, while 
indicates otherwise. Here, 
represents the 
th quantile of a standard normal distribution. Define 
, 
, and 
; they are, respectively, the number of false discoveries, correct discoveries, and the total number of discoveries. As a consequence, we have 
.
Fan et al. (2012) studied the estimation of 
under an arbitrary dependence structure of 
. One milestone finding is that if 
follows the weak dependence structure
 |
(5) |
then
 |
(6) |
almost surely, where 
denotes the cumulative distribution function of a standard normal distribution. This indicates that under the aforementioned weak dependence assumption, 
converges to a limit not depending on the covariance structure of the test statistics. It is noteworthy that this limit holds not only for 
, but also for 
.
We observe that this conclusion does not hold for the asymptotic variance of 
; the asymptotic variance of 
varies under different covariance structures among the test statistics even when they are weakly dependent. In this article, we shall derive the asymptotic variance of 
theoretically and study how it varies under different dependence structures of the test statistics numerically.
3. ASYMPTOTIC VARIANCE OF FDP
In this section, we investigate the asymptotic variance of 
within the framework introduced in Section 2. We commence by establishing the asymptotic expansion of 
and its associated asymptotic variance under a context of weak dependence. Subsequently, we demonstrate that, under mild conditions, this asymptotic variance in the presence of dependence exceeds that observed under independence. This observation implies that, in scenarios characterized by weak dependence, the asymptotic mean of 
remains unchanged, while its asymptotic variance experiences inflation. Due to space constraints and for the sake of presentation continuity, we have relegated the technical details to Web Appendix A-D.
3.1. Asymptotic expansion and variance of FDP
We begin by deriving the asymptotic expansion of the FDP within the framework of weak dependence. We introduce the following notation: for any 
, we denote 
and 
. Consider the function
 |
Within the interval 
, this function possesses a unique root, which we denote as 
. Furthermore, we define 
.
Theorem 1
Suppose
with unit variances, that is,
. Assume that
are weakly dependent as defined in Equation 5, that
, and that as p is sufficiently large, for a universal constant
,
(7)
(8)
(9) We have the following asymptotic expansion of
:
(10) where
,
,
and the remainder term
satisfies that
.
Remark 1
Theorem 1 establishes the asymptotic expansion of
, breaking it down into 3 distinct components: the asymptotic mean
, the stochastic term
that has mean 0, and an asymptotically negligible remainder term
. The asymptotic mean
represents a constant, aligning with the conclusion derived from Equation 6. Meanwhile, the stochastic term
emerges as a linear combination of the tests
, contributing to the determination of the asymptotic variance of
.
Remark 2
With the weak dependence assumption 5, the technical conditions in Theorem 1, that is, conditions 7-9, are mild, provided that
does not exhibit extreme behavior. The intuition is as follows: The number of terms summed on the left-hand side of Equation 7 is approximately
, whereas on the right-hand side, it is about
. If
, the number of terms summed on the left-hand side of Equation 7 greatly exceeds that on the right. Therefore, this condition is satisfied unless the magnitudes of
for
are generally much larger than those for
. Similar arguments apply to condition 8. As for condition 9, the summation on the left-hand side involves the fourth order of
, whereas on the right, it is related to the second order. Therefore, it is easily satisfied when most of the
approach 0 as
.
Remark 3
The development of Theorem 1 poses a significant theoretical challenge. While the expansion 10 can be obtained through a standard Taylor expansion, deriving
is technically involved. Specifically, 2 factors collectively introduce complexity into our development: (1) The leading component of
involves the summation of fourth moments of all the tests. To compare its asymptotic properties with
, we require the fourth-order Taylor expansion for each moment term. (2) The dependence structure of the tests significantly complicates our analysis of the asymptotic upper bound of
and the lower bound of
.
Following the above asymptotic expansion of 
in Theorem 1, Corollary 1 presents its asymptotic variance.
Corollary 1
With all the conditions in Theorem 1 effective, we have
(11) where
(12)
(13) with
(14) in which
is the probability density function of the bivariate normal distribution with mean
and covariance matrix
.
Corollary 1 provides an explicit formula for the asymptotic variance of FDP under weak dependence. In the special case where the test statistics are independent, that is, 
, the asymptotic variance of FDP reduces to 
. Consequently, the “additional variance” of FDP due to the dependence among the test statistics is introduced by 
. However, the sign of 
is inconclusive based on Equation 13. This leads to an interesting question: Under which scenario is 
, and thus, the variance of 
is inflated by the dependence among test statistics? To answer this question, we need to investigate the signs of the covariances between pairs of tests, that is, 
, which are included in the formula for 
. We address this question in the subsequent subsections.
3.2. Signs of the covariance between any pair of tests
In the previous subsection, we explained that the asymptotic variance of 
depends on 
, which, in turn, relies on 
. In this subsection, we investigate how the signs of 
are influenced by the dependence among the test statistics. Interestingly, we observe that the signs of 
are almost certainly determined by the memberships of i and j in 
and 
. We present the following theorem.
Theorem 2
Suppose
with unit variances, that is,
. Assume
for the index pair
. Then, we have
for
,
;
for
,
,
if
and
; and
for
,
if
and
, where
and
.
The motivation of the theorem above is to gain insights into how the variance of FDP is influenced by the dependence structure among the test statistics. However, these results stand as interesting findings in their own right, holding both theoretical and practical value. The theorem implies that the sign of the covariance between any pair 
, is almost certainly determined by the membership of i and j in 
and 
, and it relies less on the sign of 
. This conclusion applies to both weakly dependent and strongly dependent test statistics. We anticipate that this theorem may prove beneficial in other research related to multiple testing, particularly when it is in need to quantify the dependence between 
and 
. In the following discussion, we assume that 
and 
are dependent, unless stated otherwise.
Part (a) asserts that 
and 
are positively correlated when 
. This observation arises from the fact that both tests are 2-sided tests, and their corresponding test statistics have 0 expectations. Consequently, the absolute values 
and 
exhibit a positive correlation, which, in turn, results in a positive correlation between 
and 
.
Part (b) reveals that when 
, 
, and 
are negatively correlated under the condition that t is a reasonable threshold, specifically 
, and the signal of j is not too weak, that is, 
. Notably, these conditions are typically met in practical problems. The rationale behind this finding is as follows: Let 
; then, if 
were the test statistic, we would have 
. The correlations between 
and the other test statistics remain unchanged. The corresponding test 
, in the notation of 
, takes the form
 |
where we assume that 
for presentational convenience. This leads to
 |
and implies that the acceptance region for the test based on 
is contained within the rejection region of 
, given that the signal 
. Following our conclusion in Part (a), 
is positively correlated with 
, and therefore 
and 
are negatively correlated.
Part (c) provides sufficient conditions under which 
and 
are positively correlated when 
. This is the only case where 
affects the sign of the correlation between the corresponding tests. These conditions are also intuitively met in real applications. In particular, the first condition 
essentially requires that the magnitude of 
does not exceed a ratio of the minimum to the maximum signals. One special practical scenario is that if the 2 signals are reasonably strong and in the similar scale, this ratio is close to 1; thus, this condition is met. The second condition 
requires that the sign of 
coincides with the signs of the corresponding signals: The test statistics are positively correlated if the signals are of the same signs, and they are negatively correlated otherwise. To illustrate further, consider the GWAS example. Part (c) implies that the tests for the significance of the ith and jth SNPs are positively correlated if their individual effects on the disease status are both positive or negative.
3.3. Inflation of the FDP variance under dependence
In Section 3.1, we discussed that 
attributes to the FDP variance difference among varied dependence structures of the test statistics. The signs of its components are further discussed in Section 3.2. From these discussions, we conclude that the variance of FDP is inflated given that Parts (a)-(c) in Theorem 2 all hold. We summarize this result in the following theorem.
Theorem 3
Suppose
with
for
. For any
, assume that
,
, and
, where
and
. Then,
.
Theorem 3 implies that the variance of FDP is inflated if the test statistics satisfy the given assumptions on their dependence structure. On the one hand, as we have discussed in Section 3.2, these are mild conditions for practical data. On the other hand, they are only 1 set of sufficient conditions that lead to the inflation of the FDP variance. In our numerical studies, we experiment with some settings that violate the conditions in Theorem 3, and still observe the inflation of the variance of FDP; please refer to Section 5 for details. Such scenarios are not theoretically justified but may often be practically observed.
4. ESTIMATION OF THE ASYMPTOTIC VARIANCE OF FDP
In Section 3.1, we have established the asymptotic variance of FDP in the framework of weak dependence. We observe that this variance depends on the parameters 
and 
. To make this variance practically applicable, we need to estimate these parameters. Throughout, we assume that 
is known or can be estimated from other resources (see Section 2), and we focus on estimating only 
.
One direct approach is to estimate 
using their corresponding test statistics 
. However, in practical data, we often have 
, which implies that most 
’s are zeros. Estimating these “zeros” using the corresponding observed 
’s tends to diminish the efficiency of the method, as they can be viewed as noise. In our numerical studies, we consider the following estimation approach:
-
Find an estimate
for 
, which is the proportion of the number of tests that correspond to 
over the total number of tests.Estimation methods for
are available in the literature. For example, Storey and Tibshirani (2003) proposed 
with 
; the basic idea of this estimator is to pretend that p-values greater than 
are from 
. In this framework, smoothed and bootstrap estimators of 
are proposed by Storey and Tibshirani (2003) and Storey et al. (2004), respectively. Additionally, Langaas et al. (2005) proposed a non-parametric maximum likelihood estimator of 
. More recently, Wang et al. (2010) proposed a “SLIM” estimator of 
; their estimator accounted for the dependence among the test statistics. We shall adopt these estimators in our numerical studies and compare their performance in the estimation of the FDP variance. Estimate
by 
. We then collect the hypotheses with the top 
statistic values to form 
, and the rest are assigned to 
.Set
for 
; and set 
for 
; and use them to estimate the variance of 
based on Corollary 1.
5. SIMULATION STUDIES
We conduct simulation studies to validate our developments in Sections 3 and 4. In Section 5.1, we use various numerical examples to illustrate how the asymptotic variances of FDP are different for varied dependence structures among the test statistics. In Section 5.2, we illustrate how the choice of the methods for estimating 
reviewed in Section 4 affects the estimation for the variance of FDP.
5.1. Variance of FDP under varied dependence structures
Throughout our simulation studies, we generated 
with 
. We examined 3 different values for 
: 50, 100, and 200, where the indices of tests corresponding to 
were randomly sampled from 
. For 
, we set 
. We considered 3 distinct threshold values for t: 0.005, 0.02, and 0.05. The choice of 
determined the dependence structure among the test statistics, and we generated it using methods similar to those employed by Fan et al. (2012). Detailed information regarding the generation of 
is provided below.
-
We generate a random sample
, which are iid copies of 
. We consider 7 models for the joint distribution of 
.- [M1]
are generated as iid 
random variables.- [M2] Let
, where 
has diagonal elements 1 and off-diagonal elements 
.- [M3] Let
be iid 
random variables, and 
for 
with 
being 
random variables.- [M4] Let
be iid Cauchy random variables with location parameter 0 and scale parameter 1.- [M5] Let
, where 
, 
, 
, 
, 
, 
, 
; and all these random variables are independent.- [M6] Let
, where 
, 
, 
, and all these random variables are independent.- [M7] Let
, where 
, 
, and 
, and all these random variables are independent.
We compute
as the sample correlation matrix based on 
. Note that for some scenarios above, the components of 
are strongly dependent.We adopt the PFA method in Fan et al. (2012) to generate a weakly dependent
based on 
. When applying the PFA method, we used the criterion 
numerically.
With these setups, for M1, the test statistics are independent, whereas for the other models, the test statistics are weakly dependent with varied dependence structures. For each aforementioned combination of 
, t, and model, we conducted 
analysis and evaluated the corresponding sample standard deviation of 
s over 1000 repetitions (referred to as “Emp” in Table 2). We also computed the corresponding asymptotic standard deviation of 
using the values of 
and 
simulated for each model (referred to as “Asym” in Table 2). The results are summarized in Table 2.
TABLE 2.
Comparison for the standard deviation (
) of false discovery proportions.
| t |
|
Method | M1 | M2 | M3 | M4 | M5 | M6 | M7 |
|---|---|---|---|---|---|---|---|---|---|
| 50 | Asym | 4.37 | 6.11 | 5.13 | 6.91 | 6.11 | 6.12 | 6.15 | |
| Emp | 4.43 | 6.08 | 5.16 | 6.60 | 5.73 | 6.20 | 5.90 | ||
| 0.005 | 100 | Asym | 2.57 | 3.57 | 3.01 | 4.03 | 3.57 | 3.57 | 3.59 |
| Emp | 2.65 | 3.56 | 2.94 | 3.85 | 3.68 | 3.66 | 3.52 | ||
| 200 | Asym | 1.37 | 1.88 | 1.59 | 2.12 | 1.88 | 1.89 | 1.89 | |
| Emp | 1.33 | 1.83 | 1.60 | 2.06 | 1.86 | 1.79 | 1.95 | ||
| 50 | Asym | 3.92 | 6.74 | 5.23 | 7.12 | 6.75 | 6.75 | 6.80 | |
| Emp | 3.90 | 6.64 | 5.35 | 7.30 | 6.91 | 6.90 | 6.84 | ||
| 0.02 | 100 | Asym | 3.23 | 5.50 | 4.28 | 5.81 | 5.51 | 5.51 | 5.55 |
| Emp | 3.20 | 5.50 | 4.44 | 5.66 | 5.39 | 5.58 | 5.35 | ||
| 200 | Asym | 2.15 | 3.61 | 2.82 | 3.80 | 3.61 | 3.61 | 3.64 | |
| Emp | 2.14 | 3.61 | 2.84 | 3.70 | 3.44 | 3.51 | 3.40 | ||
| 50 | Asym | 2.25 | 4.33 | 3.25 | 4.42 | 4.34 | 4.34 | 4.37 | |
| Emp | 2.27 | 4.54 | 3.25 | 4.47 | 4.58 | 4.45 | 4.54 | ||
| 0.05 | 100 | Asym | 2.53 | 4.87 | 3.65 | 4.97 | 4.88 | 4.88 | 4.92 |
| Emp | 2.54 | 4.70 | 3.60 | 5.04 | 4.80 | 4.92 | 5.02 | ||
| 200 | Asym | 2.23 | 4.24 | 3.19 | 4.32 | 4.24 | 4.24 | 4.27 | |
| Emp | 2.25 | 4.25 | 3.22 | 4.30 | 4.09 | 4.17 | 4.22 |
From Table 2, we observe that for all cases, the asymptotic standard deviation is very close to the sample standard deviation of 
s over 1000 repetitions. This validation supports our conclusion in Corollary 1 and indicates that this asymptotic variance provides a good approximation of its true value in practical applications. Furthermore, for all combinations of 
and t, the standard deviation of the FDP from M1 is significantly smaller than those from the other models. This suggests that the variance of FDP is inflated when dependence is present among the test statistics. It’s noteworthy that in these examples, the conditions in Theorem 3 are not necessarily guaranteed, reinforcing our discussion in Section 3.3 and demonstrating that FDP variance inflation may occur in a wide range of real applications. Among M2-M6, we observe that M3 leads to the smallest inflation, which could be because this model is closer to the independent case, as the majority of 
’s are independent (ie, for 
).
5.2. Estimation of the asymptotic variance for FDP
In this subsection, we assess the performance of the approach in Section 4 in estimating the asymptotic variance of FDP. As we observe in Section 4, our method relies on 
, whose estimation methods are available in the literature. We incorporate these methods into our approach and compare their performance. In particular, we consider: (1) the smoothed estimate by Storey and Tibshirani (2003), named as “Smoothed”; (2) the bootstrap estimate by Storey et al. (2004), named as “Bootstrap”; (3) the non-parametric maximum likelihood estimate by Langaas et al. (2005), named as “Langaas”; and (4) the Sliding Linear Model estimate by Wang et al. (2010), named as “SLIM”. We include the asymptotic standard deviation of FDP based on the true values of the parameters, that is, 
and 
, for each case as a reference, named as “True”.
We adopt the methods in Section 5.1 to generate test statistics and apply the aforementioned methods to estimate the asymptotic variance of FDP. We only present the results for Model M4; the results for the other models show similar patterns and are provided in Web Appendix E. The mean and standard deviation of the estimated asymptotic standard deviations of FDP over 1000 replicates are displayed in Table 3. From this table, we observe that the Langaas and SLIM methods perform reasonably well in the estimation of the asymptotic standard deviation of FDPs; in particular, these methods result in estimates close to the true asymptotic standard deviation based on our Corollary 1 and lead to smaller variation. We observe that this is because in the estimation of 
, these methods are more robust to the dependence among the test statistics, whereas the Smoothed and Bootstrap methods can work well only when the test statistics are independent. We provide the average estimated number of identified alternative hypotheses, that is, 
and the corresponding standard deviation over 1000 replicates in Table 4; from this table, we observe that the performance in the estimation of 
is in the order of
TABLE 3.
Mean (
) and standard deviation (
, in parenthesis) of the estimated asymptotic standard deviation of false discovery proportions over 1000 replicates.
| t |
|
True | Smoothed | Bootstrap | Langaas | SLIM |
|---|---|---|---|---|---|---|
| 50 | 6.91 | 2.92 (2.90) | 5.35 (2.26) | 5.27 (1.11) | 6.26 (1.56) | |
| 0.005 | 100 | 4.03 | 2.76 (2.53) | 3.40 (1.19) | 3.23 (0.57) | 3.65 (0.71) |
| 200 | 2.12 | 2.31 (2.07) | 1.80 (0.40) | 1.78 (0.27) | 1.90 (0.30) | |
| 50 | 7.12 | 3.01 (2.64) | 5.34 (1.43) | 5.87 (0.97) | 6.09 (0.93) | |
| 0.02 | 100 | 5.81 | 3.22 (2.26) | 4.70 (1.04) | 4.76 (0.81) | 5.13 (0.83) |
| 200 | 3.80 | 3.15 (1.65) | 3.18 (0.65) | 3.20 (0.52) | 3.40 (0.53) | |
| 50 | 4.42 | 2.36 (2.11) | 3.80 (1.20) | 4.10 (0.70) | 3.77 (0.78) | |
| 0.05 | 100 | 4.97 | 2.75 (1.89) | 4.16 (0.70) | 4.28 (0.65) | 4.31 (0.65) |
| 200 | 4.32 | 3.11 (1.36) | 3.63 (0.62) | 3.69 (0.58) | 3.83 (0.58) |
TABLE 4.
Mean and standard deviation (in parenthesis) of the estimated number of alternative hypotheses over 1000 replicates.
| t |
|
Smoothed | Bootstrap | Langaas | SLIM |
|---|---|---|---|---|---|
| 50 | 93 (114) | 98 (73) | 85 (46) | 54 (18) | |
| 0.005 | 100 | 126 (128) | 148 (73) | 134 (45) | 103 (18) |
| 200 | 207 (148) | 247 (71) | 232 (41) | 203 (18) | |
| 50 | 93 (114) | 98 (73) | 84 (47) | 54 (18) | |
| 0.02 | 100 | 126 (128) | 148 (74) | 133 (46) | 103 (18) |
| 200 | 207 (148) | 247 (72) | 231 (43) | 202 (18) | |
| 50 | 93 (114) | 98 (73) | 84 (49) | 53 (18) | |
| 0.05 | 100 | 126 (128) | 147 (74) | 132 (49) | 100 (18) |
| 200 | 207 (148) | 247 (72) | 230 (49) | 197 (17) |
 |
which is consistent with our observation in Table 3.
6. REAL DATA ANALYSIS
We apply our method to a GWAS, specifically an expression quantitative trait locus (eQTL) mapping study. The data consist of 210 subjects from the International Hapmap Project, divided into 3 groups: (1) 90 subjects from Asia, comprising 45 Japanese from Tokyo and 45 Han Chinese from Beijing, referred to as “Grp1”; (2) 60 Utah residents with ancestry from northern and western Europe, referred to as “Grp2”; and (3) 60 Nigerians from Ibadan, referred to as “Grp3.” Further details about the data can be found in the work of Bradic et al. (2011). We adopt existing methods to construct test statistics for testing the association between the expression level of the gene CCT8 and the SNP genotypes for each group of subjects (Bradic et al., 2011; Fan et al., 2012).
The SNP genotypes can take on 1 of 3 possibilities: 
represents no polymorphism, 
indicates 1 nucleotide has polymorphism, and 
implies both nucleotides have polymorphisms. Following the approach in Bradic et al. (2011), we transform the SNP data into 2 dummy variables, denoted as 
, as follows:
for 
;
for 
;
for 
.
For subject i, let 
represent the logarithm-transformed value of the expression level of gene CCT8, and 
represent the aforementioned dummy variables for the jth SNP. We consider 2 sets of marginal linear regression models:
 |
In this analysis, we impute the missing SNP data with 0 and exclude SNPs that are identical for all subjects.
For each group of data, we combine the 2 sets of models mentioned above and follow the procedure outlined in Fan et al. (2012) to generate the test statistics, resulting in 
test statistics. To elaborate, we first employ the procedure described in Section 2 to establish an initial set of test statistics and their corresponding covariance matrix. We then utilize the dependence-adjusted procedure presented in Fan et al. (2012) to eliminate the top ten principal factors from the covariance matrix. Following this, we standardize the data to generate a new set of test statistics with unit marginal variances, effectively reducing the dependence among the test statistics.
With the resulting test statistics for each group, we estimate the asymptotic limit and standard deviation of 
based on the following setups:
For each group of test statistics, we consider 2 thresholds of p-values chosen to provide reasonable estimates for the limit of
.The asymptotic limit is estimated using Equation 6 and denoted by
.The asymptotic standard deviation is computed based on the approach outlined in Section 4. As we observed in Section 5.2 that the standard deviation estimates based on the
estimates from the “Langaas” and “SLIM” methods outperform the others, we present results only from these methods. Furthermore, to assess how dependence may affect these standard deviation estimates, we report these estimates based on the assumptions of independent and dependent test statistics, respectively. Therefore, for each group of subjects, we have 4 estimates, denoted as 
, 
, 
, and 
.
We summarize our estimation results in Table 5. From this table, we observe that assuming the presence or absence of dependence among the test statistics significantly affects the asymptotic standard deviation estimate of 
. The ratios 
and 
range from 4 to 7-folds. Moreover, the ratio of the FDP standard deviation estimate to the limit value 
ranges between 0.25 and 0.82, indicating that the uncertainty of FDP is non-negligible. These observations reinforce the statement that in FDP-related inference, it may not be sufficient to consider only the limit value of FDP (or equivalently, FDR). We suggest that it is important to correctly account for dependence; otherwise, one might have an overly optimistic conclusion about how well the FDP is controlled. For example, when 
in Grp1, the 95th percentile (say, 2 standard deviations above the mean) of 
is estimated as 
under dependence; in contrast, if we assume that the test statistics are independent, the corresponding estimate is 
. Therefore, ignoring dependence leads to an overly optimistic belief in the control of the upper bound of FDP, and such a non-negligible difference may significantly alter the statistical interpretation of the results in practice.
TABLE 5.
Estimation results for the GWAS study.
| Population | #SNP | t |
|
|
|
|
|
|---|---|---|---|---|---|---|---|
| Grp1 | 73 | 0.005 | 0.315 | 0.153 | 0.031 | 0.259 | 0.052 |
| 105 | 0.01 | 0.377 | 0.166 | 0.031 | 0.244 | 0.045 | |
| Grp2 | 166 | 0.01 | 0.199 | 0.107 | 0.018 | 0.148 | 0.025 |
| 199 | 0.02 | 0.441 | 0.125 | 0.019 | 0.167 | 0.025 | |
| Grp3 | 89 | 0.002 | 0.147 | 0.069 | 0.017 | 0.092 | 0.022 |
| 135 | 0.005 | 0.371 | 0.092 | 0.019 | 0.124 | 0.025 |
Abbreviations: FDP, false discovery proportion; GWAS, genome-wide association study; SNP, single nucleotide polymorphisms.
7. DISCUSSION
In this article, we have established theoretical results for the asymptotic variance of 
in the framework where the test statistics are normally distributed and weakly dependent. We observe that, under mild conditions, the asymptotic variance of 
under dependence is generally greater than that under independence. This observation is further supported by our numerical studies, where even under weak dependence, the variance of FDP can be numerically much larger than that under independence. Consequently, it is crucial to account for dependence when estimating the variance of FDP and report it alongside its mean. Moreover, our real data example demonstrates that the standard deviation estimate of 
under the dependence assumption is significantly different from (usually greater than) that under the independence assumption. In practical applications, we recommend presuming the presence of dependence in the estimation process, unless strong evidence suggests otherwise. Doing so will contribute to a better understanding of how well the FDP is controlled and help avoid overly optimistic conclusions.
In this article, we have examined the asymptotic uncertainty of FDP within the same framework as Fan et al. (2012). By assuming that the test statistics exhibit weak dependence, we have derived the asymptotic expansion of 
and provided an explicit formula for computing its asymptotic variance. Within this framework, we have encountered several intriguing open problems that may warrant future investigation. We present a few examples here, although numerous others exist.
First, our proposed method for estimating the variance of 
in Section 4 relies on the estimation of 
, for which we adopted existing methods from the literature. Our simulation studies in Section 5.2 have revealed that the quality of the variance estimate for 
is significantly influenced by the performance of the 
estimation method. Although the Langaas and SLIM methods have shown reasonable performance, it is worth noting that none of the existing 
estimation methods adequately account for the dependence among the test statistics. As a result, we believe that there is room for improvement in 
estimation techniques, particularly those capable of better accommodating test statistic dependence. These enhanced 
estimation methods could then be seamlessly integrated into our estimation framework presented in Section 4.
Second, our theoretical development relies on certain weak dependence assumptions, which may restrict the applicability of the method in certain scenarios. In practice, 1 possible solution is to involve using dependence-adjusted test statistics, which are often more powerful and are weakly dependent (Friguet and Causeur, 2011; Fan et al., 2012). However, it’s important to note that there are no theoretical guarantees in this context. Therefore, addressing the challenge of studying the asymptotic variance of 
under arbitrary forms of dependence would be a significant and intriguing avenue for future research.
Third, throughout this article, we have made the assumption that the test statistics follow a normal distribution. Extending our study to encompass multiple testing problems involving test statistics that follow other commonly encountered distributions, such as t, F, or chi-squared distributions, would be of significant practical and theoretical importance. As a reference, we point to recent work by Zhuo et al. (2020) on dependent t-tests. Exploring the behavior of 
under various non-normal distributions for test statistics is also an interesting direction for future research.
Supplementary Material
Web Appendices and Tables, referenced in Sections 3 and 5 are available with this paper at the Biometrics website on Oxford Academic. Code needed to reproduce the results from Sections 5 and 6 is available at the Biometrics website on Oxford Academic, and also at https://github.com/yuanjiang-osu/FDP-Biometrics/.
Acknowledgement
The authors thank the associate editor and referee for their helpful comments and suggestions.
Contributor Information
Meng Mei, Department of Statistics, Oregon State University, Corvallis, OR 97331, United States.
Tao Yu, Department of Statistics and Data Science, National University of Singapore, Singapore 117546.
Yuan Jiang, Department of Statistics, Oregon State University, Corvallis, OR 97331, United States.
FUNDING
Dr. Jiang’s research is supported in part by the National Institutes of Health grant: R01GM126549. Dr. Yu’s research is supported in part by the Singapore Ministry of Education Academic Research Tier 1 Fund: A-8000413-00-00.
CONFLICT OF INTEREST
None declared.
DATA AVAILABILITY
The real gene expression and SNP data needed to reproduce the results from Section 6 are processed from publicly available resources and are only available upon reasonable request.
References
- Basu P., Fu L., Saretto A., Sun W. (2023). An empirical bayes approach to controlling the false discovery exceedance. Journal of Business & Economic Statistics. 10.1080/07350015.2023.2277857 [DOI] [Google Scholar]
- Benjamini Y., Hochberg Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 57, 289–300. [Google Scholar]
- Benjamini Y., Yekutieli D. (2001). The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 29, 1165–1188. [Google Scholar]
- Bonferroni C. (1936). Teoria statistica delle classi e calcolo delle probabilita. Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commericiali di Firenze, 8, 3–62. [Google Scholar]
- Bradic J., Fan J., Wang W. (2011). Penalized composite quasi-likelihood for ultrahigh dimensional variable selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73, 325–349. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Clarke S., Hall P. (2009). Robustness of multiple testing procedures against dependence. The Annals of Statistics, 37, 332–358. [Google Scholar]
- Delattre S., Roquain E. (2011). On the false discovery proportion convergence under Gaussian equi-correlation. Statistics & Probability Letters, 81, 111–115. [Google Scholar]
- Delattre S., Roquain E. (2015). New procedures controlling the false discovery proportion via Romano–Wolf’s heuristic. The Annals of Statistics, 43, 1141–1177. [Google Scholar]
- Delattre S., Roquain E. (2016). On empirical distribution function of high-dimensional Gaussian vector components with an application to multiple testing. Bernoulli, 22, 302–324. [Google Scholar]
- Döhler S., Roquain E. (2020). Controlling the false discovery exceedance for heterogeneous tests. Electronic Journal of Statistics, 14, 4244–4272. [Google Scholar]
- Dudoit S., van der Laan M. J., Pollard K. S. (2004). Multiple testing. Part I. single-step procedures for control of general type I error rates. Statistical Applications in Genetics and Molecular Biology, 3, 1–69. [DOI] [PubMed] [Google Scholar]
- Efron B. (2007). Correlation and large-scale simultaneous significance testing. Journal of the American Statistical Association, 102, 93–103. [Google Scholar]
- Fan J., Han X. (2017). Estimation of the false discovery proportion with unknown dependence. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 79, 1143–1164. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fan J., Han X., Gu W. (2012). Estimating false discovery proportion under arbitrary covariance dependence. Journal of the American Statistical Association, 107, 1019–1035. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fan J., Ke Y., Sun Q., Zhou W.-X. (2019). Farmtest: factor-adjusted robust multiple testing with approximate false discovery control. Journal of the American Statistical Association, 114, 1880–1893. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Ferreira J., Zwinderman A. (2006). On the Benjamini–Hochberg method. The Annals of Statistics, 34, 1827–1849. [Google Scholar]
- Finner H., Roters M. (2002). Multiple hypotheses testing and expected number of type I errors. The Annals of Statistics, 30, 220–238. [Google Scholar]
- Friguet C., Causeur D. (2011). Estimation of the proportion of true null hypotheses in high-dimensional data under dependence. Computational Statistics & Data Analysis, 55, 2665–2676. [Google Scholar]
- Ge Y., Li X. (2012). Control of the false discovery proportion for independently tested null hypotheses. Journal of Probability and Statistics, 2012, 320425. [Google Scholar]
- Genovese C., Wasserman L. (2004). A stochastic process approach to false discovery control. The Annals of Statistics, 32, 1035–1061. [Google Scholar]
- Genovese C. R., Wasserman L. (2006). Exceedance control of the false discovery proportion. Journal of the American Statistical Association, 101, 1408–1417. [Google Scholar]
- Guo W., Romano J. (2007). A generalized Sidak-Holm procedure and control of generalized error rates under independence. Statistical Applications in Genetics and Molecular Biology, 6, 3. [DOI] [PubMed] [Google Scholar]
- Hemerik J., Solari A., Goeman J. J. (2019). Permutation-based simultaneous confidence bounds for the false discovery proportion. Biometrika, 106, 635–649. [Google Scholar]
- Hochberg Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75, 800–802. [Google Scholar]
- Holland B. S., Copenhaver M. D. (1987). An improved sequentially rejective Bonferroni test procedure. Biometrics, 43, 417–423. [Google Scholar]
- Holm S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65–70. [Google Scholar]
- Korn E. L., Troendle J. F., McShane L. M., Simon R. (2004). Controlling the number of false discoveries: application to high-dimensional genomic data. Journal of Statistical Planning and Inference, 124, 379–398. [Google Scholar]
- Langaas M., Lindqvist B. H., Ferkingstad E. (2005). Estimating the proportion of true null hypotheses, with application to dna microarray data. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67, 555–572. [Google Scholar]
- Lehmann E., Romano J. P. (2005). Generalizations of the familywise error rate. The Annals of Statistics, 33, 1138–1154. [Google Scholar]
- Owen A. B. (2005). Variance of the number of false discoveries. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67, 411–426. [Google Scholar]
- Pollard K. S., van der Laan M. J. (2004). Choice of a null distribution in resampling-based multiple testing. Journal of Statistical Planning and Inference, 125, 85–100. [Google Scholar]
- Rom D. M. (1990). A sequentially rejective test procedure based on a modified Bonferroni inequality. Biometrika, 77, 663–665. [Google Scholar]
- Sarkar S. K. (2006). False discovery and false nondiscovery rates in single-step multiple testing procedures. The Annals of Statistics, 34, 394–415. [Google Scholar]
- Šidák Z. (1967). Rectangular confidence regions for the means of multivariate normal distributions. Journal of the American Statistical Association, 62, 626–633. [Google Scholar]
- Simes R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika, 73, 751–754. [Google Scholar]
- Storey J. D. (2002). A direct approach to false discovery rates. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64, 479–498. [Google Scholar]
- Storey J. D., Taylor J. E., Siegmund D. (2004). Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 66, 187–205. [Google Scholar]
- Storey J. D., Tibshirani R. (2003). Statistical significance for genomewide studies. Proceedings of the National Academy of Sciences, 100, 9440–9445. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sun W., Cai T. (2009). Large-scale multiple testing under dependence. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 71, 393–424. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sun W., Reich B. J., Cai T. T., Guindani M., Schwartzman A. (2015). False discovery control in large-scale spatial multiple testing. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 77, 59–83. [DOI] [PMC free article] [PubMed] [Google Scholar]
- van der Laan M. J., Dudoit S., Pollard K. S. (2004). Augmentation procedures for control of the generalized family-wise error rate and tail probabilities for the proportion of false positives. Statistical Applications in Genetics and Molecular Biology, 3, 15. [DOI] [PubMed] [Google Scholar]
- Wang H.-Q., Tuominen L. K., Tsai C.-J. (2010). Slim: a sliding linear model for estimating the proportion of true null hypotheses in datasets with dependence structures. Bioinformatics, 27, 225–231. [DOI] [PubMed] [Google Scholar]
- Zhuo B., Jiang D., Di Y. (2020). Test-statistic correlation and data-row correlation. Statistics & Probability Letters, 167, 108903. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Web Appendices and Tables, referenced in Sections 3 and 5 are available with this paper at the Biometrics website on Oxford Academic. Code needed to reproduce the results from Sections 5 and 6 is available at the Biometrics website on Oxford Academic, and also at https://github.com/yuanjiang-osu/FDP-Biometrics/.
Data Availability Statement
The real gene expression and SNP data needed to reproduce the results from Section 6 are processed from publicly available resources and are only available upon reasonable request.