Weighted False Discovery Rate Control in Large-Scale Multiple Testing

Abstract

The use of weights provides an effective strategy to incorporate prior domain knowledge in large-scale inference. This paper studies weighted multiple testing in a decision-theoretic framework. We develop oracle and data-driven procedures that aim to maximize the expected number of true positives subject to a constraint on the weighted false discovery rate. The asymptotic validity and optimality of the proposed methods are established. The results demonstrate that incorporating informative domain knowledge enhances the interpretability of results and precision of inference. Simulation studies show that the proposed method controls the error rate at the nominal level, and the gain in power over existing methods is substantial in many settings. An application to a genome-wide association study is discussed.

1. Introduction

In large-scale studies, relevant domain knowledge, such as external covariates, scientific insights and prior data, is often available alongside the primary data set. Exploiting such information in an efficient manner promises to enhance both the interpretability of scientific results and precision of statistical inference. In multiple testing, the hypotheses being investigated often become “unequal” in light of external information, which may be reflected by differential attitudes towards the relative importance of testing units or the severity of decision errors. The use of weights provides an effective strategy to incorporate informative domain knowledge in large-scale testing problems.

In the literature, various weighting methods have been advocated for a range of multiple comparison problems. A popular scheme, referred to as the decision weights or loss weights approach, involves modifying the error criteria or power functions in the decision process (Benjamini and Hochberg, 1997). The idea is to employ two sets of positive constants a = {a_i : i = 1,⋯ ,m} and b = {b_i : i = 1,⋯ ,m} to take into account the costs and gains of multiple decisions. Typically, the choice of the weights a and b reflects the degree of confidence one has toward prior beliefs and external information. It may also be pertinent to the degree of preference that one has toward the consequence of one class of erroneous/correct decisions over another class based on various economical and ethical considerations. For example, in the spatial cluster analysis considered by Benjamini and Heller (2007), the weighted false discovery rate was used to reflect that a false positive cluster with larger size would account for a larger error. Another example arises from genome-wide association studies (GWAS), where prior data or genomic knowledge, such as prioritized subsets (Lin and Lee, 2012), allele frequencies (Lin et al., 2014) and expression quantitative trait loci information (Li et al., 2013), can often help to assess the scientific plausibility of significant associations. To incorporate such information in the analysis, a useful strategy is to up-weight the gains for the discoveries in preselected genomic regions by modifying the power functions in respective testing units (P˜ena et al., 2011; Sun et al., 2015; He et al., 2015). We assume in this paper that the weights have been pre-specified by the investigator. This is a reasonable assumption in many practical settings. For example, weights may be assigned according to economical considerations (Westfall and Young, 1993), external covariates (Benjamini and Heller, 2007; Sun et al., 2015) and biological insights from prior studies (Xing et al., 2010).

We mention two alternative formulations for weighted multiple testing. One popular method, referred to as the procedural weights approach, involves the adjustment of the pvalues from individual tests. In GWAS, Roeder et al. (2006) and Roeder and Wasserman (2009) proposed to utilize linkage signals to up-weight the p-values in preselected regions and down-weight the p-values in other regions. It was shown that the power to detect association can be greatly enhanced if the linkage signals are informative, yet the loss in power is small when the linkage signals are uninformative. Another useful weighting scheme, referred to as the class weights approach, involves allocating varied test levels to different classes of hypotheses. For example, in analysis of the growth curve data (Box, 1950), Westfall and Young (1993, page 186) proposed to allocate a higher family-wise error rate (FWER) to the class of hypotheses related to the primary variable “gain” and a lower FWER to the secondary variable “shape”.

We focus on the decision weights approach in the present paper. This weighting scheme is not only practically useful for a wide range of applications, but also provides a powerful framework that enables a unified investigation of various weighting methods. Specifically, the proposal in Benjamini and Hochberg (1997) involves the modification of both the error rate and power function. The formulation is closely connected to the classical ideas in compound decision theory that aim to optimize the tradeoffs between the gains and losses when many simultaneous decisions are combined as a whole. Our theory reveals that if the goal is to maximize the power subject to a given error rate, then the modifications via decision weights would lead to improved multiple testing methods with sensible procedural weights or class weights, or both. For example, in GWAS, the investigators can up-weight the power functions for discoveries in genomic regions that are considered to be more scientific plausible or biologically meaningful; this would naturally up-weight the p-values in these regions and thus yield weighting strategies similar to those suggested by Roeder and Wasserman (2009). In large clinical trials, modifying the power functions for respective rejections at the primary and secondary end points would correspond to the allocation of varied test levels across different classes of hypotheses, leading to weighting strategies previously suggested by Westfall and Young (1993).

The false discovery rate (FDR; Benjamini and Hochberg, 1995) has been widely used in large-scale multiple testing as a practical and powerful error criterion. Following Benjamini and Hochberg (1997), we generalize the FDR to weighted false discovery rate (wFDR), and develop optimal procedures for wFDR control under the decision weights framework. We first construct an oracle procedure that maximizes the weighted power function subject to a constraint on the wFDR, and then develop a data-driven procedure to mimic the oracle and establish its asymptotic optimality. The numerical results show that the proposed method controls the wFDR at the nominal level, and the gain in power over existing methods is substantial in many settings. Our optimality result marks a clear departure from existing works in the literature on covariate-assisted inference, which aims to incorporate the external information by deriving optimal procedural weights for p-values (Roeder and Wasserman, 2009; Roquain and van de Wiel, 2009) or constructing covariate-adjusted test statistics (Ferkingstad et al. 2008; Zablocki et al. 2014; Cai et al. 2016) subject to the unweighted FDR and power function.

Our research also makes a novel contribution to the theory of optimal ranking in multiple testing. Conventionally, a multiple testing procedure operates in two steps: ranking the hypotheses according to their significance levels and then choosing a cutoff along the rankings. It is commonly believed that the rankings remain the same universally at all FDR levels. For example, the ranking based on p-values or adjusted p-values in common practice is invariant to the choice of the FDR threshold. The implication of our theory is interesting, for it claims that there does not exist a ranking that is universally optimal at all test levels. Instead, the optimal ranking of hypotheses depends on the pre-specified wFDR level. That is, the hypotheses may be ordered differently when different wFDR levels are chosen. This point is elaborated in Section A of the Supplementary Material. See also Roeder and Wasserman (2009) and Roquain and van de Wiel (2009), where the FDR level α is included in constructing the procedural weights.

The rest of the article is organized as follows. Section 2 discusses a general framework for weighted multiple testing. Sections 3 and 4 develop oracle and data-driven wFDR procedures and establish their optimality properties. Practice guidelines and numerical results are presented in Section 5. Section 6 concludes the article with a discussion of related and future works. Proof of Theorem 1 is given in Section 7. Proofs of other theoretical results and additional discussions are provided in the Supplementary Material.

2. Problem Formulation

This section discusses a decision weights framework for weighted multiple testing. We first introduce the model and notation and then discuss modified error criteria and power functions.

2.1. Model and notation

Suppose that m hypotheses H₁,⋯ ,H_m are tested simultaneously based on observations X₁,⋯ ,X_m. Let θ = (θ₁,⋯ ,θ_m) ∈ {0,1}^m denote the true state of nature, where 0 and 1 stand for null and non-null cases, respectively. Assume that observations X_i are independent and distributed according to the following model

$$X_i|{\theta }_i\sim \left(1-{\theta }_i\right)F_{0i}+{\theta }_i{F}_{1i},$$ (2.1)

where F_0i and F_1i are the null and non-null distributions for X_i, respectively. Denote by f_0i and f_1i the corresponding density functions. Suppose that the unknown states θ_i are Bernoulli (p_i) variables, where p_i = P(θ_i = 1). The mixture density is denoted by f·i = (1 − pi)f0i + pif1i.

Consider the widely used random mixture model (Efron et al., 2001; Storey, 2002; Genovese and Wasserman, 2002)

$$X_i\sim F=(1-p)F_0+p{F}_1.$$ (2.2)

This model, which assumes that all observations are identically distributed according to a common distribution F, can sometimes be unrealistic in applications. In light of domain knowledge, the observations are likely to have different distributions. For example, in the context of a brain imaging study, Efron (2008) showed that the proportions of activated voxels are different for the front and back halves of a brain. In GWAS, certain genomic regions contain higher proportions of significant signals than other regions. In the adequate yearly progress study of California high schools (Rogosa, 2003), the densities of z-scores vary significantly from small to large schools. We shall develop theory and methodology for model (2.1), which allows possibly different non-null proportions and densities and is applicable to a wide range of settings.

The multi-group model considered in Efron (2008) and Cai and Sun (2009), which has been widely used in applications, is an important case of the general model (2.1). The multi-group model assumes that the observations can be divided into K groups. Let ${\mathcal{G}}_k$ denote the index set of the observations in group k, k = 1,⋅,K. For each i ∈ ${\mathcal{G}}_k$, θ_i is distributed as Bernoulli(p_k), and X_i follows a mixture distribution:

$$\left(X_i|i\in {\mathcal{G}}_k\right)\sim f_{\cdot k}=\left(1-p_k\right)f_{0k}+p_k{f}_{1k},$$ (2.3)

where f_0k and f_1k are the null and non-null densities for observations in group k. This model will be revisited in later sections. See also Ferkingstad et al. (2008), Hu et al. (2010) and Liu et al. (2016) for related works on multiple testing with groups.

2.2. Weighted error criterion and power function

This section discusses a generalization of the FDR criterion in the context of weighted multiple testing. Denote the decisions for the m tests by δ = (δ₁,⋯ ,δ_m) ∈ {0,1}^m, where δ_i = 1 indicates that H_i is rejected and δ_i = 0 otherwise. Let a_i be the weight indicating the severity of a false positive decision. For example, a_i is taken as the cluster size in the spatial cluster analyses conducted in Benjamini and Heller (2007) and Sun et al. (2015). As a result, rejecting a larger cluster erroneously corresponds to a more severe decision error. To incorporate domain knowledge in multiple testing, Benjamini and Hochberg (1997) defined the weighted FDR wFDR_BH = E{Q(a)}, where

$$Q(\mathbf{a})=\{\begin{array}{ll}\frac{{\displaystyle \sum _{i=1}^m{a}_i}\left(1-{\theta }_i\right){\delta }_i}{{\displaystyle \sum _{i=1}^m{a}_i}{\delta }_i},& \mathrm{if}{\displaystyle \sum _{i=1}^m{a}_i}{\delta }_i>0,\\ 0,& \mathrm{otherwise}.\end{array}$$ (2.4)

We consider a slightly different version of the wFDR:

$$\text{wFDR}=\frac{E\left\{{\displaystyle \sum _{i=1}^m{a}_i}\left(1-{\theta }_i\right){\delta }_i\right\}}{E\left({\displaystyle \sum _{i=1}^m{a}_i}{\delta }_i\right)}.$$ (2.5)

In Section D of the Supplementary Material, we show that the definitions in (2.4) and (2.5) are asymptotically equivalent. The main consideration of using (2.5) is to facilitate our theoretical derivations and obtain exact optimality results.

To assess the effectiveness of different wFDR procedures, we define the expected number of true positives

$$\text{ETP}=E\left({\displaystyle \sum _{i=1}^m{b}_i}{\theta }_i{\delta }_i\right),$$ (2.6)

where b_i is the weight indicating the power gain when H_i is rejected correctly. The use of b_i provides an effective scheme to incorporate domain knowledge. In GWAS, larger b_i can be assigned to pre-selected genomic regions to reflect that the discoveries in these regions are more biologically meaningful. In spatial data analysis, correctly identifying a larger cluster that contains signal may correspond to a larger b_i, indicating a greater gain.

By combining the concerns on both the error criterion and power function, the goal in weighted multiple testing is to

$$\mathbf{maximize}\mathbf{the}\mathbf{ETP}\mathbf{subject}\mathbf{to}\mathbf{the}\mathbf{constraint}\mathbf{wFDR}\le \alpha .$$ (2.7)

The optimal solution to (2.7) is studied in the next section.

3. Oracle Procedure for wFDR Control

The basic framework of our theoretical and methodological developments is outlined as follows. In Section 3.1, we assume that p_i, f_0i, and f_·i in the mixture model (2.1) are known by an oracle and derive an oracle procedure that maximizes the ETP subject to a constraint on the wFDR. Connections to the literature is given in Section 3.2. In Section 4, we develop a data-driven procedure to mimic the oracle and establish its asymptotic validity and optimality.

3.1. Oracle procedure

The derivation of the oracle procedure involves two key steps: the first is to derive the optimal ranking of hypotheses and the second is to determine the optimal threshold along the ranking that exhausts the pre-specified wFDR level. We discuss the two issues in turn. Consider model (2.1). Define the local false discovery rate (Lfdr, Efron et al. 2001) as

$${\mathrm{Lfdr}}_i=\frac{\left(1-p_i\right)f_{0i}\left(x_i\right)}{f{.}_i\left(x_i\right)}.$$ (3.1)

The wFDR problem (2.7) is equivalent to the following constrained optimization problem

$$\text{maximize}E\left\{{\displaystyle \sum _{i=1}^m{b}_i}{\delta }_i\left(1-{\mathrm{Lfdr}}_i\right)\right\}\text{subject}\text{to}E\left\{{\displaystyle \sum _{i=1}^m{a}_i}{\delta }_i\left({\mathrm{Lfdr}}_i-\alpha \right)\right\}\le 0.$$ (3.2)

Let S⁻ = {i : Lfdr_i ≤ α} and S⁺ = {i : Lfdr_i > α}. Then the constraint in (3.2) can be equivalently expressed as

$$E\left\{{\displaystyle \sum _{S^{+}}{a}_i}{\delta }_i\left({\mathrm{Lfdr}}_i-\alpha \right)\right\}\le E\left\{{\displaystyle \sum _{S^{-}}{a}_i}{\delta }_i\left(\alpha -{\mathrm{Lfdr}}_i\right)\right\}.$$ (3.3)

Consider an optimization problem which involves packing a knapsack with a capacity given by the right hand side of equation (3.3). Every available object has a known value and a known cost (of space). Clearly rejecting a hypothesis in S⁻ is always beneficial as it allows the capacity to expand, and thus promotes more discoveries. Hence the key issue boils down to how to efficiently utilize the capacity (after all hypotheses in S⁻ are rejected) to make as many discoveries as possible in S⁺. Observe that each rejection in S⁺ would simultaneously increase the power and decrease the capacity. Intuitively, we should sort all hypotheses in S⁺ in an decreasing order of the value to cost ratio (VCR). Equations (3.2) and (3.3) suggest that

$${\text{VCR}}_i=\frac{b_i\left(1-{\mathrm{Lfdr}}_i\right)}{a_i\left({\mathrm{Lfdr}}_i-\alpha \right)}.$$ (3.4)

To maximize the power, the ordered hypotheses are rejected sequentially until maximum capacity is reached.

The above considerations motivate us to consider the following class of decision rules ${\mathit{\delta }}^{*}(t)=\left\{{\delta }_i^{*}(t):i=1,\cdots ,m\right\}$, where

$${\delta }_i^{*}(t)=\{\begin{array}{ll}1,& \text{if}{b}_i\left(1-{\mathrm{Lfdr}}_i\right)>t{a}_i\left({\mathrm{Lfdr}}_i-\alpha \right),\\ 0,& \text{if}{b}_i\left(1-{\mathrm{Lfdr}}_i\right)\le t{a}_i\left({\mathrm{Lfdr}}_i-\alpha \right).\end{array}$$ (3.5)

We briefly explain some important operational characteristics of testing rule (3.5). First, if we let t > 0, then the equation implies that ${\delta }_i^{*}\left(t\right)=1$ for all i ∈ S⁻; hence all hypotheses in S⁻ are rejected as desired. (This explains why the VCR is not used directly in (3.5), given that the VCR is not meaningful in S⁻.) Second, a solution path can be generated as we vary t continuously from large to small. Along the path δ^∗(t) sequentially rejects the hypotheses in S⁺ according to their ordered VCRs. Denote by H₍₁₎,⋯ ,H_(m) the hypotheses sequentially rejected by δ^∗. (The actual ordering of the hypotheses within S⁻ does not matter in the decision process since all are always rejected.)

The next task is to choose a cutoff along the ranking to achieve exact wFDR control. The difficulty is that the maximum capacity may not be attained by a sequential rejection procedure. To exhaust the wFDR level, we permit a randomized decision rule. Denote the Lfdr values and the weights corresponding to H_(i) by Lfdr_(i), a_(i), and b_(i). Let

$$C(j)={\displaystyle \sum _{i=1}^j{a}_{(i)}}\left({\mathrm{Lfdr}}_{(i)}-\alpha \right)$$ (3.6)

denote the capacity up to jth rejection. According to the constraint in equation (3.2), we choose k = max{j : C(j) ≤ 0} so that the capacity is not yet reached when H_(k) is rejected but would just be exceeded if H_(k+1) is rejected. The idea is to split the decision point at H_(k+1) by randomization.

Let U be a Uniform (0,1) variable that is independent of the truth, the observations, and the weights. Define

$$t^{*}=\frac{b_{(k+1)}\left(1-{\mathrm{Lfdr}}_{(k+1)}\right)}{a_{(k+1)}\left({\mathrm{Lfdr}}_{(k+1)}-\alpha \right)}\mathrm{and}{p}^{*}=-\frac{C(k)}{C(k+1)-C(k)}.$$

Let ${\mathcal{I}}_A$ be an indicator, which takes value 1 if event A occurs and 0 otherwise. We propose the oracle decision rule ${\mathit{\delta }}_{OR}=\left\{{\delta }_{OR}^i:i=1,\cdots ,m\right\}$, where

$${\delta }_{OR}^i=\{\begin{array}{ll}1& \text{if}{b}_i\left(1-{\mathrm{Lfdr}}_i\right)>t^{*}{a}_i\left({\mathrm{Lfdr}}_i-\alpha \right),\\ 0& \text{if}{b}_i\left(1-{\mathrm{Lfdr}}_i\right)<t^{*}{a}_i\left({\mathrm{Lfdr}}_i-\alpha \right),\\ {\mathcal{I}}_{U<p^{*}}& \text{if}{b}_i\left(1-{\mathrm{Lfdr}}_i\right)=t^{*}{a}_i\left({\mathrm{Lfdr}}_i-\alpha \right).\end{array}$$ (3.7)

Remark 1 The randomization step is only employed for theoretical considerations to enforce the wFDR to be exactly α. Thus the optimal power can be effectively characterized. Moreover, only a single decision point at H_(k+1) is randomized, which has a negligible effect in large-scale testing problems. We do not pursue randomized rules for the data-driven procedures developed in later sections.

Let wFDR(δ) and ETP(δ) denote the wFDR and ETP of a decision rule δ, respectively. Theorem 1 shows that the oracle procedure (3.7) is valid and optimal for wFDR control.

Theorem 1 Consider model (2.1) and oracle procedure δ_OR defined in (3.7). Let ${\mathcal{D}}_{\alpha }$ be the collection of decision rules such that for any δ ∈ ${\mathcal{D}}_{\alpha }$, wFDR(δ) ≤ α. Then we have

• (i).wFDR(δ_OR) = α.
• (ii).ETP(δ_OR) ≥ ETP(δ) for all δ ∈ ${\mathcal{D}}_{\alpha }$.

3.2. Comparison with the optimality results in Spjøtvoll (1972), Benjamini and Hochberg (1997) and Storey (2007)

Spjøtvoll (1972) showed that the likelihood ratio (LR) statistic

$$T_{LR}^i=\frac{f_{0i}\left(x_i\right)}{f_{1i}\left(x_i\right)}$$ (3.8)

is optimal for the following multiple testing problem

$$\text{maximize}{E}_{\cap H_{1i}}\left({\displaystyle \sum _{i=1}^m{\delta }_i}\right)\text{subject}\text{to}{E}_{\cap H_{0i}}\left\{{\displaystyle \sum _{i=1}^m{\delta }_i}\right\}\le \alpha ,$$ (3.9)

where ∩H_0i and ∩H_1i denote the intersections of the nulls and non-nulls, respectively. The error criterion E_∩H0i $\left\{{\sum }_i{a}_i{\delta }_i\right\}$ is referred to as the intersection tests error rate (ITER).

A weighted version of problem (3.9) was considered by Benjamini and Hochberg (1997), where the goal is to

$$\text{maximize}{E}_{\cap H_{1i}}\left({\displaystyle \sum _{i=1}^m{b}_i}{\delta }_i\right)\text{subject}\text{to}{E}_{\cap H_{0i}}\left\{{\displaystyle \sum _{i=1}^m{a}_i}{\delta }_i\right\}\le \alpha .$$ (3.10)

The optimal solution to (3.10) is given by the next proposition.

Proposition 1 (Benjamini and Hochberg, 1997). Define the weighted likelihood ratio (WLR)

$$T_{IT}^i=\frac{a_i{f}_{0i}\left(x_i\right)}{b_i{f}_{1i}\left(x_i\right)}.$$ (3.11)

Then the optimal solution to (3.10) is a thresholding rule of the form ${\delta }_{IT}^i=\left(T_{IT}^i<t_{\alpha }\right)$, where t_α is the largest threshold that controls the weighted ITER at level α.

The ITER is very restrictive in the sense that the expectation is taken under the conjunction of the null hypotheses. The ITER is inappropriate for mixture model (2.1) where a mixture of null and non-null hypotheses are tested simultaneously. To extend intersection tests to multiple tests, define the per family error rate (PFER) as

$$\mathrm{PFER}(\mathit{\delta })=E\left\{{\displaystyle \sum _{i=1}^m{a}_i}\left(1-{\theta }_i\right){\delta }_i\right\}.$$ (3.12)

The power function should be modified correspondingly. Therefore the goal is to

$$\text{maximize}E\left({\displaystyle \sum _{i=1}^m{b}_i}{\theta }_i{\delta }_i\right)\text{subject}\text{to}E\left\{{\displaystyle \sum _{i=1}^m{a}_i}\left(1-{\theta }_i\right){\delta }_i\right\}\le \alpha .$$ (3.13)

The key difference between the ITER and PFER is that the expectation in (3.12) is now taken over all possible combinations of the null and non-null hypotheses. The optimal PFER procedure is given by the next proposition.

Proposition 2 Consider model (2.1) and assume continuity of the LR statistic. Let ${\mathcal{D}}_{PF}^{\alpha }$ be the collection of decision rules such that for every δ ∈ ${\mathcal{D}}_{PF}^{\alpha }$ , PFER(δ) ≤ α. Define the weighted posterior odds (WPO)

$$T_{PF}^i=\frac{a_i\left(1-p_i\right)f_{0i}\left(x_i\right)}{b_i{p}_i{f}_{1i}\left(x_i\right)}.$$ (3.14)

Denote by Q_PF (t) the PFER of ${\delta }_{PF}^i=I\left(T_{PF}^i<t\right)$. Then the oracle PFER procedure is ${\mathit{\delta }}_{PF}=\left({\delta }_{PF}^i:i=1,\cdots ,m\right)$, where ${\delta }_{PF}^i=I\left(T_{PF}^i<t_{PF}\right)$ and t_PF = sup{t : Q_PF (t) ≤ α}.

This oracle rule satisfies:

• (i).ETP(δ_PF ) = α.
• (ii).ETP(δ_PF ) ≥ ETP(δ) for all δ ∈ ${\mathcal{D}}_{PF}^{\alpha }$.

Storey (2007) proposed the optimal discovery procedure (ODP) that aims to maximize the ETP subject to a constraint on the expected number of false positives (EFP). The ODP extends the optimality result in Spjøtvoll (1972) from the intersection tests to multiple tests. However, the formulation of ODP does not incorporate weights. As a result, the ODP procedure is a symmetric rule where all hypotheses are exchangeable. Symmetric rules are in general not suitable for weighted multiple testing where it is desirable to incorporate external information and treat the hypotheses differently.

Our formulation (2.7) modifies the conventional formulations in (3.10) and (3.13) to the multiple testing situation with an FDR type criterion. These modifications lead to methods that are more suitable for large-scale scientific studies. The oracle procedure (3.7) uses the VCR (3.4) to rank the hypotheses and is an asymmetric rule. The VCR, which optimally combines the decision weights, significance measure (Lfdr) and test level α, produces a more powerful ranking than the WPO (3.14) in the wFDR problem. Section A in the supplementary material provides a detailed discussion on the ranking issue.

4. Data-Driven Procedures and Asymptotics

The oracle procedure (3.7) cannot be implemented in practice since it relies on unknown quantities such as Lfdr_i and t^∗. This section develops a data-driven procedure to mimic the oracle. We first propose a test statistic to rank the hypotheses and discuss related estimation issues. A step-wise procedure is then derived to determine the best cutoff along the ranking. Finally, asymptotic results on the validity and optimality of the proposed procedure are presented.

4.1. Proposed test statistic and its estimation

The oracle procedure utilizes the ranking based on the VCR (3.4). However, the VCR is only meaningful for the tests in S⁺ and becomes problematic when both S⁻ and S⁺ are considered. Moreover, the VCR could be unbounded, which would lead to difficulties in both numerical implementations and technical derivations. We propose to rank the hypotheses using the following statistic (in increasing values)

$$R_i=\frac{a_i\left({\mathrm{Lfdr}}_i-\alpha \right)}{b_i\left(1-{\mathrm{Lfdr}}_i\right)+a_i\left|{\mathrm{Lfdr}}_i-\alpha \right|}.$$ (4.1)

As shown in the next proposition, R_i always ranks hypotheses in S⁻ higher than hypotheses in S⁺ (as desired), and yields the same ranking as that by the VCR (3.4) for hypotheses in S⁺. The other drawbacks of VCR can also be overcome by R_i: R_i is always bounded in the interval [−1,1] and is a continuous function of the Lfdr_i.

Proposition 3 (i) The rankings generated by the decreasing values of VCR (3.4) and increasing values of R_i (4.1) are the same in both S⁻ and S⁺. (ii) The ranking based on increasing values of R_i always puts hypotheses in S⁻ ahead of hypotheses in S⁺.

Next we discuss how to estimate R_i; this involves the estimation of the Lfdr statistic (3.1), which has been studied extensively in the multiple testing literature. We give a review of related methodologies. If all observations follow a common mixture distribution (2.2), then we can first estimate the non-null proportion p and the null density f₀ using the methods in Jin and Cai (2007), and then estimate the mixture density f using a standard kernel density estimator (e.g. Silverman, 1986). If all observations follow a multi-group model (2.3), then we can apply the above estimation methods to separate groups to obtain corresponding estimates ${\widehat{p}}_k,{\widehat{f}}_{0k}$, and ${\widehat{f}}_{·k}$, k = 1,⋯ ,K. The theoretical properties of these estimators have been established in Sun and Cai (2007) and Cai and Sun (2009). In practice, estimation problems may arise from more complicated models. Related theories and methodologies have been studied in Storey (2007), Ferkingstad et al. (2008), and Efron (2008, 2010); theoretical supports for these estimators are yet to be developed.

The estimated Lfdr value for H_i is denoted by ${\widehat{\mathrm{Lfdr}}}_i$. By convention, we take ${\widehat{\mathrm{Lfdr}}}_i=1$ if ${\widehat{\mathrm{Lfdr}}}_i>1$. This modification only facilitates the development of theory and has no practical effect on the testing results (since rejections are essentially only made for small ${\widehat{\text{Lfdr}}}_i$’s). The ranking statistic R_i can therefore be estimated as

$${\widehat{R}}_i=\frac{a_i({\widehat{\mathrm{Lfdr}}}_i-\alpha )}{b_i(1-{\widehat{\mathrm{Lfdr}}}_i)+a_i|{\widehat{\mathrm{Lfdr}}}_i-\alpha |}.$$ (4.2)

The performance of the data driven procedure relies on the accuracy of the estimate ${\widehat{\text{Lfdr}}}_i$; some technical conditions are discussed in the next subsection. The finite sample performance of different Lfdr estimates are investigated in Section E.4 of the Supplementary Material.

4.2. Proposed testing procedure and its asymptotic properties

Consider ${\widehat{R}}_i$ defined in (4.2). Denote by ${\widehat{N}}_i=a_i({\widehat{\mathrm{Lfdr}}}_i-\alpha )$ the estimate of excessive error rate when H_i is rejected. Let ${\widehat{R}}_{(1)},\cdots ,{\widehat{R}}_{(m)}$ be the ordered test statistics (in increasing values). The hypothesis and estimated excessive error rate corresponding to ${\widehat{R}}_{\left(i\right)}$ are denoted by H_(i) and ${\widehat{N}}_{(i)}$. The idea is to choose the largest cutoff along the ranking based on ${\widehat{R}}_i$ so that the maximum capacity is reached. Motivated by the constraint in (3.2), we propose the following step-wise procedure.

Procedure 1 (wFDR control with general weights). Rank hypotheses according to ${\widehat{R}}_i$ in increasing values. Let $k=\max \left\{j:{\displaystyle \sum _{i=1}^j{\widehat{N}}_{(i)}}\le 0\right\}$. Reject H_(i), for i = 1,...,k.

It is important to note that in Procedure 1, ${\widehat{R}}_i$ is used in the ranking step whereas ${\widehat{N}}_i$ (or a weighted transformation of ${\widehat{\text{Lfdr}}}_i$) is used in the thresholding step. The ranking by ${\widehat{\text{Lfdr}}}_i$ is in general different from that by ${\widehat{R}}_i$. In some applications where the weights are proportional, i.e. a = c · b for some constant c > 0, then the rankings by ${\widehat{R}}_i$ and ${\widehat{\text{Lfdr}}}_i$ are identical. Specifically ${\widehat{R}}_i$ is then monotone in ${\widehat{\text{Lfdr}}}_i$. Further, choosing the cutoff based on ${\widehat{N}}_i$ is equivalent to that of choosing by a weighted ${\widehat{\text{Lfdr}}}_i$. This leads to an Lfdr based procedure (Sun et al., 2015), which can be viewed as a special case of Procedure 1.

Procedure 2 (wFDR control with proportional weights). Rank hypotheses according to ${\widehat{\mathit{Lfdr}}}_i$ in increasing values. Denote the hypotheses and weights corresponding to ${\widehat{\mathit{Lfdr}}}_{(i)}$ by H(i) and a(i). Let

$$k=\max \left\{j:{\left({\displaystyle \sum _{i=1}^j{a}_{(i)}}\right)}^{-1}{\displaystyle \sum _{i=1}^j{a}_{(i)}}{\widehat{\mathit{Lfdr}}}_{(i)}\le \alpha \right\}.$$

Reject H_(i), for i = 1,...,k.

Next we investigate the asymptotic performance of Procedure 1. We first give some regularity conditions for the weights. Our theoretical framework requires that the decision weights must be obtained from external sources such as prior data, biological insights, or economical considerations. In particular, the observed data {X_i : i = 1,⋯ ,m} cannot be used to derive the weights. The assumption is not only crucial in theoretical developments, but also desirable in practice (to avoid using data twice). Therefore given the domain knowledge, the decision weights do not depend on observed values. Moreover, a model with random (known) weights is employed for technical convenience, as done in Genovese et al. (2006) and Roquain and van de Wiel (2009). We assume that the weights are independent with each other across testing units. Formally, denote e_i the external domain knowledge for hypothesis i, we require the following condition.

Condition 1 (i) $\left(a_i,b_i|X_i,{\theta }_i,e_i\right)\stackrel{d}{\sim }\left(a_i,b_i|e_i\right)$ for 1 ≤ i ≤ m. (ii) (a_i,b_i) and (a_j,b_j) are independent for i ≠ j.

In weighted multiple testing problems, the analysis is always carried out in light of the external information e_i implicitly. The notation of conditional distribution on e_i will be suppressed when there is no ambiguity. In practice, the weights a_i and b_i are usually bounded. We need a weaker condition in our theoretical analysis.

Condition 2 (Regularity conditions on the weights.) Let C and c be two positive constants. E(sup_i a_i) = o(m),E(sup_i b_i) = o(m), $E\left(a_i^4\right)\le C$, and min{E(a_i),E(b_i)} ≥ c.

A consistent Lfdr estimate is needed to ensure the large-sample performance of the data-driven procedure. Formally, we need the following condition.

Condition 3 $\widehat{{\mathit{Lfdr}}_i}-{\mathit{Lfdr}}_i=o_P(1)$. Also, $\widehat{{\mathit{Lfdr}}_i}\stackrel{d}{\to }Lfdr$, where Lfdr is an independent copy of Lfdr_i.

Remark 2 Condition 3 is a reasonable assumption in many applications. We give a few important scenarios where Condition 3 holds. Suppose we observe z-values from random mixture model

$$Z_i\sim N\left({\mu }_i,{\sigma }_i^2\right),\left({\mu }_i,{\sigma }_i\right)\sim F(\mu ,\sigma ),$$

where (μ_i,σ_i) = (μ₀,σ₀) if θ_i = 0, (μ_i,σ_i) ≠ (μ₀,σ₀) if θ_i = 1, and F(μ,σ) is a general bivariate distribution. Let p denote the proportion of non-null cases. Then ${\widehat{p}}_{jc}$, the estimator proposed in Jin and Cai (2007), satisfies ${\widehat{p}}_{jc}\stackrel{p}{\to }p$ under mild regularity conditions. Moreover, it is known that the kernel density estimator satisfies $E\Vert \widehat{f}-f{\Vert }^2\to 0$. It follows from Sun and Cai (2007) that Condition 3 holds when the above estimators are used. If the null distribution is unknown, then we can use the method in Jin and Cai (2007) to estimate the null parameters (μ₀,σ₀). Then under certain regularity conditions, we can show that $E\Vert {\widehat{f}}_0-f_0{\Vert }^2\to 0$, and Condition 3 holds. For the multi-group model (2.3), let ${\widehat{p}}_k,{\widehat{f}}_{k0}$, and ${\widehat{f}}_k$ be estimates of p_k, f_k0, and f_k such that ${\widehat{p}}_k\stackrel{p}{\to }{p}_k,E\Vert {\widehat{f}}_{k0}-f_{k0}{\Vert }^2\to 0,E\Vert {\widehat{f}}_k-f_k{\Vert }^2\to 0,$ k = 1,⋯ ,K. Let ${\widehat{\mathrm{Lfdr}}}_i=\left(1-{\widehat{p}}_k\right){\widehat{f}}_{0k}\left(x_i\right)/{\widehat{f}}_k\left(x_i\right)$ if $i\in {\mathcal{G}}_k$. It follows from Cai and Sun (2009) that Condition 3 holds when we apply Jin and Cai’s estimators and standard kernel estimates to the groups separately.

The oracle procedure (3.7) provides an optimal benchmark for all wFDR procedures. The next theorem establishes the asymptotic validity and optimality of the data-driven procedure by showing that the wFDR and ETP levels of the data-driven procedure converge to the oracle levels as m → ∞.

Theorem 2 Assume Conditions 1–3 hold. Denote by wFDR_DD the wFDR level of the data-driven procedure (Procedure 1). Let ETP_OR and ETP_DD be the ETP levels of the oracle procedure (3.7) and data-driven procedure, respectively. Then we have

• (i). wFDR_DD = α + o(1).
• (ii).ETP_DD/ETP_OR = 1 + o(1).

Corollary 1 Suppose we choose a_i = 1 for all i. Then under the conditions of the theorem, our data-driven procedure controls the unweighted FDR at level α + o(1).

5. Practical Issues and Numerical Results

The wFDR framework provides a useful approach to integrate domain knowledge in multiple testing. However, the weights must be chosen with caution to avoid improper manipulation of results. Under our formulation, we assume that the weights are pre-specified, while stressing that the task of assigning “correct” weights is a critical issue in practice. This section first states a few practical guidelines in weighted FDR analysis, then describes general strategies in choosing weights for a range of application scenarios. Finally we illustrate the proposed methodology via an application to GWAS.

5.1. Practical guidelines and examples on choosing weights

We first state a few important practical guidelines to ensure that the weighted FDR analysis is conducted properly. The first guideline is in particular important for a valid FDR control.

1. The weights must be “external” in the sense that they do not depend on the primary data such as the z-values or p-values on which the multiple tests are performed. As a practical guideline, we recommend “choosing external weights before seeing the data” as a standard rule in all weighted FDR analyses.

2. The choice of weights requires scientific motivations or economic considerations. For a statistical analysis involving any weighting schemes, the procedure for obtaining weights must be specified beforehand in the analysis plan, and needs be disclosed and justified carefully.

3. To prevent researchers from setting weights to find desired significant results, extreme weights are in general not acceptable. It is recommended to use moderate weights to incorporate domain knowledge and carry out a sensitivity analysis afterwards to avoid manipulations of results.

Next we discuss a range of scenarios that naturally give rise to weighted FDR analyses. We use these examples to illustrate how to assign weights properly in light of various economic and scientific considerations. Moreover, we emphasize how the above practical guidelines can be implemented in respective contexts to avoid manipulations of results.

Example 1. Spatial cluster analysis.

In multiple testing, relevant domain knowledge or external data can be exploited to form more meaningful testing units such as sets or clusters. For example, in a spatial setting (Pacifico et al. 2004; Benjamini and Heller 2007; Sun et al. 2015), individual locations can be aggregated into clusters to increase the signal to noise ratio and scientific interpretability. It is natural to consider weighted error rates when clusters are heterogeneous. We state a few guidelines for the choice of weights.

• (i)The weights should be pre-specified by experts to reflect various practical and economicconsiderations based on relevant spatial covariates such as cluster sizes, population densities, poverty rates, and urban vs. rural regions.
• (ii)In cluster-wise analyses, weights may be used to modify both the error rate and powerfunction. For example, symmetric weights (i.e. a_i = b_i for all i) may be assigned to reflect that the gains and losses in the decision process are proportional. A careful investigation of the optimal ranking of clusters (Section A in the Supplementary Material) is helpful to improve the power of analysis.

Example 2. Multiple endpoints clinical trials.

In comparing treatments with multiple endpoints (e.g. Dmitrienko et al. 2003), the error rates may be modified by assigning different weights to primary and secondary end points; this promises to increase the power in discovering more important clinical findings (Young and Gries 1984; Westfall and Young 1993). Benjamini and Cohen (2016) discussed two weighting regimes in clinical trials: the first regime requires that any primary endpoint carries a larger weight than any secondary one; and the second regime in addition requires that the combined weight of the primary endpoints is greater than the combined weight of the secondary ones. To avoid manipulations of results, Benjamini and Cohen (2016) suggested that a weighted FDR analysis should obey the following rules:

• (i)The tradeoffs between primary and secondary end points should be evaluated prior toconducting the trial.
• (ii)The weights should be given before seeing the data.
• (iii)The details of obtaining weights should be revealed in the trial protocol.

Example 3. Prioritized subset analysis.

In GWAS data analysis, finding diseasecausing SNPs in the immense data is very challenging. Weighted FDR analysis provides a powerful approach to integrate genomic knowledge in inference with prioritized SNP subsets. For example, Roeder et al. (2006) and Zablocki et al. (2014) assigned higher weights to pre-selected genomic regions to increase the power in detecting disease-associated SNPs that are more biologically plausible. In large-scale multiple testing with prioritized subset, we have the following requirements regarding the choice of weights:

• (i).It is not allowed to use the primary data (e.g. significant levels of disease association) to derive weights; such practice would distort the null distribution of p-values and lead to the failure of the wFDR procedure.
• (ii).The weights must be obtained based on either prior knowledge regarding the biological importance of an association, or external knowledge such as the linkage disequilibrium (LD) correlations between SNPs. This requirement can effectively avoid selection bias because the weights are conditionally independent from the significance levels of SNP associations (cf. our Condition 1). See also Section 5.(a) in Benjamini et al. (2009) for related discussions.
• (iii).To avoid further aggravating the multiplicity issue in large-scale testing problems, the weights should only be used to modify the power function. The unweighted FDR criterion is recommended to remain unchanged. This ensures greater power to reject more biologically relevant hypotheses without increasing the FDR (cf. our Corollary 1). See also Theorem 1 in Genovese et al. (2006) and Section 5.(a) in Benjamini et al. (2009) for related theoretical results and discussions.

5.2. Decision weights vs. procedural weights

The formulation (2.7) is different from the “procedural weights” approach, for example, in Roeder et al. (2006) and Zablocki et al. (2014), where the goal is to exploit external information by constructing weights to increase the total number of rejections (covariateassisted inference). The decision weights framework integrates the external information in a very different way; the goal is to improve the interpretability of results instead of statistical power (although the power may be gained as a byproduct, see Section E.5 in the Supplementary Material). Under the wFDR framework, the hypotheses are of “unequal” importance in light of domain knowledge. For instance, in spatial cluster analysis, a false positive cluster with larger size would account for a larger error; hence it is not sensible to use the conventional FDR definition that treats all testing units equally. The wFDR framework is different from those in Genovese et al. (2006), Ferkingstad et al. (2008) and Cai et al. (2016), where the gains and losses in various decisions are exchangeable, and conventional FDR and power definitions are suitable.

5.3. Application to Framingham Heart Study (FHS)

In this section, we implement the proposed method for analyzing a data set from Framingham Heart Study (Fox et al., 2007; Jaquish, 2007). A brief description of the study, the implementation of our methodology, and the results are discussed in turn.

The goal of the study is to decipher the genetic architecture behind the cardiovascular disorders for the Caucasians. Started in 1948 with nearly 5,000 healthy subjects, the study is currently in its third generation of the participants. The biomarkers responsible for the cardiovascular diseases, for e.g., body mass index (BMI), weight, blood pressure, and cholesterol level, were measured longitudinally. Since the mutation SNPs (within the block of DNA) are commonly passed on to descendants, it is a standard practice to collect data over several generations. The specific goal of our study is to identify disease-associated SNPs in the second generation while paying more attention to those that remain stable over the first two generations. Identifying significant SNPs (or SNP blocks) that are preserved across generations can improve both biological plausibility and statistical replicability. In our study design, we use 399 subjects in the first generation to serve as the baseline to construct weights, and use the information from the 578 subjects in the second generation (with 310 males and 268 females) to conduct the primary analysis. It is important to note that the goal of our weighted FDR analysis is different from that in Roeder et al. (2006) and Zablocki et al. (2014), where prior likelihood of association with the phenotype is utilized for constructing “procedural weights” to improve the unweighted power. In contrast, we use decision weights to modify the power function so that discoveries in certain preselected regions receive a higher priority. The main consideration is not to increase the total number of discoveries, but to focus on findings are more relevant to our study goal (identifying stable mutations across generations). In our study it is not sensible to use the conventional power definition where the discoveries from all regions are treated equally.

We consider the BMI as the response variable and develop a dynamic model to detect the SNPs associated with the BMI. Let Y_i(t_ij) denote the response (BMI) from the i-th subject at time t_ij, j = 1,...,T_i. Consider the following model for longitudinal traits:

$$Y_i\left(t_{ij}\right)=f\left(t_{ij}\right)+{\beta }_k{G}_{ik}+{\gamma }_{i0}+{\gamma }_{i1}{t}_{ij}+{ϵ}_i\left(t_{ij}\right),$$ (5.1)

where f(·) is the general effect of time that is modeled by a polynomial function of suitable order, selected by AIC or BIC model selection criterion, β_k is the effect of the k-th SNP on the response and G_ik denotes the genotype of the i-th subject for the k-th SNP. We also consider the random intercepts and random slopes, denoted γ_0i and γ_1i, respectively, for explaining the subject-specific response trajectories. A bivariate normal distribution for γ_i = (γ_0i,γ_1i) is assumed. Moreover, we assume that the residual errors are normally distributed with zero mean, and covariance matrix Σ_i with an order-one auto-regressive structure. We fit model (5.1) for each SNP and obtain the estimate of the genetic effect ${\widehat{\beta }}_k$. If we reject the null hypothesis H₀ : β_k = 0 vs. H₁ : β_k ≠ 0, then we conclude that the k-th SNP has a significant association with the BMI. Since we have nearly 5 million SNPs, the false discovery rate needs to be controlled for making scientifically meaningful inference. For each k, we take standardized ${\widehat{\beta }}_k$ as our z-scores and obtain the estimated ranking statistic ${\widehat{R}}_k$ as described in (4.2).

To incorporate external domain knowledge, we construct weights by utilizing the pvalues obtained from first generation participants. This information is secondary in the sense that the first generation data are not directly used in multiple testing but are only employed to provide supplementary support in inference. In the FHS study, there are 399 subjects in the first generation for which we have obtained the trait values (such as BMI) and genetic information on the same set of SNPs. Using the p-values obtained from the first generation data, we partition the SNPs into three groups: less than 0.001, between 0.001 and 0.01, and greater than 0.01. The groups are denoted ${\mathcal{G}}_j$, j = 1,2,3. We implement the proposed wFDR method using the p-values computed from second generation participants, with group-wise weights b_i = c_j if i ∈ ${\mathcal{G}}_j$. Note that b_i are the weights in respective power functions with a larger b_i indicating a higher priority or a stronger belief. Since assigning a value to the biological importance of an association can be quite subjective, we propose to use different combinations of weights to investigate the sensitivity of testing results:

• Setting 1: a_i = 1 for all i, b_i = c_j if i ∈ G_j, (c₁,c₂,c₃) = (1,1,1)

• Setting 2: a_i = 1 for all i, b_i = c_j if i ∈ G_j, (c₁,c₂,c₃) = (4,2,1)

• Setting 3: a_i = 1 for all i, b_i = c_j if i ∈ G_j, (c₁,c₂,c₃) = (10,5,1).

Setting 1 corresponds to an unweighted FDR analysis. Settings 2 and 3 reflect our belief that the correct rejections from groups 1 and 2 are more “powerful” or more “relevant” when compared to those from group 3. This is sensible as one of the goals in the FHS study is to discover significant SNPs that remain stable across generations. In our analysis, the weights a_i = 1 are chosen for all SNPs; this ensures that the FDR will not be increased by weighting. We only modify the weights b_i in the power functions to reflect that discoveries in certain pre-selected regions are of greater importance. In practice b_i may be derived from expert opinion to incorporate scientific or economic considerations. We stress that in GWAS, the details of the process in choosing weights, which requires a separate discussion or justification, should be revealed.

We implement the proposed wFDR method to select most significant SNPs. The results for the sensitivity analysis with different choices of weights are summarized in Table 1. The table shows the groups sizes, the group-wise threshold levels, and the number of SNPs selected from different “groups” for each weight combination at two different FDR levels α = 0.10 and α = 0.05.

Table 1:

Number of rejections and thresholds by varying choice of weights between groups.

Choice of Weights
Group	Total	(1, 1, 1)	Threshold	(4, 2, 1)	Threshold	(10, 5, 1)	Threshold
α = 0.10
${\mathcal{G}}_1$	1043	3	6.69 × 10−4	5	1.70 × 10−3	53	3.17 × 10−2
${\mathcal{G}}_2$	5277	23	9.01 × 10−4	33	1.52 × 10−3	57	4.53 × 10−3
${\mathcal{G}}_3$	220392	2019	2.92 × 10−3	1997	2.88 × 10−3	1663	2.31 × 10−3
Overall	226712	2045	–	2035	–	1773	–
α = 0.05
${\mathcal{G}}_1$	1043	1	5.22 × 10−6	2	3.95 × 10−4	4	7.43 × 10−4
${\mathcal{G}}_2$	5277	13	4.22 × 10−4	19	5.20 × 10−4	21	6.61 × 10−4
${\mathcal{G}}_3$	220392	159	1.17 × 10−4	147	1.06 × 10−4	131	1.01 × 10−4
Overall	226712	173	–	168	–	156	–

From Table 1 we can see that the unweighted analysis [i.e. the weights combination (1, 1, 1)] consistently selects higher number of SNPs than the other two weights combinations. Meanwhile, it selects fewer number of SNPs from groups 1 and 2. Here we have aimed to prioritize certain pre-selected regions and discover significant SNPs that are more biologically relevant. In our analysis, groups 1 and 2 are believed to be more “informative.” At α = 0.10, it is interesting to note that the number of rejections from groups 1 and 2 are increased by several folds but the number of rejections from group 3 is only decreased by a small proportion.

In Table 1, we also report the group-wise thresholds for the p-values. The value of the threshold is computed as the maximum of the p-values of the hypotheses rejected in the specific group and weight combination. We can see that the threshold for group 3 decreases in weights, which indicates that the rejection criteria become more stringent with higher weights. In contrast, the thresholds for groups 1 and 2 increase with weights, which allows for more discoveries in pre-selected regions. All these above observations are sensible and in agreement with the intuitions of our proposed methodology.

5.4. Simulation

In all simulation studies, we consider a two-point normal mixture model

$$X_i\sim (1-p)N(0,1)+pN\left(\mu ,{\sigma }^2\right),i=1,\cdots ,m.$$ (5.2)

The nominal wFDR is fixed at α = 0.10. We consider the comparison of different methods under the scenario where there are two groups of hypotheses and within each group the weights are proportional.

The proposed method (Procedure 1 in Section 4.2) is denoted by 4 DD. Other methods to be compared include:

1. The wFDR method proposed by Benjamini and Hochberg (1997); denoted by 2 BH97. In simulations where a_i = 1 for all i, BH97 reduces to the well-known step-up procedure in Benjamini and Hochberg (1995), denoted by BH95.

2. A stepwise wFDR procedure, which rejects hypotheses along the WPO (3.14) ranking sequentially and stops at $k=\max \left\{j:{\displaystyle \sum _{i=1}^j{\widehat{N}}_{(i)}}\le 0\right\}$, with ${\widehat{N}}_{(i)}$ defined in Section 4.2. The method is denoted by ◦ WPO. Following similar arguments in the proof of Theorem 2, we can show that the WPO method controls the wFDR at the nominal level asymptotically. This is also verified by our simulation results. Meanwhile, we expect that the WPO method will be outperformed by the proposed method (4 DD), which operates along the more efficient VCR ranking.

3. The adaptive z-value method in Sun and Cai (2007), denoted by + AZ. AZ is valid and optimal in the unweighted case but suboptimal in the weighted case.

To save space, this section only presents results on group-wise weights. Our setting is motivated by our application to GWAS, where the hypotheses can be divided into groups: those in preselected regions and those in other regions. It is desirable to assign varied weights to separate groups to reflect that the discoveries in preselected regions are more biologically meaningful. In Section E.2 of the Supplementary Material, we compare our methods with existing methods using general weights a_i and b_i that are generated from probability distributions. We also provide additional numerical results such as the comparison of various wFDR definitions, the finite sample performance of Lfdr, and the impacts of weights on the power of different wFDR procedures.

The first simulation study investigates the effect of weights. Consider two groups of hypotheses with group sizes m₁ = 3000 and m₂ = 1500. In both groups, the non-null proportion is p = 0.2. The null and non-null distributions are N(0,1) and N(1.9,1), respectively. We fix a_i = 1 for all i. Hence BH97 reduces to the method proposed in Benjamini and Hochberg (1995), denoted by BH95. The wFDR reduces to the regular FDR, and all methods being considered are valid for FDR control. For hypotheses in group 1, we let c₁ = a_i/b_i. For hypotheses in group 2, we let c₂ = a_i/b_i. We choose c₁ = 3 and vary c₂. Hence the weights are proportional within respective groups and vary across groups. The second simulation study investigates the impacts of signal strength. The results are presented in Section E.1 of the Supplementary material.

In the simulation, we apply the four methods above to the simulated data set and obtain the wFDR and ETP levels by averaging the multiple testing results over 200 replications. In Figure 1, we plot the wFDR levels and ETP of different methods as functions of c₂, which is varied over [0.1,0.8]. Panel (a) shows that all methods control the wFDR under the nominal level, and the BH97 method is conservative. Panel (b) shows that the proposed method dominates all existing methods. The proposed method is followed by the WPO method, which outperforms all unweighted methods (AZ and BH95) since b_i, the weights in the power function, are incorporated in the testing procedure. The BH97 (or BH95) has the smallest ETP. As c₂ approaches 1 or the weights a_i and b_i equalizes, the relative difference of the various methods (other than BH95) becomes less.

Figure 1:

Comparison under group-wise weights: □ BH97 (or BH95), o WPO, Δ DD (proposed), and + AZ. The efficiency gain of the proposed method increases as c₁ and c₂ become more distinct.

6. Discussion

The FDR provides a practical and powerful approach to large-scale multiple testing problems and has been widely used in a wide range of scientific studies. The wFDR framework extends the FDR paradigm to integrate useful domain knowledge in simultaneous inference. The weights must be chosen with caution to avoid improper manipulation of results.

In the multiple testing literature, procedural, decision, and class weights are often viewed as distinct weighting schemes and have been mostly investigated separately. Although this paper focuses on the decision weights approach, the decision-theoretic framework enables a unified investigation of other weighting schemes. For example, a comparison of the LR (3.8) and WLR (3.11) demonstrates how the LR statistic may be adjusted optimally to account for the decision gains or losses. This shows that procedural weights may be derived in the decision weights framework. Moreover, the difference between the WLR (3.11) and WPO (3.14) shows the important role that p_i plays in multiple testing. In particular the WPO (3.14) provides important insights on how prior beliefs may be incorporated in a decision weights approach to derive appropriate class weights. To see this, consider the multi-class model (2.3). Following the arguments in Cai and Sun (2009), we can conclude that in order to maximize the power, different FDR levels should be assigned to different classes. Similar suggestions for varied class weights have been made in Westfall and Young (1993, pages 169 and 186). These examples demonstrate that the decision weights approach provides a powerful framework to derive both procedural weights and class weights.

Our formulation requires that the weights must be pre-specified based on external domain knowledge. It is of interest to extend the work to the setting where the weights are unknown. Due to the variability in the quality of external information, subjectivity of investigators, and complexity in modeling and analysis, a systematic study of the issue is beyond the scope of the current paper. The optimal choice of weights depends on statistical, economic and scientific concerns jointly. Notable progresses have been made, for example, in Roeder and Wasserman (2009) and Roquain and van de Wiel (2009). However, these methods are mainly focused on the weighted p-value approach under the unweighted FDR criterion, hence do not apply to the framework in Benjamini and Hochberg (1997). Moreover, the optimal decision rule in the wFDR problem in general is not a thresholding rule based on the adjusted p-values. Much work is still needed to derive decision weights that would optimally incorporate domain knowledge in large-scale studies.

7. Proofs

This section proves Theorem 1. The proofs of other results are given in the Supplementary Material.

Proof of Part (i) of Theorem 1.

To show that wFDR(δ_OR) = α, we only need to establish hat

$$E_{U,\mathit{a},\mathit{b},\mathit{X}}\left\{{\displaystyle \sum _{i=1}^m{a}_i}{\delta }_{OR}^i\left({\mathrm{Lfdr}}_i-\alpha \right)\right\}=0,$$

where the notation E_U,a,b,X denotes that the expectation is taken over U,a,b, and X. According to the definitions of the capacity function C(·) and threshold t^∗, we have

$${\displaystyle \sum _{i=1}^m{a}_i}{\delta }_{OR}^i\left({\mathrm{Lfdr}}_i-\alpha \right)=C(k)+I\left(U<p^{*}\right)\{C(k+1)-C(k)\}.$$

It follows from the definition of p^* that $E_{U|\mathit{a},\mathit{b},\mathit{X}}\left\{{\sum }_{i=1}^m{a}_i{\delta }_{OR}^i\left({\mathrm{Lfdr}}_i-\alpha \right)\right\}=C(k)+\{C(k+1)-C(k)\}{p}^{*}=0$, where the notation E_U|a,b,X indicates that the expectation is taken over U while holding (a,b,X) fixed. Therefore

$$E_{U,\mathit{a},\mathit{b},\mathit{X}}\left\{{\displaystyle \sum _{i=1}^m{a}_i}{\delta }_{OR}^i\left({\mathrm{Lfdr}}_i-\alpha \right)\right\}=0,$$ (7.1)

and the desired result follows.

Proof of Part (ii) of Theorem 1.

Let δ^* be an arbitrary decision rule such that wFDR(δ^∗) ≤ α. Then

$$E_{\mathit{a},\mathit{b},\mathit{X}}\left\{{\displaystyle \sum _{i=1}^m{a}_i}E\left({\delta }_i^{*}|\mathit{a},\mathit{b},\mathit{x}\right)\left({\mathrm{Lfdr}}_i-\alpha \right)\right\}\le 0.$$ (7.2)

The notation $E\left({\delta }_i^{*}|\mathit{x},\mathit{a},\mathit{b}\right)$ means that the expectation is taken to average over potential randomization conditional on the observations and weights.

Let ${\mathcal{I}}^{+}=\left\{i:{\delta }_{OR}^i-E\left({\delta }_i^{*}|\mathit{x},\mathit{a},\mathit{b}\right)>0\right\}$ and ${\mathcal{I}}^{-}=\left\{i:{\delta }_{OR}^i-E\left({\delta }_i^{*}|\mathit{x},\mathit{a},\mathit{b}\right)<0\right\}$. For i ∈ ${\mathcal{I}}^{+}$, we have ${\delta }_{OR}^i=1$ and hence b_i(1 − Lfdr_i) ≥ t^∗a_i(Lfdr_i − α). Similarly for i ∈ ${\mathcal{I}}^{-}$, we have ${\delta }_{OR}^i=0$ and so b_i(1 − Lfdr_i) ≤ t^∗a_i(Lfdr_i − α). Thus

$${\displaystyle \sum _{i\in {\mathcal{I}}^{+}\cup {\mathcal{I}}^{-}}\left\{{\delta }_{OR}^i-E\left({\delta }_i^{*}|\mathit{x},\mathit{a},\mathit{b}\right)\right\}}\left\{b_i\left(1-{\mathrm{Lfdr}}_i\right)-t^{*}{a}_i\left({\mathrm{Lfdr}}_i-\alpha \right)\right\}\ge 0.$$

Note that ${\delta }_{OR}^i$ is perfectly determined by X except for (k + 1)th decision. Meanwhile,b_(k=0)(1-Lfdr_(k+1) – t*a_(k+1) (Lfdr_(k+1) –α)=0 by our choice of t^∗. It follows that

$$E_{\mathit{a},\mathit{b},\mathit{X}}\left[{\displaystyle \sum _{i=1}^m\left\{E\left({\delta }_{OR}^i|\mathit{x},\mathit{a},\mathit{b}\right)-E\left({\delta }_i^{*}|\mathit{x},\mathit{a},\mathit{b}\right)\right\}}\left\{b_i\left(1-{\mathrm{Lfdr}}_i\right)-t^{*}{a}_i\left({\mathrm{Lfdr}}_i-\alpha \right)\right\}\right]\ge 0.$$ (7.3)

Recall that the power function is given by $\mathrm{ETP}(\mathit{\delta })=E\left\{{\sum }_{i=1}^{m}E\left({\delta }_i|\mathit{x},\mathit{a},\mathit{b}\right)b_i\left(1-{\mathrm{Lfdr}}_i\right)\right\}$ for any decision rule δ. Combining equations (7.1) – (7.3) and noting that t^∗ > 0, we claim that ETP(δ_OR) ≥ ETP(δ^∗) and the desired result follows.