Directional false discovery rate control in large-scale multiple comparisons

Abstract

The advance of high-throughput biomedical technology makes it possible to access massive measurements of gene expression levels. An important statistical issue is identifying both under-expressed and over-expressed genes for a disease. Most existing multiple-testing procedures focus on selecting only the non-null or significant genes without further identifying their expression type. Only limited methods are designed for the directional problem, and yet they fail to separately control the numbers of falsely discovered over-expressed and under-expressed genes with only a unified index combining all the false discoveries. In this paper, based on a three-classification multiple testing framework, we propose a practical data-driven procedure to control separately the two directions of false discoveries. The proposed procedure is theoretically valid and optimal in the sense that it maximizes the expected number of true discoveries while controlling the false discovery rates for under-expressed and over-expressed genes simultaneously. The procedure allows different nominal levels for the two directions, exhibiting high flexibility in practice. Extensive numerical results and analysis of two large-scale genomic datasets show the effectiveness of our procedure.

1. Introduction

In bioinformatics such as gene expression microarray experiments and brain image studies, it is often crucial to study each gene or brain location and decide whether it is ‘significant’ or not. In these studies, the number of genes or brain locations is very large. If each gene or brain location is associated with one hypothesis, then a large number of hypotheses are tested simultaneously, which is referred to as large-scale multiple testing in the literature. Since the pioneering work of [3] that introduced the concept of controlling the false discovery rate (FDR), there has been an increasing amount of literature on multiple testing in recent years. See, for example, [1,2,6,8,14,19,24,27] and the references therein.

Conventionally, various multiple testing methods have been proposed for dividing genes into null and non-null genes. However, in many bioinformatics applications, such approach may be insufficient to meet the need of effective gene diagnosis, and it is often required to make further directional decisions for the selected non-null genes. For instance, in a typical comparative genomics studies with high-throughput technologies, the dataset is obtained from comparing the expression levels of genes for the healthy and cancer subjects [7,12]. The goal of the study is not only to isolate significant genes related to cancer but also to distinguish between under-expressed and over-expressed genes. Another example can be found in financial market, where monthly returns from a large number of mutual funds are routinely collected. Investors are interested in selecting skilled mutual funds with true capabilities to make profits by using the well-known alpha factor. By convention, a positive alpha indicates skilled performance, a negative alpha indicates unskilled performance, and a zero alpha indicates no fluctuations. As a result, it is important to accurately identify skilled fund managers and unskilled fund managers [28]. These inspire us to investigate such directional multiple testing problems with directional alternatives.

It should be noted that conventional multiple testing methods designed for binary classification can also be applied to the considered directional problem by making directional decisions according to the sign of the test statistics. However, as pointed out by [15], the BH procedure even some advanced FDR-control methods such as the robust FDR method of [4], the q-value of [22,23], and the empirical Bayes estimate of [11] would result in higher FDR in one direction once the null distribution is highly skewed. There has been some work focusing on the directional multiple testing problem [16,18,20,29]. However, these methods mainly focus on controlling the type III errors that misclassify under-expressed genes as over-expressed or vice-versa based on one single directional FDR index, which may be inappropriate especially in cases when one direction of the two alternatives is more important than the other. The main reason is that even if the defined directional FDR is controlled, the separate error rate in each direction may not be well controlled. Also, such a framework only involves a unified nominal type III error, and does not allow different nominal errors for each direction separately.

In this paper, we propose a directional multiple testing approach for controlling the directional FDRs simultaneously in both directions. Based on a three-classification multiple testing framework, we first give the definitions of the directions FDRs for both directions (e.g. under expressed and over expressed genes). Oracle and data-driven procedures are proposed to maximize the expected number of true rejections while controlling the two individual directional FDRs at nominal levels specified beforehand. Theoretically, we show that our procedure is valid and optimal for the considered directional multiple testing problem with simultaneous FDR-control. Extensive numerical results also show the effectiveness of our proposed methods for directional FDR-control, especially in cases with different separate nominal levels of the two directions as well as different non-null proportions and strengths.

The rest of this article is organized as follows. Section 2 formulates the directional multiple testing problem for controlling two directional FDRs simultaneously. Section 3 introduces the proposed oracle and data-driven procedures and establishes their validity and optimality for directional FDR control. Section 4 presents extensive simulation results. The analysis of two microarray datasets are given in Section 5. Section 6 concludes this article. Theoretical details are provided in the Appendix.

2. Problem formulation

Suppose $\mathit{X}=(X_1,\dots ,X_m)$ are independent and identically distributed (i.i.d.) observations from a random mixture model:

$$X_i={\mu }_i+{\epsilon }_i,$$ (1)

where ${\mu }_i$ is the unknown parameter of interest and ${\epsilon }_i\sim N(0,{\sigma }^2)$. We further assume that ${\mu }_i$s are generated from the following mixture model

$$\begin{array}{c}{\mu }_i\mid {\theta }_i\sim ((1-|{\theta }_i|){\delta }_0({\mu }_i)+I({\theta }_i=1)h_1({\mu }_i)+I({\theta }_i=-1)h_2({\mu }_i)\\ {\displaystyle {\theta }_i\stackrel{i.i.d}{\sim }\mathrm{Multinoulli}(p_0,p_1,p_{-1}),\sum _{k=0,1,-1}{p}_k=1,i=1,\dots ,m}\end{array}$$ (2)

where $\mathit{\theta }=({\theta }_1,\dots ,{\theta }_m)\in \{0,1,-1{\}}^m$ denotes the set of states (i.e. null/over-expressed/under-expressed) of each gene with $p_0/p_1/p_{-1}$ being the corresponding probabilities, and $h_1(\cdot )$ and $h_2(\cdot )$ are the density functions for respectively the over- and under-expressed states with supports $(0,+\mathrm{\infty })$ and $(-\mathrm{\infty },0)$. Also, ${\delta }_0(\cdot )$ is the dirac delta function whose value is zero everywhere except at zero, and whose integral over the entire real line is equal to one. Note that variations of model (1)–(2) have been widely applied in the field of large-scale multiple testing [5,10,19,24,25]. Conventional multiple testing problems primarily focus on the nondirectional hypotheses $H_{i,0}:{\mu }_i=0$ vs. $H_{i,1}:{\mu }_i\ne 0$, $i=1,\dots ,m$, or equivalently, $H_{i,0}:{\theta }_i=0$ vs. $H_{i,1}:{\theta }_i\ne 0,i=1,\dots ,m$.

In this paper, in order to distinguish over-expressed genes from under-expressed ones, we consider $H_{i,0}:{\mu }_i=0$ vs. $H_{i,1}:{\mu }_i>0$ vs. $H_{i,-1}:{\mu }_i<0$, $i=1,\dots ,m$, which is essentially equivalent to the following three-classification multiple testing problem

$$H_{i,0}:{\theta }_i=0\mathrm{vs}.H_{i,1}:{\theta }_i=1\mathrm{vs}.H_{i,-1}:{\theta }_i=-1,i=1,\dots ,m.$$ (3)

Denote by $\mathit{\delta }=({\delta }_1,\dots ,{\delta }_m)\in \{0,1,-1{\}}^m$ a set of decision rules for problem (3), where ${\delta }_i=0,$ 1 or -1 implies respectively that the ith gene is claimed as null, over- or under-expressed. Based on $\mathit{\delta }$ and $\mathit{\theta }$, the results of multiple testing can be summarized as in Table 1. Various indies can then be defined for error control. For instance, the total marginal false discovery rate (FDR) is defined as

$$\mathrm{tmFDR}=\frac{E(N_{01}+N_{21}+N_{02}+N_{12})}{E(R_1+R_2)}.$$

The quantity is equivalent to the standard marginal FDR in the binary classification problem. Similarly, the type III error rate can also be defined as $\frac{E(N_{21}+N_{12})}{E(R_1+R_2)}$.

Table 1.

Classification of tested hypothesis.

	${\delta }_i=0$	${\delta }_i=1$	${\delta }_i=-1$	Total
${\theta }_i=0$	$N_{00}$	$N_{01}$	$N_{02}$	$m_0$
${\theta }_i=1$	$N_{10}$	$N_{11}$	$N_{12}$	$m_1$
${\theta }_i=-1$	$N_{20}$	$N_{21}$	$N_{22}$	$m_2$
Total	$R_0$	$R_1$	$R_2$	m

In practice, we are often interested in controlling false discoveries of $H_{i,-1}$ and $H_{i,1}$ separately, a goal that the tmFDR and type III error fail to achieve. For instance, controlling the tmFDR can only ensure that the total error rate is controlled, and it is very possible that one of the false discoveries of $H_{i,-1}$ and $H_{i,1}$ is not controlled well, especially for skewed data. To solve this problem, we define respectively the marginal FDR of over- and under-expressed genes as

$$\mathrm{mFD}{\mathrm{R}}_1=\frac{E\{\sum _{i=1}^{m}I({\theta }_i\ne 1)I({\delta }_i=1)\}}{E[\sum _{i=1}^{m}I({\delta }_i=1)]}=\frac{E(N_{01}+N_{21})}{E(R_1)}$$

and

$$\mathrm{mFD}{\mathrm{R}}_{-1}=\frac{E\{\sum _{i=1}^{m}I({\theta }_i\ne -1)I({\delta }_i=-1)\}}{E[\sum _{i=1}^{m}I({\delta }_i=-1)]}=\frac{E(N_{02}+N_{12})}{E(R_2)}.$$

By controlling the $\mathrm{mFD}{\mathrm{R}}_1$ and $\mathrm{mFD}{\mathrm{R}}_{-1}$ at satisfactory levels simultaneously, one can achieve effective error control for both $H_{i,-1}$ and $H_{i,1}$.

To measure the power of $\mathit{\delta }$, we also define the total expected number of true positives (ETP) as

$$\mathrm{ETP}=E\{\sum _{i=1}^m[(I({\theta }_i=1)I({\delta }_i=1)+I({\theta }_i=-1)I({\delta }_i=-1))]\},$$

which equals the total number of tests that are correctly identified by $\mathit{\delta }$ into each direction. Hence, our considered problem can be formulated into a constrained optimization problem:

$$\mathrm{find}\mathrm{the}\mathit{\delta }\mathrm{that}\mathrm{maximize}\mathrm{ETP}\mathrm{subject}\mathrm{to}\{\begin{array}{l}\mathrm{mFD}{\mathrm{R}}_1\le {\alpha }_1\\ \mathrm{mFD}{\mathrm{R}}_{-1}\le {\alpha }_{-1}\end{array}$$ (4)

for given error levels $0<{\alpha }_1,{\alpha }_{-1}<1$.

Remark 2.1

Besides the proposed directional mFDR, variations of FDR can be employed to measure misclassification error. For example, the directional FDR can be defined as

$$\mathrm{FD}{\mathrm{R}}_1=E\left\{\frac{N_{01}+N_{21}}{max(R_1,1)}\right\},\mathrm{FD}{\mathrm{R}}_{-1}=E\left\{\frac{N_{02}+N_{12}}{max(R_2,1)}\right\}.$$

Results from [13] imply that $\mathrm{FD}{\mathrm{R}}_1=\mathrm{mFD}{\mathrm{R}}_1+O(m^{-1/2})$ and $\mathrm{FD}{\mathrm{R}}_{-1}=\mathrm{mFD}{\mathrm{R}}_{-1}+O(m^{-1/2})$, which means that the two measures are asymptotically equivalent. Following previous literatures, marginal FDR will be used in this paper for technical convenience when obtaining optimality results.

Remark 2.2

Another advantage of problem (4) is that the different ${\alpha }_1$ and ${\alpha }_{-1}$ enable fine control over the FDR of different directions. For example, if one direction is more important than the other, it may be necessary to choose a more stringent ${\alpha }_k$ when classifying genes into that direction. In cases when it is unclear how to choose the ${\alpha }_k$s individually, it is easy to show that the optimal decision rule of problem (4) is also valid for controlling the tmFDR at level $\alpha =\underset{k}{max}{\alpha }_k$.

3. Methodology

3.1. The oracle procedure

Define $T_k(X_i)=P({\theta }_i\ne k|X_i)$, k = 0, 1, −1. Then under model (1)–(2), problem (4) is equivalent to maximizing

$$\begin{array}{rl}\mathrm{ETP}& =E\left\{E\{\sum _{i=1}^m[(I({\theta }_i=1)I({\delta }_i=1)+I({\theta }_i=-1)I({\delta }_i=-1))]\mid \mathit{X}\}\right\}\\ & =E\{\sum _{i=1}^{m}I({\delta }_i=1)P({\theta }_i=1\mid X_i)+I({\delta }_i=-1)P({\theta }_i=-1\mid X_i)\}\\ & =E\{\sum _{i=1}^{m}I({\delta }_i=1)(1-T_1(X_i))+I({\delta }_i=-1)(1-T_{-1}(X_i))\}\end{array}$$

subject to

$$\{\begin{array}{l}E\{\sum _{i=1}^m[I({\delta }_i=1)(T_1(X_i)-{\alpha }_1)]\}\le 0\\ E\{\sum _{i=1}^m[I({\delta }_i=-1)(T_{-1}(X_i)-{\alpha }_{-1})]\}\le 0.\end{array}$$

By Lagrange multiplier technique, problem (4) can be solved by minimizing the following unconstrained optimization problem with penalized objective function

$$L(\mathit{\delta },\mathit{\lambda })=\sum _{i=1}^m\sum _{k=1,-1}[I({\delta }_i\ne k)(1-T_k(X_i))+{\lambda }_{k}I({\delta }_i=k)(T_k(X_i)-{\alpha }_k)],$$

since any $\mathit{\delta }$ that minimizes $L(\mathit{\delta },\mathit{\lambda })$ conditionally on the observed test statistics will also minimize $E\{L(\mathit{\delta },\mathit{\lambda })\}$, where $\mathit{\lambda }=({\lambda }_1,{\lambda }_{-1})$ are the penalty parameters. Note that ${\lambda }_1$ and ${\lambda }_{-1}$ can be considered as the cost of missed discovery relative to false positive, in which $L(\mathit{\delta },\mathit{\lambda })$ can be viewed as a generalization of the loss function in the compound decision theory [24].

Given ${\lambda }_1,{\lambda }_{-1}>0$, denote by ${\mathit{\delta }}^{\mathit{\lambda }}=\{{\delta }_1^{\mathit{\lambda }},\dots ,{\delta }_m^{\mathit{\lambda }}\}$ the minimizer of $L(\mathit{\delta },\mathit{\lambda })$, and by

$${\delta }_i^{\mathit{\lambda }}=\{\begin{array}{ll}k,& \text{if}{R}_{k,i}\le min(0,R_{k^{\prime },i}),k,k^{\prime }=\pm 1,k\ne k^{\prime }\\ 0,& \text{otherwise.}\end{array}$$ (5)

its ith element, where $R_{k,i}={\lambda }_k\{T_k(X_i)-{\alpha }_k\}-\{1-T_k(X_i)\}$. The following proposition describes the behavior of ${\mathit{\delta }}^{\mathit{\lambda }}$.

Proposition 3.1

Consider model (1)–(2) and suppose that $\underset{k}{min}{p}_k>0$. Then, for ${\mathit{\delta }}^{\mathit{\lambda }}$ in (5):

• (a) ${\mathit{\delta }}^{\mathit{\lambda }}$ minimizes $E\{L(\mathit{\delta },\mathit{\lambda })\};$
• (b)
  define $N_k(\mathit{\lambda })=E[I({\delta }_i^{\mathit{\lambda }}=k)\{T_k(X_i)-{\alpha }_k\}]$ and

  $${\check{\lambda }}_{k,t}=inf\{\lambda \le {\check{\lambda }}_{k,t-1}:N_k({\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{\mathit{k}\mathbf{,}\mathit{t}\mathbf{-}\mathbf{1}})\le 0\}k=1,-1,$$

  where $t\ge 1$, ${\check{\lambda }}_{k,0}=\mathrm{\infty }$, and ${\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{\mathit{k}\mathbf{,}\mathit{t}\mathbf{-}\mathbf{1}}$ is the $\mathit{\lambda }$ with ${\lambda }_k=\lambda$ and ${\lambda }_{k^{\prime }}={\check{\lambda }}_{k^{\prime },t-1},k^{\prime }\ne k.$ Suppose that ${\alpha }_1+{\alpha }_{-1}\le 1$ and $\mathbf{0}\in \{(N_1(\mathit{\lambda }),N_{-1}(\mathit{\lambda })):{\lambda }_1\ge 0,{\lambda }_2\ge 0\}$. Then the sequence $\{{\check{\lambda }}_{k,t}\}$ is convergent, and $N_k({\mathit{\lambda }}^{\ast })=0$ for $k=1,-1,$ where ${\mathit{\lambda }}^{\ast }=({\lambda }_1^{\ast },{\lambda }_{-1}^{\ast })$ and ${\lambda }_k^{\ast }=\underset{t\to \mathrm{\infty }}{lim}{\check{\lambda }}_{k,t}.$

Remark 3.1

The condition $\underset{k}{min}{p}_k>0$ in Proposition 3.1 ensures that the proportion of features in each direction is non-zero. The quantity $N_k(\mathit{\lambda })$ derived from the constraint on ${\mathrm{mFDR}}_k$ can be seen as a measure of the unused directional misclassification error by ${\mathit{\delta }}^{\mathit{\lambda }}$ to maximize the number of discoveries. The condition $\mathbf{0}\in \{(N_1(\mathit{\lambda }),N_{-1}(\mathit{\lambda })):{\lambda }_1\ge 0,{\lambda }_2\ge 0\}$ ensures that there exists some $\mathit{\lambda }$ such that ${\mathrm{mFDR}}_k({\mathit{\delta }}^{\mathit{\lambda }})$ exactly attains ${\alpha }_k$ for k = 1, −1. The condition ${\alpha }_1+{\alpha }_{-1}\le 1$ is mild as it allows a wide range of ${\alpha }_k$, for example, $0\le {\alpha }_k\le 1.$

Now, the oracle procedure for the directional error control problem (4) can be defined. The following Theorem 3.1 gives the definition of the procedure and shows its validity and optimality for directional mFDR-control. Here, validity means that our procedure is able to control the two directional FDRs at desired levels, and optimality means that the ETP of our procedure is the smallest among all valid procedures.

Theorem 3.1

Consider model (1)–(2) and suppose that $\underset{k}{min}{p}_k>0$. With ${\mathit{\lambda }}^{\ast }$ given in Proposition 3.1, define

$${\mathit{\delta }}^{{\mathit{\lambda }}^{\ast }}=\{{\delta }_1^{{\mathit{\lambda }}^{\ast }},\dots ,{\delta }_m^{{\mathit{\lambda }}^{\ast }}\}.$$

If ${\alpha }_1$ and ${\alpha }_{-1}$ satisfy the conditions in Proposition 3.1(b), then:

• (a) ${\mathrm{mFDR}}_k({\mathit{\delta }}^{{\mathit{\lambda }}^{\ast }})={\alpha }_k,k=1,-1$;
• (b)
  for any $\mathit{\delta }$ satisfying ${\mathrm{mFDR}}_k(\mathit{\delta })\le {\alpha }_k,k=1,-1$,

  $$\mathrm{ETP}({\mathit{\delta }}^{{\mathit{\lambda }}^{\ast }})\ge \mathrm{ETP}(\mathit{\delta }).$$

3.2. The data-driven procedure

The proposed oracle procedure cannot be applied in practice because it relies on the unknown quantities $T_k$s. To solve this problem, we now develop a data-driven procedure for the three-group problem and investigate the estimation of the unknown quantities.

It is straightforward that $T_k(X_i)$ can be rewritten as

$$\begin{array}{rl}{T}_k(X_i)& =1-P({\theta }_i=k|X_i)\\ & =1-\frac{g(X_i|{\theta }_i=k)P({\theta }_i=k)}{g(X_i)}\\ & =1-\frac{p_k{g}_k(X_i)}{g(X_i)},\end{array}$$

where $g(X_i)$ is the marginal probability density function (p.d.f.) of $X_i$ and $g_k(X_i)=g(X_i|{\theta }_i=k)$ is the conditional p.d.f. of $X_i$ given ${\theta }_i=k$. First, the estimate of $g(X_i)$, denoted as $\hat{g}(X_i)$, can be obtained with conventional kernel-based methods [21]. For the estimation of $p_k{g}_k(X_i)$, a useful observation here is that $p_k{g}_k(X_i)$ can be estimated without estimating $p_k$ and $g_k(X_i)$ separately. Specifically, denote by

$$h(\mu )=\frac{p_1}{1-p_0}{h}_1(\mu )+\frac{p_{-1}}{1-p_0}{h}_{-1}(\mu )$$

the p.d.f. of ${\mu }_i$ when ${\theta }_i\ne 0$. It is then easy to check that

$$p_1{g}_1(X_i)=(1-p_0){\int }_0^{+\mathrm{\infty }}g(X_i|{\theta }_i\ne 0,\mu )h(\mu )\mathrm{d}\mu$$

and

$$p_{-1}{g}_{-1}(X_i)=(1-p_0){\int }_{-\mathrm{\infty }}^{0}g(X_i|{\theta }_i\ne 0,\mu )h(\mu )\mathrm{d}\mu .$$

Noticing that $g(X_i|{\theta }_i\ne 0,\mu )$ is the p.d.f. of $N(\mu ,{\sigma }_i^2)$, once $h(\mu )$ is estimated, $p_k{g}_k(X_i)$ can be estimated immediately by numerical approximation. Motivated by [26], a deconvoluting kernel estimator of $h(\mu )$ is given by

$$\hat{h}(\mu )=\frac{1}{2\pi (1-{\hat{p}}_0)}{\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}{e}^{-\mathrm{it\mu }}[\hat{\mathrm{\Psi }}(t)/{\mathrm{\Psi }}_{ϵ}(t)-{\hat{p}}_0]{\mathrm{\Psi }}_K(\mathrm{\tau t})\mathrm{d}t,$$

where $\hat{\mathrm{\Psi }}(t)$ is the empirical characteristic function of $X_i$, ${\mathrm{\Psi }}_{ϵ}(t)$ and ${\mathrm{\Psi }}_K$ are respectively the characteristic functions of the error distribution and a kernel $K_t$, τ is the bandwidth parameter, and ${\hat{p}}_0$ is the estimate of $p_0$ proposed by [17]. We choose the sinc kernel $K_t=(\mathrm{\pi t}{)}^{-1}\sin t$ with ${\mathrm{\Psi }}_K(t)=I(|t|\le 1))$ for normal distribution in this paper. To select the optimal choice of τ, the bandwidth selection method of [9] is applied to minimize the bootstrap-based approximated MISE of h. It should also be noted that in practical use, the standard estimator $max\{0,\hat{h}(\mu )\}$ is used instead of $\hat{h}(\mu )$.

Based on the above estimates, the plug-in statistic of $T_k(X_i)$ can be obtained, denoted as ${\hat{T}}_k(X_i)$. The data-driven procedure for the directional error control problem (4) can now be constructed. Similar to the oracle problem, let ${\hat{\delta }}_i^{\mathit{\lambda }}$ be the ${\delta }_i^{\mathit{\lambda }}$ defined in (5) with $T_k$ replaced by ${\hat{T}}_k$ and ${\hat{N}}_k(\mathit{\lambda })=(1/m)\sum _{i=1}^m[I({\hat{\delta }}_i^{\mathit{\lambda }}=k)\{{\hat{T}}_k(X_i)-{\alpha }_k\}]$. Then we define

$${\hat{\lambda }}_{k,t}=inf\{\lambda \le {\hat{\lambda }}_{k,t-1}:{\hat{N}}_k({\hat{\mathit{\lambda }}}_{\mathit{k}\mathbf{,}\mathit{t}\mathbf{-}\mathbf{1}})\le 0\}k=1,-1,$$

where $t\ge 1$, ${\hat{\lambda }}_{k,0}=\mathrm{\infty }$, and ${\hat{\mathit{\lambda }}}_{\mathit{k}\mathbf{,}\mathit{t}\mathbf{-}\mathbf{1}}$ is the $\mathit{\lambda }$ with ${\lambda }_k=\lambda$ and ${\lambda }_{k^{\prime }}={\hat{\lambda }}_{k^{\prime },t-1},k^{\prime }\ne k.$ The convergence of the sequence $\{{\hat{\lambda }}_{k,t},t\ge 1\}$ can be proved similar to Proposition 3.1. Denote by ${\hat{\lambda }}_k^{\ast }$ the converged value. The data-driven decision rule for the directional error control problem (4) can then be defined as

$${\hat{\mathit{\delta }}}^{{\hat{\mathit{\lambda }}}^{\ast }}=\{{\hat{\delta }}_1^{{\hat{\mathit{\lambda }}}^{\ast }},\dots ,{\hat{\delta }}_m^{{\hat{\mathit{\lambda }}}^{\ast }}\},$$

where ${\hat{\mathit{\lambda }}}^{\ast }=({\hat{\lambda }}_1^{\ast },{\hat{\lambda }}_{-1}^{\ast }).$ An efficient algorithm is given in Algorithm 1 to obtain ${\hat{\mathit{\delta }}}^{{\hat{\mathit{\lambda }}}^{\ast }}$.

Theorem 3.2 shows that the data-driven decision rule ${\hat{\mathit{\delta }}}^{{\hat{\mathit{\lambda }}}^{\ast }}$ is asymptotically valid and optimal for directional mFDR-control.

Theorem 3.2

Consider model (1)–(2) and suppose that all the conditions in Theorem 3.1 hold. Let ${\hat{p}}_0$ and $\hat{g}$ be the estimates of $p_0$ and g. Suppose that

• (C1) ${\hat{p}}_0\le 1$ almost surely, and $E|{\hat{p}}_0-p_0{|}^2=o(m^{-\kappa })$ for some $\kappa >0$.
• (C2)The OC proportion $1-p_0$ satisfies $m^{-{\rho }_2}\le 1-p_0\le 1-m^{-{\rho }_2}$ for some ${\rho }_2\in (0,1)$.
• (C3) $E\parallel \hat{g}(X_i)-g(X_i){\parallel }^2\to 0$ holds as $m\to \mathrm{\infty }$.
• (C4) $h(\mu )$ is continuous, bounded and twice differentiable, and $\int h^{″}(\mu )\mathrm{d}\mu$ is bounded.
• (C5) $K(t)$ is a bounded even probability density function with $\int t^{2}K(t)\mathrm{d}t$ being bounded. Given τ, ${\sup }_t|{\mathrm{\Psi }}_K(t)/{\mathrm{\Psi }}_{ϵ}(t/\tau )|$ and $\int |{\mathrm{\Psi }}_K(t)/{\mathrm{\Psi }}_{ϵ}(t/\tau )|\mathrm{d}t$ are bounded.

Then,

• (a) ${\mathrm{mFDR}}_k({\hat{\mathit{\delta }}}^{{\hat{\mathit{\lambda }}}^{\ast }})={\alpha }_k+o(1),k=1,-1$;
• (b) $\mathrm{ETP}({\hat{\mathit{\delta }}}^{{\hat{\mathit{\lambda }}}^{\ast }})/\mathrm{ETP}({\mathit{\delta }}^{{\mathit{\lambda }}^{\ast }})=1+o(1).$

Remark 3.2

Conditions (C1)–(C3) are mild and satisfied by using the aforementioned estimators. Condition (C4) is a regular one for deconvolution to ensure that $h(\mu )$ is well estimated. Condition (C5) concerns the choice of the kernel function, implying that any kernel satisfying this condition can be utilized. It can be easily checked that Condition (C5) is satisfied with the sinc kernel.

4. Simulation studies

This section investigates the numerical performance of the proposed oracle and data-driven procedures. The observations are generated from m = 5000 features according to model (1)–(2). All the results are obtained based on 500 replicated simulations. For comparison purposes, the following procedures are considered:

1. our proposed oracle and data-driven procedures;

2. the Benjamini-Hochberg (BH) procedure of [3]. Note that the BH procedure is based on nondirectional hypotheses. Thus, to make it useable for the directional multiple-testing problem (3), upon rejecting $H_{i,0}$, we select $H_{i,-1}$ if $X_i<0$ and $H_{i,1}$ if $X_i>0$;

3. the local false discovery rates (Lfdr) procedure of [24]. Its modification for applying to the directional multiple-testing problem (3) is similar to BH;

4. the Bayes directional false discovery rate (BDFDR) procedure of [20]. The BDFDR procedure was designed for the directional problem, but under an assumption that the null hypothesis $H_{i,0}$ is never true, which is often invalid in practice. Considering that the null hypothesis cannot be ignored, we modify the posterior probabilities of negative alternatives $S_i^{-}=P({\theta }_i=-1|X_i)$ and positive alternatives $S_i^{+}=P({\theta }_i=1|X_i)$ in [20] respectively as $T_1$ and $T_{-1}$ in our paper, and then implement the BDFDR procedure.

The following four settings are considered:

• Case IThe nominal positive and negative mFDR levels are fixed at ${\alpha }_1={\alpha }_{-1}=0.1$, the positive and negative proportions are fixed at $p_1=p_{-1}=0.125$, and the p.d.f.s of ${\mu }_i$ are chosen as $h_1({\mu }_i)=h_2(-{\mu }_i)=\mathrm{Gam}(A,1,0.5)$, where $\mathrm{Gam}(A,B,C)$ denotes the p.d.f. of the gamma distribution with shape parameter A, location parameter B and scale parameter C. The shape parameter A varies from 2 to 5.
• Case II

  ${\alpha }_1=2{\alpha }_{-1}=0.1$, $p_1=p_{-1}=0.125$ and $h_1({\mu }_i)=h_2(-{\mu }_i)=\mathrm{Gam}(A,1,0.5)$. The shape parameter A varies from 2 to 5.

• Case III

  ${\alpha }_1=2{\alpha }_{-1}=0.1$, $p_1=4p_{-1}=0.2$ and $h_1({\mu }_i)=h_2(-{\mu }_i)=\mathrm{Gam}(A,1,0.5)$. The shape parameter A varies from 2 to 5.

• Case IV

  ${\alpha }_1=2{\alpha }_{-1}=0.10$, $h_1({\mu }_i)=\mathrm{Gam}(4,1,0.5)$, $h_2(-{\mu }_i)=\mathrm{Gam}(2,1,0.5)$, $p_1$ varies from 0.025 to 0.225 and $p_{-1}=1-p_0-p_1$ also varies.

Note that in all cases considered, the null proportion $p_0$ is fixed at 0.75. In Case I, the non-null signal strengths and proportions as well as the nominal mFDR levels all have symmetric structures, and at least one of them is asymmetric in Cases II–IV. The results of Cases I–IV are presented respectively in Figures 1–4. In each figure, the actual $\mathrm{mFD}{\mathrm{R}}_1$, $\mathrm{mFD}{\mathrm{R}}_{-1}$, $\mathrm{tmFDR}$ and $\mathrm{ETP}$ values of the considered methods are plotted in turn. To conduct a fair comparison, in each case, after the nominal ${\alpha }_1$ and ${\alpha }_{-1}$ are fixed in the proposed oracle and data-driven procedures, the nominal total error in the other methods is set as the induced nominal total error in our methods.

Figure 1.

Comparison of procedures for directional FDR control under Case I.

Figure 4.

Comparison of procedures for directional FDR control under Case IV.

From Figure 1, we can observe that in cases when the non-null signal strengths and proportions as well as the nominal mFDR levels all have symmetric structures, the proposed oracle and data-driven procedures, Lfdr and BDFDR procedures could all control the directional mFDR at the nominal level, while the BH procedure is conservative in the sense that its directional mFDRs are significantly smaller than the nominal level. Meanwhile, these methods perform similarly in terms of ETP, except that the BH procedure has a slightly worse performance.

Different from Case I, ${\alpha }_1$ and ${\alpha }_{-1}$ are set to be different in Case II, which often happens in situations when the signals in one direction is more important than those in the other. The results are shown in Figure 2. From the plots, we can observe that the performance of the proposed oracle and data-driven procedures is not affected by such asymmetry in the sense that the two procedures could still control the ${\mathrm{mFDR}}_1$ and ${\mathrm{mFDR}}_{-1}$ at respectively their nominal levels. By contrast, the other considered procedures all fail to do so in the sense that their directional FDRs are all significantly larger or smaller than the nominal levels.

Figure 2.

Comparison of procedures for directional FDR control under Case II.

Based on Case II, Figure 3 further presents the results of Case III with different positive and negative signal proportions. It can be observed from the plots that our proposed procedures significantly outperform the other three alternatives. Specifically, in cases when the nominal mFDR level in the direction with fewer signals is more stringent, the FDR levels in this direction of these three methods are much larger than the nominal level. Finally, Figure 4 presents the results when the two-sided signal proportions and strengths as well as the nominal mFDR levels are all different. It can be seen from the plots that our proposed oracle and data-driven procedures are still valid and optimal for directional mFDR control. The performance of the alternatives is seriously affected by the asymmetry, especially for the direction with weaker and fewer signals.

Figure 3.

Comparison of procedures for directional FDR control under Case III.

5. Applications to microarray analysis

In this section, we apply the proposed data-driven procedure to the analysis of the breast cancer and HIV microarray datasets analyzed by [10,17]. For comparison, the BH, Lfdr and BDFDR procedures in the simulation are also considered.

5.1. Analysis of breast cancer data

The breast cancer microarray data, contains 15 patients diagnosed with breast cancer, among which 7 patients are with the BRCA1 mutation and the other 8 are with the BRCA2 mutation. The tumor of each patient was analyzed on a separate microarray, and the microarrays were reported on the same set of m = 3226 genes. For the ith gene, the two-sample t test was implemented to compare the BRCA1 responses with the BRCA2 responses and to obtain the t score, which was then transformed into z score, denoted as $X_i$.

Based on $X_i,i=1,\dots ,m$, the considered procedures can be implemented. We set ${\alpha }_1$ = ${\alpha }_{-1}=0.1$ for our data-driven procedure, and set the nominal total mFDR level at $\alpha =0.1$ for the other three procedures. The results are summarized in Table 2. From the table, it can be seen that the numbers of identified genes of our procedure and the BDFDR procedure are almost the same. Also note that all the 6 under-expressed genes from our proposed data-driven procedure are also identified by BDFDR, while all the 10 over-expressed genes from BDFDR are also identified by our data-driven procedure. Moreover, according to [17], the estimated non-null proportion is very small but nonzero (i.e. ${\hat{p}}_0=0.9872$). In such cases, the Lfdr procedure reports no rejections and thus fail to identify any significant genes. Also, as claimed by [17], the Lfdr procedure reports rejections only when $\alpha \ge \mathrm{0.91.}$ By contrast, the BH procedure reports over 100 significant genes in total.

Table 2.

Number of identified genes in the breast cancer data.

Method	Nominal mFDR	Over-expressed	Under-expressed
Data-driven	${\alpha }_1={\alpha }_{-1}=0.1$	11	6
Data-driven	${\alpha }_1=0.08,{\alpha }_{-1}=0.15$	3	20
BDFDR	$\alpha =0.1$	10	8
Lfdr	$\alpha =0.1$	0	0
BH	$\alpha =0.1$	52	55

Another advantage of the proposed procedure is that the two nominal directional mFDR levels, ${\alpha }_1$ and ${\alpha }_{-1}$, can be properly adjusted depending on the practical need. For instance, if breast tumors with over-expressed BRCA1 mutations compared with BRCA2 mutations are not of primary interest, i.e. under-expressed BRCA1 mutations genes are more useful for diagnosis, the nominal directional mFDR levels can be adjusted as, for example, ${\alpha }_1=0.08$ and ${\alpha }_{-1}=0.15$. The results are also shown in Table 2, from which we observe that more under-expressed genes and less over-expressed genes will be identified in such cases. In conclusion, the above results indicate that our proposed procedure leads to reliable directional identification of differentially expressed genes.

5.2. Analysis of HIV data

The second example is the HIV microarray data. Similarly, the proposed data-driven procedure with ${\alpha }_1$ = ${\alpha }_{-1}=0.05$ and the other methods with $\alpha =0.05$ are applied to identify under-expressed and over-expressed genes. The results are shown in Table 3. From the table, it can be observed that the proposed data-driven procedure and the BDFDR procedure report more significant genes than Lfdr and BH. Compared with BDFDR, our proposed data-driven procedure reports a few more over-expressed genes and less under-expressed genes. In practice, if over-expressed genes or under-expressed genes are our major concern, our data-driven procedure can be flexibly implemented by adjusting the nominal directional mFDR levels, for example, ${\alpha }_1=0.1,{\alpha }_{-1}=0.01$ or ${\alpha }_1=0.01,{\alpha }_{-1}=0.1$. The corresponding results are also listed in Table 3.

Table 3.

Number of selected genes in the HIV microarray data.

Method	Nominal mFDR	Over-expressed	Under-expressed
Data-driven	${\alpha }_1={\alpha }_{-1}=0.05$	195	306
Data-driven	${\alpha }_1=0.1,{\alpha }_{-1}=0.01$	283	86
Data-driven	${\alpha }_1=0.01,{\alpha }_{-1}=0.1$	103	521
BDFDR	$\alpha =0.05$	177	330
Lfdr	$\alpha =0.05$	74	36
BH	$\alpha =0.05$	16	2

6. Conclusions

In this paper, we focused on the problem of directional large-scale multiple testing and proposed oracle and data-driven procedures on basis of a three-classification multiple testing framework. The proposed procedures enable simultaneous FDR-control for both the two directions with possibly different nominal FDR levels, exhibiting high flexibility. We proved theoretically that the proposed procedures are valid and optimal for directional FDR-control in the sense that they maximize the ETP while controlling the two directional FDRs at their nominal levels. We ran extensive simulations and real-data analysis and showed that our procedures outperforms the alternatives significantly in terms of directional FDR control.

The methods were developed under the assumption that the observations $X_i$s are independent of each other. Developing directional multiple testing procedures for general dependence structures is still an open problem, which requires further research. Besides the dependence of the observations, it is assumed that the states ${\theta }_i$s are independent, which may also be invalid in practice. For instance, in microarray experiments, genes from the same biological pathway may share similar significance patterns. There has been some work to model such dependence using a hidden Markov model [25,29]. The extension to a three-state HMM should be fruitful for directional FDR-control. In addition, when domain knowledge (e.g. biological theory or prior experimental results) is available, it can be applied to weight the observed test statistics to further improve the power of the proposed procedures, which will be left for future study.

Appendix. Proofs.

To simplify the notation, we suppress the dependence of $T_k(X_i)$ on $X_i$ in the proofs, and denote it as $T_{k,i}$.

Proof Proo of Proposition 3.1 —

(a) To derive the oracle procedure that minimizes $L(\mathit{\delta },\mathit{\lambda })$, it suffices to minimize each of the terms

$$\sum _{k=1,-1}\{I({\delta }_i\ne k)(1-T_{k,i})+{\lambda }_{k}I({\delta }_i=k)(T_{k,i}-{\alpha }_k)\}$$

for $i=1,\dots ,m$. It is easy to check that the minimizer is ${\delta }_i^{\mathit{\lambda }}$ defined in Equation (5) for $i=1,\dots ,m$. As a result, for any $\mathit{\delta }\in \{-1,0,1{\}}^m$,

$$L({\mathit{\delta }}^{\mathit{\lambda }},\mathit{\lambda })\le L(\mathit{\delta },\mathit{\lambda }),$$

where ${\mathit{\delta }}^{\mathit{\lambda }}=\{{\delta }_1^{\mathit{\lambda }},\dots ,{\delta }_m^{\mathit{\lambda }}\}$. Taking the expectation on both sides, we have $E\{L({\mathit{\delta }}^{\mathit{\lambda }},\mathit{\lambda })\}\le E\{L(\mathit{\delta },\mathit{\lambda })\}$ for all $\mathit{\delta }\in \{-1,0,1{\}}^m$.

(b) First, we need to show that $N_k(\mathit{\lambda })$ is non-increasing in ${\lambda }_k$ but non-decreasing in ${\lambda }_{k^{\prime }},k^{\prime }\ne k$. Define

$$A_{{\lambda }_k}=\{T_{k,i}\le \frac{{\alpha }_k{\lambda }_k+1}{{\lambda }_k+1}\}$$

and

$$B_{{\lambda }_k}=\{{\lambda }_k(T_{k,i}-{\alpha }_k)+T_{k,i}<\underset{k^{\prime }\ne k}{min}{\lambda }_{k^{\prime }}(T_{k^{\prime },i}-{\alpha }_{k^{\prime }})+T_{k^{\prime },i}\}.$$

Then we have

$$N_k(\mathit{\lambda })=E\{I_{A_{{\lambda }_k}}{I}_{B_{{\lambda }_k}}(T_{k,i}-{\alpha }_k)\}.$$

Given ${\lambda }_k^1>{\lambda }_k^2>0,$ it can be easily concluded that $A_{{\lambda }_k^1}\subseteq A_{{\lambda }_k^2}$ and $B_{{\lambda }_k^1}\subseteq B_{{\lambda }_k^2}$. Furthermore, we have

$$N_k({\mathit{\lambda }}_1)-N_k({\mathit{\lambda }}_2)=E\{(I_{A_{{\lambda }_k^1}}{I}_{B_{{\lambda }_k^1}}-I_{A_{{\lambda }_k^2}}{I}_{B_{{\lambda }_k^2}})I(T_{k,i}\ge {\alpha }_k)(T_{k,i}-{\alpha }_k)\}\le 0,$$

where ${\mathit{\lambda }}_i,i=1,2$ is the $\mathit{\lambda }$ with its kth component ${\lambda }_k={\lambda }_k^i$. In other words, $N_k(\mathit{\lambda })$ is non-increasing in ${\lambda }_k$. Similarly, we can prove that $N_k(\mathit{\lambda })$ is non-decreasing in ${\lambda }_{k^{\prime }}.$

According to lemma 1 in [30], there exists a ${\mathit{\lambda }}^{\ast \ast }$ such that the constructed sequences $\{{\check{\lambda }}_{k,t},t\ge 1\}$ satisfy the relationships ${\check{\lambda }}_{k,1}\ge {\check{\lambda }}_{k,2}\ge \cdots \ge {\lambda }_k^{\ast \ast }$ and $N_k({\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t}^{\prime })=0,k=1,-1,t\ge 1,$ where ${\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t}^{\prime }$ is the $\mathit{\lambda }$ with ${\lambda }_k={\check{\lambda }}_{k,t}$ and ${\lambda }_{k^{\prime }}={\check{\lambda }}_{k^{\prime },t-1},k^{\prime }\ne k.$ Then, it follows from the monotone convergence theorem that $\{{\check{\lambda }}_{k,t},t\ge 1\}$ will converge to a number, denoted as ${\lambda }_k^{\ast }$. Let ${\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_t=({\check{\lambda }}_{1,t},{\check{\lambda }}_{-1,t})$, then we have

$$N_k({\mathit{\lambda }}^{\ast })=\underset{t\to \mathrm{\infty }}{lim}{N}_k({\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_t)=\underset{t\to \mathrm{\infty }}{lim}{N}_k({\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t}^{\prime })=0,k=1,-1.$$

Proof Proof of Theorem 3.1 —

(a) Given the fact that $X_1,\dots ,X_m$ are identically distributed, for each k, $N_k({\mathit{\lambda }}^{\ast })=0$ is equivalent to ${\mathrm{mFDR}}_k({\mathit{\delta }}^{{\mathit{\lambda }}^{\ast }})={\alpha }_k,k=1,-1$.

(b) For any $\mathit{\delta }$ satisfying ${\mathrm{mFDR}}_k(\mathit{\delta })\le {\alpha }_k,k=1,-1$, we have

$$\begin{array}{rl}& E\left[\sum _{i=1}^m\sum _{k=1,-1}\{(1-T_{k,i})-I({\delta }_i^{{\mathit{\lambda }}^{\ast }}=k)(1-T_{k,i})\}\right]\\ & E[\sum _{i=1}^m\sum _{k=1,-1}\{I({\delta }_i^{{\mathit{\lambda }}^{\ast }}\ne k)(1-T_{k,i})+{\lambda }_k^{\ast }I({\delta }_i^{{\mathit{\lambda }}^{\ast }}=k)(T_{k,i}-{\alpha }_k)\}]\\ & =E\{L({\mathit{\delta }}^{{\mathit{\lambda }}^{\ast }},{\mathit{\lambda }}^{\ast })\}\le E\{L(\mathit{\delta },{\mathit{\lambda }}^{\ast })\}\\ & =E[\sum _{i=1}^m\sum _{k=1,-1}\{I({\delta }_i\ne k)(1-T_{k,i})+{\lambda }_k^{\ast }I({\delta }_i=k)(T_{k,i}-{\alpha }_k)\}]\\ & \le E\left[\sum _{i=1}^m\sum _{k=1,-1}\{(1-T_{k,i})-I({\delta }_i=k)(1-T_{k,i})\}\right]\end{array}$$

As a result, we have

$$\mathrm{ETP}({\mathit{\delta }}^{{\mathit{\lambda }}^{\ast }})\ge \mathrm{ETP}(\mathit{\delta }).$$

Proof Proof of Theorem 3.2 —

(a) Define ${\tilde{N}}_k(\mathit{\lambda })=\frac{1}{m}\sum _{i=1}^{m}I({\delta }_i^{\mathit{\lambda }}=k)(T_{i,k}-{\alpha }_k)$. By the weak law of large numbers, we have (a) ${\tilde{N}}_k({\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1})\stackrel{P}{\to }{N}_k({\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1})$ holds. For each k = 1, −1, if we fix ${\lambda }_{k^{\prime }},k^{\prime }\ne k$, then ${\hat{N}}_k(\mathit{\lambda })$ is a function of ${\lambda }_k$, and its continuous version, denoted as ${\hat{N}}_k^C(\mathit{\lambda })$, can be constructed by linear interpolation. ${\hat{N}}_k^C(\mathit{\lambda })$ is continuous in ${\lambda }_k$ and monotone. Its inverse function, denoted as ${\hat{N}}_k^{C,-1}(\mathit{\lambda })$, is also well defined, continuous and monotone. Then we have (b) ${\hat{N}}_k({\hat{\mathit{\lambda }}}_{k,t-1})-{\hat{N}}_k^C({\hat{\mathit{\lambda }}}_{k,t-1})\stackrel{P}{\to }0$ and (c) ${\hat{\lambda }}_{k,t}-{\hat{N}}_k^{C,-1}({\hat{\mathit{\lambda }}}_{k0,t-1})\stackrel{P}{\to }0$ hold for k = 1, −1, where ${\hat{\mathit{\lambda }}}_{k0,t-1}$ is the $\mathit{\lambda }$ with the kth component 0 and the other the same as the counterparts of ${\hat{\mathit{\lambda }}}_{k,t-1}$. Suppose that ${\hat{\lambda }}_{k^{\prime },t-1}\stackrel{P}{\to }{\check{\lambda }}_{k^{\prime },t-1}$ for all $k^{\prime }\ne k$. Then we have (d) ${\hat{N}}_k^C({\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1})-{\hat{N}}_k^C({\hat{\mathit{\lambda }}}_{k,t-1})\stackrel{P}{\to }0$ and (e) ${\hat{N}}_k^{C,-1}({\hat{\mathit{\lambda }}}_{k0,t-1})-{\hat{N}}_k^{C,-1}({\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k0,t-1})\stackrel{P}{\to }0$ immediately, where ${\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k0,t-1}$ is the $\mathit{\lambda }$ with the kth component 0 and the rest the same as the counterparts of ${\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}$.

To prove Theorem 3.2, the following two results shall be discussed:

1. ${\hat{N}}_k({\hat{\mathit{\lambda }}}_{k,t-1})\stackrel{P}{\to }{\tilde{N}}_k({\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1})$ holds for any ${\lambda }_k>0$;

2. ${\hat{N}}_k^{C,-1}({\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k0,t-1})\stackrel{P}{\to }{\check{\lambda }}_{k,t}$ and ${\hat{\lambda }}_{k,t}\stackrel{P}{\to }{\check{\lambda }}_{k,t},t\ge 1.$

To prove result (1), it suffices to prove that

$$</p><p>E{\{I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k)({\hat{T}}_{k,i}-{\alpha }_k)-I({\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)(T_{k,i}-{\alpha }_k)\}}^2=o(1),$$

and ${\hat{N}}_k({\hat{\mathit{\lambda }}}_{k,t-1})\stackrel{P}{\to }{\tilde{N}}_k({\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1})$ can be proved accordingly, which will be shown later. We have

$$\begin{array}{rl}P({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k,{\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}\ne k)& \le P({\hat{T}}_{k,i}\le \frac{{\alpha }_k{\lambda }_k+1}{{\lambda }_k+1},{\hat{T}}_{k,i}>\frac{{\alpha }_k{\lambda }_k+1}{{\lambda }_k+1})\\ & +P({\hat{T}}_{k,i}>\frac{{\alpha }_k{\lambda }_k+1}{{\lambda }_k+1},T_{k,i}\le \frac{{\alpha }_k{\lambda }_k+1}{{\lambda }_k+1})\\ & +P\{{\lambda }_k({\hat{T}}_{k,i}-{\alpha }_k)+{\hat{T}}_{k,i}\le \underset{k^{\prime }\ne k}{min}{\hat{\lambda }}_{k^{\prime },t-1}({\hat{T}}_{k^{\prime },i}-{\alpha }_{k^{\prime }})+{\hat{T}}_{k^{\prime },i},\\ & {\lambda }_k(T_{k,i}-{\alpha }_k)+T_{k,i}>\underset{k^{\prime }\ne k}{min}{\check{\lambda }}_{k^{\prime },t-1}({\hat{T}}_{k^{\prime },i}-{\alpha }_{k^{\prime }})+{\hat{T}}_{k^{\prime },i}\}\\ & +P\{{\lambda }_k({\hat{T}}_{k,i}-{\alpha }_k)+{\hat{T}}_{k,i}>\underset{k^{\prime }\ne k}{min}{\hat{\lambda }}_{k^{\prime },t-1}({\hat{T}}_{k^{\prime },i}-{\alpha }_{k^{\prime }})+{\hat{T}}_{k^{\prime },i},\\ & {\lambda }_k(T_{k,i}-{\alpha }_k)+T_{k,i}\le \underset{k^{\prime }\ne k}{min}{\check{\lambda }}_{k^{\prime },t-1}(T_{k^{\prime },i}-{\alpha }_{k^{\prime }})+T_{k^{\prime },i}\}\\ & =o(1)+o(1)=o(1)\end{array}$$

and similarly

$$</p><p>P({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}\ne k,{\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)=o(1).$$

Then we have

$$\begin{array}{rl}& E{\{I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k)({\hat{T}}_{k,i}-{\alpha }_k)-I({\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)(T_{k,i}-{\alpha }_k)\}}^2\\ & \le E\{({\hat{T}}_{k,i}-T_{k,i}{)}^{2}I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k,{\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)\}+E\{({\hat{T}}_{k,i}-{\alpha }_k{)}^{2}I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k,{\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}\ne k)\}\\ & +E\{(T_{k,i}-{\alpha }_k{)}^{2}I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}\ne k,{\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)\}\\ & \le E\{({\hat{T}}_{k,i}-T_{k,i}{)}^2\}+P({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k,{\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}\ne k)+P({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}\ne k,{\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)\\ & =o(1)+o(1)+o(1)=o(1).\end{array}$$

By the Cauchy–Schwarz inequality, we have

$$\begin{array}{rl}& E\{|I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k)({\hat{T}}_{k,i}-{\alpha }_k)-I({\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)(T_{k,i}-{\alpha }_k)|\}\\ & \le {\left[E{\{I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k)({\hat{T}}_{k,i}-{\alpha }_k)-I({\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)(T_{k,i}-{\alpha }_k)\}}^2\right]}^{1/2}=o(1)\end{array}$$

and

$$\begin{array}{rl}& E[\{I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k)({\hat{T}}_{k,i}-{\alpha }_k)-I({\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)(T_{k,i}-{\alpha }_k)\}\\ & \times \{I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k)({\hat{T}}_{k,i}-{\alpha }_k)-I({\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)(T_{k,i}-{\alpha }_k)\}]\\ & \le {\left[E{\{I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k)({\hat{T}}_{k,i}-{\alpha }_k)-I({\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)(T_{k,i}-{\alpha }_k)\}}^2\right]}^{1/2}\\ & \times {\left[E{\{I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k)({\hat{T}}_{k,i}-{\alpha }_k)-I({\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)(T_{k,i}-{\alpha }_k)\}}^2\right]}^{1/2}=o(1).\end{array}$$

Then the following two results can be obtained immediately:

$$\begin{array}{rl}& \left|E\left[1/m\sum _{i=1}^m\{I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k)({\hat{T}}_{k,i}-{\alpha }_k)-I({\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)(T_{k,i}-{\alpha }_k)\}\right]\right|\\ & \le E\{|I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k)({\hat{T}}_{k,i}-{\alpha }_k)-I({\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)(T_{k,i}-{\alpha }_k)|\}=o(1)\end{array}$$

and

$$\begin{array}{rl}& \mathrm{Var}\left[1/m\sum _{i=1}^m\{I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k)({\hat{T}}_{k,i}-{\alpha }_k)-I({\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)(T_{k,i}-{\alpha }_k)\}\right]\\ & \le (1/m)E{\{I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k)({\hat{T}}_{k,i}-{\alpha }_k)-I({\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)(T_{k,i}-{\alpha }_k)\}}^2\\ & +(1-1/m)E[\{I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k)({\hat{T}}_{k,i}-{\alpha }_k)-I({\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)(T_{k,i}-{\alpha }_k)\}\\ & \times \{I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}_{k,t-1}}=k)({\hat{T}}_{k,i}-{\alpha }_k)-I({\delta }_i^{{\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1}}=k)(T_{k,i}-{\alpha }_k)\}]=o(1).\end{array}$$

Now we can conclude that ${\hat{N}}_k({\hat{\mathit{\lambda }}}_{k,t-1})\stackrel{P}{\to }{\tilde{N}}_k({\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1})$.

Proof of (2): To prove result (2), it suffices to prove that

$${\hat{N}}_k^C({\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1})-{\hat{N}}_k({\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k,t-1})\stackrel{P}{\to }0,$$

which follows from (1), (a), (b) and (d). Thus, we have

$${\hat{N}}_k^{C,-1}({\stackrel{\mathbf{ˇ}}{\mathit{\lambda }}}_{k0,t-1})\stackrel{P}{\to }{\check{\lambda }}_{k,t}.$$

This result, together with (c) and (e) show that ${\hat{\lambda }}_{k,t}\stackrel{P}{\to }{\check{\lambda }}_{k,t},t\ge 1.$

Now we proceed to prove Theorem 3.2. When t = 1, we have ${\hat{\lambda }}_{k^{\prime },t-1}={\check{\lambda }}_{k^{\prime },t-1}=\mathrm{\infty }$for $k^{\prime }\ne k$, and thus ${\hat{\lambda }}_{k,t}\stackrel{P}{\to }{\check{\lambda }}_{k,t}.$ Then it is easy to derive that

$${\hat{\lambda }}_{k,t}\stackrel{P}{\to }{\check{\lambda }}_{k,t},t\ge 1.$$

Taking the limitations on both sides, we have ${\hat{\lambda }}_k^{\ast }\stackrel{P}{\to }{\lambda }_k^{\ast }.$ Then similar to the proof of (1), we have $P({\hat{\delta }}_{\mathrm{Sj}}^{{\hat{\mathit{\lambda }}}^{\ast }}=k,{\delta }_{\mathrm{Sj}}^{{\mathit{\lambda }}^{\ast }}\ne k)=o(1)$ and $P({\hat{\delta }}_{\mathrm{Sj}}^{{\hat{\mathit{\lambda }}}^{\ast }}\ne k,{\delta }_{\mathrm{Sj}}^{{\mathit{\lambda }}^{\ast }}=k)=o(1)$. Then we have

$$E\{|I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}^{\ast }}=k)-I({\delta }_i^{{\mathit{\lambda }}^{\ast }}=k)|\}\le P({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}^{\ast }}=k,{\delta }_i^{{\mathit{\lambda }}^{\ast }}\ne k)+P({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}^{\ast }}\ne k,{\delta }_i^{{\mathit{\lambda }}^{\ast }}=k)=o(1).$$

Based on the above results, it can be easily shown that

$$\begin{array}{rl}|E\{1/m\sum _{i=1}^m(T_{k,i}-{\alpha }_k)I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}^{\ast }}=k)\}|& =|E[(T_{k,i}-{\alpha }_k)\{I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}^{\ast }}=k)-I({\delta }_i^{{\mathit{\lambda }}^{\ast }}=k)\}]|\\ & \le E\{|I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}^{\ast }}=k)-I({\delta }_i^{{\mathit{\lambda }}^{\ast }}=k)|\}=o(1)\end{array}$$

and

$$E\{1/m\sum _{i=1}^{m}I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}^{\ast }}=k)\}=E\{1/m\sum _{i=1}^{m}I({\delta }_i^{{\mathit{\lambda }}^{\ast }}=k)\}+o(1)>0,$$

based on which we can derive that ${\mathrm{mFDR}}_k({\hat{\mathit{\delta }}}^{{\hat{\mathit{\lambda }}}^{\ast }})={\alpha }_k+o(1)$. Meanwhile, we have

$$|E[1/m\sum _{i=1}^m(1-T_{k,i})\{I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}^{\ast }}=k)-I({\delta }_i^{{\mathit{\lambda }}^{\ast }}=k)\}]|\le E\{|I({\hat{\delta }}_i^{{\hat{\mathit{\lambda }}}^{\ast }}=k)-I({\delta }_i^{{\mathit{\lambda }}^{\ast }}=k)|\}=o(1),$$

based on which we can derive that $\mathrm{ETP}({\hat{\mathit{\delta }}}^{{\hat{\mathit{\lambda }}}^{\ast }})/\mathrm{ETP}({\mathit{\delta }}^{{\mathit{\lambda }}^{\ast }})=1+o(1).$ Till now, we have finished the proof of Theorem 3.2.

Funding Statement

This work was supported by the National Key R&D Program of China [2022YFA1003801; 2021YFA1000101; 2021YFA1000102], National Natural Science Foundation of China [12201382; 12071144; 71931004], Basic Research Project of Shanghai Science and Technology Commission [22JC1400800], Anhui Provincial Natural Science Foundation [2308085MA10], Natural Science Foundation of Anhui Provincial Universities [KJ2021A1040].

Disclosure statement

No potential conflict of interest was reported by the author(s).