Practical guidelines for assessing power and false discovery rate for a fixed sample size in microarray experiments

SUMMARY

One major goal in microarray studies is to identify genes having different expression levels across different classes/conditions. In order to achieve this goal, a study needs to have an adequate sample size to ensure the desired power. Due to the importance of this topic, a number of approaches to sample size calculation have been developed. However, due to the cost and/or experimental difficulties in obtaining sufficient biological materials, it might be difficult to attain the required sample size. In this article we address more practical questions for assessing power and false discovery rate (FDR) for a fixed sample size. The relationships between power, sample size and FDR are explored. We also conduct simulations and a real data study to evaluate the proposed findings.

1. INTRODUCTION

The development of microarray technologies has revolutionized biomedical research. Microarrays allow scientists to simultaneously estimate the expression levels of thousands of genes in a given sample. Such high-dimensional data demand and motivate novel statistical approaches for the experimental design, data analysis and interpretation. One main objective of microarray studies is to identify genes with different expression levels between two or more conditions. Because thousands of tests are conducted simultaneously, there is a significant multiple comparison issue and it is not appropriate in general to control the false positives on a per comparison basis. Instead, the errors need to be controlled at a more stringent level. Two commonly used approaches are controlling the family-wise error (FWE) rate or the false discovery rate (FDR). Control of FWER in microarray studies can be conservative and usually has low power. In contrast, FDR measures the proportion of false positives among the identified genes. It is less conservative and more commonly used in microarray data analysis.

The power of a specific study is affected by the following factors: the proportion of differentially expressed genes, the distribution of effect sizes, the variation across samples in each condition, and the sample size [2]. Sample size is probably the most important factor in study design. Due to its importance, a number of approaches to sample size calculation have been proposed in the literature [3–12]. These studies have found that, to achieve good power while controlling the false positives at a low level (e.g. low FDR), a large sample size is usually required. For example, under the rule that the number of genes called significant is the same as the number of non-null genes in the population, Tibshirani [13] observed that the sample size should be increased to 100 in order to get the FDR down to 5%, depending on the proportion of genes truly changed at two-fold. Similar ideal sample sizes were also reported by Jung [5], Pounds and Cheng [9] and others. However, due to the cost and/or experimental difficulties in obtaining biological materials, it might be difficult to attain such a large sample size.

In this article, in addition to re-addressing the sample size calculation problem, we present some practical guidelines for assessing power and FDR for a fixed sample size (or for the largest sample size available within the budget) in microarray experiments by studying the following two questions:

1. For a fixed sample size and a desired FDR level, what is the maximum power achievable?

2. For a fixed sample size and a desired power level, what is the minimum FDR achievable?

In addition, we address the relationship between power and FDR and the determination of an appropriate FDR level to employ in practice. As an alternative to the common practice of declaring all genes with corresponding test statistics above a given threshold as significant, we introduce the concept of quantile thresholding where a fixed proportion of the genes that have the largest test statistics are declared significant. The rest of this paper is organized as follows. In Section 2, we introduce the notation, describe the model and briefly review the history of FDR and its new developments. We re-address the sample size calculation problem in Section 3, and study practical questions for assessing power and FDR in Sections 4 and 5 with extensive simulations. Finally, we analyze a real data set to evaluate the proposed findings in Section 6, and conclude the article in Section 7 with some discussion.

2. T-STATISTICS and FALSE DISCOVERY RATE

Given a microarray experiment with m genes, one major goal is to identify differentially expressed genes between different conditions. Let x_ij (j = 1, …, n₁) and y_ij (j = 1, …, n₂) denote the observed expression levels of gene i under conditions 1 and 2, respectively. With proper normalization, we assume that x_ij and y_ij are normally distributed with means μ_i1 and μ_i2 and standard deviations σ_i1 and σ_i2. To identify differentially expressed genes is then equivalent to testing H_i0: μ_i1 = μ_i2 versus H_i1: μ_i1 ≠ μ_i2. We consider the following two-sample t-tests,

$$T_i=\frac{{\overline{x}}_i-{\overline{y}}_i}{s_i\sqrt{1/n_1+1/n_2}},i=1,\dots ,m,$$

where ${\overline{x}}_i={\sum }_{j=1}^{n_1}{x}_{ij}/n_1$ and ${\overline{y}}_i={\sum }_{j=1}^{n_2}{y}_{ij}/n_2$, and s_i is the pooled standard deviation defined as $s_i={(({\sum }_{j=1}^{n_1}{(x_{ij}-{\overline{x}}_i)}^2+{\sum }_{j=1}^{n_2}{(y_{ij}-{\overline{y}}_i)}^2)/(n_1+n_2-2))}^{1/2}$. Noting that although we have assumed a common variance for simplicity of exposition, the following analysis applies to unequal variances or other t-tests (e.g. paired t-tests) as well.

Let n = n₁ + n₂ denote the total number of arrays and λ = n₁/n the allocation proportion for condition 1. Note that λ = 1/2 represents a balanced design. When n is large, T_i is approximately normally distributed with mean $\sqrt{n\lambda (1-\lambda )}{\delta }_i$ and variance 1, where δ_i = (μ_i1 − μ_i2)/σ_i is the so-called effect size. Thus, by assuming a reasonably large n (for example n ≥ 10) for the moment so that the approximation holds well, we can restate the hypothesis as H_i0: δ_i = 0 versus H_i1: δ_i ≠ 0. When n is very small, we recommend the use of the exact t-distribution for the T-statistic rather than a simple normal approximation.

Table I summarizes the various outcomes of testing m hypotheses, of which m₀ represents the total number of true null hypotheses and m₁ represents the total number of false null hypotheses. The quantity V is the number of false positives and R is the total number of rejections. In multiple testing, the ratio of the false discoveries, V/R, is often of interest. Benjamini and Hochberg [1] defined the false discovery rate (FDR) as

Table I.

Outcomes when testing m hypotheses.

	Accept	Reject	Total
Null true	U	V	m0
Alternative true	T	S	m1
Total	W	R	m = m0 + m1

$$\text{FDR}=E\left(\frac{V}{R}|R>0\right)P(R>0).$$ (1)

Noting that in microarray studies, the scientists are rarely interested in the situation where no genes are selected (i.e., R = 0), Storey [15] introduced the positive false discovery rate (pFDR), by removing the term P(R > 0) in (1). In the asymptotic setting, FDR and pFDR are equivalent and they possess the same asymptotic properties. This can be observed by noting that $\underset{m\to \infty }{lim}P(R>0)=1$ for any nontrivial threshold. Both FDR and pFDR consider the expectation of the ratio V/R. Recently, the quantity E(V)/E(R), called the proportion of false positives by Fernando et al. [16] or the decisive false discovery rate (dFDR) by Bickel [17], has also been introduced. We refer to it as dFDR for the rest of the article. dFDR is meaningful as long as P(R > 0) ≠ 0. Although dFDR fails to describe the simultaneous fluctuations in V and R, it has some desirable properties such as it can be optimized using decision theory without the independence assumption [17]. It is also interesting to note that under the weak dependence criterion such as finite blocks and ergodic dependence, there is little difference between FDR and dFDR because $\underset{m\to \infty }{lim}\mid \text{FDR}-\text{dFDR}\mid =0$ [18].

Let (size m₀) denote the set of true null hypotheses and (size m₁) the set of false null hypotheses. Let π₀ = m₀/m denote the proportion of true null hypotheses. Let H_i = 0 if the ith null hypothesis is true, and H_i = 1 otherwise. We reject the null hypothesis H_i0 if its corresponding p-value, p_i, is smaller than or equal to a given threshold α ∈ (0, 1). Then it is easy to see that

$$\text{dFDR}(\alpha )=\frac{{\sum }_{i\in {\mathcal{M}}_0}P(p_i\le \alpha \mid H_i=0)}{{\sum }_{i\in {\mathcal{M}}_0}P(p_i\le \alpha \mid H_i=0)+{\sum }_{i\in {\mathcal{M}}_1}P(p_i\le \alpha \mid H_i=1)}.$$ (2)

Note that (2) holds under any dependence structure among the test statistics, as long as the marginal distribution for each test statistic is maintained. As mentioned above, since there is little practical difference between FDR, pFDR and dFDR when m is large and the dependence between genes is not strong, we will not distinguish them in this article since microarray data usually contain thousands of genes. Let β_i = P(p_i ≤ α|H_i = 1) denote the power for the ith hypothesis test, and β = β_i/m₁ the average power. Because the p-values corresponding to the true null hypotheses are uniformly distributed, we have

$$\text{FDR}(\alpha )=\frac{m_0\alpha }{m_0\alpha +{\sum }_{i\in {\mathcal{M}}_1}{\beta }_i}=\frac{{\pi }_0\alpha }{{\pi }_0\alpha +(1-{\pi }_0)\beta }.$$ (3)

3. SAMPLE SIZE CALCULATION

In this section, we derive the required sample size for achieving the desired power β while controlling the level of FDR at γ. Based on equation (3), we have

$$\alpha =\frac{\gamma (1-{\pi }_0)}{(1-\gamma ){\pi }_0}\beta .$$ (4)

Recall that for a reasonably large n, T_i is approximately normally distributed with mean $\sqrt{n\lambda (1-\lambda )}{\delta }_i$ and variance 1. Therefore, T_i ~ N(0, 1) for i ∈ , and $T_i\sim N(\sqrt{n\lambda (1-\lambda )}{\delta }_i,1)$ for i ∈ . For two-sided tests, we reject H_i0 if |T_i| > z_1−α/2, where z_α is the αth quantile of N(0, 1). Let Φ(·) denote the cumulative distribution function of N(0, 1). Using the fact that z_1−α = −z_α, we have ${\beta }_i(\alpha )=\mathrm{\Phi }(\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{\alpha /2})+\mathrm{\Phi }(-\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{\alpha /2})$ and thus

$$\beta (\alpha )=\frac{1}{m_1}\sum _{i\in {\mathcal{M}}_1}\left(\mathrm{\Phi }(\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{\alpha /2})+\mathrm{\Phi }(-\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{\alpha /2})\right).$$ (5)

Note that the second term in equation (5) is the minor term as long as the quantity $\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid$ is nontrivial. Combining (5) and (4), we have

$$\alpha =\frac{\gamma (1-{\pi }_0)}{(1-\gamma ){\pi }_0}\frac{1}{m_1}\sum _{i\in {\mathcal{M}}_1}\left(\mathrm{\Phi }(\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{\alpha /2})+\mathrm{\Phi }(-\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{\alpha /2})\right).$$ (6)

The required sample size can then be obtained by solving equation (6), which can be done using numerical methods such as the bisection method [5] or a simple grid search. The R code for implementing the bisection method is available from the authors upon request. In practice, to calculate the required sample size, we need to estimate the proportion of true null hypotheses and the corresponding effect sizes in the set (see next section for more details). Throughout the article, we take the smallest integer larger than n whenever necessary.

Equation (6) suggests that the required sample size is inversely proportional to λ(1 − λ). This implies that the most efficient design to achieve a desired power is the balanced design, i.e., λ = 1/2. It is also easy to see that the required sample size decreases as the effect sizes increase. For the special case that |δ_i| ≡ δ > 0 for all i ∈ , by ignoring the minor term in equation (5) we have n ≈ (z_β − z_α/2)²/(λ(1 − λ)δ²), which is the same result found in Jung [5].

4. POWER CALCULATION

In this section, we calculate the maximum power achievable, denoted by β_max, when the sample size n and the level of FDR γ are given. For β(α) in equation (5), we show in Appendix A that

Lemma 1

(i) β(α) is a strictly increasing function of α ∈ [0, 1]; and (ii) β(α)/α is a strictly decreasing function of α ∈ [0, 1].

By (ii) and the fact that FDR(α) = π₀/[π₀ + (1 − π₀)β(α)/α], FDR(α) is a strictly increasing function of α ∈ [0, 1] as long as π₀ ≠ 1. This suggests that the assigned level of FDR should not be larger than π₀. Because both β(α) and FDR(α) are strictly increasing functions of α, the power is also a strictly increasing function of FDR. Therefore, for any given level of FDR γ, there exists a unique β_max such that β_max = β (α_max), where α_max satisfies FDR(α_max) = γ. This implies that, to find β_max, it suffices to find the unique maximum threshold α_max. From equation (3), we have

$$\frac{{\pi }_0{\alpha }_{\mathit{max}}}{{\pi }_0{\alpha }_{\mathit{max}}+(1-{\pi }_0)\beta ({\alpha }_{\mathit{max}})}=\gamma ,$$ (7)

where

$$\beta ({\alpha }_{\mathit{max}})=\frac{1}{m_1}\sum _{i\in {\mathcal{M}}_1}\left(\mathrm{\Phi }(\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{{\alpha }_{\mathit{max}}/2})+\mathrm{\Phi }(-\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{{\alpha }_{\mathit{max}}/2})\right).$$

By solving α_max in equation (7) using numerical methods mentioned in Section 3, the maximum power achievable can be estimated as

$${\widehat{\beta }}_{\mathit{max}}=\beta ({\widehat{\alpha }}_{\mathit{max}})=\frac{(1-\gamma ){\widehat{\pi }}_0}{\gamma (1-{\widehat{\pi }}_0)}{\widehat{\alpha }}_{\mathit{max}},$$ (8)

where π̂₀ is an estimate of the proportion of true null hypotheses.

4.1. Simulation study

To explore the effects of n, λ, π₀ and δ_i on the relationship between β_max and FDR, we consider the following four scenarios (plotted in Figure 1): (A) To explore the effect of n, we consider n = 10, 20, 40, 80 with λ = 0.5, π₀ = 0.8 and {δ_i, i ∈ } ~ U[0, 2]; (B) To explore the effect of λ, we consider λ = 0.1, 0.2, 0.3, 0.5 with n = 20, π₀ = 0.8 and {δ_i, i ∈ } ~ U[0, 2]; (C) To explore the effect of π₀, we consider π₀ = 0.4, 0.6, 0.8, 0.95 with n = 20, λ = 0.5 and δ_i ~ U[0, 2]; and (D) To explore the effect of δ_i, we draw {δ_i, i ∈ } from U[0, 2], U[1, 2], 1, 1.5 with n = 20, λ = 0.5 and π₀ = 0.8. Note that without loss of generality, by symmetry we have assumed that δ_i > 0 for any i ∈ . Further, we set m = 2000 throughout the simulations since it has little impact on the relationship between β_max and FDR.

Figure 1.

Plots of the power versus FDR. Four curves (solid, dashed, dotted, dash-dotted) correspond to four different values of n (10, 20, 40, 80) in panel A, four different values of λ (0.1, 0.2, 0.3, 0.5) in panel B, four different values of π₀ (0.4, 0.6, 0.8, 0.95) in panel C, and four different sets of {δ_i, i ∈} (U[0, 2], 1, U[1, 2], 1.5) in panel D, respectively.

In all settings β_max increases as FDR increases as expected because power is an increasing function of FDR. For the same FDR level, Panel A shows that although β_max increases with the sample size as expected, the increase in power levels off when n becomes larger, which implies that there is little benefit for further increasing the sample size after a certain size. Panel B re-states the fact that a balanced design is always more efficient. Panels C and D indicate that β_max increases when the proportion of true null hypotheses decreases or when the level of effect sizes increases. Another interesting finding from these results is that most curves of β_max on FDR are concave with a clear elbow, especially when the power is nontrivial. To balance the trade-off between power and FDR, we suggest the use of an FDR near the elbow point, which provides an effective power while controlling FDR at a low level. Accurate determination of the FDR level can be critical. One approach based on decision theory is to maximize the quantity c₁β_max(FDR) − c₂FDR, where c₁ ≥ 0 is the benefit of the achieved power, and c₂ ≥ 0 is the cost paid for the falsely discovered hypotheses. In addition, though m has little impact on the relationship between β_max and FDR, in practice a larger FDR might be have to chosen to accommodate the typical microarray studies having the order of 54000 genes. More research is required to determine the FDR level for an efficient control.

To check the validation of (8) in practice, we conduct simulations to evaluate the performance of β̂_max by reporting β̂_max − β_max as a measure of accuracy. Set m = 2000 as before. Our first simulation study investigates the effect of δ̂_i by assuming that a consistent estimate of π₀ is available at this moment [19]. Noting that we estimate the effect size as (x̄_i − ȳ_i)/s_i which is approximately normally distributed with mean δ_i and variance 1/n₁ + 1/n₂ (see Section 2), we simulate δ̂_i from N(δ_i, 1/n₁ + 1/n₂) where the true effect size δ_i is drawn from U[0, 2]. We consider a balanced design for simplicity and consider three different values of n (10, 30, and 50). We further consider two different levels of FDR (0.01 and 0.05), and three different values of π₀ (0.6, 0.8, and 0.95). We run 5000 simulations and report the mean values of β̂_max − β_max and their standard deviations in Table II. In general, the FDR level has little impact on the accuracy of β̂_max, whose accuracy decreases as the sample size reduces. We observe little difference between β̂_max and β_max when the sample size is at least moderately large. For a small sample size, e.g. n = 10, β̂_max is slightly larger than β_max. In addition, for each given sample size, β̂_max becomes more variable when π₀ is approaching 1.

Table II.

Simulation results for the mean values of β̂_max − β_max and their standard deviations (parenthesis) in various settings.

π0	FDR	n = 50	n = 30	n = 10
0.6	0.01	0.0001 (0.007)	0.004 (0.008)	0.075 (0.009)
	0.05	0.0000 (0.007)	0.002 (0.009)	0.066 (0.011)
0.8	0.01	0.0003 (0.010)	0.007 (0.012)	0.069 (0.011)
	0.05	−0.0001 (0.010)	0.003 (0.012)	0.074 (0.014)
0.95	0.01	0.0004 (0.020)	0.013 (0.024)	0.051 (0.018)
	0.05	0.0003 (0.021)	0.006 (0.024)	0.070 (0.022)

Our second simulation study does not assume the existence of a consistent estimate of π₀. We set π₀ = 0.8 for illustration. Noting that most existing estimators for π₀ in the literature are conservative and overestimate π₀ [20, 23], we consider two different values of π̂₀ at 0.85 (i.e., m̂₁ = 300) and 0.9 (i.e., m̂₁ = 200). In each simulation, we draw m̂₁ samples from {δ̂_i, i ∈ } without replacement. All other settings are the same as before. Let β̃_max denote the estimated power, and we report the mean values of β̃_max − β_max and their standard deviations in Table III. We also list in Table III the corresponding results for π̂₀ = 0.8 from the previous simulation study for comparison. Similar to previous results, the FDR level has little impact on the accuracy of β̃_max. We observe that when the inconsistency of π̂₀ increases (i.e., π̂₀ − π₀ increases), the mean difference between β̃_max and β_max stays similar but the variation increases. Overall, β̃_max is fairly robust to the choice of π̂₀.

Table III.

Simulation results for the mean values of β̃_max − β_max and their standard deviations (parenthesis) in various settings.

π̂0	FDR	n = 50	n = 30	n = 10
0.8	0.01	0.0003 (0.010)	0.007 (0.012)	0.069 (0.011)
	0.05	−0.0001 (0.010)	0.003 (0.012)	0.074 (0.014)
0.85	0.01	−0.0003 (0.016)	0.006 (0.017)	0.069 (0.013)
	0.05	−0.0002 (0.016)	0.002 (0.018)	0.074 (0.017)
0.9	0.01	−0.0001 (0.024)	0.006 (0.025)	0.069 (0.017)
	0.05	−0.0007 (0.023)	0.002 (0.025)	0.075 (0.022)

5. FALSE DISCOVERY RATE CALCULATION

In this section, we calculate the minimum level of FDR achievable, denoted by FDR_min, when the sample size n and the desired power β are given. Similar arguments to those in Section 4 indicate that FDR is also a strictly increasing function of power, and thus there exists a unique FDR_min such that the desired power β is achieved. An easy way to calculate FDR_min is through FDR_min = FDR(α_β) = π₀α_β/[π₀α_β + (1 − π₀)β], where α_β is the unique solution of

$$\sum _{i\in {\mathcal{M}}_1}(\mathrm{\Phi }(\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{\alpha /2})+\mathrm{\Phi }(-\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{\alpha /2}))=m_1\beta .$$

Under this setting, α_β can be solved using numerical methods as before. The relationship between FDR_min and power is also readable from Figure 1. In general, to achieve a desired power, FDR_min increases with the proportion of true null hypotheses, but decreases with the effect sizes and sample size.

Let α_i, i ∈ , denote the threshold of the ith hypothesis test so as to have a detection power β, and α_β denote the common threshold such that the average power equals β. For the special case that |δ_i| ≡ δ > 0, the α_i are all the same and thus α_β = α_i/m₁. In general, this relationship does not hold between α_β and {α_i, i ∈ } when the |δ_i| are not all the same. By ignoring the minor term in equation (5) (or similarly for a one-sided test), α_i has an explicit form as ${\alpha }_i/2=\mathrm{\Phi }(-\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{\beta })$. Define ᾱ = α_i/m₁, ${\beta }_i(\overline{\alpha })=\mathrm{\Phi }(\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{\overline{\alpha }/2})$ and β (ᾱ) = β_i(ᾱ)/m₁. In Appendix B we show that

Lemma 2

(i) When β + ᾱ/2 ≤ 1, we have β (ᾱ) ≥ β and thus ᾱ ≥ α_β; (ii) When β + α_β/2 > 1, we have β (ᾱ) < β and thus ᾱ < α_β.

Furthermore we have FDR(ᾱ) ≥ FDR(α_β) if β + ᾱ/2 ≤ 1, and FDR(ᾱ) < FDR(α_β) if β + ᾱ/2 > 1 by noting that FDR(α) is an increasing function of α. In practice, it is common that scientists are more interested in validating a small number of genes (e.g., top 10 or top 50 genes) than all the genes inferred to be differentially expressed. This implies that using a quantile threshold can also be of interest. Let α_(k) denote the kth smallest value in {α_i, i ∈ }, ${\beta }_i({\alpha }_{(k)})=\mathrm{\Phi }(\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{{\alpha }_{(k)}/2})+\mathrm{\Phi }(-\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{{\alpha }_{(k)}/2})$, then #(β_i(α_(k)) ≥ β) = k. That is, when α_(k) serves as a common threshold for all the tests, the detection power is at least β for each of the top k significant genes with α_i ≤ α_(k). For the special case that α_med = median{α_i, i ∈ }, we have P(β_i(α_med) ≥ β) = 0.5, which implies that the detection power is at least β for half of the genes with δ_i ≥ median{δ_i, i ∈ }.

6. REAL STUDY

We use a well-studied data set, the colon cancer data set [21] containing n₁ = 22 normal colon tissue samples and n₂ = 40 colon tumor samples with expression levels from 2000 genes in each sample, to evaluate the proposed findings. The main objective of this colon cancer study is to identify important genes that can distinguish colon tumors from normal tissues. We follow the same normalization steps as those in Huang and Pan [22]. That is, to remove possible array effects, we standardize the data in each array by subtracting the median expression level of the array and then dividing this value by the difference of its third quantile and its first quantile of the expression levels.

We use Storey’s method [23] to estimate the proportion of true null hypotheses. Noting that the alternative p-values are more likely to be small, for a reasonably large p₀ ∈ (0, 1), the majority of p-values larger than p₀ should correspond to the true null hypotheses. This suggests a conservative estimate of π₀ as π̂₀ = #{p_i > p₀}/m(1−p₀). In this study, p₀ is chosen as the median of all p-values as in Ge et al. [24]. Now since the sample sizes for both tissues are at least moderate, for simplicity, we calculate the p-values using the normal approximation. From the data set, we get π̂₀ = 0.616 and thus m̂₁ = 768. We then treat the largest 768 values of observed |δ̂_i| as the true {δ_i, i ∈ }, where δ̂_i = (x̄_i − ȳ_i)/s_i.

The plots of β̂_max on FDR for various combinations of (n₁, n₂) are presented in Figure 2. It is clear that the power increases as the total sample size increases when the ratio n₁/n₂ stays the same. The dash-dotted line with n1 = n2 = 31 is always above the solid line. This demonstrates again that a balanced design is most efficient. It is also interesting to note that under the same level of FDR, the power with (n1, n2) = (6, 56) is even lower than that with (n1, n2) = (11, 20), which indicates that an extremely unbalanced design is not recommended in practice unless necessary. As an illustration to determine the FDR level in the solid line, if we choose c₁ = 1 − π̂₀ and c₂ = π̂₀, an efficient control suggests an FDR level at 0.098.

Figure 2.

Plots of the power versus FDR for the colon cancer data set. Five curves (solid, dashed, dotted, dash-dotted and long-dashed) correspond to five different pairs of (n₁, n₂): (22, 40), (11, 20), (6, 10), (31, 31) and (6, 56), respectively.

Figure 3 displays the pattern of FDR on power when the quantile threshold, α_(k), is employed. We consider α₍₁₀₀₎, α_med, α₍₅₀₀₎ and α_β, where α_β is the threshold such that the average power is exactly β (Section 5). As expected, FDR(α_(k)) increases with k for any given power level. It is interesting to see that α_med has similar performance as α_β, which suggests that α_med can serve as a proxy of α_β in practice. In addition, when only a small number of top genes, e.g. k = 100, are of interest to scientists, we can control the FDR at a very satisfactory level.

Figure 3.

Plots of FDR versus the power when the quantile thresholds are used. Four curves (solid, dashed, dotted and dash-dotted) correspond to four different thresholds (α_β, α_med, α₍₁₀₀₎ and α₍₅₀₀₎).

7. DISCUSSION

Our work is motivated by the fact that the sample size required for achieving good power and low FDR in microarray studies is usually large and hard to obtain, due to the cost and/or other experimental difficulties. Instead of sample size calculation under FDR control, we have studied several practical questions for assessing power and FDR for a fixed sample size. The relationships between power, sample size and FDR are explored. We hope our methods can help with microarray study designs. Our methods can be further used in the post-experimental stage, such as to determine the appropriate level of FDR control or to use a quantile threshold to follow up the top genes in the validation study.

For simplicity of exposition, we have focused on experiments with two conditions. Our methods can be generalized to more than two conditions or to more general settings. We have further assumed that there is little difference between FDR, pFDR and dFDR. The relationship between power and FDR is explored through the concept of dFDR. We note that the results in Sections 3, 4 and 5 are still valid when the test statistics are correlated with each other by noting that (2) holds in general. When the weak dependence of Storey et al. [18] does not hold, we can not claim that FDR and dFDR are asymptotically equivalent and thus the relationship between power and FDR may no longer hold. Further research is needed to explore the relationship between power and FDR (or pFDR) in the situations where the data are strongly correlated. In addition, we have assumed that the sample size is reasonably large (e.g. n > 10) such that the T -statistics are approximately normally distributed. When only a small sample size is available, we recommend the use of the exact t-distribution for the T -statistic rather than a simple normal approximation. Simulations (not shown) indicate that the patterns are also similar.

APPENDIX A: PROOF OF LEMMA 1

1. The first result is made trivial by noting that both $\mathrm{\Phi }(\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{\alpha /2})$ and $\mathrm{\Phi }(-\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{\alpha /2})$ are increasing functions of α ∈ [0, 1].

2. Let φ(·) denote the density function of N(0, 1). By Hung et al. [25] or Sackrowitz and Samuel-Cahn [26], it is easy to see that the density function of p_i is

   $$\begin{array}{l}f(p_i)=\frac{\phi (\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{p_i/2})}{2\phi (z_{p_i/2})}+\frac{\phi (-\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{p_i/2})}{2\phi (z_{p_i/2})}\\ \propto exp\left(-\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid z_{p_i/2}\right)+exp\left(\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid z_{p_i/2}\right).\end{array}$$

Note that f(p_i) ≡ 1 for any i ∈ . When i ∈ , f(p_i) is a strictly decreasing function of p_i ∈ [0, 1] by noting that z_pi/2 is a strictly increasing function of p_i ∈ [0, 1] and g(x) = e^x+e^−x is strictly decreasing on x ∈ (−∞, 0). This implies that β_i(α)/α is a strictly decreasing function of α ∈ [0, 1] for i ∈ , and so is β (α)/α.

APPENDIX B: PROOF OF LEMMA 2

We prove the result (i) first. For ease of notation, denote ξ_i = α_i/2 and ξ̄ = ξ_i/m. The result (i) is then equivalent to proving that ${\sum }_{i\in {\mathcal{M}}_1}\mathrm{\Phi }(\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid +z_{\overline{\xi }})\ge m_1\beta$ when β + ξ̄ ≤ 1. Denote ξ_i = ξ̄ + d_i where d_i ∈ (− ξ̄, 1 − ξ̄). Noting that $\sqrt{n\lambda (1-\lambda )}\mid {\delta }_i\mid =z_{\beta }-z_{{\xi }_i}$, to prove (i), it suffices to prove that

$$\sum _{i\in {\mathcal{M}}_1}\mathrm{\Phi }(z_{\beta }+z_{\overline{\xi }}-z_{\overline{\xi }+d_i})\ge m_1\beta .$$

When m₁ = 2, let g(d) = Φ(z_β + z_ξ̄ − z_ξ̄+d) + Φ(z_β+ z_ξ̄ − z_ξ̄−d) − 2β and we need to prove that g(d) ≥ 0 for any d ∈ (−ξ̄, 1 − ξ̄). Without loss of generality, we assume d ≥ 0. Denote z_ξ̄+d = y, then d = Φ(y) − ξ̄. We have

$$\frac{\partial d}{\partial y}=\frac{\partial }{\partial y}{\int }_{-\infty }^y\phi (t)dt=\phi (y).$$

This implies that (∂/∂d)z_ξ̄+d = 1/φ(z_ξ̄+d) and similarly (∂/∂d)z_ξ̄−d = −1/φ(z_ξ̄−d). Thus

$$\begin{array}{l}\frac{\partial }{\partial d}g(d)=-\frac{\phi (z_{\beta }+z_{\overline{\xi }}-z_{\overline{\xi }+d})}{\phi (z_{\overline{\xi }+d})}+\frac{\phi (z_{\beta }+z_{\overline{\xi }}-z_{\overline{\xi }-d})}{\phi (z_{\overline{\xi }-d})}\\ \triangleq \frac{e^{-B_1/2}-e^{-B_2/2}}{2\pi \phi (z_{\overline{\xi }+d})\phi (z_{\overline{\xi }-d})},\end{array}$$

where $B_1={(z_{\beta }+z_{\overline{\xi }}-z_{\overline{\xi }-d})}^2+z_{\overline{\xi }+d}^2$ and $B_2={(z_{\beta }+z_{\overline{\xi }}-z_{\overline{\xi }+d})}^2+z_{\overline{\xi }-d}^2$. Noting that z_β+z_ξ̄ ≤ 0 since β + ξ̄ ≤ 1, we have

$$B_1-B_2=2(z_{\beta }+z_{\overline{\xi }})(z_{\overline{\xi }+d}-z_{\overline{\xi }-d})\le 0$$

since z_ξ̄+d ≥ z_ξ̄−d for any d ≥ 0. Therefore, (∂/∂d)g(d) ≥ 0 and thus g(d) ≥ g(0) = 0.

For m₁ = 3, we define g(d₁, d₂) = Φ(z_β + z_ξ̄ − z_ξ̄+d1) + Φ(z_β + z_ξ̄ − z_ξ̄+d2) + Φ(z_β + z_ξ̄ − z_{ξ̄−(d1+d2)}) − 3β. Without loss of generality, we assume (d₁ ≥ 0, d₂ ≥ 0) or (d₁ < 0, d₂ < 0). For d₁d₂ < 0, we can classify it to case 1 by reordering the terms of g(d₁, d₂) when d₁+d₂ > 0, or case 2 otherwise. Similar arguments as above lead to g(d₁, d₂) ≥ 0 for both (d₁ ≥ 0, d₂ ≥ 0) and (d₁ < 0, d₂ < 0).

Denote ${\mathcal{M}}_1^{+}=\{i:d_i\ge 0,i\in {\mathcal{M}}_1\}$ and ${\mathcal{M}}_1^{-}=\{i:d_i<0,i\in {\mathcal{M}}_1\}$. And let $L^{+}=\#\left\{{\mathcal{M}}_1^{+}\right\}$ and $L^{-}=\#\left\{{\mathcal{M}}_1^{-}\right\}$. Clearly, L⁺+L⁻ = m₁. Thus by the fact that ${\sum }_{i\in {\mathcal{M}}_1^{+}}{d}_i+{\sum }_{i\in {\mathcal{M}}_1^{-}}{d}_i=0$, we have

$$\begin{array}{l}\sum _{i\in {\mathcal{M}}_1}\mathrm{\Phi }(z_{\beta }+z_{\overline{\xi }}-z_{\overline{\xi }+d_i})=\sum _{i\in {\mathcal{M}}_1^{+}}\mathrm{\Phi }(z_{\beta }+z_{\overline{\xi }}-z_{\overline{\xi }+d_i})+\sum _{i\in {\mathcal{M}}_1^{-}}\mathrm{\Phi }(z_{\beta }+z_{\overline{\xi }}-z_{\overline{\xi }+d_i})\\ \ge (L^{+}-1)\beta +\mathrm{\Phi }(z_{\beta }+z_{\overline{\xi }}-z_{\overline{\xi }+{\sum }_{i\in {\mathcal{M}}_1^{+}}{d}_i})+(L^{-}-1)\beta +\mathrm{\Phi }(z_{\beta }+z_{\overline{\xi }}-z_{\overline{\xi }+{\sum }_{i\in {\mathcal{M}}_1^{-}}{d}_i})\\ \ge m_1\beta .\end{array}$$

Therefore, β(ᾱ) ≥ β.

The proof of the result (ii) is shown to be essentially the same as in (i) by noting that z_β + z_ξ̄ > 0 for β + ξ̄ > 1, and thus omitted.