CALCULATING AVERAGE POWER FOR THE BENJAMINI-HOCHBERG PROCEDURE

SUMMARY

Using exact, analytic results for the average power of the Benjamini-Hochberg (1995) procedure, we provide example power analyses useful for scientists planning studies involving multiple comparisons. The power results are based on the distribution of the p-value under the alternative for the Pearson’s χ², and for the Hotelling-Lawley trace, the Wilks’ lambda, and the Pillai-Bartlett trace, all tests for the general linear multivariate model. Detailed example power analyses are given for a planned mammography experiment with categorical data and a study that tests the association of a single nucleotide polymorphism with insulin resistance and visceral adiposity.

1. INTRODUCTION

A Type I error is a rejection of the null hypothesis when the null hypothesis is true. Typically, the Type I error rate is denoted by α, which is often fixed at 0.05. When multiple tests are performed in the experiment, the family-wise error rate (FWER) is the probability that at least one Type I error occurs during hypothesis testing. The FWER increases as a function of the number of hypotheses tested in the experiment. The increasing chance of making an error is called the multiple comparison problem (Curran-Everett, 2000).

The multiple comparison problem arises in clinical trials, laboratory studies, microarray experiments, and observational studies. A variety of statistical procedures have been suggested to control the multiple comparison problem. See Westfall et al. (1999) for an overview.

Benjamini and Hochberg (1995) introduced a multiple comparison technique that controls the expected value of the false discovery rate (FDR). This is the proportion of all rejections that are rejections of null hypotheses, i.e. rejections that are made in error. Pounds and Cheng (2005) provide an approximate method for power under the Benjamini and Hochberg (1995) procedure. Exact power for the Benjamini and Hochberg (1995) procedure is derived in Glueck et al. (2008). In this paper, we give two detailed examples of how to conduct power analyses for the Benjamini and Hochberg (1995) procedure using methods from Glueck et al. (2008). The two tests we consider in this paper are among the most commonly used tests in statistics, namely the Pearson’s χ² and the F test for the general linear multivariate model (GLMM). Since we will use approximate inputs for Glueck et al. (2008), we will assess the accuracy of the exact power computation. We will also provide guidance as to when it is appropriate to use the methods of this paper.

In Section 2 we discuss power calculations for the Benjamini and Hochberg (1995) procedure. In Section 3 we review previous work on the distribution of p-values under the null and alternative, and derive the approximate distributions of the p-value under the alternative for the Pearson χ² test and for three tests for the general linear multivariate model. In Section 4 we discuss the accuracy of the power approximations for the Benjamini and Hochberg (1995) procedure. In Section 5 we provide the results of two example power analyses, one using the Pearson χ² for a mammography experiment and one using the general linear multivariate model for genetic association with diabetes related outcomes. In Section 6 we discuss the advantages and limitations of the example power analyses and further areas of research.

2. BENJAMINI AND HOCHBERG PROCEDURE AND POWER

Given α ∈ [0, 1], null hypotheses H_0i, i = 1, 2 …, m, with independent but not necessarily identically distributed p-values P_i, and corresponding order statistics P_(i) (the p-values P_i sorted in non-decreasing order), the Benjamini and Hochberg (1995) procedure produces a non-decreasing sequence of rejection bounds b_i = iα^*/m ∈ [0, 1]. When k is the largest number for which P_(k) ≤ b_k, the k null hypotheses H_0(i), i = 1, 2, …, k, k ≤ m, are rejected, and the other null hypotheses are not rejected.

Benjamini and Liu (1999) define average power as the expected value of the ratio of correct rejections, K − J, and the number of non-true null hypotheses, m − n, so

$$E\left(\frac{K-J}{m-n}\right)={\displaystyle \sum _k}{\displaystyle \sum _j}\left(\frac{k-j}{m-n}\right)\text{ Pr }(K=k;J=j).$$ (2.1)

As summarized in Table 1, given m hypotheses with m − n true alternatives, fixed α and known test statistics and distribution of p-values, Glueck et al. (2008) derive the exact joint distribution of the total number of rejections and the number of false rejections in terms of sums of the joint distribution function of the order statistics of the p-values. Glueck et al. (2008) thus calculate the exact average power given in Equation (2.1).

Table 1.

Summary of possible hypothesis rejection and non-rejection scenarios. m is the total number of hypotheses tested. K is the total number of rejections of which J are false rejections.

	Decision
	Do Not Reject	Reject
Null Hypothesis True	n − J	J	n
Alternative Hypothesis True	(m − n) − (K − J)	K − J	m − n
	m − K	K	m

3. DISTRIBUTION OF p-VALUES UNDER THE ALTERNATIVE

In order to calculate average power using the method of Glueck et al. (2008), it is necessary to know the exact distribution of the un-ordered p-values under the alternative. Denote P_HA to be the p-value when the alternative hypothesis is true. Thus, for a right-tailed test with a generic test statistic X, P_HA = P(X ≥ X_OBS | HA is true). Note then even if the alternative hypothesis is true the p-value is still calculated assuming a true null hypothesis. In general, the distribution of the P_HA has been considered by authors such as Hung et al. (1997), who derived the result for the normal case, Sackrowitz and Samuel-Cahn (1999), who considered expected p-values, and Bhattacharya and Habtzghi (2002), who focused on median p-values. Although Ruppert et al. (2007) gave the general form of the distribution of P_HA, the distribution of P_HA has not been derived for the Pearson χ² test of independence nor for multivariate F tests for the GLMM. We derive the distribution of P_HA for those two cases in the next two sections.

3.1 PEARSON’S χ² TEST

Pearson’s χ² test (Pearson, 1904) is commonly used in clinical research situations to assess whether two binary outcome variables are independent. Kroll (1989) and Agresti (1992) provide a thorough summary of the statistical literature on the 2×2 table. The observed data for two binary outcome random variables can be summarized as in Table 2. The underlying population probabilities under the null hypothesis are defined in Table 3.

Table 2.

Observed data for two binary outcome random variables. Here n₁₁ is the number of subjects for whom both variable 1 and variable 2 take on the values 1. n₁₀ is the number where variable 2 is 1 and variable 1 is 0, and n₀₁ and n₀₀ are defined similarly. The total number of observations is N = n₁₁ + n₁₀ + n₀₁ + n₀₀. The row and column marginals are N₁₊ = n₁₁ + n₁₀, N₀₊ = n₀₁ + n₀₀, N₊₁ = n₁₁ + n₀₁ and N₊₀ = n₁₀ + n₀₀.

		Variable 1
		1	0
Variable 2	1	n11	n10	N1+
	0	n01	n00	N0+
		N+1	N+0	N

Table 3.

Population probabilities under the null hypothesis for two binary random variables. π₁₁ is the probability that a subject will have both Variable 1 and Variable 2 take on the value 1. π₁₀, π₀₁, and π₀₀ are defined similarly.

		Variable 1
		1	0
Variable 2	1	π11	π10	π1+
	0	π01	π00	π0+
		π+1	π+0

The null hypothesis for Pearson’s χ² test of independence (Agresti, 1990, p. 47) is

$$H_0:{\pi }_{\mathit{\text{ij}}}={\pi }_{i+}{\pi }_{+j}.,$$ (3.1.1)

for all i, j ∈ {0, 1}.

The usual test statistic is the Pearson’s χ² test (Pearson, 1904). As in Upton (1982) let

$$Q=\frac{N{(n_{11}{n}_{00}-n_{01}{n}_{10})}^2}{N_{1+}{N}_{0+}{N}_{+1}{N}_{+0}}.$$ (3.1.2)

We will assume fixed marginal totals. Under this assumption, when the null hypothesis is true, the exact probability distribution of the test statistic (and of the 2 × 2 table) is the hypergeometric distribution (Agresti, 1992, p. 134, Equation 1.2). For ease of calculation, under the null, the exact distribution is typically approximated by the central χ² with one degree of freedom. The approximation is usually accurate enough when the expected cell counts are all greater than 5 (Rosner, 2006, p. 396). While the exact test allows one-sided hypothesis testing, the Pearson’s χ² is always a two-sided test. The approximate two-sided p-value for the Pearson’s χ² test is given by

$$P=1-F_{{\chi }^2}[Q;1].$$ (3.1.3)

Since n₁₁, n₀₀, n₀₁ and n₁₀ are not observed until an experiment has occurred the p-value is a random variable.

For power analysis, we are interested in the distribution of P_HA. The alternative hypothesis is

$$H_A:{\pi }_{\mathit{\text{ij}}}={\delta }_{\mathit{\text{ij}}}.,$$ (3.1.4)

for all i, j ∈ {0, 1}, where the population probabilities under the alternative hypothesis are shown in Table 4. Define

$$\omega =N{\displaystyle \sum _{j=0}^1}{\displaystyle \sum _{i=0}^1}\frac{{({\pi }_{\mathit{\text{ij}}}-{\delta }_{\mathit{\text{ij}}})}^2}{{\pi }_{\mathit{\text{ij}}}}.$$ (3.1.5)

Under the alternative hypothesis, the exact distribution of the test statistic is the noncentral hypergeometric distribution (Agresti, 1992, p. 134, Equation 1.2). This can be approximated by a noncentral χ² (1,ω) distribution (Cohen, 1988). For 0 < p < 1, an asymptotic form of the cumulative distribution function of P_HA is given by

$$F(p)=1-F_{{\chi }^2}[F_{{\chi }^2}^{-1}(1-p;1);1,\omega ],$$ (3.1.6)

which is a special case of Ruppert et al.[(2007, Equation 5]. Note that under a true null hypothesis Equation (3.1.6) reduces to a continuous uniform(0,1) distribution.

Table 4.

Population probabilities under the alternative hypothesis for two binary random variables. δ₁₁ is the probability that a subject will have both Variable 1 and Variable 2 take on the value 1. δ₁₀, δ₀₁, and δ₀₀ are defined similarly.

		Variable 1
		1	0
Variable 2	1	δ11	δ10	δ1+
	0	δ01	δ00	δ0+
		δ+1	δ+0

In order to confirm the result given in (3.1.6), we performed an enumeration experiment to calculate the empirical cumulative distribution function for the approximate two-sided p-value for the Pearson’s χ² test. The overall sample size, N and the marginal totals were fixed and all possible tables, i, within the fixed N and marginal totals were enumerated. Table probabilities under the null hypothesis were calculated using the central hypergeometric distribution (Agresti, 1992, p. 134, Equation 1.2). To calculate the probability of Table i under the alternative, the tables were listed in ascending order using the test statistic for each table. The table probability under the alternative was then calculated as

$$X_i=F_{{\chi }^2}[Q_i;1,\omega ]-F_{{\chi }^2}[Q_{i-1};1,\omega ]$$ (3.1.7)

with Q given in (3.1.2) and ω given in (3.1.5). The table probability under the alternative for the table with the largest test statistic was calculated as 1 − F_χ² [Q_max(i)−1; 1,ω]. Let i index the tables, with corresponding p-values p_i and probabilities under the alternative given by X_i. The empirical cumulative distribution function was calculated as

$$G(p)={\displaystyle \sum _{i:p_i\le p}}{X}_i.$$ (3.1.8)

We compared the empirical cumulative distribution function to the approximate cumulative distribution of the p-value (3.1.6) using the following steps. 1) Fix N, N₁₊, N₀₊, N₊₀, and N₊₁; 2) Fix the null hypothesis so that the odds ratio is 1; 3) Fix the alternative hypothesis so that the odds ratio is greater than 1. Note that values for N and the alternative hypotheses were chosen to provide a range of non-centrality parameters, ω, which is a function of the sample size and the effect size; 4) Enumerate all possible tables with the fixed setup; 5) Calculate p-values for each table; 6) Calculate probabilities for each table under the null and alternative hypothesis; 7) Calculate the empirical distribution function and compare it to the theoretical distribution function.

Figure (1) shows the empirical and theoretical p-value distributions when the odds ratio under the alternative was 1.2. The approximate distribution of P_HA increases in accuracy as the sample size increases. As shown in Figure (1), the empirical cumulative distribution function is a step function, while the approximate cumulative distribution function is continuous. As the sample size increases, the step size decreases, and the empirical function is better approximated by approximate cumulative distribution function.

Figure 1.

Comparison of the empirical and theoretical χ² p-value distributions under the alternative hypothesis when the odds ratio under the alternative hypothesis is 1.2. Population parameters under the null were π₁₁ = 0.25, π₁₀ = 0.25, π₀₁ = 0.25, π₀₀ = 0.25 and under the alternative were δ₁₁ = 0.26, δ₁₀ = 0.24, δ₀₁ = 0.24, δ₀₀ = 0.26.

Although Figure (1) shows convergence between the approximate and empirical cumulative distribution function, the functions are significantly different by the Kolmogorov-Smirnov test (Daniel 1990). To further explore the effect of odds ratio on the strength of the approximation, we also considered odds ratios under the alternative hypothesis of 2.3 and 5.4. The accuracy of the approximation depended on both the sample size and the size of the odds ratio under the alternative. For example, the Kolmogorov-Smirnov test showed no difference between the empirical and approximate p-value distribution for N = 500, and odds ratio = 5.4; N = 500, odds ratio = 2.3, and N = 250, and odds ratio = 2.3. However, for N = 500, and odds ratio = 1.2, and for N = 100, and odds ratio = 2.3, the empirical and approximate distribution results were significantly different. The results of our enumeration experiment were similar to those observed by Bayarri and Berger (2000), who compared the approximate and exact cumulative distribution functions under the null. Unfortunately, as is shown later, the divergence between the empirical and approximate p-value distributions can have adverse effects on power estimates.

3.2 MULTIVARIATE F TESTS

The general linear multivariate model (GLMM) is used when multiple outcome measurements are taken for each subject. Because there is no uniformly most powerful test for the GLMM, a variety of test statistics are in use. Three of the most common tests are Wilks’ lambda (W), the Pillai-Bartlett trace (B), and the Hotelling-Lawley trace (L) (see Equations 7.20, 7.21 and 7.22 for definitions). Although there are no exact results, the distributions of the test statistics under the null (given in Equation 7.19) are well approximated by a central F distribution (Muller et al. 1992). Under the alternative, Muller and Peterson (1984) give non-central F approximations (shown in Equation 7.24).

For power analysis, we are interested in the distribution of P_HA. Let 𝒯(z) be the scalar transformation of test statistic z (details in Appendix A), and 𝒟(z) be the denominator degrees of freedom for the F approximation for that test. The approximate p-value, which is a random variable, is given by

$$P(z)=1-F_F[𝒯(z);\mathit{\text{ab}},𝒟(z)].$$ (3.2.1)

The approximate cumulative distribution function of P_HA is

$$F(p)=1-F_F\{F_F^{-1}[1-p;\mathit{\text{ab}},𝒟(z)];\mathit{\text{ab}},𝒟(z),\omega (z)\},$$ (3.2.2)

for 0 < p < 1. This is a special case of Equation (5), Ruppert et al. (2007).

A simulation experiment showed that the approximation for the cumulative distribution function of the p-value given in Equation 3.2.2 is very accurate. For the simulation, we fixed N, β, and Σ, and repeated these steps 10,000 times: 1) generating ℰ so that [row_i (ℰ)]′ ~ N (0, Σ), independently; 2) calculating Y = Xβ + ℰ; 3) performing the Pillai-Bartlett trace, Wilks’ lambda and the Hotelling-Lawley trace, and 4) finding the empirical cumulative distribution function of the p-values. As shown in Figure 2, the approximate and empirical cumulative distribution functions essentially coincide. Similar results were obtained for the Pillai-Bartlett trace and the Hotelling-Lawley Trace. The one-sample Kolmogorov-Smirnov test (Daniel 1990) showed that there was no difference between the empirical and exact cumulative distribution functions for all but the smallest values of ω(z) and the smallest sample sizes.

Figure 2.

Comparison of empirical and theoretical p-value distributions for Wilks’ Lambda for sample sizes of N = 30, 90 and 150 with δ = 0.2, where δ measures the difference between treatment group means.

4. ASSESSING ACCURACY OF THE POWER APPROXIMATION

The results of Glueck et al. (2008) give exact average power. Using the previously described power formulas with approximate P_HA distributions produces approximate average power estimates.

A simulation experiment showed that the average power for the Benjamini and Hochberg (1995) procedure was accurate to the second decimal place using the F test and to the first decimal place using Pearson’s χ². We verified the accuracy of the power approximation using the following steps: 1) We fixed whether the null or the alternative hypothesis was true within and across experiments, the sample size, and parameters under the null and alternative hypotheses for three independent experiments. These parameters were fixed to provide a variety of non-centrality parameters, ω, which is a function of the sample size and the effect size; 2) We used a stochastic approach to generate data, a test statistic (either χ² or one of the multivariate F tests), and a p-value for each experiment; 3) We conducted the Benjamini and Hochberg (1995) procedure to determine the number of hypothesis rejections; 4) We counted the number of rejections of the null when the null was actually true; 5) We repeated this process 10,000 times for the F. For the χ² we formed all possible three table combinations given fixed N and fixed table margins. 6) We calculated empirical average power as

$$E\left(\frac{K-J}{m-n}\right)={\displaystyle \sum _k}{\displaystyle \sum _j}\left(\frac{k-j}{m-n}\right){\widehat{p}}_{j,k},$$ (4.1)

where

$${\widehat{p}}_{j,k}=\frac{\#\text{ of times }K=k\text{ and }J=j}{\mathrm{10,000}},$$ (4.2)

for the F and

$${\widehat{p}}_{j,k}=\frac{\#\text{ of times }K=k\text{ and }J=j}{\text{Number of three table combinations}},$$ (4.3)

for the χ², i.e., the empirical joint probability mass is a function of k, the number of rejections, and j, the number of false rejections. 7) We calculated the distribution of P_HA given in Equation (3.1.6), for the χ², or Equation (3.2.2), for the appropriate general linear multivariate model test; 8) We used the p-value distribution to calculate the theoretical average power result by the methods of Glueck et al. (2008). The theoretical result is exact when the p-value distribution is exact. Because the distributions of P_HA given in Equation (3.1.6) and Equation (3.2.2) is approximate, our calculations of the theoretical average power will also be approximate.

The results of an example simulation for the Pearson’s χ² test are shown in Table (5). The largest difference between the empirical average power and theoretical average power for any χ² test was 0.131. The inaccuracy in the power result is due to errors in the approximation for the distribution of P_HA. The power approximation is accurate for small and large values of power, but inaccurate for values around 0.5.

Table 5.

Comparison of empirical and theoretical average power for the Benjamini-Hochberg (1995) procedure. Results are for three independent χ² hypothesis tests. The sample size for each experiment is N ∈ {100, 250, 500}. The odds ratio under the alternative is δ ∈ {1.2, 2.3 and 5.4}.

δ	N	Empirical Average Power	Theoretical Average Power	Absolute Difference
1.2	100	0.064	0.028	0.036
	250	0.076	0.045	0.031
	500	0.096	0.077	0.019
2.3	100	0.567	0.436	0.131
	250	0.896	0.876	0.020
	500	0.996	0.994	0.002
5.4	100	0.992	0.978	0.014
	250	1.000	1.000	0.000
	500	1.000	1.000	0.000

The results of an example simulation for the three multivariate F tests, and its comparison to the theoretical results are given in Tables (6), (7) and (8). The empirical average power and theoretical average power did not differ widely for any case examined. The largest difference between the empirical average power and theoretical average power for any F test was 0.069 and occurred in the smallest sample size. The majority of cases had absolute differences of less than 0.01. Given the uncertainty of parameter selection for power analysis, errors in the estimation of average power on the order of 0.01 should make no difference in clinical or experimental design.

Table 6.

Comparison of empirical and theoretical average power for the Benjamini and Hochberg (1995) procedure. Results are for three independent general linear multivariate models. Each hypothesis is tested with the Wilks’ lambda. The sample size for each experiment is N ∈ {30, 90, 150}.The difference between treatment group means under the alternative is δ ∈ {0.2, 0.5 and 0.8}.

δ	N	Empirical Average Power	Theoretical Average Power	Absolute Difference
0.2	30	0.043	0.046	0.003
	90	0.153	0.157	0.004
	150	0.318	0.315	0.003
0.5	30	0.350	0.342	0.008
	90	0.953	0.944	0.009
	150	0.999	0.998	0.001
0.8	30	0.861	0.825	0.036
	90	1.000	1.000	0.000
	150	1.000	1.000	0.000

Table 7.

Comparison of empirical and theoretical average power for the Benjamini and Hochberg (1995) procedure. Results are for three independent general linear multivariate models. Each hypothesis is tested with the Pillai-Bartlett trace. The sample size for each experiment is N ∈ {30, 90, 150}. The difference between treatment group means under the alternative is δ ∈ {0.2, 0.5 and 0.8}.

δ	N	Empirical Average Power	Theoretical Average Power	Absolute Difference
0.2	30	0.036	0.047	0.011
	90	0.148	0.157	0.009
	150	0.312	0.314	0.002
0.5	30	0.311	0.333	0.022
	90	0.951	0.933	0.018
	150	0.998	0.997	0.001
0.8	30	0.838	0.769	0.069
	90	1.000	1.000	0.000
	150	1.000	1.000	0.000

Table 8.

Comparison of empirical and theoretical average power for the Benjamini and Hochberg (1995) procedure. Results are for three independent general linear multivariate models. Each hypothesis is tested with the Hotelling-Lawley trace. The sample size for each experiment is N ∈ {30, 90, 150}. The difference between treatment group means under the alternative is δ ∈ {0.2, 0.5 and 0.8}.

δ	N	Empirical Average Power	Theoretical Average Power	Absolute Difference
0.2	30	0.054	0.045	0.009
	90	0.158	0.156	0.002
	150	0.323	0.316	0.007
0.5	30	0.376	0.348	0.028
	90	0.954	0.953	0.001
	150	0.999	0.999	0.000
0.8	30	0.875	0.864	0.011
	90	1.000	1.000	0.000
	150	1.000	1.000	0.000

5. EXAMPLE POWER ANALYSES FOR THE BENJAMINI AND HOCHBERG PROCEDURE

We conducted two example power analyses using the Benjamini and Hochberg (1995) procedure. The first one is for a set of independent χ² tests, and the second is for a set of independent multivariate tests for the general linear multivariate model. We give detailed power analyses for each experiment.

5.1 PEARSON’S χ² TEST

The first example is for a proposed meta-analysis, designed to answer the question whether hormone replacement therapy (HRT) affects the detection of cancer by digital mammography. Meta-analysis is useful because it combines several, smaller studies into a study that has a larger number of subjects and as a result has increased power to detect the difference of interest. Hormone replacement therapy (HRT) has been implicated both in increasing the number of breast cancers (Collaborative Group, 1997), and in making breasts more mammographically dense (Greendale et al. 1999), which can hide developing cancers. Lewin et al. (2002), Skaane et al. (2003) and Pisano et al. (2005) compared the diagnostic accuracy of digital and film mammography. Although none of these clinical trials were designed to look at the effect of hormone replacement therapy on breast cancer detection, the data from these trials could be used to answer the question.

Data for women with breast cancer in each study was used to form three 2 × 2 tables. The tables show women with cancer cross-classified by two binary variables: use or non-use of HRT, and detection or non-detection by digital mammography. Each 2 × 2 table will yield a Pearson’s χ² test. The null hypothesis is no association between hormone replacement therapy and whether the cancer is detected by digital mammography.

With three studies, the risk of making at least one Type 1 error is more than 0.14. All the studies are independent, as they were conducted in different locations, with different subjects and investigators. Thus, we can use the Benjamini and Hochberg (1995) procedure to control the multiple comparisons problem. We then use the methods of Glueck et al. (2008) to calculate the exact average power for Benjamini and Hochberg (1995) procedure. To conduct an average power analysis, there are four main steps.

First, we need to choose α^*, which is the level at which we wish to control the false discovery rate. Recall that the false discovery rate is the expected number of rejections when the null is true, divided by the total number of rejections. By analogy to the Type 1 error rate, this is conventionally set at α^* = 0.05.

Second, to calculate power using the methods of Glueck et al. (2008), we need to specify the number of hypothesis tests for which the null is actually true. In this case, since we have three studies that all test the same hypothesis, either all the null hypotheses are true, or none of them are. It makes no sense to conduct the study if we truly believe that all the null hypotheses are true, so we will assume that the number of hypothesis tests for which the null is true is zero. When no null hypotheses are true, average power can be interpreted as the fraction of hypotheses that should be rejected that are rejected. Note that the assumption of none of the three null hypotheses being true is not essential to use the methods of Glueck et al. (2008) but is made for logical reasons.

Third, we need to specify the sample size for each hypothesis test. There were N = 42, 31 and 335 total breast cancer cases in Lewin et al. (2002), Skaane et al. (2003) and Pisano et al. (2005) respectively. Digital mammography detected 27, 23 and 185, respectively.

Finally, we need to specify the population proportions under both the null and the alternative hypotheses. It is important to note here that we never know these answers well until the experiment is run. The best we can do is to estimate the population parameters from previous studies, or from clinical knowledge. In any power analysis, it is important to note where we are uncertain, and to conduct sensitivity analyses, to see if the sample size is fairly stable to different input parameters. We show here how we estimated the proportions under the null and alternative for this example.

In the Collaborative Group (1997) paper, 33% of the 53,865 women in the main analysis had used hormone replacement therapy at some point. In the Pisano et al. (2005) study, 55% of the cancers were detected digitally. We fixed the margins at these values. Assuming no association between the rows and columns, π_ij = π_i+π_+j. Thus, under the null, we obtain π₁₁ = 0.18, π₁₀ = 0.15, π₀₁ = 0.37, π₀₀ = 0.30 as reasonable values for the population parameters.

With the fixed margins, we calculated the power for clinically interesting odds ratios. If we choose the alternative hypotheses of δ₁₁ = 0.10, δ₁₀ = 0.23, δ₀₁ = 0.45, δ₀₀ = 0.22 for all three experiments, women who take hormone replacement therapy are 0.21 times as likely to get their cancer detected by a digital mammogram, and the average power is 0.67. Similarly, if we choose an the alternative hypotheses of δ₁₁ = 0.07, δ₁₀ = 0.26, δ₀₁ = 0.48, δ₀₀ = 0.19, women who take hormone replacement therapy are 0.08 times as likely to get their cancer detected by a digital mammogram, and the average power is 0.86. This means that with the sample sizes observed in Lewin et al. (2002),Skaane et al. (2003) and Pisano et al. (2005), we would expect to reject 86% of the three hypotheses.

As in single hypothesis power analysis, one wants average power to be as high as possible. The interpretation of a 86% average power is that there is a large chance of getting all three rejections in the meta-analysis, while controlling the false discovery rate at 0.05.

5.2 MULTIVARIATE F TESTS

As a second example, we now give an example of using an average power analysis to choose sample size for an experiment with three independent general linear models. Links between visceral fat accumulation, increased insulin sensitivity and Type 2 diabetes have been explored. (Lewis et al. 2002). Visceral fat accumulation and insulin sensitivity may be linked to Type 2 diabetes by a single nucleotide polymorphism (SNP).

A typical genome wide association study can result at least 300,000 hypothesis tests, and a large multiple comparison problem. Because the exact power methods of Glueck et al. (2008) work only for a small number of hypotheses, assessing average power for a genome wide association study is beyond the scope of this paper. Instead, we examine another common problem, a confirmatory study.

Suppose a genome wide association study has identified three independently inherited SNPs which are strongly associated with the outcomes of interest. We assume that each SNP is expressed in a dominant genetic manner. We will also assume that each study subject will only have no more than one of the three SNPs. This is a reasonable assumption if the SNPs are rare in the population. With control of the false discovery rate, we wish to assess the association between each SNP and the bivariate outcomes of visceral fat accumulation and insulin sensitivity, controlling for age. Since visceral fat and insulin sensitivity are measured in different scales we will also assume they have been scaled so that are of the same order of magnitude.

Suppose we collect information on visceral fat volume and insulin sensitivity in four groups of different people: group 1 with SNP 1; group 2 with SNP 2; group 3 with SNP 3; and the last group with none of the implicated SNPs. We wish to compare the difference in average visceral adiposity and insulin sensitivity between each group.

We now use the methods of Glueck et al. (2008) to calculate the exact average power for Benjamini and Hochberg (1995) procedure. To conduct an average power analysis, there are four main steps.

First, we choose α^* = 0.05 to control the false discovery rate. Second, we specify the number of hypothesis tests for which the null is actually true. Here, we suppose that each SNP is in fact related to the outcomes, and thus no null is true. Third, we set up the models, and null and alternative hypotheses. We calculate average power as a function of the parameters and sample sizes, and use the average power values to choose a sample size for the study.

With x_i indicating the i^th subject’s age, the X and β matrices for each of the SNP models are:

$$\underset{N\times 3}{{\mathit{X}}_{\mathit{i}}}=\left[\begin{array}{ccc}1\hfill & 0\hfill & x_1\hfill \\ ·\hfill & ·\hfill & ·\hfill \\ 1\hfill & 0\hfill & x_j\hfill \\ 0\hfill & 1\hfill & x_{j+1}\hfill \\ 0\hfill & 1\hfill & x_{j+2}\hfill \\ ·\hfill & ·\hfill & ·\hfill \\ 0\hfill & 1\hfill & x_N\hfill \end{array}\right],$$ (5.2.1)

with associated β matrix given by

$$\underset{3\times 2}{{\mathit{\beta }}_{\mathit{i}}}=\left[\begin{array}{cc}\hfill {\beta }_{0,\text{SNP},\text{Visceral Fat}}\hfill & \hfill {\beta }_{0,\text{SNP},\text{Insulin Sensitivity}}\hfill \\ \hfill {\beta }_{0,\text{Normal},\text{Visceral Fat}}\hfill & \hfill {\beta }_{0,\text{Normal},\text{Insulin Sensitivity}}\hfill \\ \hfill {\beta }_{1,\text{Visceral Fat}}\hfill & \hfill {\beta }_{1,\text{Insulin Sensitivity}}\hfill \end{array}\right].$$ (5.2.2)

The null hypothesis that there is no difference between the normal population, and those carrying SNP i is

$$H_{0i}:\mathit{C}{\mathit{\beta }}_{\mathit{i}}\mathit{U}=\mathbf{0}$$ (5.2.3)

The C, U and Σ matrices for this hypothesis are

$$\underset{1\times 3}{\mathit{C}}=\left[\begin{array}{ccc}1\hfill & -1\hfill & 0\hfill \end{array}\right],$$ (5.2.4)

$$\underset{2\times 2}{\mathit{U}}=\left[\begin{array}{cc}1\hfill & 0\hfill \\ 0\hfill & 1\hfill \end{array}\right],$$ (5.2.5)

$${\displaystyle \underset{2\times 2}{\mathbf{\Sigma }}}=\left[\begin{array}{cc}1\hfill & 0\hfill \\ 0\hfill & 1\hfill \end{array}\right].$$ (5.2.6)

As with one hypothesis power, multiple hypothesis average power depends on the hypothesis being tested and the sample size. In this case it also depends on what assumption is made regarding the prevalence of individuals with a SNP of interest in each study. Define κ to be the proportion of the study population that has one of the SNP being studied. Let ϕ represent the difference between the normal population, and those with any SNP in both visceral adiposity and insulin sensitivity. Also, define N to be the sample size for each hypothesis test.

Figure 3 shows the average power curve for three single SNP models. Here, κ = 0.10. The difference,ϕ, in both visceral adiposity and insulin sensitivity between the normal population and those with a SNP was allowed to vary between 0 and 0.8. With sample size of 500 per hypothesis, κ of 0.10 and ϕ of 0.25, the average power is nearly 50%. With sample size of 500 per hypothesis, κ of 0.10 and ϕ of 0.4, the average power is over 90%. Thus the proposed genetics studies should provide adequate average power to answer the question as to which of the SNPs are associated with visceral adiposity and insulin resistance, while controlling the false discovery rate at 0.05. The interpretation of average power for the Benjamini and Hochberg (1995) procedure is the fraction of hypotheses that should be rejected that are rejected. Power of 90% shows that on average, all three hypotheses will be rejected 90% of the time.

Figure 3.

Average power curve for the Benjamini and Hochberg (1995) procedure for three independent studies using the general linear multivariate model. Here the Hotelling-Lawley trace, the Pillai-Bartlett trace and Wilks’ lambda test coincide. The hypothesis for each study is that there is no association between a single nucleotide polymorphism and visceral adiposity and insulin sensitivity. The false discovery rate was set at 0.05. The population proportion of individuals in each study of individuals with the SNP of interest was set at 0.10. The difference between the normal population and those with any SNP in both visceral adiposity and insulin sensitivity was varied to examine the effect on power.

For this example power analysis, the values of κ and ϕ were set somewhat arbitrarily. In a grant application, one would use previously published literature, or clinical knowledge to choose reasonable values of κ and ϕ. A graph similar to Figure 3 is useful for clinicians and granting agencies so that they can see the effects of changes in population value and sample sizes on the power of the study.

6. DISCUSSION

This paper develops the necessary components for performing average power calculations for small numbers of hypotheses when using the Benjamini and Hochberg (1995) procedure to adjust for multiple comparisons. The Benjamini and Hochberg (1995) procedure can be used in clinical trials, experimental studies and grant proposals whenever the hypotheses are independent. The examples presented here show how to conduct power and sample size analysis based on the Benjamini and Hochberg (1995) procedure for two commonly used tests. These methods also would be useful to investigators submitting grants and as another way to perform a meta analysis. We hope that our examples will aid study designers and statisticians in both computing and interpreting average power results.

All power analysis depends on knowledge of population parameters that are in fact impossible to know before the study is completed. The best power analysis uses information from previously published studies, or from clinical knowledge. Because there is uncertainty in these estimates, even the best power analysis is subject to error. The error in the power estimates is compounded by using approximations to the distributions of P_HA.

This paper discusses average power for the Pearson’s χ² test of independence and three tests for the general linear multivariate model. The methods of this paper can be extended to calculate Benjamini and Hochberg (1995) power for other statistical tests. This paper assumes independent hypotheses that are necessary for the Benjamini and Hochberg (1995) procedure. Often however, hypotheses are correlated. While not explored in this paper, Benjamini and Yekutieli (2001) explores using the Benjamini and Hochberg (1995) procedure under conditions of non independent hypotheses. The methods of this paper could thus be extended using Benjamini and Yekutieli (2001) to allow for situations where the assumption of hypothesis independence can be relaxed.

Table 10.

Denominator degrees of freedom for three multivariate test statistics.

Name, (z)	Denominator degrees of Freedom, 𝒟 (z)
Wilks’ Lambda (W)	g [(N − r) − (b − a + 1)/2] − (ab − 2) /2
Pillai-Bartlett Trace (PB)	s [(N − r) − b + s]
Hotelling-Lawley Trace (HL)	s [(N − r) − b − 1] + 2

APPENDIX

In this appendix, we give definitions, and approximate distributions under the null and the alternative for three multivariate tests. The discussion is condensed from Muller et al., (1992) and is reproduced here for the readers’ convenience.

The general linear multivariate model is

$$\mathit{Y}=\mathit{X}\mathit{\beta }+\mathbf{ℰ},$$ (7.1)

where Y and ℰ are N × p, X is N × q, and β is q × p with rank(X)= r. We assume that each row of ℰ is independent and identically distributed as N(0, Σ) (Muller et al 1992).

Define two contrast matrices, C and U, and a hypothesis matrix θ so that

$$\underset{a\times b}{\mathit{\theta }}=\mathit{C}\mathit{\beta }\mathit{U}.$$ (7.2)

The multivariate general linear null hypothesis is

$$H_0:\mathit{\theta }={\mathit{\theta }}_{\mathbf{0}},$$ (7.3)

All test statistics are functions of the observed data. Let s = min(a, b). As in Muller et al (1992), define

$$\widehat{\mathit{\beta }}={(\mathit{X}\prime \mathit{X})}^{-}\mathit{X}\prime \mathit{Y}$$ (7.4)

$$\widehat{\mathit{\theta }}=\mathit{C}\widehat{\mathit{\beta }}\mathit{U}$$ (7.5)

$$\widehat{\mathit{H}}={(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)}^{\prime }{[\mathit{C}{(\mathit{X}\prime \mathit{X})}^{-}\mathit{C}\prime ]}^{-1}(\widehat{\mathit{\theta }}-{\mathit{\theta }}_0)$$ (7.6)

$$\widehat{\mathbf{\Sigma }}=\mathit{Y}’[\mathit{I}-{(\mathit{X}\prime \mathit{X})}^{-}\mathit{X}\prime ]\mathit{ Y }[\mathit{N}-\mathit{r}]$$ (7.7)

$$\widehat{\mathit{E}}=\mathit{M}\prime \widehat{\mathbf{\Sigma }}\mathit{M}(N-r)$$ (7.8)

$$\widehat{\mathit{T}}=\widehat{\mathit{H}}+\widehat{\mathit{E}}.$$ (7.9)

The alternative hypothesis is of the form

$$H_A:\mathit{\theta }={\mathit{\theta }}_A.$$ (7.10)

Under the alternative hypothesis, we calculate

$$\mathit{H}=({\mathit{\theta }}_A-{\mathit{\theta }}_0)\prime {[\mathit{C}{(\mathit{X}\prime \mathit{X})}^{-}\mathit{C}\prime ]}^{-1}({\mathit{\theta }}_A-{\mathit{\theta }}_0)$$ (7.11)

$$\mathit{E}=\mathit{M}\prime \mathbf{\Sigma }\mathit{M}(N-r)$$ (7.12)

$$\mathit{T}=\mathit{H}+\mathit{E}.$$ (7.13)

Define the test statistic for Wilks’ lambda as

$$\widehat{W}=\left|{\widehat{\mathit{E}}\widehat{\mathit{T}}}^{-1}\right|.$$ (7.14)

Define the test statistic for the Pillai-Bartlett trace as

$$\widehat{B}=\text{trace}({\widehat{\mathit{H}}\widehat{\mathit{T}}}^{-1}).$$ (7.15)

Define the test statistic for the Hotelling-Lawley trace as

$$\widehat{L}=\text{trace}({\widehat{\mathit{H}}\widehat{\mathit{E}}}^{-1}).$$ (7.16)

A scalar measurement of multivariate association, 𝒯 (ẑ) for each observed test statistic z ∈ {Ŵ, B̂, L̂} appears in Table 9. Muller et al., (1992) gave single F approximations under the null and the alternative for the multivariate tests. Let

$$g=\sqrt{\left[\frac{(a^2{b}^2-4)}{(a^2+b^2-5)}\right]}.$$ (7.17)

The denominator degrees of freedom for three multivariate tests are given in Table 11. Define for each test statistic z

$$F(\widehat{z})=\frac{𝒯(\widehat{z})/\mathit{\text{ab}}}{[1-𝒯(\widehat{z})]/𝒟(\widehat{z})},$$ (7.18)

Then for each multivariate test under the null,

$$F(\widehat{z})~F_F[\mathit{\text{ab}},𝒟(\widehat{z})],$$ (7.19)

where F_F [ab, 𝒟(z)] is the central F distribution with numerator degrees of freedom ab and denominator degrees of freedom 𝒟(z). Under the alternative hypothesis, the distribution of test statistic becomes noncentral. For the purposes of defining the non-centrality parameter, define the test statistic under the alternative for Wilks’ lambda as

$$W=|{\mathit{\text{ET}}}^{-1}|,$$ (7.20)

the test statistic under the alternative for the Pillai-Bartlett trace as

$$B=\text{trace}({\mathit{\text{HT}}}^{-1}),$$ (7.21)

and the test statistic under the alternative for the Hotelling-Lawley trace as

$$L=\text{trace}(\mathit{H}{\widehat{\mathit{E}}}^{-1}).$$ (7.22)

A scalar measurement of multivariate association under the alternative, 𝒯 (z) for each test statistic z ∈ {W, B, L} appears in Table 11. Define the non-centrality parameter ω as

$$\omega (z)=\frac{𝒯(z)}{[1-𝒯(z)]/𝒟(z)},$$ (7.23)

This is the value that the scalar test statistic would have taken on if we observed θ̂=θ_A and Σ̂=Σ_A For each multivariate test under the alternative we have

$$F(\widehat{z})~F_F[\mathit{\text{ab}},𝒟(z),\omega (z)],$$ (7.24)

where F_F [ab, 𝒟(z), ω (z)] is the noncentral F distribution with ab numerator degrees of freedom, 𝒟(z) denominator degrees of freedom, and noncentrality w (z).

Table 9.

Scalar measurements of multivariate association for three multivariate tests.

Name, (z)	Transformation, 𝒯 (ẑ)
Wilks’ Lambda (Ŵ)	1 − $1-{\widehat{W}}^{\frac{1}{g}}$
Pillai-Bartlett Trace (B̂)	B̂/s
Hotelling-Lawley Trace (L̂)	(L̂/s) (1 + L̂/s)−1

Table 11.

One-to-one transformations for three multivariate test statistics.

Name, (z)	Transformation, 𝒯 (z)
Wilks’ Lambda (W)	1 − $1-W^{\frac{1}{g}}$
Pillai-Bartlett Trace (B)	B/s
Hotelling-Lawley Trace (L)	(L/s) (1 + L/s)−1