Optimality of the Holm procedure among general step-down multiple testing procedures

Abstract

We study the class of general step-down multiple testing procedures, which contains the usually considered procedures determined by a nondecreasing sequence of thresholds (we call them threshold step-down, or TSD, procedures) as a parametric subclass. We show that all procedures in this class satisfying the natural condition of monotonicity and controlling the family-wise error rate (FWER) at a prescribed level are dominated by one of them – the classical Holm procedure. This generalizes an earlier result pertaining to the subclass of TSD procedures (Lehmann and Romano, Testing Statistical Hypotheses, 3rd ed., 2005). We also derive a relation between the levels at which a monotone step-down procedure controls the FWER and the generalized FWER (the probability of k or more false rejections).

1 Introduction

Multiple testing procedures (MTPs) have many applications. Among major applications are those in biology and medicine, such as DNA microarray experiments, clinical trials with multiple endpoints, functional neuroimaging. Their needs have largely motivated the intensive development of the theory and methods of multiple hypothesis testing in the last decade. For a brief introduction to MTPs in the context of microarray experiments, see Dudoit et al. (2003). More information can be found in Hochberg and Tamhane (1987), Shaffer (1995), Hsu (1996), and Benjamini et al. (2004).

The problem of multiple testing arises when we want to simultaneously test several, say m, hypotheses about the probability distribution from which our data are drawn. We assume that associated with each hypothesis H_i is a p-value, P_i, which is a random variable, such that 0 ≤ P_i ≤ 1 and, if H_i is true,

$$\text{pr}\{P_i\le x\}\le x,0\le x\le 1.$$ (1)

The observed p-value measures the strength of the evidence against the corresponding hypothesis, typically provided by an appropriate test statistic: the smaller the p-value, the stronger the evidence. For a more detailed discussion of p-values, see, e.g., Lehmann and Romano (2005b, pp. 63–64).

From now on we will assume that hypothesis H_i is true if and only if (1) holds.

In the rest of the Introduction we will outline the notions necessary for our main result, as well as the result itself. Rigorous formulations will be given in Section 2 and Section 3.

A multiple testing procedure, say ℳ, is a decision rule that, based on the vector of observed p-values (p-vector), selects some hypotheses for rejection; the corresponding p-values are said to be ℳ-significant, or just significant, if this causes no ambiguity.¹

We assume that an MTP has two basic properties: it is cutting (a significant p-value cannot exceed an insignificant one) and symmetric (permuting components of a p-vector does not affect which of them will be found significant).

Our main result pertains to monotone step-down procedures. The natural condition of monotonicity means that if some p-values are replaced by smaller ones, the number of rejected hypotheses can only increase. The fact that ℳ is a step-down MTP essentially means: whether or not a p-value is ℳ-significant depends only on this and smaller p-values.

We consider those monotone step-down procedures for which the probability of rejecting at least one true hypothesis (the family-wise error rate, or FWER) does not exceed a chosen level α (0 < α < 1), whatever the joint distribution of the p-values. Our main result states that all those MTPs are dominated by one of them – the classical Holm procedure. Domination means that whenever such a procedure ℳ rejects some hypotheses, the Holm procedure, based on the same p-values, rejects those hypotheses too.

A particular case of this result pertaining to the algorithmically defined class of what we call threshold step-down (TSD) procedures (see Subsection 2.3 below) was obtained earlier by Lehmann and Romano (2005b, Chapter 9).

The remaining part of the paper is organized as follows. In Section 2 we give necessary definitions, mainly following Gordon (2007a) and (Gordon 2007b) (with some changes in terminology). In Section 3 the main result is rigorously formulated and proven. In Section 4 we apply it to the study of generalized family-wise error rates of a monotone step-down procedure.

2 Multiple testing procedures and the FWER

2.1 Multiple testing procedures

A p-value-only-based multiple testing procedure (in the sequel – multiple testing procedure, or MTP) is a Borel measurable mapping ℳ: I^m → 2^K_m from the unit cube I^m = [0, 1]^m to the set 2^K_m of all subsets of K_m = {1, 2, … ,m}. Applying ℳ to a vector p = (p₁,…, p_m) of observed p-values (p-vector), we obtain a subset ℳ(p) of K_m; the inclusion i ∈ ℳ(p) means that, given p-values p₁,…, p_m, ℳ rejects hypothesis H_i, or equivalently, the ith p-value is ℳ-significant.

Example 2.1

The classical Bonferroni procedure, which we denote by Bon f^α (0 < α < 1), is defined by Bon f^α(p) := {i ∈ K_m : p_i ≤ α/m}, p ∈ I^m.

A multiple testing procedure ℳ is symmetric, if for any p-vector p and any one-to-one mapping (permutation) σ:K_m → K_m, denoting by σ(p) such p-vector p′ that $p_{\sigma (i)}^{\prime }=p_i$ for all i, we have

$$ℳ(\sigma (\mathbf{p}))=\sigma (ℳ(\mathbf{p})).$$ (2)

Informally speaking, this means that if we list the hypotheses (and their p-values) in a different order, the procedure ℳ will reject the same hypotheses as before.

Procedure ℳ is cutting, if it satisfies the following condition: whenever, given p-vector p, p_i is ℳ-significant and p_j is not, we have p_i ≤ p_j. In fact, in this case we even have p_i < p_j, provided ℳ is symmetric; indeed, symmetry implies that if p_i = p_j, then the corresponding hypotheses H_i and H_j are either both rejected or both accepted. (To see this, use (2) choosing the permutation σ that transposes i and j.)

All MTPs considered below are assumed to be symmetric and cutting.

Procedure ℳ determines a function s_ℳ: I^m → L_m = {0, 1, 2,…,m} defined as follows: sℳ(p) is the number of rejected hypotheses, given p-vector p (i.e., s_ℳ(p) = Card ℳ(p)). The function s_ℳ(·) is symmetric (permuting components of the p-vector p does not affect s_ℳ(p), which follows from (2)) and, in turn, allows to uniquely recover procedure ℳ: given p, ℳ rejects hypotheses with the s_ℳ(p) smallest p-values. Note that the function s_ℳ(·), in view of its symmetry, is completely determined by its values on the simplex

$${\mathit{\text{Simp}}}^m=\{\mathbf{t}=(t_1,\dots ,t_m)\in {\mathbf{R}}^m:0\le t_1\le \dots \le t_m\le 1\};$$

the same is true for ℳ. The restriction of s_ℳ(·) to Simp^m can be any Borel measurable function s(·) on Simp^m with the following property:

$$\text{if}\mathbf{t}\in {\mathit{\text{Simp}}}^m\text{and}r:=s(\mathbf{t})\text{satisfies}1\le r\le m-1,\text{then}{t}_r<t_{r+1}.$$

This property is equivalent to the fact that declaring the first s(t) components of t ∈ Simp^m significant will not result in equalities between significant and insignificant components.

For a more detailed discussion, see Gordon (2007a).

2.2 Comparison of procedures

Let ℳ and ℳ′ be two multiple testing procedures. Following Liu (1996), we say that procedure ℳ′ dominates procedure ℳ, if for any p-vector p = (p₁, p₂,…,p_m) we have ℳ′(p) ⊃ ℳ(p), i.e., ℳ′ rejects all hypotheses H_i rejected by ℳ (and maybe some others). In this case we write ℳ′ ⪰ ℳ.

It is easy to see that ℳ′ ⪰ ℳ if and only if s_ℳ′(t) ≥ s_ℳ(t) for all t ∈ Simp^m.

Note that the partial order ⪰ between MTPs is not linear: both relations ℳ′ ⪰ ℳ and ℳ ⪰ ℳ′ may be false. Therefore, given an arbitrary class of MTPs, one should not expect that it contains a procedure that dominates every procedure in the class.

2.3 Step-down procedures

A multiple testing procedure ℳ is a step-down procedure if, given t ∈ Simp^m, whether or not i ∈ ℳ(t) (i.e., s_ℳ(t) ≥ i), depends only on t₁,…, t_i. In other words, it is required that if t, t′ ∈ Simp^m, $t_j=t_j^{\prime }$ for all j = 1, 2,…, i, and s_ℳ(t) ≥ i, then also s_ℳ(t′) ≥ i.

The threshold step-down (TSD) procedure generated by u ∈ Simp^m (notation: TSD_u), given a nondecreasing sequence of p-values t = (t₁, t₂,…,t_m) ∈ Simp^m, declares t_i significant if and only if t_j ≤ u_j for all j = 1,…,i.

Remark 2.1

A TSD procedure is a step-down procedure.

Example 2.2

The Bonferroni procedure Bon f^α (0 < α < 1) is TSD_u with u_i = α/m, i = 1,…,m.

Example 2.3

The Holm (1979) procedure Holm^α (0 < α < 1) is TSD_u with u_i = α/(m − i + 1), i = 1,…,m.

Example 2.4

The procedure introduced by Benjamini and Liu (1999) is TSD_u with

$$u_i=\text{min}\left(1,\frac{mq}{{(m-i+1)}^2}\right),1\le i\le m(0<q<1).$$

Given two m-dimensional vectors x and y, we write x ≥ y if x_i ≥ y_i, i = 1,…,m.

Remark 2.2

TSD_u′ ⪰ TSD_u if and only if u′ ≥ u.

Note that the class of step-down procedures is broader than that of TSD procedures.

Example 2.5

Let m = 2 and a, ε ∈ (0, 1). Let procedure ℳ act as follows. Given 0 ≤ t₁ ≤ t₂ ≤ 1, t₁ is ℳ-significant if and only if t₁ ≤ ε; t₂ is ℳ-significant if and only if t₁ ≤ ε and at₁ + (1 − a)t₂ ≤ ε. Then ℳ is a well-defined MTP (see the end of Subsection 2.1); moreover, ℳ is a step-down procedure, but not a TSD one.

This example can be modified in many ways and generalized to arbitrary m > 1.

2.4 Monotone procedures

We call a multiple testing procedure, ℳ, monotone, if for any p, p′ ∈ I^m relation p′ ≤ p implies: s_ℳ(p′) ≥ s_ℳ(p).

To show that a given procedure is monotone, it suffices to establish the above implication only for pairs of p-vectors that belong to Simp^m, which is usually simpler.

Proposition 1

Procedure ℳ is monotone, if and only if relation t′ ≤ t (t, t′ ∈ Simp^m) implies: s_ℳ(t′) ≥ s_ℳ(t).

This statement, in a slightly different formulation, was proven in (Gordon 2007b). It shows that the definition of monotonicity given in that work is equivalent to the definition given above.

Corollary 1

Any TSD procedure is monotone.

Remark 2.3

Procedure ℳ in Example 2.5 is monotone.

2.5 Family-wise error rate

The family-wise error rate (FWER) introduced by Tukey (1953) measures the probability with which some type I errors occur.

Let ℳ be a multiple testing procedure and P a probability distribution on the unit cube I^m. Let P = (P₁, P₂,…,P_m) be a random vector with distribution P. Denote by T the set (depending on P) of all i ∈ K_m = {1, 2,…,m} for which the ith hypothesis

$$H_i:\text{pr}\{P_i\le x\}\le x\text{for}\text{all}x\in [0,1]$$ (3)

is true. Let

$$\text{FWER}(ℳ,P):=\text{pr}\{ℳ(\mathbf{P})\cap T\ne \mathrm{\varnothing }\}.$$ (4)

In other words, FWER(ℳ, P) is the probability for ℳ, given a set of m random values P_i with joint distribution P, to reject at least one true hypothesis H_i.

Procedure ℳ is said to control the FWER at level α, if for all probability distributions P on I^m we have FWER(ℳ, P) ≤ α. This is true, if and only if the quantity

$$\text{FWER}(ℳ):=\underset{P}{\text{sup}}\text{FWER}(ℳ,P),$$ (5)

where the supremum is taken over all probability distributions P on I^m, does not exceed α.

The quantity (5) may be viewed as the exact level at which ℳ controls the FWER.

3 Optimality of the Holm procedure

The Holm procedure Holm^α (see Example 2.3), given t ∈ Simp^m, finds t_i significant if and only if

$$t_j\le \frac{\alpha }{m-j+1}\text{for}\text{all}j,1\le j\le i.$$

Proposition 2

(Holm 1979). FWER(Holm^α) ≤ α.

Therefore, Holm^α is a monotone step-down procedure (see Remark 2.1 and Corollary 1) controlling the FWER at level α. The following statement – the main result of this work – shows that Holm^α dominates any procedure with similar properties.

Theorem 1

Let ℳ be a monotone step-down multiple testing procedure with

$$\text{FWER}(ℳ)\le \alpha <1.$$ (6)

Then ℳ ⪯ Holm^α.

Proof

Let ℳ be a monotone step-down procedure, such that (6) is true. Suppose that, given a nondecreasing sequence of p-values

$$\mathbf{t}=(t_1,\dots ,t_m),0\le t_1\le t_2\le \dots \le t_m\le 1,$$ (7)

procedure ℳ finds the kth of them, t_k, significant. How large can the number t_k be?

Set l := m − k + 1, τ := min{t_k, 1/l}, and

$$\beta :=l\tau ,$$ (8)

so that β ≤ 1.

Define l random variables

$$P_k,P_{k+1},\dots ,P_m$$ (9)

as follows. First, choose at random an integer z equal to one of the numbers k, k + 1,…,m, 0 with probabilities τ, τ, …, τ, 1 − β, respectively. Then generate, independently of each other (given z), the random numbers (9), so that if z ≠ 0, P_z is uniformly distributed on the interval [0, τ], while the others are uniformly distributed on [τ, 1]; if z = 0, all P_i, i = k, k + 1,…,m, are uniformly distributed on [τ, 1].

It follows from the construction that the distribution of each random variable P_i, i = k, k + 1,…,m, is supported on [0, 1] and has a constant density on each of the two intervals [0, τ] and [τ, 1]. On the first of them the density equals 1, hence the same is true for the other (because the integral of the density is 1). Therefore,

$$\text{each}{P}_i,i=k,k+1,\dots ,m,\text{is}\text{uniformly}\text{distributed}\text{on}[0,1].$$ (10)

Also, we have

$$\underset{k\le i\le m}{\text{min}}{P}_i\le \tau ,\text{if}z\ne 0.$$ (11)

Furthermore, let P₁,…, P_k−1 be such random variables that 0 ≤ P_i ≤ t_i, i = 1,…,k − 1. These inequalities and (11), where τ ≤ t_k, imply that the ordered random variables P_(i) satisfy inequalities

$$P_{(i)}\le t_i,i=1,2,\dots ,k,\text{if}z\ne 0.$$ (12)

(We use the following property of nondecreasing rearrangements: given a₁,…, a_m ∈ R and c ∈ R, the inequality a_(i) ≤ c is equivalent to the existence of at least i values of j, such that a_j ≤ c.)

The random vector P = (P₁,…, P_m), whose distribution on the unit cube [0, 1]^m we denote by P, has the following properties:

1. According to (10), hypotheses H_k,H_k+1,…,H_m (see (3)) are true.

2. By (12), with probability at least β we have

   $$P_{(i)}\le t_i,i=1,2,\dots ,k.$$ (13)

Since the procedure ℳ found the first k terms t_i in (7) significant, being a step-down procedure, ℳ will still find them significant if the remaining terms t_k+1, t_k+2,…, t_m are replaced by 1’s. Because ℳ is monotone, the first k terms of the sequence (P₍₁₎,…, P_(m)) will also be found significant, if (13) holds. According to (ii), this occurs with probability at least β. By (i), no more than k − 1 hypotheses H_i are false, so by rejecting k or more hypotheses procedure ℳ commits at least one type I error. We, therefore, have FWER(ℳ, P) ≥ β, and consequently FWER(ℳ) ≥ β. Comparing this with (6), we see that β ≤ α. In particular, β < 1, so that recalling (8) we obtain τ = t_k and

$$t_k\le \frac{\alpha }{m-k+1}.$$

Because, given p-vector (7), procedure ℳ finds all t_j, j = 1,…, k, significant, it follows that inequalities

$$t_j\le \frac{\alpha }{m-j+1},j=1,\dots ,k,$$

are all true; this means that the kth p-value t_k is Holm^α-significant.

We have shown that if, for a given t ∈ Simp^m, its kth component t_k is ℳ-significant, then it is Holm^α-significant as well. Therefore, ℳ ⪯ Holm^α.

Corollary 2

(Lehmann and Romano 2005b, Chapter 9) Let ℳ = TSD_u, where u ∈ Simp^m. If FWER(ℳ) ≤ α < 1, then u_i ≤ α/(m − i + 1), i = 1,…,m.

Proof

A TSD procedure is a step-down procedure; according to Corollary 1, it is monotone. Hence, Theorem 1 applies, and ℳ ⪯ Holm^α. In view of Remark 2.2, this implies inequalities between respective thresholds of the two procedures.

The following example shows that the condition that ℳ is a step-down procedure cannot be removed from Theorem 1.

Example 3.1

Let ℳ be a procedure that, given a p-vector t ∈ Simp^m, rejects all hypotheses, if t_i ≤ α for all i, and accepts all hypotheses otherwise. This procedure, as is easy to see, is monotone and controls FWER at level α. Nevertheless, the relation ℳ ⪯ Holm^α is not true: if t_i = c (α/m < c < α), i = 1, 2,…,m, then ℳ rejects all hypotheses, while Holm^α rejects none.

4 Generalized FWER of a step-down procedure

The concept of generalized FWER – the probability of rejecting k or more true hypotheses – was introduced by Victor (1982) and studied in the context of TSD procedures by Hommel and Hoffman (1988), Lehmann and Romano (2005a), and Gordon (2007a). We use notation from the latter work similar to (4) and (5): for a multiple testing procedure ℳ, a probability distribution P on the unit cube I^m, and an integer k (1 ≤ k ≤ m), we set

$${\text{FWER}}_k(ℳ,P):=\text{pr}\{\text{Card}(ℳ(\mathbf{P})\cap T)\ge k\},$$

where P is a random vector with distribution P and T is the set (depending on P) of all i ∈ {1, 2,…,m} for which the hypothesis (3) is true. Furthermore, we set

$${\text{FWER}}_k(ℳ):=\underset{P}{\text{sup}}{\mathrm{\text{FWER}}}_k(ℳ,P).$$

This quantity may be interpreted as the exact level at which ℳ controls FWER_k.

Obviously, FWER_k(ℳ) ≤ FWER₁(ℳ) ≡ FWER(ℳ) for all k (1 ≤ k ≤ m). It turns out that for monotone step-down MTPs this inequality can be significantly strengthened.

Theorem 2

Let ℳ be a monotone step-down procedure, such that FWER(ℳ) ≤ α (α < 1). Then for all k (1 ≤ k ≤ m)

$${\mathrm{FWER}}_k(ℳ)\le C_k\alpha ,$$ (14)

where

$$C_k=\{\begin{array}{cc}4k/{(k+1)}^2,\hfill & \text{if}k\text{is odd};\hfill \\ 4/(k+2),\hfill & \text{if}k\text{is even}.\hfill \end{array}$$ (15)

The upper bound C_kα in (14) is sharp.

Proof

By Theorem 1, ℳ ⪯ Holm^α, so that FWER_k(ℳ) ≤ FWER_k(Holm^α). It is shown in Gordon (2007a, Example 5.1) that FWER_k(Holm^α) = C_kα, where C_k is given by (15). This proves (14). The upper bound C_kα cannot be replaced by a smaller one, because it is attained when ℳ= Holm^α.

Remark 4.1

Note that the sharp upper bound in (14) does not depend on m (the number of hypotheses, m ≥ k). Also note that 1 = C₁ = C₂ > C₃ > C₄ > … and that C_k < 4/k, the two sides being asymptotically equivalent for large k.