Simultaneous confidence intervals that are compatible with closed testing in adaptive designs

Summary

We describe a general method for finding a confidence region for a parameter vector that is compatible with the decisions of a two-stage closed test procedure in an adaptive experiment. The closed test procedure is characterized by the fact that rejection or nonrejection of a null hypothesis may depend on the decisions for other hypotheses and the compatible confidence region will, in general, have a complex, nonrectangular shape. We find the smallest cross-product of simultaneous confidence intervals containing the region and provide computational shortcuts for calculating the lower bounds on parameters corresponding to the rejected null hypotheses. We illustrate the method with an adaptive phase II/III clinical trial.

1. Introduction

For experiments designed to make inference about a parameter vector θ = (θ₁, … , θ_K), it is common to find confidence intervals for all of the individual θ_k such that the simultaneous coverage probability is at least 1 − α. Sometimes, though, an experimenter will only attempt to assert that an individual parameter exceeds a specific value, say θ_k > δ_k. If this cannot be achieved in such a way that the probability of making at least one incorrect rejection in a family of hypotheses H_k = {θ_k ⩽ δ_k} (k = 1, … , K) is no greater than α, the experimenter will not assert anything about θ_k. The latter method of inference is used in so-called closed test procedures (Marcus et al., 1976), and its advantage is often greater power.

For experiments conducted in a single stage, Hayter & Hsu (1994) showed how simultaneous 100(1 − α)% confidence intervals can be constructed to be compatible with some commonly used closed test procedures, in the sense that a null hypothesis H_k is rejected at familywise level α if and only if the confidence interval for θ_k excludes all values for which H_k is true. Often, these intervals are scarcely more informative than the test decisions. For example, for one-sided problems where larger parameter values are more beneficial, no 100(1 − α)% lower confidence bound for any individual θ_k can exceed δ_k unless all hypotheses H₁, … , H_K can be rejected at familywise level α.

In this article we derive confidence intervals for adaptive experiments. Our motivating example is a seamless phase II/III clinical trial, although the method is not limited to this setting. Such trials consist of a first stage in which K experimental treatments, indexed by T₁= {1, … , K}, are compared with a common control and, after an interim analysis, a second stage in which only a subset of treatments, indexed by T₂ ⊆ T₁, are compared with the control. The state-of-the-art methodology for this problem (Bauer & Kieser, 1999; Posch et al., 2005; Bretz et al., 2009) is a hybrid of the closure principle of Marcus et al. (1976) and a p-value combination which goes back to Fisher (1932). This methodology allows any subset of treatments to be chosen at interim, based on all trial data and external factors. Other adaptations, such as sample size re-estimation, are also possible. A serious concern, though, is that there is no established method for constructing confidence intervals. As emphasized in the International Conference on Harmonisation’s E9 guideline (ICH E9 Expert Working Group, 1999, p. 1932), ‘Estimates of treatment effect should be accompanied by confidence intervals, whenever possible, and the way in which these will be calculated should be identified.’

Posch et al. (2005) proposed 100(1 − α)% simultaneous confidence intervals following such a trial. Unfortunately, their intervals are not guaranteed to be compatible with the closed test procedure. Here, we construct intervals that are compatible. As in the one-stage case, an inevitable shortcoming of these intervals is that they are not always substantially more informative than the original test decisions. We will show that this problem is mitigated to some extent by the adaptive nature of the experiment.

2. Fundamental Methodology

2·1. Closure principle

The closure principle of Marcus et al. (1976) is a general method for multiple hypothesis testing. A formal description is given in Finner & Strassburger (2002), and we adopt similar notation here. Let $\mathcal{P}=\{P_{{\theta }^{\ast }}:{\theta }^{\ast }\in \Theta \}$ be a family of probability measures defined on a common sample space (Ω, $\mathcal{F}$), where Θ is a multi-dimensional parameter space. Suppose that we wish to test a family of null hypotheses $\mathcal{H}=\{H_i:i\in \mathcal{I}\}$, where H_i ⊂ Θ for each i in some index set $\mathcal{I}$. Let $\psi =\{{\psi }_i:i\in \mathcal{I}\}$ denote a multiple test of $\mathcal{H}$, with each component ψ_i taking value 0 or 1 corresponding to nonrejection or rejection of H_i, respectively. It is often desirable to ensure that

$$\underset{{\theta }^{\ast }\in \Theta }{\sup }{P}_{{\theta }^{\ast }}\left(\bigcup _{i\in I\left({\theta }^{\ast }\right)}\{{\psi }_i=1\}\right)⩽\alpha ,$$ (1)

where $I\left({\theta }^{\ast }\right)=\{i\in \mathcal{I}:{\theta }^{\ast }\in H_i\}$ is the index set of true hypotheses under θ*. In other words, the probability of rejecting at least one true null hypothesis is bounded by α. This is known as strong control of the familywise error rate. The closure principle can be used to ensure (1). We are required to find, for each $I\subseteq \mathcal{I}$ such that $H_I={\bigcap }_{i\in I}{H}_i$ is nonempty, a local level-α test φ_I for the intersection hypothesis H_I; that is, we require

$$\underset{{\theta }^{\ast }\in H_I}{\sup }{P}_{{\theta }^{\ast }}\left({\phi }_I=1\right)⩽\alpha ,$$ (2)

where φ_I takes values in {0, 1} with the usual interpretation. If we define ${\psi }_i={\min }_{I:H_I\ne \varnothing ,H_I\subseteq H_i}\left({\phi }_I\right)$, then (1) holds. This can be very useful, as in many applications it is easy to find tests satisfying (2), whereas validating (1) directly is hard.

2·2. Combination test

Fisher (1932) discussed combining independent p-values to test a single null hypothesis. For convenience and brevity, we will only consider two-stage designs. We define a p-value combination function Q: [0, 1]² ↣ [0, 1] that is left-continuous and nondecreasing in both its arguments and is uniformly distributed provided that both arguments are themselves independent and uniformly distributed. An example is

$$Q\left(u,v\right)=1-\Phi \left[2^{1∕2}\left\{{\Phi }^{-1}\left(1-u\right)+{\Phi }^{-1}\left(1-v\right)\right\}\right],$$ (3)

where Φ denotes the standard normal distribution function.

Such a combination function lends itself to a two-stage adaptive closed test, ψ, for a family of null hypotheses, $\mathcal{H}$. An important application, discussed in Bretz et al. (2009), is a seamless phase II/III confirmatory clinical trial. We henceforth restrict attention to a parameter θ = (θ₁, … , θ_K) taking values in parameter space $\Theta ={ℝ}^K$ and a family of null hypotheses $\mathcal{H}=\{H_k:k\in T_1\}$ where T₁ = {1, … , K} and H_k = {θ_k ⩽ δ_k} (k ∈ T₁) for some constants ${\delta }_1,\dots ,{\delta }_K\in ℝ$. The θ_k (k ∈ T₁) might correspond to the mean effects of K different treatments, for example. By defining local tests φ_I (I ⊆ T₁) via a combination function $\mathcal{Q}$, it is possible to make data-dependent modifications to the trial design at an interim analysis (cf. Bauer & Kieser, 1999; Hommel, 2001; Brannath et al., 2002). For instance, attention can be focused on a subset T₂ ⊆ T₁ of the initial hypotheses of interest; changes can be made to sample sizes, allocation ratios, etc.

2·3. Two-stage closed test procedure

Assume that the full first-stage trial data are represented by a random vector $X\in {ℝ}^n$ with distribution function G(x; θ). Prior to starting the trial, one must specify a combination function $\mathcal{Q}$ and, for each I ⊆ T₁, a first-stage test of $H_I={\bigcap }_{i\in I}$ H_i with an associated p-value function $p_I^{\left(1\right)}:{ℝ}^n\to [0,1]$ that satisfies ${\sup }_{{\theta }^{\ast }\in H_I}{\int }_{\{p_I^{\left(1\right)}\left(x\right)⩽u\}}\mathrm{d}G\left(x;{\theta }^{\ast }\right)⩽u$ for all $u\in [0,1]$. The second-stage design is unspecified.

At the interim analysis, the experimenter defines a second-stage design, d, by choosing a subset of the original hypotheses, indexed by T₂ ⊆ T₁, to continue studying in the second stage, along with second-stage sample sizes and, for each I ⊆ T₁, a second-stage hypothesis test for H_I. See below for a proposal for choosing second-stage tests for H_I where I ⊈ T₂. We assume that the design d is allowed to depend on the unblinded first-stage data x without prespecifying an adaptation rule. Let Y denote the data collected at the second stage, taking values in ${ℝ}^m$, and let $p_{I,x,d}^{\left(2\right)}\left(y\right)$ (I ⊆ T₁) denote the p-value functions of the second-stage tests. Because the tests used in the second stage depend on the first-stage data x and the chosen design d, the p-value functions will in general depend on both.

Let F_x,d(y; θ) denote the distribution function of the second-stage data, given the chosen design d and interim data x. We assume that for all x, d and I ⊆ T₁, the second-stage p-values $p_{I,x,d}^{\left(2\right)}$ satisfy ${\sup }_{{\theta }^{\ast }\in H_I}{\int }_{\{p_{I,x,d}^{\left(2\right)}\left(y\right)⩽u\}}\mathrm{d}{F}_{x,d}\left(y;{\theta }^{\ast }\right)⩽u$ for all u ∈ [0, 1]. The distribution F_x,d is assumed to be known, i.e., not merely specified up to a null set, for all x and d, a condition that can be formalized by assuming an appropriate regression model (Brannath et al., 2012). See § 3·2 for a numerical example.

At the final analysis, for each I ⊆ T₁, the test decision is φ_I = 1 if and only if $\mathcal{Q}\{p_I^{\left(1\right)},p_{I,x,d}^{\left(2\right)}\}⩽\alpha$. As shown in Brannath et al. (2012), this combination test for H_I controls the Type I error rate at level α.

We assume that only data for the hypotheses indexed by T₂ are collected in the second stage and propose setting $p_I^{\left(2\right)}=p_{I\cap T_2}^{\left(2\right)}$ for I ⊈ T₂, where we drop the indices x and d for simplicity and set $p_{\varnothing }^{\left(2\right)}=1$ by convention. Such second-stage p-values have the required distribution under H_I∩T2 and hence also under H_I.

We emphasize that while Type I error control is guaranteed even if the second-stage design is initially open-ended, in the design of actual clinical trials it is crucial to perform detailed planning based on likely first-stage outcomes. The added flexibility is necessary because it is impossible to foresee all eventualities in extremely complex areas such as clinical drug development.

3. Confidence regions

3·1. Partitioning the parameter space

A standard approach to deriving a 100(1 − α)% confidence set for θ is to perform a level-α test of each elementary hypothesis {θ = θ*} (θ* ∈ Θ) and include all θ* corresponding to nonrejected hypotheses (see, e.g., Lehmann, 1986, p. 90). To ensure compatibility with closed testing, the key idea (Stefansson et al., 1988; Hayter & Hsu, 1994; Finner & Strassburger, 2002) is to partition the parameter space into disjoint regions

$${\Theta }_I=\{{\theta }^{\ast }\in \Theta :{\theta }_i^{\ast }⩽{\delta }_i,i\in I;{\theta }_i^{\ast }>{\delta }_i,i\in T_1\backslash I\}\left(I\subseteq T_1\right)$$

and apply different tests in each of the disjoint Θ_I. If, for each I ⊆ T₁, we let {φ_I (θ*): θ* ∈ Θ} denote a family of tests with

$$\underset{{\theta }^{\ast }\in \Theta }{\inf }{P}_{{\theta }^{\ast }}\{{\phi }_I\left({\theta }^{\ast }\right)=0\}⩾1-\alpha ,$$ (4)

where φ_I (θ*) takes values in {0, 1} with the usual interpretation, we can apply the following general result from Hsu (1996, p. 234).

Lemma 1. A level-100(1 − α)% confidence set for θ is

$$C=\bigcup _{I\subseteq T_1}\left[\{{\theta }^{\ast }\in \Theta :{\phi }_I\left({\theta }^{\ast }\right)=0\}\cap {\Theta }_I\right].$$ (5)

Our aim is to find families of tests such that C is compatible with the two-stage closed test procedure. This requires us to augment our specification of $p_{I\cap T_j}^{\left(j\right)}\left(j=1,2;I\subseteq T_1\right)$ with a family of p-values $\{p_{I\cap T_j}^{\left(j\right)}\left({\theta }^{\ast }\right):{\theta }^{\ast }\in \Theta \}$ where, under {θ = θ*}, the distribution of $p_I^{\left(1\right)}\left({\theta }^{\ast }\right)$ and $p_{I\cap T_2}^{\left(2\right)}\left({\theta }^{\ast }\right)$ meet conditions as outlined for $p_I^{\left(1\right)}$ and $p_{I\cap T_2}^{\left(2\right)}$ in § 2·3. Additionally, if we treat the data as fixed and view each family as a function $p_{I\cap T_j}^{\left(j\right)}:\Theta \to [0,1]$, then unless $I\cap T_j=\varnothing ,p_{I\cap T_j}^{\left(j\right)}\left({\theta }^{\ast }\right)$ is constant in all arguments ${\theta }_i^{\ast }$ such that i ∉ I ∩ T_j, and is left-continuous and nondecreasing in all arguments ${\theta }_i^{\ast }$ such that i ∈ I ∩ T_j, with $p_{I\cap T_j}^{\left(j\right)}\left({\theta }^{\ast }\right)=p_{I\cap T_j}^{\left(j\right)}$ for any θ* such that ${\theta }_i^{\ast }={\delta }_i$ for all i ∈ I ∩ T_j. Furthermore, we assume that

$$\underset{{\theta }_i^{\ast }\to \infty ,i\in T_2}{\lim }{p}_{\varnothing }^{\left(2\right)}\left({\theta }^{\ast }\right)=1.$$ (6)

Proposition 1. Inserted into (5), the following families of hypothesis tests give rise to a 100(1 − α)% confidence set for θ, denoted by C, that is compatible with the two-stage closed test procedure, i.e., ψ_k = 1 if and only if H_k ∩ C = ∅: for ∅ ≠ I ⊆ T₁ and θ* ∈ Θ,

$${\phi }_I\left({\theta }^{\ast }\right)=\{\begin{array}{cc}1,\hfill & Q\{p_I^{\left(1\right)}\left({\theta }^{\ast }\right),p_{I\cap T_2}^{\left(2\right)}\left({\theta }^{\ast }\right)\}⩽\alpha ,\hfill \\ 0,\hfill & Q\{p_I^{\left(1\right)}\left({\theta }^{\ast }\right),p_{I\cap T_2}^{\left(2\right)}\left({\theta }^{\ast }\right)\}>\alpha ,\hfill \end{array}\phantom{\}}$$ (7)

and {φ_∅(θ*): θ* ∈ Θ } is any family of tests satisfying (4).

Proof. See the Appendix.

There will be no unique collection of families of p-values satisfying the aforementioned distributional and monotonicity constraints. Rather, the families must be specified in a two-stage procedure in an analogous way to the p-values in § 2·3. As will become clear from the example below, for many commonly encountered scenarios and when I ∩ T_j ≠ ∅, the choice of $\{p_{I\cap T_j}^{\left(j\right)}\left({\theta }^{\ast }\right):{\theta }^{\ast }\in \Theta \}$ will be obvious from the choice of $p_{I\cap T_j}^{\left(j\right)}$. As a simple example, suppose that $p_{\left\{k\right\}}^{\left(j\right)}$ is the p-value from a one-sided z-test of the null hypothesis {θ_k ⩽ δ_k} using the stage-j data only. Then the natural choice for $p_{\left\{k\right\}}^{\left(j\right)}\left({\theta }^{\ast }\right)$ is the one-sided p-value from a standard z-test of $\{{\theta }_k⩽{\theta }_k^{\ast }\}$ using the same stage-j data.

While for I ∩ T_j ≠ ∅ there will often be a natural choice for $p_{I\cap T_j}^{\left(j\right)}\left({\theta }^{\ast }\right)$, it is unclear how φ_∅(θ*) and $p_{\varnothing }^{\left(2\right)}\left({\theta }^{\ast }\right)$ should be chosen. A reasonable suggestion is given below.

Corollary 1. Define $p_{\varnothing }^{\left(j\right)}\left({\theta }^{\ast }\right)=p_{T_j}^{\left(j\right)}\left({\theta }^{\ast }\right)$ for j = 1,2. The following is a 100(1 − α)% confidence region for θ that is compatible with the two-stage closed test procedure:

$$C_1=\bigcup _{I\subseteq T_1}\left[{\theta }^{\ast }\in {\Theta }_I:Q\{p_I^{\left(1\right)}\left({\theta }^{\ast }\right),p_{I\cap T_2}^{\left(2\right)}\left({\theta }^{\ast }\right)\}>\alpha \right].$$ (8)

The properties of a region defined by (8) are best illustrated by a specific example.

3·2. Example

Posch et al. (2005) considered a clinical trial where three active treatments, indexed by T₁ = {A, B, C}, are compared with a placebo using a two-stage adaptive design. The individual null hypotheses of interest are H_k = {θ_k ⩽ 0} (k ∈ T₁), where θ_k = π_k − π₀ denotes the difference between the success probabilities of treatment k and placebo. Denote the observed success rate of treatment k in stage j by ${\widehat{\pi }}_{k,j}\left(k\in T_1\cup \left\{0\right\};j=1,2\right)$, where treatment 0 corresponds to a placebo.

At the design stage, the inverse normal combination function (3) is specified and n₁ = 140 first-stage patients are recruited to each treatment arm. Approximately, the ${\widehat{\theta }}_{k,1}={\widehat{\pi }}_{k,1}-{\widehat{\pi }}_{0,1}\left(k\in T_1\right)$ are multivariate normal with $E\left({\widehat{\theta }}_{k,1}\right)={\theta }_k,\mathrm{var}\left({\widehat{\theta }}_{k,1}\right)=\{{\widehat{\pi }}_{k,1}\left(1-{\widehat{\pi }}_{k,1}\right)+{\widehat{\pi }}_{0,1}\left(1-{\widehat{\pi }}_{0,1}\right)\}∕n_1$ and positive correlations. Based on this assumption, Simes (1986) tests are used for each intersection hypothesis; that is, $p_{\left\{k\right\}}^{\left(1\right)}=1-\Phi \left[{\widehat{\theta }}_{k,1}{\left\{\mathrm{var}\left({\widehat{\theta }}_{k,1}\right)\right\}}^{-1∕2}\right]$ for k ∈ T₁ and, for |I|>1, $p_I^{\left(1\right)}={\min }_{k\in I}{p}_{\left\{k\right\}}^{\left(1\right)}\mid I\mid ∕R\left(k,I\right)$, where R(k, I) denotes the rank of $p_{\left\{k\right\}}^{\left(1\right)}$ among $\{p_{\left\{i\right\}}^{\left(1\right)}:i\in I\}$. The natural way of augmenting these p-values is to define $p_{\left\{k\right\}}^{\left(1\right)}\left({\theta }^{\ast }\right)=1-\Phi \left[\left({\widehat{\theta }}_{k,1}-{\theta }_k^{\ast }\right){\left\{\mathrm{var}\left({\widehat{\theta }}_{k,1}\right)\right\}}^{-1∕2}\right]$ for k ∈ T₁ and $p_I^{\left(1\right)}\left({\theta }^{\ast }\right)={\min }_{k\in I}{p}_{\left\{k\right\}}^{\left(1\right)}\left({\theta }^{\ast }\right)\mid I\mid ∕R\left(k,I,{\theta }^{\ast }\right)$ for |I| > 1, where R(k, I, θ*) denotes the rank of $p_{\left\{k\right\}}^{\left(1\right)}\left({\theta }^{\ast }\right)$ among $\{p_{\left\{i\right\}}^{\left(1\right)}\left({\theta }^{\ast }\right):i\in I\}$.

Suppose that the unblinded first-stage results are ${\widehat{\pi }}_{0,1}=0\cdot 21,{\widehat{\pi }}_{A,1}=0\cdot 22$, ${\widehat{\pi }}_{B,1}=0\cdot 3$ and ${\widehat{\pi }}_{C,1}=0\cdot 36$. The experimenter decides that treatments A and C are not to be considered in the second stage owing to lack of efficacy and safety concerns, respectively. A further n₂ = 140 patients are recruited to both treatment B and placebo. A family of p-values with $p_{\left\{B\right\}}^{\left(2\right)}\left({\theta }^{\ast }\right)=1-\Phi \left[\right({\widehat{\theta }}_{B,2}-{\theta }_B^{\ast }\left){\left\{\mathrm{var}\right({\widehat{\theta }}_{B,2}\left)\right\}}^{-1∕2}\right]$ is chosen, where ${\widehat{\theta }}_{B,2}={\widehat{\pi }}_{B,2}-{\widehat{\pi }}_{0,2}$.

Now suppose that the second-stage results are ${\widehat{\pi }}_{0,2}=0\cdot 19$ and ${\widehat{\pi }}_{B,2}=0\cdot 31$. The p-values from the elementary hypotheses are $p_{\left\{A\right\}}^{\left(1\right)}=0\cdot 419$, $p_{\left\{B\right\}}^{\left(1\right)}=0\cdot 0412$, $p_{\left\{C\right\}}^{\left(1\right)}=0\cdot 00241$ and $p_{\left\{B\right\}}^{\left(2\right)}=0\cdot 00961$. Therefore $p_{\{A,B,C\}}^{\left(1\right)}=3p_{\left\{C\right\}}^{\left(1\right)}$, $p_{\{A,B\}}^{\left(1\right)}=2p_{\left\{B\right\}}^{\left(1\right)}$ and $p_{\{B,C\}}^{\left(1\right)}=2p_{\left\{C\right\}}^{\left(1\right)}$. As ${\min }_{I\subseteq }{T}_1,B\in I\mathcal{Q}(p_I^{\left(1\right)},p_B^{\left(2\right)})\le 0\cdot 025$, H_B can be rejected at familywise level 0·025. Both H_A and H_C fail to be rejected, as $\mathcal{Q}\{p_{\left\{k\right\}}^{\left(1\right)},1\}=1$ for k = A,C. A compatible 97·5% confidence region for θ is given by

$$\bigcup _{I\subseteq T_1}\left\{{\theta }^{\ast }\in {\Theta }_I:Q\{p_I^{\left(1\right)}\left({\theta }^{\ast }\right),p_B^{\left(2\right)}\left({\theta }^{\ast }\right)\}>0.025\right\},$$ (9)

where $p_{\varnothing }^{\left(1\right)}\left({\theta }^{\ast }\right)$ is defined as $p_{T_1}^{\left(1\right)}\left({\theta }^{\ast }\right)$ for all θ* ∈ Θ.

The region (9) will have a complicated three-dimensional shape. However, in terms of making inference on θ_B, its crucial features can be seen by taking two cross-sections, as displayed in Fig. 1. As $p_I^{\left(1\right)}\left({\theta }^{\ast }\right)$ is nondecreasing in ${\theta }_C^{\ast }$ for all I ⊆ T₁, we know that for any γ ∈ (-∞, 0), the cross-section at ${\theta }_C^{\ast }=\gamma$ is contained in the cross-section at ${\theta }_C^{\ast }=0$. Similarly, for any γ ∈ (0, ∞), the cross-section at ${\theta }_C^{\ast }=\gamma$ is contained in the limit of the cross-section of the region as ${\theta }_C^{\ast }\to \infty$. One can see immediately from Fig. 1 that for any ϵ > 0, the 97·5% confidence region fails to exclude all parameter vectors θ* such that ${\theta }_B^{\ast }⩽ϵ$. In other words, the lower confidence bound on θ_B provides no more information than the decision of the closed test procedure.

Fig. 1.

Cross-sections of confidence regions of the form (9) for making inference on the second-stage parameter of interest, θ_B, in the example of § 3·2: (a) two cross-sections of the 97·5% confidence region; (b) two cross-sections of the 95% confidence region.

For confidence intervals that are compatible with single-stage closed test procedures (Hayter & Hsu, 1994; Strassburger & Bretz, 2008; Guilbaud, 2008), a necessary condition for obtaining informative lower confidence bounds for parameters corresponding to the rejected null hypotheses is that ψ_k =1 for all k ∈ T₁. In the adaptive setting, this is no longer a necessary condition. For example, repeating the above test procedure at level α=0·05, the compatible 95% confidence region analogous to (9) is also summarized in Fig. 1. Here it appears, and indeed can be verified by considering all values of ${\theta }_A^{\ast }$, that there does exist some ϵ > 0 such that the confidence region excludes all parameter vectors θ* for which ${\theta }_B^{\ast }⩽ϵ$. We will show that for the two-stage adaptive setting, a necessary condition for informative lower confidence bounds on parameters corresponding to the rejected null hypotheses is that ψ_k =1 for all k ∈ T₂. However, as can be seen from Fig. 1, this condition is not sufficient.

3·3. A two-stage, single-step confidence region

Posch et al. (2005) proposed the following 100(1 − α)% confidence region:

$$C_2=\left\{{\theta }^{\ast }\in \Theta :Q\{p_{T_1}^{\left(1\right)}\left({\theta }^{\ast }\right),p_{T_2}^{\left(2\right)}\left({\theta }^{\ast }\right)\}>\alpha \right\}.$$ (10)

They note that the resulting confidence intervals are not compatible with the closed test procedure described in § 2·3 (Posch et al., 2005, p. 3702). Nevertheless, the region (10) can be used to generate an alternative multiple test. More generally, any 1 − α confidence set C generates a multiple test for a family of hypotheses $\mathcal{H}$, whereby $H_k\in \mathcal{H}$ is rejected if and only if H_k ∩ C = ∅. This guarantees strong control of the familywise error rate (1). The multiple test generated by (10) can be thought of as single-step in the sense that rejection or nonrejection of a null hypothesis does not take into account the decision for any other hypothesis. If H_k is rejected, informative lower bounds will be available for θ_k regardless of the test decisions for all other hypotheses.

4. Computation of confidence intervals

4·1. Least-favourable parameter configurations

In the above example, marginal inference on θ_B was achieved by considering least-favourable parameter configurations for θ_k, k ∈ T₁ \ {B}. This idea can be generalized to find 100(1 − α)% simultaneous confidence intervals containing (8) or (10).

Definition 1. For j = 1, 2, k ∈ T₁ and I ⊆ T_j, the locally least-favourable jth-stage p-value function for H_k in Θ_I, $p_{k,I}^{\left(j\right)}:ℝ\to [0,1]$, is defined for I ≠ ∅ as $p_{k,I}^{\left(j\right)}\left(\vartheta \right)=p_I^{\left(j\right)}\left(\xi \right)$, where ξ =(ξ₁, … , ξ_K) with ξ_i =δ_i for i ≠ k and ξ_k = ϑ. Additionally, for j = 1, 2,

$$p_{k,\varnothing }^{\left(j\right)}\left(\vartheta \right)=\underset{{\xi }_i\to \infty ,i\in T_j\backslash \left\{k\right\}}{\lim }{p}_{T_j}^{\left(j\right)}\left(\xi \right)\left({\xi }_k=\vartheta \right).$$ (11)

Proposition 2. The smallest Cartesian product of intervals, ×_k∈T1(l_k, ∞), that contains the confidence region (8) has l_k = min_I⊆T1 l_k,I, where for k ∈ I,

$$l_{k,I}=\{\begin{array}{cc}\infty \hfill & \left({\phi }_I=1\right),\hfill \\ \text{sup}\left\{\vartheta :Q\{p_{k,I}^{\left(1\right)}\left(\vartheta \right),p_{k,I\cap T_2}^{\left(2\right)}\left(\vartheta \right)\}⩽\alpha \right\}\hfill & \left({\phi }_I=0\right),\hfill \end{array}\phantom{\}}$$ (12)

and for k ∉ I,

$$l_{k,I}=\max \left({\delta }_k,\text{sup}\left\{\vartheta :Q\{p_{k,I}^{\left(1\right)}\left(\vartheta \right),p_{k,I\cap T_2}^{\left(2\right)}\left(\vartheta \right)\}⩽\alpha \right\}\right).$$ (13)

Furthermore, these intervals are compatible with the two-stage closed test procedure, i.e., ψ_k = 1 if and only if H_k ∩ ×_k∈T1(l_k, ∞)=∅.

Proof. See the Appendix.

In general, to find each interval requires one-dimensional root finding for each I ⊆ T₁, a calculation that is O(2^K). However, substantial shortcuts are available for reducing the computational burden.

4·2. Efficient computation of confidence bounds

There are two possible scenarios at the end of the closed test procedure: either ψ_k = 1 for all k ∈ T₂, or at least one H_k (k ∈ T₂) fails to be rejected. In the latter case, there exists some I ⊆ T₁ with I ∩ T₂ ≠ ∅ such that for any k ∈ T₂,

$$\alpha <Q\left(p_I^{\left(1\right)},p_{I\cap T_2}^{\left(2\right)}\right)=Q\left\{p_{k,I}^{\left(1\right)}\left({\delta }_k\right),p_{k,I\cap T_2}^{\left(2\right)}\left({\delta }_k\right)\right\}$$

and therefore l_k ⩽ l_k,I ⩽ δ_k. Due to the compatibility of the intervals with the closed test procedure, if ψ_k = 1, then l_k = δ_k; if ψ_k = 0, then l_k < δ_k.

If ψ_k =1 for all k ∈ T₂, then l_k ⩾ δ_k for all k ∈ T₂. Additionally, we can use the fact that for all k ∈ T₂ and I ⊆ T₁ with I ∩ T₂ ≠ ∅, we know from (12) and (13) that l_k,I = ∞; so, when finding l_k =min_I⊆T1 l_k,I in Proposition 2, the minimum can be taken over a much smaller number of l_k,I. The following algorithm finds the lower bounds for all parameters corresponding to the rejected hypotheses.

Step 1. Perform the closed test procedure. If ψ_k′ = 0 for some k′ ∈ T₂, then l_k = δk for ψ_k =1 and l_k < δ_k for ψ_k =0. If ψ_k =1 for all k ∈ T₂, go to Step 2.

Step 2. Find $p_{\mathrm{M}}={\max }_{\varnothing \ne I\subseteq T_1\backslash T_2}{p}_I^{\left(1\right)}$. If T₁ \ T₂ = ∅, then p_M =0.

Step 3. For k ∈ T₂,

$$l_k=\max \left[{\delta }_k,\text{sup}\left\{\vartheta :Q\left[\max \{p_{\mathrm{M}},p_{k,\varnothing }^{\left(1\right)}\left(\vartheta \right)\},p_{k,\varnothing }^{\left(2\right)}\left(\vartheta \right)\right]⩽\alpha \right\}\right].$$

The cost of computing the intervals for θ_k (k ∈ T₂) in Step 3 is linear in the number of parameters. Step 1 is O(2^|T₁|), but a shortcut of O(|T₁|²) is given in Brannath & Bretz (2010). Step 2 is O(2^|T₁\T₂|), but a shortcut of size |T₁ \ T₂| is available, provided there exists an ordering i₁, … , i_k of T₁ \ T₂ such that for each u ∈ {1, … , k}, $p_J^{\left(1\right)}⩽p_L^{\left(1\right)}$ for all J ⊆ L ⊆ {i_u, … , i_k} with i_u ∈ J. This is because we only have to check $p_{\{i_u,\dots ,i_k\}}$ for u =1, … , k. Many common multiple test procedures, such as those based on Dunnett (1955) tests or weighted Bonferroni tests, satisfy this condition, with the ordering i₁, … , i_k following the ordering of the univariate test statistics or the weighted elementary p-values (Brannath & Bretz, 2010).

4·3. Lower bounds for parameters corresponding to retained hypotheses

Consider k ∈ T₂ such that ψ_k = 0. We know that l_k < δ_k, and therefore we need only consider l_k,I such that k ∈ I. However, since in general l_k,I < ∞, finding the minimum such lower bound will still have a computational cost that is exponential in the number of parameters.

For k ∈ I ⊆ T₁ \ T₂, we have $p_{k,I\cap T_2}^{\left(2\right)}\left(\vartheta \right)=p_{k,\varnothing }^{\left(2\right)}\left(\vartheta \right)$ and know from (11) and (6) that this is equal to 1. Many commonly used combination functions, including (3), have the property that v = 1 implies $\mathcal{Q}\left(u,v\right)=1$. In this case, l_k = −∞ for all k ∈ T₁ \ T₂.

4·4. Lower bounds for the two-stage single-step procedure

Posch et al. (2005) showed that the region (10) is contained in a rectangle, ${\times }_{k\in T_1}({\stackrel{‒}{l}}_k,\infty )$, where

$${\stackrel{‒}{l}}_k=\text{sup}\left\{\vartheta :Q\{p_{k,\varnothing }^{\left(1\right)}\left(\vartheta \right),p_{k,\varnothing }^{\left(2\right)}\left(\vartheta \right)\right\}⩽\alpha \}.$$ (14)

The computation of each interval requires only a one-dimensional search for a root, and overall computation will be linear in the number of parameters.

4·5. Example continued

Recall from § 3·2 that T₂ = {B} and ψ_B = 1. Proceeding to Step 2 of the above algorithm, p_M =0·419. In this case we need just one iteration in Step 3, because

$$Q\left[\max \{0.419,p_{B,\varnothing }^{\left(1\right)}\left(0\right)\},p_{B,\varnothing }^{\left(2\right)}\left(0\right)\right]=0.0360>0.025,$$

and therefore the 97·5% confidence interval for θ_B is (0, ∞), consistent with Fig. 1. This example emphasizes that there is a price to pay for the additional power of the closed test as opposed to the single-step procedure of § 3·3 with, by (14),

$${\stackrel{‒}{l}}_B=\text{sup}\left\{\vartheta :Q\{p_{B,\varnothing }^{\left(1\right)}\left(\vartheta \right),p_{B,\varnothing }^{\left(2\right)}\left(\vartheta \right)\}⩽0.025\right\}=0.0159.$$

While this agrees with the assertion θ_B > 0 in this specific case, it is invalid to claim it as a 97·5% lower confidence bound if the closed test procedure of § 2·3 had been planned. One can see that for any α > 0·036, the 100(1 − α)% confidence interval for treatment B that is compatible with the closed test procedure has a positive lower bound. For example, the 95% lower confidence bound is l_B = 0·0112, consistent with Fig. 1. Again, if the region (10) had been specified pre-trial, the 95% lower confidence bound (14) would have been ${\stackrel{‒}{l}}_B=0\cdot 0252$.

5. Confidence bounds for closed tests based on the conditional error rate

Consider again the two-stage closed test procedure of § 2·3. As an alternative to combination tests, Koenig et al. (2008) used the conditional error approach (Proschan & Hunsberger, 1995) to derive local tests φ_I (I ⊆ T₁). The only difference is that instead of prespecifying a combination function Q and first-stage p-value $p_I^{\left(1\right)}$, one must prespecify a measurable conditional error function $A_I:{ℝ}^n\to [0,1]$ such that

$$\underset{{\theta }^{\ast }\in H_I}{\text{sup}}{\int }_{{ℝ}^n}{A}_I\left(x\right)\mathrm{d}G\left(x;{\theta }^{\ast }\right)⩽\alpha$$

and, at the final analysis, φ_I =1 if and only if $p_{I\cap T_2}^{\left(2\right)}⩽A_I\left(x\right)$.

To produce a compatible 100(1 − α)% confidence region for θ, each A_I (I ⊆ T₁) must be augmented with a family of conditional error functions {A_I(θ*) : θ* ∈ Θ} such that ${\int }_{{ℝ}^n}{A}_I\left({\theta }^{\ast }\right)\left(x\right)\mathrm{d}G\left(x;{\theta }^{\ast }\right)⩽\alpha$ and, for fixed $x\in {ℝ}^n$, A_I(θ*) is constant in all arguments ${\theta }_i^{\ast }$ with i ∉ I and is left-continuous and nonincreasing in all arguments ${\theta }_i^{\ast }$ with i ∈ I. Furthermore, A_I (θ*)= A_I for all θ* ∈ Θ such that ${\theta }_i^{\ast }={\delta }_i$ for i ∈ I. The second-stage p-values $p_{I\cap T_2}^{\left(2\right)}\left(I\subseteq T_1\right)$ must be augmented with a family $\{p_{I\cap T_2}^{\left(2\right)}\left({\theta }^{\ast }\right):{\theta }^{\ast }\in \Theta \}$ as described in § 3·1.

Müller & Schäfer (2004) propose defining A_I = sup_θ*∈HI E_θ*(ϕ_I | X), where ϕ_I is a pre-planned fixed sample level-α test for H_I. In many situations the natural choice for A_I(θ*) will be obvious from A_I. For example, if ϕ_I is the decision function for a Dunnett (1955) test of H_I = ⋂_k∈I{θ_k ⩽ δ_k}, then it is natural to choose A_I (θ*) = E_θ*(ϕ_I,θ* | X) where ϕ_I,θ* is the decision function for a Dunnett test of ${\bigcap }_{k\in I}\{{\theta }_k⩽{\theta }_k^{\ast }\}$ which can be derived via a corresponding translation of the test statistics.

Using the arguments of Propositions 1 and 2, it can be shown that, analogously to (8), a compatible 100(1 − α)% confidence region for θ is

$$\bigcup _{I\subseteq T_1}\left\{{\theta }^{\ast }\in {\Theta }_I:p_{I\cap T_2}^{\left(2\right)}\left({\theta }^{\ast }\right)>A_I\left({\theta }^{\ast }\right)\right\},$$

where $p_{\varnothing }^{\left(2\right)}\left({\theta }^{\ast }\right)$ and A_∅(θ*) are set equal to $p_{T_2}^{\left(2\right)}\left({\theta }^{\ast }\right)$ and A_T1(θ*) respectively. Also, the largest compatible 100(1 − α)% confidence lower bounds are l_k =min_I⊆T1 l_k,I, where for k ∈ I,

$$l_{k,I}=\{\begin{array}{cc}\infty \hfill & \left({\phi }_I=1\right),\hfill \\ \text{sup}\{\vartheta :p_{k,I\cap T_2}^{\left(2\right)}\left(\vartheta \right)⩽A_{k,I}\left(\vartheta \right)\}\hfill & \left({\phi }_I=0\right),\hfill \end{array}\phantom{\}}$$

and for k ∉ I, $l_{k,I}=\max [{\delta }_k,\text{sup}\{\vartheta :p_{k,I\cap T_2}^{\left(2\right)}\left(\vartheta \right)⩽A_{k,I}\left(\vartheta \right)\left\}\right]$ with A_k,I(ϑ) defined analogously to $p_{k,I}^{\left(1\right)}\left(\vartheta \right)\left(k\in T_1;I\subseteq T_1\right)$ in Definition 1.

6. Concluding remarks

The lower confidence bounds (12)–(13) provide more information about the location of θ than the decisions of the closed test procedure of § 2·3. The utility of this additional information will depend strongly on the context. In practice, the primary concern will often be to find lower bounds for the components of θ corresponding to the rejected null hypotheses. As this can be achieved using an algorithm that is O(K²), application to large-scale simultaneous inference problems is, in principle, feasible. However, these lower bounds will only be informative if all hypotheses considered in the second stage of testing are rejected, and even this may be insufficient. In practice, therefore, the lower bounds (12)–(13) are only likely to be useful in relatively small-scale problems. Furthermore, in situations where informative lower confidence bounds are deemed to be more important than the possibility of rejecting as many individual null hypotheses as possible, it would be sensible to use the intervals (14) instead of applying the closed test procedure. For large-scale simultaneous inference problems, an approach based on controlling the false coverage-statement rate (Benjamini & Yekutieli, 2005) may be more appropriate than aiming for a high simultaneous coverage probability.

Extensions to more than two stages and to allow early rejection of hypotheses are straightforward with an appropriate combination function in place of (3). An open question is how best to choose φ_∅(θ*) and $p_{\varnothing }^{\left(2\right)}\left({\theta }^{\ast }\right)$. The tests we use in region (8) are a natural choice but may not be the most powerful.

Appendix

Proof of Proposition 1. With the assumptions in § 3·1, all tests of the form (7) satisfy condition (4), and therefore C is a 100(1 − α)% confidence set for θ. By the monotonicity conditions imposed on the p-values, we have $p_{I\cap T_j}^{\left(j\right)}\left({\theta }^{\ast }\right)⩽p_{I\cap T_j}^{\left(j\right)}$ for all θ* ∈ Θ_I (j = 1,2; I ≠ ∅; I ⊆ T₁), so that Θ_I ∩ C = ∅ if and only if $\mathcal{Q}(p_I^{\left(1\right)},p_{I\cap T_2}^{\left(2\right)})⩽\alpha$. Therefore, ψ =1 if and only if min_I⊆T1,k∈I $\mathcal{Q}(p_I^{\left(1\right)},p_{I\cap T_2}^{\left(2\right)})⩽\alpha$ if and only if ⋃_I⊆T1,k∈I ΘI ∩ C = ∅. Since ⋃_I⊆T1,k∈I Θ_I = H_k, we have compatibility.

Proof of Proposition 2. First, note the key property that $p_{k,I\cap T_j}^{\left(j\right)}\left(\vartheta \right)⩾p_{I\cap T_j}^{\left(j\right)}\left({\theta }^{\ast }\right)$ for all θ* ∈ Θ_I with ${\theta }_k^{\ast }⩽\vartheta \left(I\subseteq T_1;k\in T_1;j=1,2\right)$.

To show that C₁⊆×_k∈T1 (l_k, ∞), consider any θ* ∈ Θ \ ×_k∈T1 (l_k, ∞). We must have θ* ⊆ Θ_I for some I ⊆ T₁ and ${\theta }_k^{\ast }⩽l_k$ for some k ∈ T₁. If k ∈ I, then ${\theta }_k^{\ast }⩽\min \left({\delta }_k,l_{k,I}\right)$, and (12) implies that $\alpha ⩾\mathcal{Q}\{p_{k,I}^{\left(1\right)}\left({\theta }_k^{\ast }\right),p_{k,I\cap T_2}^{\left(2\right)}\left({\theta }_k^{\ast }\right)\}⩾\mathcal{Q}\{p_I^{\left(1\right)}\left({\theta }^{\ast }\right),p_{I\cap T_2}^{\left(2\right)}\left({\theta }^{\ast }\right)\}$. The same inequality follows from $l_{k,I}⩾{\theta }_k^{\ast }>{\delta }_k$ and (13) if k ∉ I. Therefore, θ* ∉ C₁ and C₁ ⊆ ×_k∈T1 (l_k, ∞).

To show that no smaller interval (l_k + ϵ, ∞) is possible for any ϵ > 0, we must find some θ* ∈ C₁ with ${\theta }_k^{\ast }\in \left(l_k,l_k+ϵ\right)$. Consider a subset I ⊆ T₁ such that l_k = l_k,I and therefore $\mathcal{Q}\{p_{k,I}^{\left(1\right)}\left(\vartheta \right),p_{k,I\cap T_2}^{\left(2\right)}\left(\vartheta \right)\}>\alpha$ for all ϑ > l_k. If k ∈ I or, equivalently, l_k < δ_k, take any ${\theta }_k^{\ast }\in \left(l_k,\min \{{\delta }_k,l_k+ϵ\}\right)$. If k ∉ I or, equivalently, l_k ⩾ δ_k, take any ${\theta }_k^{\ast }\in \left(l_k,l_k+ϵ\right)$. Now consider a parameter vector ${\xi }^{I,k}=({\xi }_1^{I,k},\dots ,{\xi }_K^{I,k})$, where ${\xi }_k^{I,k}={\theta }_k^{\ast }$, ${\xi }_i^{I,k}={\delta }_i$ for k ≠ i ∈ I, and ${\xi }_i^{I,k}>{\delta }_i$ for i ∉ I ∪ {k}. All such parameter vectors ξ^I,k are contained in Θ_I, and

$$\alpha <Q\left\{p_{k,I}^{\left(1\right)}\left({\theta }_k^{\ast }\right),p_{k,I\cap T_2}^{\left(2\right)}\left({\theta }_k^{\ast }\right)\right\}=\underset{{\xi }_i^{I,k}\to \infty ,i\notin I\cup \left\{k\right\}}{\lim }Q\left\{p_I^{\left(1\right)}\left({\xi }^{I,k}\right),p_{I\cap T_2}^{\left(2\right)}\left({\xi }^{I,k}\right)\right\}.$$

Thus there exists some such ξ^I,k ∈ C₁ and hence C₁ is not contained in this smaller product of intervals.

Finally, H_k ∩×_k∈T1 (l_k, ∞) = ∅ if and only if l_k,I ⩾ δ_k for I ⊆ T₁. if and only if $\mathcal{Q}\{p_{k,I}^{\left(1\right)}\left({\delta }_k\right)$, $p_{k,I\cap T_2}^{\left(2\right)}\left({\delta }_k\right)\}=\mathcal{Q}\{p_I^{\left(1\right)},p_{I\cap T_2}^{\left(2\right)}\}⩽\alpha$ for I ⊆ T₁ and k ∈ I, if and only if ψ_k = 1.