Simultaneous confidence intervals for an extended Koch-Röhmel design in three-arm non-inferiority trials

Abstract

Three-arm ‘gold-standard’ non-inferiority trials are recommended for indications where only unstable reference treatments are available and the use of a placebo group can be justified ethically. For such trials, several study designs have been suggested that use the placebo group for testing ’assay sensitivity’, that is, the ability of the trial to replicate efficacy. Should the reference fail in the given trial, then non-inferiority could also be shown with an ineffective experimental treatment and hence becomes useless. In this article, we extend the so-called Koch-Röhmel design where a proof of efficacy for the experimental treatment is required in order to qualify for the non-inferiority test. While the efficacy of the experimental treatment is an indication of assay sensitivity, it does not guarantee that the reference is sufficiently efficient to let the non-inferiority claim be meaningful. It has, therefore, been suggested to adaptively test the non-inferiority only if the reference demonstrates superiority to placebo and otherwise to test $\delta$ -superiority of the experimental treatment over placebo, where $\delta$ is chosen in such a way that it provides proof of non-inferiority with regard to the reference’s historical effect. In this article, we extend the previous work by complementing its adaptive test with compatible simultaneous confidence intervals. Confidence intervals are commonly used and suggested by regulatory guidelines for non-inferiority trials. We show how to adopt different approaches to simultaneous confidence intervals from the literature to the setting of three-arm non-inferiority trials and compare these methods in a simulation study. Finally, we apply these methods to a real clinical trial example.

1. Introduction

We consider three-arm trials including an experimental treatment ( $E$ ), an established reference ( $R$ ) and a placebo (see, e.g. Koch and Röhmel ¹ ). Typical indications for such trials are, for example, asthma or depression, where the use of a placebo is acceptable from an ethical point of view, and where the effect of the reference has normally a high variance. New treatments are desired to obtain better effects for certain patient collectives or to reduce side effects. An effect that is not inferior to that of the reference is then regarded as successful. To assure assay sensitivity, a placebo group is included in the trial.

Denote by ${\mu }_i$ the effect in group $i$ , where $i\in \{E,R,P\}$ . We will use the following notations for the hypotheses of interest:

$$\begin{array}{rl}{H}_{EP}^S& :{\mu }_E-{\mu }_P\le 0\text{(to show superiority of }E\text{ over }P\text{)}.\\ H_{ER}^N& :{\mu }_E-{\mu }_R\le -{\delta }_0\text{(to show non-inferiority of }E\text{ compared to }R\text{ with margin }{\delta }_0>0\text{)}.\\ H_{EP}^{{\delta }_1}& :{\mu }_E-{\mu }_P\le {\delta }_1\text{(to show superiority by }{\delta }_1>0\text{ of }E\text{ over }P\text{)}.\end{array}$$

Without loss of generality, we can assume that ${\mu }_P=0$ . The non-inferiority margin is normally chosen as ${\delta }_0=r{\mu }_R^h$ , where $r\in (0,1)$ and ${\mu }_R^h$ is the historical reference effect, that is, the effect observed in earlier studies. A common choice is $r=1/2$ or smaller. Note, that any other choice of ${\delta }_0$ , for example, by clinical considerations, is also possible and can be represented by $r{\mu }_R^h$ for some $r$ . We will come to the choice of ${\delta }_1$ immediately.

The gold-standard for testing in three-arm trials was proposed by Koch and Röhmel ¹ and consists of a hierarchical procedure rejecting first $H_{EP}^S$ to assure assay sensitivity and then rejecting $H_{ER}^N$ to confirm efficacy of the new treatment. There are several possibilities to proceed after the second rejection. An important issue is the uncertainty about the strength of the reference in the study. If actually ${\mu }_R=0$ , then the rejection of $H_{ER}^N$ is both easy and worthless (see Hauschke and Pigeot ² ). Since it is unsatisfying to spend level on showing superiority of the reference over placebo, an idea is to consider the hypothesis $H_{EP}^{{\delta }_1}$ instead. This targets the clinical relevance of the new treatment and implicitly assures non-inferiority to the historical reference effect. We propose to choose ${\delta }_1=(1-r){\mu }_R^h$ . If the effect of the reference in the study corresponds to the historical effect, that is, ${\mu }_R={\mu }_R^h$ , then non-inferiority implies ${\mu }_E>{\mu }_R-{\delta }_0=(1-r){\mu }_R^h$ . On the other hand, if the reference in the study is weak, then the rejection of $H_{EP}^{{\delta }_1}$ assures likewise that ${\mu }_E>{\delta }_1=(1-r){\mu }_R^h$ . Hence, non-inferiority to the (historical) reference with margin $r{\mu }_R^h={\delta }_0$ is shown.

In Brannath et al., ³ a flexible extension of the Koch-Röhmel design was proposed, which is in the spirit of the above argumentation. A hierarchical test of the sequence $H_{EP}^S$ , $H_{ER}^N$ , $H_{EP}^{{\delta }_1}$ was introduced, and success of the study was interpreted in dependence of the reference strength. If ‘ $R>P$ ’, then success is reached if $H_{ER}^N$ is rejected. If ‘ $R\le P$ ’, then success means rejecting $H_{EP}^{{\delta }_1}$ . The test of the hypothesis $H_{RP}^S:{\mu }_R-{\mu }_P\le 0$ served as a filter (‘ $R>P$ ’ or ‘ $R\le P$ ’) for the interpretation of the study results. It was shown that the overall probability for erroneously declaring success can be controlled at level $\alpha$ without a confirmatory test of the reference effect.

It is well known that confidence intervals provide more information than hypothesis testing. They are particularly important for non-inferiority trials. For example, we can quantify the amount by which ${\mu }_E$ is superior to ${\mu }_P$ , or even superior to ${\mu }_P+{\delta }_1$ . We may also be able to improve the non-inferiority to the reference or even show superiority, that is, prove that ${\mu }_E-{\mu }_R>0$ . In this article, we extend the idea of Brannath et al. ³ by introducing three sets of simultaneous confidence intervals (SCIs) for the effects ${\mu }_E-{\mu }_P$ and ${\mu }_E-{\mu }_R$ . The first set of proposed SCIs is based on stepwise confidence intervals introduced by Hsu and Berger. ⁴ We will find an intrinsic filter describing the strength of the reference, which is perfectly compatible with the interpretation of these simultaneous confidence bounds. Thus, we obtain an intuitive procedure to decide about the success of the study and obtain information via a SCI. Another set of SCIs will be based on the work of Schmidt and Brannath, ⁵ who introduced informative confidence intervals in hierarchical testing. Finally, we will compare these two sets of SCIs to the single step SCIs which can easily be derived for the aforementioned effects.

The article is organized as follows. We introduce the SCIs in Section 2 and give numerical results on their performance in Section 3. We apply the different intervals to data of clinical trial in Section 4 and end with a discussion in Section 5.

2. Description of the SCIs

2.1. SCIs with intersection union filter

We assume that all observations are normally distributed with common standard deviation $\sigma >0$ . Denote by $n_i$ the sample size of group $i\in \{E,R,P\}$ . Then the observed mean in group $i$ is $X_i\sim N({\mu }_i,{\sigma }^2/n_i)$ . Define the univariate confidence bounds

$${\ell }_{EP}=X_E-X_P-z_{\alpha }\sigma \sqrt{n_E^{-1}+n_P^{-1}}\text{and}{\ell }_{ER}=X_E-X_R-z_{\alpha }\sigma \sqrt{n_E^{-1}+n_R^{-1}}$$

where $z_{\alpha }={\mathrm{\Phi }}^{-1}(1-\alpha )$ is the quantile of the standard normal distribution. According to Hsu and Berger, ⁴ simultaneous lower confidence bounds for $\theta =({\theta }_1,{\theta }_2)=({\mu }_E-{\mu }_P,{\mu }_E-{\mu }_R)$ with coverage probability $(1-\alpha )$ are given by

$$L_{EP}=\{\begin{array}{ll}{\ell }_{EP}& \text{if }{\ell }_{EP}<0\\ 0& \text{if }{\ell }_{EP}\ge 0,{\ell }_{ER}<-{\delta }_0\\ L_{min}:=min\{{\ell }_{EP},{\ell }_{ER}+{\delta }_0\}& \text{if }{\ell }_{EP}\ge 0,{\ell }_{ER}\ge -{\delta }_0\end{array}$$ (1)

and

$$L_{ER}=\{\begin{array}{ll}-\mathrm{\infty }& \text{if }{\ell }_{EP}<0\\ {\ell }_{ER}& \text{if }{\ell }_{EP}\ge 0,{\ell }_{ER}<-{\delta }_0\\ L_{min}-{\delta }_0=min\{{\ell }_{EP}-{\delta }_0,{\ell }_{ER}\}& \text{if }{\ell }_{EP}\ge 0,{\ell }_{ER}\ge -{\delta }_0\end{array}$$ (2)

These SCIs are compatible with the rejection decisions of the Koch-Röhmel design, that is, with a hierarchical test of $H_{EP}^S$ and $H_{ER}^N$ as by Maurer et al. ⁶ The first line of (1) and (2) corresponds to the case where the hypothesis $H_{EP}^S$ cannot be rejected. In this case, the study is considered a failure. The second line is the case where $H_{EP}^S$ is rejected, but $H_{ER}^N$ is not. We also consider this case as failure, even though non-inferiority is not of interest in the case of a weak reference. However, if the reference fails then $H_{ER}^N$ should be rejected easily. The most important situation occurs in the third line of (1) and (2), where both hypotheses $H_{EP}^S$ and $H_{ER}^N$ are rejected. The confidence bounds $L_{EP}$ and $L_{ER}$ are then both determined by either information on the effect ${\mu }_E-{\mu }_P$ via ${\ell }_{EP}$ or on the effect ${\mu }_E-{\mu }_R$ via ${\ell }_{ER}$ , where the boundary between these two options can easily be calculated. Under the last condition of (1) and (2):

$$L_{min}={\ell }_{ER}+{\delta }_0⟺X_R-X_P\ge z_{\alpha }\sigma (\sqrt{n_E^{-1}+n_P^{-1}}-\sqrt{n_E^{-1}+n_R^{-1}})+{\delta }_0$$ (3)

It appears unnatural at first sight that $L_{ER}$ , which is a lower bound for ${\theta }_1$ , is given by an estimate of ${\theta }_2$ in some cases, and vice versa. However, there is a transparent way of interpreting the bounds: If the reference is ‘strong’ in the sense of (3), then the confidence bound for the effect of $E$ versus $R$ is of interest, which is reflected by the fact that $L_{min}={\ell }_{ER}+{\delta }_0$ and hence the level is fully exploited for the parameter ${\mu }_E-{\mu }_R$ . In the other case, where the reference is ‘weak’ so that the inequality in (3) is not satisfied, then $L_{min}={\ell }_{EP}$ , that is, the level is exploited for ${\mu }_E-{\mu }_P$ . This makes sense, because the difference of the new treatment to the weak reference is not of interest and non-inferiority to the historical reference is targeted indirectly via a bound for the effect of $E$ versus $P$ . In summary, the study can be judged as successful if either the reference is strong and $L_{ER}\ge -{\delta }_0=-r{\mu }_R^h$ or the reference is weak and $L_{EP}\ge {\delta }_1=(1-r){\mu }_R^h$ . This inference can be made without needing to test the reference effect directly. Thus, equation (3) serves as an intrinsic filter for the interpretation of the study results.

Formally, the definition of $L_{min}$ is an application of nested intersection union (IU) tests of hypotheses $H_{\mathrm{\Lambda }}^{\vartheta }:\mathrm{\Lambda }\le \vartheta$ at level $\alpha$ for the parameter $\mathrm{\Lambda }=min\{{\mu }_E-{\mu }_P,{\mu }_E-{\mu }_R+{\delta }_0\}$ . The value of $\mathrm{\Lambda }$ equals ${\mu }_E-{\mu }_P$ if and only if ${\mu }_R\le {\mu }_P+{\delta }_0$ , otherwise it quantifies ${\mu }_E-{\mu }_R+{\delta }_0$ . This corresponds to the philosophy of a weak versus strong reference. Since the SCIs (1) and (2) and the filter (3) are the application of nested IU tests, we will call the filter IU filter.

The procedure of defining SCIs and applying them to define the success of the study can be illustrated graphically using the notation of Schmidt and Brannath. ⁵ This is done in Figure 1. The test of the hypothesis family $(H_{EP}^{\vartheta }{)}_{\vartheta \le 0}$ (at level $\alpha$ ) leads to a confidence bound $L_{EP}={\ell }_{EP}<0$ if $H_{EP}^S$ cannot be rejected, or $L_{EP}=0$ , otherwise. In the first case, the procedure stops with $L_{ER}=-\mathrm{\infty }$ , while in the latter case, the family of hypotheses $(H_{ER}^{\vartheta }{)}_{\vartheta \le -{\delta }_0}$ is tested (at level $\alpha$ ), resulting in a confidence bound $-\mathrm{\infty }<L_{ER}\le -{\delta }_0$ . If $L_{ER}=-{\delta }_0$ , that is, if $H_{ER}^N$ is rejected, then the level $\alpha$ is shifted further to the hypothesis family $(H_{\mathrm{\Lambda }}^{\vartheta }{)}_{\vartheta >0}$ , improving the confidence bounds $L_{EP}$ and $L_{ER}$ simultaneously.

Figure 1.

Formal description of the simultaneous confidence intervals. The hypotheses are $H_{EP}^{\vartheta }:{\mu }_E-{\mu }_P\le \vartheta$ , $H_{ER}^{\vartheta }:{\mu }_E-{\mu }_R\le \vartheta$ and $H_{\mathrm{\Lambda }}^{\vartheta }:\mathrm{\Lambda }\le \vartheta$ for $\mathrm{\Lambda }=min\{{\mu }_E-{\mu }_P,{\mu }_E-{\mu }_R+{\delta }_0\}$ . The intersection union (IU) filter ‘ $R>P$ ’ is given in (3).

This procedure modifies the hierarchical test design introduced by Brannath et al., ³ where the hypothesis $H_{EP}^{{\delta }_1}$ is tested after rejection of $H_{EP}^S$ and $H_{ER}^N$ . Here, more information about the size of both effects ${\mu }_E-{\mu }_P$ and ${\mu }_E-{\mu }_R$ can be gained via a confidence interval for $\mathrm{\Lambda }$ . Another difference is that the filter ‘ $R<P$ ’ is now given via (3), while it was a level- $\alpha$ -test of the hypothesis $H_{RP}:{\mu }_R-{\mu }_P\le 0$ in Brannath et al. ³ The new filter has a remarkable advantage. Indeed, it was discussed by Brannath et al. ³ that the rather complex strategy in Figure 1 can be replaced by an intuitive strategy for many situations. Actually, we can give an intuitive picture for all situations if we use the new filter. The corresponding graph is shown in Figure 2. After rejection of the gatekeeper $H_{EP}^S$ , it is decided via the filter whether to look at the lower bound for ${\mu }_E-{\mu }_R$ or the lower bound for ${\mu }_E-{\mu }_P$ . In the first case $R>P$ , the study is successful if $L_{ER}:={\ell }_{ER}\ge -{\delta }_0$ , in the second case, $R<P$ , it is successful if $L_{EP}:={\ell }_{EP}\ge {\delta }_1$ . Additionally, one can report $L_{EP}=max\{0,L_{ER}+{\delta }_0\}$ in the first case, or respectively, $L_{ER}=L_{EP}-{\delta }_0$ in the second case. Although the procedure in Figure 2 is data-driven via the filter, one can show that it maintains the given coverage probability $1-\alpha$ , because it is equivalent to the procedure in Figure 1, which is based on the intervals from Hsu and Berger ⁴ (see the Appendix). Therefore, also the probability of erroneously stating success of the study is controlled.

Figure 2.

Intuitive graphical description and interpretation of the simultaneous confidence intervals. Notations as in Figure 1.

2.2. Informative SCIs

Next, we present another way to calculate simultaneous lower confidence bounds for $({\mu }_E-{\mu }_P,{\mu }_E-{\mu }_R)$ with coverage probability $(1-\alpha )$ by employing the method of Schmidt and Brannath. ⁵ They derived the so-called informative SCIs in hierarchical testing. A confidence bound is informative, if it does not stick to the border of the null-hypothesis interval with positive probability, in case the null hypothesis is rejected.

The procedure for deriving the simultaneous lower confidence bounds, denoted by $L_{EP}^{inf}$ and $L_{ER}^{inf}$ in the following, is illustrated in Figure 3 and works as follows: First, fix a value $0<q<1$ . Then, $H_{EP}^S:{\mu }_E-{\mu }_P\le 0$ is tested at full level $\alpha$ . If $H_{EP}^S$ cannot be rejected, the procedure stops and reports $L_{EP}^{inf}={\ell }_{EP}$ and $L_{ER}^{inf}=-\mathrm{\infty }$ . If $H_{EP}^S$ can be rejected, the full level $\alpha$ is passed to testing $H_{ER}^N$ . If $H_{ER}^N$ cannot be rejected, the procedure stops and reports $L_{EP}^{inf}=0$ and $L_{ER}^{inf}={\ell }_{ER}$ . If however, also $H_{ER}^N$ is rejected, we test each $H_{ER}^{\vartheta }:{\mu }_E-{\mu }_R\le \vartheta$ at level $q^{\vartheta +{\delta }_0}\alpha$ . The lower confidence bound $L_{ER}^{inf}$ for ${\mu }_E-{\mu }_R$ will then be the first $\vartheta$ for which $H_{ER}^{\vartheta }$ cannot be rejected at the respective level. It can be shown that this bound is the unique solution to

$$1-\mathrm{\Phi }\left\{\frac{X_E-X_R-\vartheta }{\sigma \sqrt{n_E^{-1}+n_R^{-1}}}\right\}=q^{\vartheta +{\delta }_0}\alpha .$$

Figure 3.

Graphical description of the algorithm to determine informative SCIs for ${\mu }_E-{\mu }_P$ and ${\mu }_E-{\mu }_R$ , based on the notation and procedure of Schmidt and Brannath. ⁵

In the next step, we pass the remaining $\alpha$ -level ( $1-q^{L_{ER}^{inf}+{\delta }_0}\alpha$ ) to the hypotheses $H_{EP}^{\vartheta }$ and determine the lower confidence bound for ${\mu }_E-{\mu }_P$ as $L_{EP}^{inf}=max\{0,{\tilde{\ell }}_{EP}\}$ , where

$${\tilde{\ell }}_{EP}=X_E-X_P-z_{q^{L_{ER}^{inf}+{\delta }_0}\alpha }\sigma \sqrt{n_E^{-1}+n_P^{-1}}$$

Note, that in the cases where $H_{EP}^S$ or $H_{ER}^N$ cannot be rejected, these SCIs coincide with those presented in the previous section. Only in the case, where E is superior to P and non-inferior to R, the two procedures to calculate SCIs differ.

Note, that the intervals introduced here are not informative in the sense indicated at the beginning of this subsection. This is due to the fact, that if $H_{EP}^S$ is rejected, but $H_{ER}^N$ is not, which might be the case with positive probability, the confidence bound for ${\mu }_E-{\mu }_P$ is 0 by construction. In this case, the study would fail in the sense that it was not able to show a sufficient treatment effect of the experimental treatment. Since the SCIs introduced here are based on the informative SCIs introduced by Schmidt and Brannath, ⁵ we will refer to them as informative SCIs in the following.

While the SCIs introduced in Section 2.1 offered an intrinsic filter to determine whether ‘ $R>P$ ’ or ‘ $R\le P$ ’, this is not the case for the SCIs introduced in this section. For the interpretation of the confidence intervals, respectively, the study results, we will use the superiority filter presented by Brannath et al., ³ which declares ‘ $R>P$ ’ if

$$\frac{X_R-X_P}{\sigma \sqrt{n_R^{-1}+n_P^{-1}}}\ge z_{\alpha }$$ (4)

that is if $H_{RP}^S$ can be rejected. The interpretation then works similar to the case of Section 2.1. After rejection of $H_{ER}^S$ we can declare a successful study, if either ‘ $R>P$ ’ (i.e. $H_{RP}^S$ can be rejected) and $L_{ER}^{inf}>-{\delta }_0$ , or if ‘ $R\le P$ ’ (i.e. $H_{RP}^S$ cannot be rejected) and $L_{EP}^{inf}>{\delta }_1$ . Furthermore, $L_{EP}^{inf}$ , respectively, $L_{ER}^{inf}$ can additionally be reported.

2.3. Single-step SCIs

We now briefly introduce simultaneous single-step confidence intervals for the effects considered. To this end, we consider the test statistics

$$T_{EP}=\frac{X_E-X_P}{\sigma \sqrt{n_E^{-1}+n_P^{-1}}},T_{ER}^N=\frac{X_E-X_R+{\delta }_0}{\sigma \sqrt{n_E^{-1}+n_R^{-1}}}$$

For large sample sizes $n_E,n_R$ and $n_P$ and for ${\mu }_E={\mu }_P$ and ${\mu }_E-{\mu }_R=-{\delta }_0$ (i.e. $H_{EP}^S\cap H_{ER}^N$ is true), the vector of test statistics $(T_{EP},T_{ER}^N{)}^{\mathrm{\prime }}$ is asymptotically multivariate normally distributed with mean vector $\mu =(0,0)$ and covariance matrix $\mathrm{\Sigma }=\left(\begin{array}{cc}1& \rho \\ \rho & 1\end{array}\right)$ with $\rho =\sqrt{\frac{c_P{c}_R}{(1+c_P)(1+c_R)}}$ for $c_R=n_R/n_E$ and $c_P=n_P/n_E$ . Let $d_{\alpha }$ be the equicoordinate $1-\alpha$ quantile of this distribution. With this definition, it is well known (cf. Hsu ⁷ ) that

$$L_{EP}^S=X_E-X_P-d_{\alpha }\sigma \sqrt{n_E^{-1}+n_P^{-1}}\text{and}{L}_{ER}^S=X_E-X_R-d_{\alpha }\sigma \sqrt{n_E^{-1}+n_R^{-1}}$$

give simultaneous lower confidence bounds for $({\mu }_E-{\mu }_P,{\mu }_E-{\mu }_R)$ with coverage probability $(1-\alpha )$ . As for the intervals introduced in Section 2.2, we need a filter for the interpretation of study results, when using the single-step SCIs introduced here. We will also use the superiority filter (4) and declare success of the study if either ‘ $R>P$ ’ (i.e. $H_{RP}^S$ can be rejected) and $L_{ER}^S>-{\delta }_0$ , or if ‘ $R\le P$ ’ (i.e. $H_{RP}^S$ cannot be rejected) and $L_{EP}^S>{\delta }_1$ .

3. Properties of SCIs and probability of success

3.1. Optimal sample sizes

Brannath et al. ³ presented a method to calculate the minimal necessary sample size to obtain a certain probability of success (i.e. probability of correctly declaring the study a ‘success’) which is based on a procedure described by Schlömer and Brannath. ⁸ By ‘success’ we mean that we can show either non-inferiority of the new treatment to the reference in the case where the filter is satisfied (denoted by ‘Success ER’ in the following), or to show ${\delta }_1$ -superiority of the new treatment over placebo in the case where the filter is not satisfied (denoted by ‘Success EP’). The overall success probability (‘Total PoS’) is the sum of Success ER and Success EP (for further explanation see Section 3.2). The algorithm to determine the optimal sample sizes is based on the observation, that the hypotheses $H_{EP}^{\vartheta }:{\mu }_E-{\mu }_P\le \vartheta ,H_{ER}^{\vartheta }:{\mu }_E-{\mu }_R\le \vartheta$ and $H_{RP}^{\vartheta }:{\mu }_R-{\mu }_P\le \vartheta$ can be tested using test statistics which are multivariate normally distributed. Given specific sample size allocations (specified via the ratios $c_R=n_R/n_E$ and $c_P=n_P/n_E$ ) a probability of success, a significance level $\alpha$ and assumptions on the effects of E, R and P, the total sample size needed can then be derived via the cumulative distribution function of the respective multivariate normal distribution. In order to obtain the optimal sample sizes, the required total sample size N is optimized over the sample size allocations.

The method presented by Brannath et al. ³ can also be applied to our setup with the IU filter (3). We want to compare the optimal sample sizes derived for this setup to those derived for the superiority filter (4). To compare the two approaches, we make the following assumptions:

• desired power $90\mathrm{\%}$ for the success of the study,

• one-sided FWER $\alpha =2.5\mathrm{\%}$ ,

• independent normal observations with common standard deviation $\sigma =2$ ,

• the effect of the new treatment is equal to the historical reference effect, ${\mu }_E-{\mu }_P=1$ .

We consider three different scenarios for the effect of the reference over the placebo where we assume a non-inferiority margin of ${\delta }_0=0.5$ , which corresponds to half of the historical reference effect:

• Scenario 1: the reference is as good as observed historically, ${\mu }_R-{\mu }_P=1$ .

• Scenario 2: the reference is only half as good as observed historically, ${\mu }_R-{\mu }_P=0.5$ .

• Scenario 3: the reference fails and is as good as placebo, ${\mu }_R-{\mu }_P=0$ .

Furthermore, we consider the setup of scenario 1 but we assume a non-inferiority margin of ${\delta }_0=0.25$ , which corresponds to 25% of the historical reference effect:

• Scenario 4: the reference is as good as observed historically, ${\mu }_R-{\mu }_P=1$ , ${\delta }_0=0.25$ .

Scenarios 2 and 3 are particularly relevant if the reference treatment is assumed to be ‘unstable’. Unstable reference effects are observed in indications such as depression, where there is a rather large uncertainty about the mean treatment effect to be expected in a specific trial, inter alia due to individual variability in treatment effects (see also EMA ⁹ and Kaiser et al. ¹⁰ ).

Table 1 and Figure 4 show the optimal sample sizes for the approach derived by Brannath et al. ³ (‘superiority filter’) and for the procedures involving SCIs as introduced in Section 2. As we can see, the informative SCIs require only slightly larger sample sizes than the procedure of Brannath et al. ³ throughout all scenarios. The single step SCIs lead to the largest sample sizes for most values of $v$ . For the scenario in which the reference presents with the full historical effect, the sample size needed for the IU filter is higher than that needed for the informative SCIs. For the other scenarios, in which the reference effect is assumed smaller than the historical reference effect, the required sample size for the IU filter is smaller than that of the informative SCIs.

Table 1.

Optimal sample sizes for the flexible non-inferiority design comparing the SCIs introduced in Section 2 to the procedure of Brannath et al. ³ without SCIs (see text for details).

Scenario		Superiority filter	IU filter	Informative	Single step
1	$n_E$	345	356	349	402
	$n_R$	350	348	348	406
	$n_P$	102	145	104	100
	$N$	797	849	801	908
2	$n_E$	185	227	159	134
	$n_R$	182	75	216	253
	$n_P$	303	285	313	323
	$N$	670	587	688	710
3	$n_E$	341	306	348	397
	$n_R$	44	33	52	44
	$n_P$	339	325	346	399
	$N$	724	661	746	840
4	$n_E$	1352	1355	1345	1587
	$n_R$	1354	1354	1362	1593
	$n_P$	107	123	108	106
	$N$	2813	2832	2815	3286

SCIs: simultaneous confidence intervals; IU: intersection union.

Figure 4.

Optimal sample sizes for the flexible non-inferiority design comparing the simultaneous confidence intervals (SCIs) introduced in Section 2 to the procedure of Brannath et al. ³ without SCIs (see text for details). The $x$ -axis is the ratio $v=$ reference effect/historical reference effect. Because of numerical instabilities, the values for the informative SCIs were smoothed. The grey line at $v=1$ indicates the scenario, where the reference is assumed to be as strong as historically observed.

For all methods, we see that the sample size for the reference group increases in $v$ most of the time, while the sample size for the placebo group is decreasing in $v$ . This is due to the fact that for increasing $v$ , the focus of the procedures is shifted from the superiority comparison of experimental treatment versus placebo (which is more relevant for small $v$ , i.e. a weak reference) to the non-inferiority comparison of experimental versus reference treatment (which is more relevant for large $v$ , i.e. a strong reference).

We can see in the lower left panel of Figure 4, that there is a ‘bump’ in the sample size of the reference group for $v$ around 0.5 for the method without SCIs as well as the single step and informative SCIs. This is due to the fact that in this region, the superiority filter, which is used for the interpretation of the study results with theses procedures, begins to declare ‘ $R>P$ ’, which directs the focus from testing $H_{EP}$ towards $H_{ER}$ . Hence, a higher sample size in the reference group is beneficial in these situations.

Usually, we are interested to keep the placebo group as small as possible. We can see from the lower right panel of Figure 4 that the single step intervals require substantially larger placebo group sample sizes than the other procedures for small values $v$ . For values of $v$ larger than 0.6 we observe, that the UI intervals require larger sample sizes in the placebo group than the other methods. The informative intervals seem to offer a good compromise with sample sizes near those of the procedure of Brannath et al., ³ while offering the benefit of SCIs.

Comparing the results of scenario 4 to those of scenario 1, we observe that choosing a smaller non-inferiority margin leads to the expected effect of inflating the required sample sizes substantially. However, it can be seen that also in this setup the additional sample size required for the informative and IU intervals compared to the procedure without SCIs is negligible.

Note, that usually we would only consider values of $v\le 1$ , that is, the reference was considered at most as good as historically observed in the sample size planning. This seems natural in the setup of possibly unstable references. One of the reviewers made us aware of the fact, that values of $v>1$ could still be considered in a setting with equivalence tests. Therefore, Figure 4 also shows values of $v>1$ . In the given scenario, where the effect of the new treatment is assumed to equal the historical reference effect, sample sizes substantially increase for such $v$ , since showing non-inferiority to reference for higher reference-effects is harder.

As seen in Table 1 and Figure 4, the optimal sample size is very sensitive to the assumed reference effect in the current trial. Since there might be some uncertainty about the ratio $v$ of the reference effect in the study divided by the historical reference effect, Brannath et al. ³ proposed an ‘hybrid’ Bayesian-frequentist approach to the sample size calculation. In this approach, several ratios for $v$ are considered and weighted with an assumed probability of their occurrence. More general, denote the density of the probability distribution of $v$ on $[0,1]$ with $f(v)$ . Then the weighted success probability is given by

$$S={\int }_0^1{S}_{v}f(v)dv$$ (5)

where $S$ is the targeted success probability and $S_v$ is the success probability if $v$ is the true ratio. Brannath et al. ³ stated that $S$ is often referred to as ‘probability of success’ (cf. Kunmann et al. ¹¹ ) or ‘assurance’ (cf. Stallard et al. ¹² ). A very simple application of this approach is to define a grid of values for $v$ and assign probabilities for each of the values. To illustrate this approach, we use a simple example: Since we do not expect the reference to fail completely, we expect $v=1$ with some probability $p$ and we assume with probability $1-p$ that $v=3/4$ . The probability of success in (5), which is targeted at $90\mathrm{\%}$ , as before, is then given by

$$S=p{S}_1+(1-p)S_{3/4}$$ (6)

Figure 5 shows the optimal sample sizes obtained with this approach for $p\in [0,1]$ . We see that for all values of $p$ the informative intervals yield sample sizes close to those of the procedure of Brannath et al. ³ We can also see that the IU intervals yield to a substantially larger placebo group, compared to the other methods.

Figure 5.

Optimal sample sizes for the flexible non-inferiority design comparing the simultaneous confidence intervals (SCIs) introduced in Section 2 to the procedure of Brannath et al. ³ without SCIs (see text for details). The $x$ -axis is the probability $p$ that $v=1$ . $v=3/4$ is assumed with probability $1-p$ . Because of numerical instabilities, the values for the informative SCIs were smoothed.

Brannath et al. ³ indicated that ‘in practical applications, the choice of the values for $v$ and $p$ should be driven by knowledge from prior studies. I.e. the values chosen for $v$ should reflect true ratios observed in prior studies and $p$ should be driven by the believes as to how likely the respective values for $v$ will be observed in the planned trial’.

3.2. Example

To illustrate the simultaneous confidence intervals proposed in Sections 2.1 to 2.3, as well as their interpretation on our notion of ‘success’, we give an example of different outcomes. We assume the setting of scenario 1 of the previous section, resulting in the following optimal sample sizes for the IU filter: $n_E=356$ , $n_R=348$ and $n_P=145$ .

Without loss of generality, the observed placebo mean is $X_P=0$ . Consider Table 2(a), where the lower confidence bounds $L_{EP}$ and $L_{ER}$ are calculated via (1) and (2) for different observed means $X_E$ and $X_R$ . Furthermore, it is indicated if the filter (3) is satisfied. Note, that with the given assumptions, ‘ $R>P$ ’ can be stated if and only if $X_R>0.591$ . If this filter is satisfied, $L_{ER}$ is of major interest and $L_{EP}$ is reported only additionally (in the table in parentheses). If the filter is not satisfied, then $L_{EP}$ is of major interest. In Table 2(b) and (c), the informative and single-step confidence bounds are calculated. Furthermore, it is indicated if the filter (4) is satisfied. With the given assumptions, ‘ $R>P$ ’ can be stated if and only if $X_R>0.387$ . For the informative SCIs $q=0.01$ was chosen. For more information on the choice of $q$ see Section 3.3.

Table 2.

Example for the determination of the SCI with different filters and the resulting success.

(a) IU filter with corresponding SCIs. Success ER is satisfied if the IU filter holds and $L_{ER}\ge -0.5$ . Success EP is satisfied if the IU
filter does not hold and $L_{EP}>0.5$ . See text for further explanations.
$X_E$	$X_R$	IU filter	${\ell }_{EP}$	${\ell }_{ER}$	$L_{EP}$	$L_{ER}$	Success EP	Success ER
1.00	1.00	Yes	0.614	$-0.295$	(0.205)	$-0.295$	No	Yes
1.00	0.50	No	0.614	0.205	0.614	(0.114)	Yes	No
1.00	0.30	No	0.614	0.404	0.614	(0.114)	Yes	No
0.80	0.30	No	0.414	0.205	0.414	( $-0.086$ )	No	No
(b) Superiority filter with informative SCIs ( $q=0.01$ ). Success ER is satisfied if the superiority filter holds and $L_{ER}^{inf}\ge -0.5$ .
Success EP is satisfied if the superiority filter does not hold and $L_{EP}^{inf}>0.5$ . See text for further explanations.
$X_E$	$X_R$	Superiority filter	$L_{EP}^{inf}$	$L_{ER}^{inf}$	Success EP	Success ER
1.00	1.00	Yes	0.561	$-0.340$	No	Yes
1.00	0.50	Yes	0.607	0.063	No	Yes
1.00	0.30	No	0.611	0.228	Yes	No
0.80	0.30	No	0.407	0.063	No	No
(c) Superiority filter and single-step SCIs. Success ER is satisfied if the superiority filter holds and $L_{ER}^S\ge -0.5$ . Success EP is
satisfied if the superiority filter does not hold and $L_{EP}^S>0.5$ . See text for further explanations.
$X_E$	$X_R$	Superiority filter	$L_{EP}^S$	$L_{ER}^S$	Success EP	Success ER
1.00	1.00	Yes	0.560	$-0.337$	No	Yes
1.00	0.50	Yes	0.560	0.163	No	Yes
1.00	0.30	No	0.560	0.363	Yes	No
0.80	0.30	No	0.360	0.163	No	No

SCIs: simultaneous: confidence: intervals; : IU: intersection union.

In the first rows of Table 2(a), the observed mean in the experimental group equals the historical reference effect. In the first row, $X_R>0.591$ , hence the IU filter is satisfied and the focus is on the bound $L_{ER}={\ell }_{ER}$ . Success is given via the column named Success ER in the table, namely if $L_{ER}\ge -{\delta }_0$ , which is the case here. In the second row, the IU filter is not satisfied, therefore, success is given by $L_{EP}={\ell }_{EP}$ and is defined via Success EP, namely that $L_{EP}\ge {\delta }_1$ . One has proven that the new treatment has a relevant effect, although the reference is only half as strong as historically. In the fourth example, the filter is also not satisfied. Now the effect of the experimental treatment is lower, therefore no success can be attested. This rule is stricter than the simple Koch-Röhmel design, where non-inferiority of $E$ versus $R$ would be sufficient. Here one can even show superiority of $E$ versus $R$ with ${\ell }_{ER}$ , however, this has no merit because the reference is weak and $L_{ER}$ is only a shifted version of $L_{EP}$ and thus does not give additional information.

In the first two rows of Table 2(b) and (c), the superiority filter assesses ‘ $R>P$ ’ and the respective confidence bounds $L_{ER}^{inf}$ and $L_{ER}^S$ are both larger than $-{\delta }_0$ leading to a successful outcome (Success ER). In the last two rows, the observed mean $X_R$ is fairly small, leading the superiority filter to declare ‘ $R\le P$ ’. For the third row, $L_{EP}^{inf}$ and $L_{EP}^S$ are both larger than ${\delta }_1=0.5$ leading to a Success EP. In the last row, $L_{EP}^{inf}$ and $L_{EP}^S$ both fall below that threshold, resulting in no claim of success. While both, the informative and the single-step confidence intervals come to the same conclusions, we can see that they differ in their size. In this example, we observe that we always have $L_{EP}^{inf}>L_{EP}^S$ . This is particularly useful in the third row, because superiority over placebo is the main conclusion here. Note that there is no such trend for the bounds for the comparison of E and R.

3.3. Probability of success

We now compare the probability of success between the SCIs presented in this article and the method of Brannath et al. ³ using the superiority filter (4) but without calculating SCIs. With the comparison to the latter procedure, we can assess the ‘cost’ (in terms of probability of success) for the additional information the SCIs provide. We ran a simulation with 100,000 repetitions of observations in the settings of scenarios 1 and 2 of Section 3.1. We counted how often the filters were satisfied and, in each case, how often the relevant success occurred (i.e. proof of non-inferiority if the filter is valid, proof of ${\delta }_1$ -superiority if the filter is not valid) and what the median of the confidence bounds is. Note, that the mean of $L_{ER}$ cannot be reported because this bound is minus infinity in case the gatekeeper (superiority of $E$ vs. $P$ ) fails to be shown. The mean of $L_{EP}$ , however, did not differ strongly from its median, so we show here only the median as a measure of central tendency of the confidence bounds. However, failure of the gatekeeper was almost never a problem. For the informative SCIs we investigated different choices for $q$ . Here, we only present the results for $q=0.01$ , since they yielded the best results.

The results of the simulation are presented in Table 3 and Figures 6 and 7. We see that there is only very little loss in probability of success when comparing the informative SCIs interpreted with the help of the superiority filter, to the procedure of Brannath et al. ³ using the superiority filter without SCIs. The maximum observed loss in probability of success observed in our simulations was about $1\mathrm{\%}$ , which is quite low considering the benefit in interpretation the SCIs offer. Throughout all simulations, we observed that the single-step confidence intervals interpreted with the help of the superiority filter yielded a lower probability of success than the procedure of Brannath et al. ³ or the informative SCIs. The loss in probability of success compared to using the informative SCIs amounted up to $\sim 10\mathrm{\%}$ in some scenarios. Comparing the SCIs with IU filter to the procedures using the superiority filter, we see that in scenario 1, where we the sample size is optimized under the assumption that the reference is as strong as historically observed, the IU filter offers substantial less probability of success for most reference effects (ratio of $v$ =reference effect/historical reference effect $>0.2$ ). The loss in probability of success was as high as $13\mathrm{\%}$ for $v=0.65$ . In scenario 2, however, where the sample size is optimized under the assumption, that the reference will only show half it’s historical effect in the present trial, the SCIs with IU filter show best results in terms of probability of success. However, the observed loss for the superiority filter is at most around $6\mathrm{\%}$ , which is smaller than the loss the IU filter procedure presented with in scenario 1. From this we can conclude, that if small reference effects are expected, and the sample size is planned accordingly, the SCIs based on the IU filter should be chosen. In case of large expected reference effects the informative SCIs offer the best performance.

Table 3.

Characteristics of simultaneous confidence intervals for the settings of scenarios 1 and 2 of Section 3.1 for different ratios of $v$ =reference effect/historical reference effect. Without SCI is the probability of success for the procedure introduced by Brannath et al., ³ using the superiority filter but no SCIs. For the informative and single-step intervals, the superiority filter was used for interpretation of study results.

(a) Scenario 1 (sample size planning with $v=1$ ). Sample size: $N=n_E+n_R+n_P=356+348+145=849$ .
	Filter satisfied		Probability of success
v	IU	Superiority	Without SCI	IU	Informative	Single step
1.50	100.0	100.0	2.5	2.5	2.5	1.3
1.25	100.0	100.0	37.9	37.9	37.9	28.1
1.00	98.1	99.9	91.2	89.5	91.2	86.0
0.75	79.2	96.7	96.9	85.5	96.9	96.6
0.50	32.7	71.6	82.2	73.2	81.9	78.8
0.25	4.3	24.2	72.4	71.7	71.9	63.1
0.00	0.1	2.5	72.0	72.0	71.8	62.1
(b) Scenario 2 (sample size planning with $v=0.5$ ). Sample size: $N=n_E+n_R+n_P=227+75+285=587$ .
	Filter satisfied		Probability of success
v	IU	Superiority	Without SCI	IU	Informative	Single step
1.50	100.0	100.0	2.5	2.5	2.5	1.3
1.25	100.0	99.8	15.3	15.4	15.3	9.6
1.00	99.5	97.1	45.7	46.8	45.6	34.6
0.75	95.0	82.4	74.6	78.4	73.8	64.2
0.50	75.0	49.0	83.0	88.4	81.4	75.0
0.25	38.4	15.9	81.3	84.5	79.8	73.3
0.00	10.5	2.5	80.6	81.2	79.6	72.1

SCIs: simultaneous: confidence: intervals; : IU: intersection union.

Figure 6.

Probability of success, success with valid filter (Power ER) and success without valid filter (Power EP) from simulations in scenario 1 (left column) and scenario 2 (right column). The $x$ -axis is the ratio $v=$ reference effect/historical reference effect. The vertical line corresponds to the ratio $v$ which the sample size was calculated for.

Figure 7.

Median lower confidence limits from simulations in scenario 1 (left column) and scenario 2 (right column). The $x$ -axis is the ratio $v=$ reference effect/historical reference effect.

Note, that the probability of success decreases for values of $v$ which are larger than those assumed in the sample size calculation (vertical line in Figure 6). This is due to the fact that for increasing $v$ it becomes harder to prove non-inferiority of experimental treatment to the (increasingly efficacious) reference. If we wanted to protect ourselves against the potential power loss associated with values of $v$ which are larger than assumed during the planning of the trial, we could think of using adaptive designs, which would allow for a recalculation of the sample size during the course of the trial. However, this is not in the scope of this article. For values of $v$ which exceed $1.5$ , the experimental treatment is inferior to the reference by more than the pre-specified non-inferiority margin. Therefore, a claim of ‘success’ would in this case constitute a type-I-error. We can see that in our simulations, the probability for this to happen was bounded by 2.5%. This is in line with the fact that we know from theoretical derivations, that the FWER is controlled with all of the presented procedures. Following a reviewers suggestion, we simulated the FWER for several scenarios under the null hypothesis. The results are provided in Supplemental Material. The FWER was controlled in all setups considered.

4. Example – Application to trial data

Next, we apply the introduced confidence intervals to data of a three-arm study on major depressive disorder by Higuchi et al., ¹³ which is also discussed by Hida and Tango ¹⁴ and Brannath et al. ³ We will first show the example in it’s original version, as shown by Higuchi et al. ¹³ as well as the application of the adaptive test strategy of Brannath et al. ³ After that, we will apply the IU and the informative ( $q=0.01$ ) confidence intervals to the same data.

Higuchi et al. ¹³ investigated the efficacy and safety of 6-week treatment with duloxetine (E) to those of paroxetine (R) and placebo (P) in a double-blinded, randomized, active controlled, parallel-group study. It’s primary endpoint was the HAM-D17 change from baseline at 6 weeks and the statistical analysis was planned to test the superiority of duloxetine over placebo and the non-inferiority of duloxetine compared to paroxetine in hierarchical order. The non-inferiority margin was set as ${\delta }_0=2.5$ . The following mean decreases in the HAM-D17 were observed: $10.2\pm 6.1$ (mean $\pm$ sd) in the duloxetine group, $9.4\pm 6.9$ in the paroxetine group and $8.3\pm 5.8$ in the placebo group with sample sizes $n_E=147,n_R=148\mathrm{and}{n}_P=145$ .

The resulting (unadjusted) 97.5% lower confidence interval for the difference in means between duloxetine and placebo is in this case given by (0.53, $\mathrm{\infty }$ ) (i.e. ${\ell }_{EP}=0.53$ ), indicating superiority of duloxetine over placebo. The (unadjusted) 97.5% lower confidence interval for the difference in means between duloxetine and paroxetine can be calculated to be ( $-$ 0.69, $\mathrm{\infty }$ ) (i.e. ${\ell }_{ER}=-0.69$ ) and excludes $-{\delta }_0=-2.5$ indicating non-inferiority of duloxetine compared to paroxetine. However, the superiority of paroxetine over placebo could not be established, because the lower 97.5% confidence interval for this comparison is given by ( $-$ 0.37, $\mathrm{\infty }$ ) and therefore includes 0. Higuchi et al. ¹³ concluded that non-inferiority of duloxetine compared to paroxetine did not have assay sensitivity.

Brannath et al. ³ applied their adaptive test strategy to the data, employing the superiority filter (4) to determine, whether paroxetine was sufficiently efficacious. They used ${\delta }_1=2.5$ for the investigation of superiority of E versus P. As seen above duloxetine can be shown to be superior to placebo (i.e. $H_{EP}^S$ is rejected). However, using the superiority filter (4), ‘ $R>P$ ’ cannot be concluded, which is why as a next step they tested for ${\delta }_1$ -superiority of duloxetine over placebo ( $H_{EP}^{{\delta }_1}$ ). Since the lower 97.5% confidence interval for this comparison includes ${\delta }_1=2.5$ , $H_{EP}^{{\delta }_1}$ could not be rejected, and the study was not successful in the sense that there was not enough evidence to declare duloxetine sufficiently efficacious.

For illustrative purposes, Brannath et al. ³ investigated how their procedure would perform, if the observed mean change in the duloxetine group would have been $12.2$ with all other parameters unchanged. Under this assumption, duloxetine can be shown to be superior to placebo (i.e. $H_{EP}^S$ is rejected) and the superiority filter still concludes ‘ $R\le P$ ’. However, one now has ${\ell }_{EP}=2.53$ , leading to a claim of ${\delta }_1$ superiority of duloxetine to placebo, which in case can be interpreted as a successful study. Brannath et al. ³ however noted that the confidence intervals reported ‘while appropriate for deriving the test decision in the hierarchical test, are no simultaneous confidence intervals, and therefore do not have simultaneous coverage probability’.

We now want to apply the simultaneous confidence intervals introduced in this manuscript to the original data of this study. Applying the intuitive approach outlined in Figure 2, we note that $H_{EP}:{\mu }_E-{\mu }_P\le 0$ can be rejected. The IU and the superiority filters both conclude ‘ $R\le P$ ’ and the IU simultaneous confidence bounds are given as $L_{EP}={\ell }_{EP}=0.53$ and $L_{ER}={\ell }_{EP}-{\delta }_0=-1.97$ . Since $L_{EP}$ covers the superiority margin ${\delta }_1=2.5$ , we cannot establish ‘success’ of the study. The informative confidence bounds are gives as $L_{EP}^{inf}=0.528$ and $L_{ER}^{inf}=-1.67$ . Since $L_{EP}^{inf}$ also covers the superiority margin ${\delta }_1=2.5$ , we cannot establish ‘success’ of the study with the informative SCIs either. However, the simultaneous confidence bounds $L_{EP}$ and $L_{ER}$ resp. $L_{EP}^{inf}$ and $L_{ER}^{inf}$ allow for an interpretation on the magnitude of treatment effects, which was not possible before.

Applying the confidence intervals to the hypothetical case where we assume that the observed mean change in the duloxetine group is $12.2$ with all other parameters unchanged, we obtain ${\ell }_{EP}=2.53$ and ${\ell }_{ER}=-0.69$ . Again, $H_{EP}:{\mu }_E-{\mu }_P\le 0$ can be rejected and the IU and the superiority filter both conclude ‘ $R\le P$ ’. The IU simultaneous confidence bounds are now given as: $L_{EP}={\ell }_{EP}=2.53$ and $L_{ER}={\ell }_{EP}-{\delta }_0=0.03$ . Since $L_{EP}>{\delta }_1$ we now conclude ${\delta }_1$ superiority of duloxetine over placebo, and, therefore, a ‘success’ of the study. For the informative SCIs, we obtain $L_{EP}^{inf}=2.53$ and $L_{ER}^{inf}=-0.59$ . Since $L_{EP}^{inf}>{\delta }_1$ we also conclude ${\delta }_1$ superiority of duloxetine over placebo, and, therefore, a ‘success’ of the study with the informative SCIs. While Brannath et al. ³ came to the same conclusions on the significance tests, the proposed confidence bounds offer the advantage, that we can derive conclusions on the size of the effects of duloxetine compared to placebo and paroxetine.

5. Summary and discussion

Brannath et al. ³ introduced a flexible extension to the three-arm ‘gold-standard’ non-inferiority design, in which it is possible to declare success of the study, even if the reference fails. We complemented the previous work with compatible simultaneous confidence intervals, thus providing more information than hypothesis testing alone. We introduced three methods to calculate confidence intervals in the given design: stepwise confidence intervals based on those introduced by Hsu and Berger, ⁴ ‘informative’ SCIs based on the work of Schmidt and Brannath ⁵ and single-step confidence intervals. While the informative and the single-step intervals rely on a ‘filter’ for the interpretation of study results, the intervals based on Hsu and Berger ⁴ offer an ‘intrinsic’ filter (IU filter) guaranteeing FWER control for all sample sizes.

We did multiple simulations in which we compared the different procedures for calculating SCIs and also the procedure by Brannath et al. ³ in different scenarios. We saw that substantial power losses may occur with the IU filter compared to the superiority filter. However, when the reference effect is anticipated to be smaller than historically observed (and the study is planned accordingly) the SCIs with the IU filter might perform better than the superiority filter. We observed only very little loss in the probability of success for the informative SCIs compared to the procedure by Brannath et al. ³ which does not yield SCIs at all.

Finally, we applied the different SCIs to data from a real clinical trial, exemplifying the gain of calculating SCIs, which lies is the possibility to make statements of the size of the effects.

As mentioned in Section 3.3, an extension of the presented method to adaptive designs might be desirable to protect against the potential loss in power, if the reference effect is larger than expected during the sample size calculation. A further extension of the proposed methods, not discussed in this article either, includes the application to sequential designs to allow for interim testing.

Appendix: Equivalence between formal and intuitive procedure

We will show that the confidence bounds $L_{EP}$ and $L_{ER}$ that are defined by the intuitive Figure 2 are identical to the bounds defined in Figure 1, that is, by formulas (1) and (2). Note that the quantities ${\ell }_{EP}$ and ${\ell }_{ER}$ are equivalent to the test statistics for $H_{EP}^S$ and $H_{ER}^N$ , respectively, in the sense that $H_{EP}^S$ is rejected if and only if ${\ell }_{EP}\ge 0$ , and $H_{ER}^N$ is rejected if and only if ${\ell }_{EP}\ge -{\delta }_0$ . We will discuss all cases.

If ${\ell }_{EP}<0$ , then $H_{EP}^S$ is not rejected and $L_{EP}={\ell }_{EP}<0,L_{ER}=-\mathrm{\infty }$ with both the formal and the intuitive design, because the first family of hypotheses $(H_{EP}^{\vartheta }{)}_{\vartheta \le 0}$ is not rejected completely and both procedures stop.

If ${\ell }_{EP}\ge 0$ and ${\ell }_{ER}<-{\delta }_0$ , then $H_{EP}^S$ is rejected and $H_{ER}^N$ is accepted so that $L_{EP}=0$ and $L_{ER}={\ell }_{ER}$ with the formal design. In the intuitive design, we can – informally written – see that ‘ $R>E>P$ ’, that is, the filter is satisfied. More formally, we obtain from the definitions of ${\ell }_{EP}$ and ${\ell }_{ER}$ that

$$X_R-X_P={\ell }_{EP}-({\ell }_{ER}+{\delta }_0)+z_{\alpha }\sigma (\sqrt{n_E^{-1}+n_P^{-1}}-\sqrt{n_E^{-1}+n_R^{-1}})+{\delta }_0.$$

Because ${\ell }_{EP}\ge 0$ and ${\ell }_{ER}+{\delta }_0<0$ , it follows that the filter (3) holds. Therefore, $L_{ER}={\ell }_{ER}$ and $L_{EP}=max\{L_{ER}+{\delta }_0,0\}=max\{{\ell }_{ER}+{\delta }_0,0\}=0$ , as in the formal design.

If ${\ell }_{EP}\ge 0$ and ${\ell }_{ER}\ge -{\delta }_0$ , then we reject both $H_{EP}^S$ and $H_{ER}^N$ with the formal procedure. The confidence bounds are $L_{EP}={\ell }_{ER}+{\delta }_0,L_{ER}={\ell }_{ER}$ if the filter (3) holds and $L_{EP}={\ell }_{EP},L_{ER}={\ell }_{EP}-{\delta }_0$ if the filter does not hold. Consider now the intuitive design. If the filter holds, then $L_{ER}={\ell }_{ER}$ and $L_{EP}=max\{L_{ER}+{\delta }_0,0\}={\ell }_{ER}+{\delta }_0$ . If the filter does not hold, then $L_{EP}={\ell }_{EP}$ and $L_{ER}=L_{EP}-{\delta }_0={\ell }_{EP}-{\delta }_0$ . All bounds correspond to those of the formal design in the respective cases.