A stochastically curtailed single-arm phase II trial design for binary outcomes

Abstract

Phase II clinical trials are a critical aspect of the drug development process. With drug development costs ever increasing, novel designs that can improve the efficiency of phase II trials are extremely valuable.

Phase II clinical trials for cancer treatments often measure a binary outcome. The final trial decision is generally to continue or cease development. When this decision is based solely on the result of a hypothesis test, the result may be known with certainty before the planned end of the trial. Unfortunately, there is often no opportunity for early stopping when this occurs.

Some existing designs do permit early stopping in this case, accordingly reducing the required sample size and potentially speeding up drug development. However, more improvements can be achieved by stopping early when the final trial decision is very likely, rather than certain, known as stochastic curtailment. While some authors have proposed approaches of this form, these approaches have various limitations.

In this work we address these limitations by proposing new design approaches for single-arm phase II binary outcome trials that use stochastic curtailment. We use exact distributions, avoid simulation, consider a wider range of possible designs and permit early stopping for promising treatments. As a result, we are able to obtain trial designs that have considerably reduced sample sizes on average.

1. Introduction

Most novel treatments are found to be inefficacious, which makes the average development cost associated with each successful treatment extremely high (Wong et al. 2019). Furthermore, trials themselves are expensive to run (Martin et al. 2017), and the nature of evaluating treatment response in oncology trials means that results are not immediately available, meaning that trials can take substantial time to complete. This makes novel designs that can improve the efficiency of clinical research extremely valuable.

Phase II clinical trials for cancer treatments often have a binary primary outcome, based on change in tumour size as measured by the RECIST criterion (Eisenhauer et al. 2009), and typically contain only a single arm. The aim of such a phase II trial is to gain enough information to decide whether a treatment should be carried forward for further testing (a go decision) or abandoned (a no go decision). In general, if a sufficient number of (positive) responses are observed, a go decision is made and some corresponding null hypothesis is rejected, otherwise a no go decision is made and the corresponding null hypothesis is not rejected. The most simple design to evaluate a treatment with a binary outcome is the single-stage design, described by A’Hern (A’Hern 2001). In a single-stage design, a fixed number of participants are recruited and once the trial is completed, a go or no go decision is made based on the number of responses.

A number of designs have been proposed that can reduce the expected sample size (ESS) of a single-arm binary outcome trial compared to a single-stage design.

For single-arm trials with a binary outcome, the most simple trial design is the single-stage trial described by A’Hern (A’Hern 2001), which is comprised of N participants. The trial is deemed a success if the final number of responses exceeds a specified boundary r.

Extending this approach, Simon’s design (Simon 1989) adds an interim analysis after n₁ participants, at which point the trial proceeds to recruit a further N – n₁ participants only if the number of responses is greater than a pre-specified r₁, otherwise it stops for a no go decision. Mander and Thompson extended this to permit both a go and no go decision at the interim (Mander and Thompson 2010; Simon 1989). Chi and Chen extended the Simon design by permitting stopping as soon as a go or no go decision is certain, known as non-stochastic curtailment (Chi and Chen 2008). Consequently, we will refer to this design as the “NSC” design.

It is possible to end a trial early not only when a go decision is either certain or no longer possible, as in NSC above, but also when a go decision is either likely or unlikely. This is known as stochastic curtailment (SC). One approach to SC is based on the concept of conditional power (CP). Conditional power (or “assumed conditional power” (Kunzmann et al. 2020)) is the probability of rejecting some null hypothesis (and making a go decision), conditional on an anticipated treatment effect and the current number of participants and responses. The idea of CP can be used in conjunction with SC in the following way: if the CP is below some specified lower threshold, or exceeds some specified upper threshold, then a trial will end for a no go or go decision respectively. Ayanlowo and Redden (Ayanlowo and Redden 2007) and Kunz and Kieser (Kunz and Kieser 2012) proposed designs that allow early stopping for a no go decision if the CP is below some lower threshold, that is, by allowing SC. Denote this lower threshold θ_F.

Another design of interest is the sequential probability ratio test (SPRT) of Wald (Wald 1947). It contains upper and lower stopping boundaries for every possible number of participants. These upper and lower boundaries do not converge to the same value as the number of participants increases, and consequently the design has no maximum sample size.

We present two novel designs that use SC to obtain upper and lower stopping boundaries for a trial. The first design applies SC to what would otherwise be a single-stage design, while the second applies SC to what would otherwise be a Simon-type design. Novel aspects include permitting stopping for a go decision and avoiding simulation by using the exact distribution of a trial’s outcomes. We describe how the proposed designs differ from existing designs and compare the them to existing designs in a variety of ways. We summarise the results and make recommendations for investigators.

The remainder of the paper is organised as follows: Section 2 covers the background of curtailed designs and the limitations of existing similar designs. This section also introduces the details of the proposed designs, including information about the concept of conditional power, technical aspects about how we search for specific examples of the designs and how to compare such examples. Section 3 compares the proposed designs to existing designs, through both a real data example and other examples. The paper concludes with a discussion in Section 4.

2. Materials and methods

2.1. Background on curtailed designs

NSC and SC are typically described in terms of continuous monitoring, where the data are analysed after each participant’s results become available. This may be considered a special case of sequential monitoring, which describes any trial in which interim results are analysed. Sequential analysis is methodologically well established (Jennison and Turnbull 2000; Whitehead 1997). Continuous monitoring has been proposed not only in the single-arm approaches of Chi and Chen, Ayanlowo and Redden (second stage) and Kunz and Kieser, but also in designs for randomised binary outcome trials (Carsten and Chen 2016; Chen et al. 2018; Law et al. 2020). In terms of practicality, continuous monitoring may be easier when a trial’s recruitment rate is low (Wason et al. 2019), which is often the case in application: Campbell et al. (Campbell et al. 2007) found that early participant recruitment was slower than expected in 77 (63%) of 122 reviewed trials, and a review of 151 randomised controlled trials by Walters et al. (Walters et al. 2017) reported a median recruitment rate of 0.92 participants per centre per month. Furthermore, Campbell et al. (Campbell et al. 2007) found that only 38 (31%) of 122 trials reached their intended sample size and 66 (54%) requested a trial extension.

Continuous monitoring may be expected to be specified at the trial design stage; see for example, Todd et al. (Todd et al. 2016) and McCabe et al. (McCabe et al. 2020). However, continuous monitoring and subsequent curtailment may also take place in trials where no such monitoring is specified in advance. In particular, authors may acknowledge the use of curtailment (and thus continuous monitoring) in a trial without using such terms in the corresponding manuscript. For example, Simon’s design may be curtailed to make either a no go decision (Mego et al. 2016; Necchi et al. 2014; Santana et al. 0000; Wagner et al. 2015) or a go decision (Moskowitz et al. 2013; Stein et al. 2013; Yoon et al. 2016; Yu et al. 2017). A trial may be curtailed without prior planning, including when the endpoint length is long (Pedersen et al. 2015; Sepulveda-Sanchez et al. 2017). Investigators may even use SC informally, ending a Simon design when trial success is possible but “unlikely” (Odia et al. 2015). We give further details of all these trials in the Appendix.

The examples cited above also suggest that continuous monitoring and subsequent curtailment is more common than citations of the associated methodological literature indicates. An important additional consequence of unplanned monitoring is that any resulting inference may result in biased point estimates and confidence intervals with low coverage (Atkinson and Brown 1985; Whitehead 1986). Accordingly, by not anticipating and accounting for continuous monitoring and curtailment at the design stage, investigators are taking inferential risks. This lack of planning also has costs in terms of the ability to report accurate ESSs and trial duration at the design stage, which may directly affect expected financial costs and/or the ability to act quickly upon making a go or no go decision.

Define a design realisation as a particular instance of a design, for example, a single-stage design where a go decision will be made if more than five responses are observed from a total of 20 participants. A design realisation must be feasible, that is, it must satisfy some chosen type-I error-rate and power. A feasible design that is the best-performing design realisation with respect to some single optimality criterion is known as the optimal design realisation for that criterion.

In the setting of multiple optimality criteria, Jung et al. (Jung et al. 2004) determined the “best” design realisations by creating a loss function that was a weighted combination of two optimality criteria: maximum sample size and ESS under some null hypothesis. The authors describe a design realisation as admissible if it has the smallest expected loss of all considered design realisations, that is, it is superior to all other considered design realisations, for some weighted combination of optimality criteria. Our interest generally lies in finding the collection of design realisations that comprise the admissible designs across all combinations of weights. Mander et al. (Mander et al. 2012) extend the concept of the loss function and admissible designs by incorporating a third component to the loss function: ESS under an alternative hypothesis.

2.2. Limitations of existing designs

Some aspects of the existing curtailed designs described above have scope for improvement. Firstly, the operating characteristics of curtailed designs have often been estimated using simulation. However, such estimates are subject to simulation error, with the exact distribution of each trial’s possible outcomes remaining unknown. Secondly, the approaches of Ayanlowo and Redden (Ayanlowo and Redden 2007) and Kunz and Kieser (Kunz and Kieser 2012) use fixed or uniformly distributed thresholds for CP, which may reduce the number of meaningful design realisations searched over. Thirdly, in both Ayanlowo and Redden and Kunz and Kieser, rather than taking curtailment into account when searching for the optimal design (for some definition of “optimal”), the optimal non-curtailed design is found and then SC is applied to it. This again means that a narrower range of possible designs is examined. A second consequence of this is that the type-I error-rate and power is decreased compared to the selected optimal non-curtailed design (as SC is only permitted to make a no go decision). This further reduces the number of possible design realisations, as many will not reach the required type-I error-rate and power once curtailment has been applied. Finally, the design approaches of Ayanlowo and Redden and Kunz and Kieser give equations for evaluating CP, but these equations do not fully account for the early stopping caused by SC.

2.3. Proposed designs

The experimental treatment is assumed to have a true response rate p ∈ [0, 1] (i.e., each patient outcome is assumed to be distributed as Bern(p)). We test the null hypothesis H₀ : p ≤ p₀ against the alternative hypothesis H₁ : p > p₀. We assume that results from participants are independent and identically distributed. Consequently, the number of responses observed at each stage are also independent. For all analyses described in this paper, the test statistic used to undertake the hypothesis test after the first m participants is the current number of responses S(m).

The trial is powered to a level 1 − β under p = p₁, and the type-I error-rate is controlled to α when p = p₀. Available results (Shan et al. 2017) on the monotonicity of the power function in designs of the type considered here means that the type-I error-rate is then controlled to α over all of H₀ (i.e., for all p < p₀) and power is at least 1 − β for all p ≥ p₁. Commonly, the value of p₀ is chosen to be the greatest response rate that is deemed typical for standard of care, while p₁ is chosen to be the smallest response rate that is large enough to warrant further study.

We propose two designs where the trial may stop not only for a no go decision if the CP is below some lower threshold, that is, CP < θ_F but also for a go decision if the CP is greater than some upper threshold denoted θ_E, that is, CP > θ_E. The first is a Simon-type design that allows SC after each participants’ results, for either a go or no go decision, and contains an interim analysis in the design to which SC is added. This will be referred to as the “SC” design. The second design we propose can be understood as an otherwise single-stage design that incorporates SC. This will be referred to as the “m-stage” design. The latter design will also be alterable to allow analyses that are less frequent than continuous monitoring, while still reducing the ESS. The m-stage design is the single-arm analogue to the two-arm design introduced by Law et al. (Law et al. 2020), which permits SC for go and no go decisions.

Note that it will ultimately be possible for all go and no go decisions to be concatenated into N-length vectors of stopping boundaries e = (e₁, e₂, …, e_N) and f = (f₁, f₂, …, f_N) respectively, using e_i = ∞ and f_i = −∞ at any points i ∈ [1, N] where stopping is not permitted/possible. Thus it is possible to characterise a realisation of any design type using e and f only. This demonstrates why the comparisons conducted between designs are fair: both previous and our newly proposed designs amount to methodologies for specifying e and f. This also means that for any single-arm design, all possible combinations of number of participants and responses so far can be represented in an easy-to-understand grid. Examples of this are shown in Figure 1, for the following designs: single-stage, Simon, Mander and Thompson, NSC, SC and m-stage. Here, all possible points, that is, all possible participant and response combinations, and whether at each point the trial will continue or stop for a go or no go decision, are shown.

Figure 1.

Illustrative diagrams of different trial designs, showing potential points where the study would end, known as terminal points. m: Number of participant results so far. S(m): Number of responses so far. All trials have N = 8, r = 4, with r₁ = 1 in the two-stage designs and e₁ = 3 in Mander and Thompson’s design. We may assume that (θ_F, θ_E) in the SC design are such that (e₅, e₆, e₇, e₈) = (5, 5, 5, 5), (f₂, f₄, f₆, f₇, f₈) = (0, 1, 2, 3, 4) and that (θ_F, θ_E) in the m-stage design are such that (e₄, e₅, e₇, e₈) = (4, 4, 5, 5), (f₂, f₃, f₆, f₇, f₈) = (0, 1, 2, 3, 4).

An objection to curtailment for a go decision could be that one would wish to obtain more data if a treatment appears promising. However, the current abundance of possible treatments to be tested among relatively few participants makes this argument less compelling than in the past.

The proposed designs use binding stopping rules, as do six designs to which the proposed designs are compared (Ayanlowo and Redden 2007; Chi and Chen 2008; Kunz and Kieser 2012; Mander and Thompson 2010; Simon 1989; Wald 1947), including a number of designs that use continuous monitoring. If binding stopping rules are used, it should be made clear that crossing a stopping boundary necessitates stopping the trial, as such stopping has already been agreed upon in the design stage (Pallmann et al. 2018). If binding stopping rules are deemed in advance to be difficult to implement in conjunction with continuous monitoring, less frequent monitoring can be employed (Section 2.5).

2.4. Obtaining exact distributions

For any design, there are a number of points (S(m), m) at which the trial would stop. Define these points “terminal points”, and $𝒯$ as the set of all such points. The terminal points can be determined using the CP at each point in the trial (see Appendix). The ESS for response rate p can then be obtained by multiplying the number of participants m at each terminal point by the probability of reaching that point:

$$ESS\left(p\right)={\displaystyle \sum _{m=1}^N{\displaystyle \sum _{S\left(m\right)=0}^m\text{𝕀}\left[\left\{S\left(m\right),m\right\}\in 𝒯\right]}}mU\left(S\left(m\right),m|p,\mathbf{\text{e}},\mathbf{\text{f}}\right)$$ (1)

where U(S(m), m|p, e, f) denotes the probability of reaching the point (S(m), m) in a particular trial given true response rate p and vectors of stopping boundaries e and f. The indicator term ensures that only terminal points are considered. Consequently, for any type of design, all that is required to find the ESS is the probability of reaching each terminal point in $𝒯$. The sample size of a trial can also be described in terms of quantiles, including the median, in the following way: sort the sample sizes of the terminal points, that is, each m in each $\left\{S\left(m\right),m\right\}\in 𝒯$, in ascending order. The corresponding probabilities U(S(m), m|p, e, f) then comprise the cumulative density function, which can be used to calculate quantiles of the required sample size.

The probability of rejecting H₀ is

$$R\left(p\right)={\displaystyle \sum _{m=1}^N{\displaystyle \sum _{S\left(m\right)=0}^m\text{𝕀}\left[\left\{S\left(m\right),m\right\}\in 𝒯\right]}}\text{𝕀}\left[S\left(m\right)\ge e_m\right]U\left(S\left(m\right),m|p,\mathbf{\text{e}},\mathbf{\text{f}}\right)$$

for some response rate p. This equation is similar to Equation (1) in that only terminal points are considered, and here a further indicator term ensures that only terminal points resulting in a go decision, that is, rejecting H₀, are considered. R(p) is then the sum of the probabilities of reaching each terminal point that results in a go decision, for some response rate p. The type-I error-rate and power are obtained by finding R(p₀) and R(p₁) respectively. Being able to obtain this information means that both the exact distribution and the operating characteristics of the trial outcomes are known.

2.5. Conditional power

We define conditional probability CP(p, S(m), m) as the probability of rejecting H₀ conditional on being at point (S(m), m), when the true response rate is p. Define the conditional power as the conditional probability when p = p₁, that is, CP(p₁, S(m), m). From here we refer only to conditional power rather than conditional probability and reiterate that “CP” is used to refer to conditional power. No sample size re-estimation takes place.

For the NSC design we have derived the following equation for calculating CP(p₁, S(m), m) exactly:

$$CP\left(p_1,S\left(m\right),m\right)=\left\{\begin{array}{ll}0,\hfill & \text{if}m-S\left(m\right)>N-r-1\text{or}\hfill \\ \hfill & \left(m-S\left(m\right)>n_1-r_1-1\text{and}m\le n_1\right)\hfill \\ {\displaystyle \sum _{j=r-S\left(m\right)}^{n_1-m-1}\left[A\left(j,r_1\right){\displaystyle \sum _{i=r-S\left(m\right)}^{N-\left(j+m+1\right)-1}A\left(i,r\right)}\right],}\hfill & \text{if}m-S\left(m\right)\le n_1-r_1-1\text{and}m\le n_1\hfill \\ {\displaystyle \sum _{i=r-S\left(m\right)}^{N-m-1}A\left(i,r\right)}\hfill & \text{if}m-S\left(m\right)\le N-r-1\text{and}m>n_1\hfill \\ 1,\hfill & \text{if}S\left(m\right)>r\hfill \end{array}\right\}$$ (2)

where

$$A\left(x,y\right)=\left(\begin{array}{c}x\\ y-S\left(m\right)\end{array}\right)p_1^{y-S\left(m\right)+1}{\left(1-p_1\right)}^{x-\left\{y-S\left(m\right)\right\}}.$$

A(x, y) describes the joint probability of observing y − S(m) + 1 responses out of x results, where the last observation is a response.

Equation (2) can be understood component by component: if S(m) > r, success is guaranteed, and so CP is equal to 1. If failure is guaranteed, CP is equal to 0. This occurs during the first stage if the number of responses cannot exceed r₁, or equivalently, if the number of non-responses reaches n₁ − r₁, that is, if m − S(m) > n₁ − r₁ − 1. Similarly, failure is guaranteed during the second stage if the number of responses cannot exceed r, or equivalently, if the number of non-responses reaches N − r, that is, if m − S(m) > N − r − 1. If the final decision is to reject H₀, the final observed result must be a response. Consequently, at the point (S(m), m), m > n₁, rejecting H₀ involves observing r − S(m) or more responses in the proceeding results but one, and then observing a response in the final result. The third component of Equation (2) sums the probability of reaching the required number of responses for all valid total numbers of results. The second component of Equation (2) works in the same way, and additionally accounts for the explicit interim analysis at n₁.

The exact CP for the NSC design can also be written recursively as

$$CP\left(p_1,S\left(m\right),m\right)=\left\{\begin{array}{ll}0,\hfill & \text{if}m-S\left(m\right)>N-r-1\hfill \\ \hfill & \text{or}\left(m-S\left(m\right)>n_1-r_1-1\text{and}m\le n_1\right)\hfill \\ D,\hfill & \text{if}m-S\left(m\right)\le N-r-1\hfill \\ \hfill & \text{or}\left(m-S\left(m\right)\le n_1-r_1-1\text{and}m\le n_1\right)\hfill \\ 1,\hfill & \text{if}S\left(m\right)>r\hfill \end{array}\right\},$$ (3)

where

$$D=p_{1}CP\left(p_1,S\left(m+1\right),m+1\right)+\left(1-p_1\right)CP\left(p_1,S\left(m\right),m+1\right).$$

For a single-stage trial incorporating NSC, the CP can be obtained using these equations by omitting the conditions relating to r₁ and n₁. In its recursive form, it can be seen that the CP at any point in a trial is a function of the CP at points with at least the same number of responses and more participants, among such points that are possible to reach in the trial.

For designs that incorporate NSC, the trial stops and a no go decision is taken if CP = 0. The trial stops and a go decision is taken if CP = 1. The CP at any point can be obtained using Equation (2) directly. For designs that incorporate SC, the trial will additionally end at any point where 0 < CP < θ_F or θ_E < CP < 1, for some specified θ_F, θ_E ∈ [0, 1], θ_F < θ_E. As the CP is a function of later points in the trial, the predetermined decision to end a trial at any point where 0 < CP < θ_F causes the CP of such points to become zero. Conversely, points where θ_E < CP < 1 then have a CP of one. This in turn affects the CP of earlier points in trial. As such, when incorporating SC, it is logical to calculate CP at each point using a recursive equation, one value at a time, starting at the point (S(m) = r, m = N − 1), where CP(p₁, r, N − 1) = p₁ by definition. All “earlier” points in the trial, i.e., points such that m < N − 1, are either a function of CP(p₁, r, N − 1) or are terminal points. For points with more responses or more participants, CP(p₁, a, N) = 0 if a ≤ r, a ∈ ℤ⁰⁺, and CP(p₁, a, b) = 1 for any a > r and any b ≥ a, b ≤ N. Thus for the SC design, the CP at each point can be obtained using the following equation:

$$CP\left(p_1,S\left(m\right),m\right)=\left\{\begin{array}{ll}0,\hfill & \text{if}D<{\theta }_F\text{or}m-S\left(m\right)>N-r-1\text{or}\hfill \\ \hfill & \left(m-S\left(m\right)>n_1-r_1-1\text{and}m\le n_1\right)\hfill \\ D,\hfill & \text{if}{\theta }_F\le D\le {\theta }_E\text{and}\hfill \\ \hfill & m-S\left(m\right)>N-r-1\text{or}\left(m-S\left(m\right)>n_1-r_1-1\text{and}m\le n_1\right)\hfill \\ 1,\hfill & \text{if}D>{\theta }_E\text{or}S\left(m\right)>r\hfill \end{array}\right\}$$ (4)

Equation (4) can be used to obtain the CP for the m-stage design by omitting the conditions relating to r₁ and n₁:

$$CP\left(p_1,S\left(m\right),m\right)=\left\{\begin{array}{ll}0,\hfill & \text{if}D<{\theta }_F\text{or}m-S\left(m\right)>N-r-1\hfill \\ D,\hfill & \text{if}{\theta }_F\le D\le {\theta }_E\text{and}\left\{m-S\left(m\right)>N-r-1\right\}\hfill \\ 1,\hfill & \text{if}D>{\theta }_E\text{or}S\left(m\right)>r\hfill \end{array}\right\}.$$ (5)

Continuous monitoring may not always be possible in practice. We describe a design approach for SC using sequential monitoring in terms of specified block sizes B, though it is also possible to specify the number of stages instead. An interim analysis is undertaken after every block of B participants, at which point the number of responses is compared to corresponding lower and upper stopping boundaries. The recursive equation used to calculate CP, Equation (4), may still be used, with D now generalised to handle blocks of size B:

$$D\left(B\right)={\displaystyle \sum _{i=0}^B{p}_1^i}{\left(1-p_1\right)}^{B-i}CP\left(p_1,S\left(m+i\right),m+B\right).$$

With this generalisation, Equation (4) can now be used to obtain CP for every possible number of responses for n ∈ {B, 2B, …, N} participants, from which lower and upper stopping boundaries can be obtained.

2.6. Design search

We seek a set of values from which ordered pairs of θ_F and θ_E will be created and searched over to find optimal or admissible design realisations, for single optimality criteria or weighted multiple-optimality criteria, respectively. One could use a uniformly distributed set of possible thresholds to some specified degree of coarseness. However, in trials that use curtailment, the CP values are not uniformly distributed; instead, most of the mass is close to zero or one. To account for the lack of a uniform distribution, we propose searching over a set of thresholds chosen based on the CP at each point in each possible trial. We obtain every possible value of CP(p₁, S(m), m), including zero and one, for a given combination of {r, N} (m-stage) or {r₁, n₁, r, N} (SC design). We allow an upper and lower limit for θ_F and θ_E respectively, termed ${\theta }_{F_{MAX}}$ and ${\theta }_{E_{MIN}}$. Then, without loss of generality, a trial-specific set of thresholds can be defined as $\theta =\{CP\left(p_1,S\left(m\right),m\right):S\left(m\right)=0,\dots ,r,m=1,\dots ,N:\left\{r,N\}\in ℛ,CP\le {\theta }_{F_{MAX}}|\mathrm{CP}\ge {\theta }_{E_{MIN}}\right\}$, where $ℛ$ is the family containing all possible sets {r, N} (or {r₁, n₁, r, N}).

We suggest setting an upper bound for the lower threshold of ${\theta }_{F_{MAX}}=p_1$ and a lower bound for the upper threshold of ${\theta }_{E_{MIN}}=0.95$ An upper limit for the number of CP values used in each trial set in $ℛ$ is given by Θ is was set to 10⁶. The final rejection boundary r was constrained to the rounded interval [⌊Np₀⌋,⌈Np₁⌉] in the design search, using the rationale of A’Hern (A’Hern 2001). As an alternative, constraining r to the boundaries of Wald’s SPRT (Wald 1947) was also examined. We search over all combinations of θ_F, θ_E ∈ θ, N and r (and r₁ and n₁ for the SC design) subject to the above constraints and desired type-I error-rate and power.

The designs that minimise ESS(p₀) and ESS(p₁) respectively are termed the p₀- and p₁-optimal designs. The designs that minimise ESS(p₀) and ESS(p₁) respectively among the subset of designs that minimise N are termed the p₀- and p₁-minimax designs. These terms are analogous to the terms H₀-and H₁-optimal and H₀- and H₁-minimax used by Mander and Thompson (Mander and Thompson 2010). The ESSs of the p₀- and p₁-optimal and p₀- and p₁-minimax admissible designs of the proposed designs will be compared to those of Simon’s design, Mander and Thompson and the NSC design, and additionally to those of the designs found using the SPRT of Wald (Wald 1947) in the case of the p₀- and p₁-optimal criteria. More details regarding the design search are given in the Appendix.

2.7. Comparison of existing and proposed designs

Figure 2 shows one way of understanding the relationships between existing single- and two-stage designs and the two proposed designs. Our proposed designs are denoted by a grey background. The designs CC+AR, CC+KK, Simon+AR and Simon+KK were examined by Kunz and Kieser (Kunz and Kieser 2012), and are described more fully in Section 3.1. This figure shows that the proposed m-stage and SC designs provide a general framework for both stochastic and non-stochastic curtailment in single-stage and two-stage designs, and furthermore, introduce SC for a go decision. The designs achieve this generality though the thresholds θ_F and θ_E. Key differences between existing designs (Simon, Mander and Thompson, NSC, Kunz and Kieser and Ayanlowo and Redden) and our proposed designs are shown in an alternative manner in Table A1, and in a taxonomy of possible two-stage designs in Table A2 in the Appendix.

Figure 2.

CC: Chi and Chen. AR: Ayanlowo and Redden. KK: Kunz and Kieser. *The approach of Ayanlowo and Redden uses θ_F ∈ {0.05, 0.10}. **The approach of Kunz and Kieser uses θ_F ∈ {0, 0.01, …, 1}.

2.8. The loss function

The expected loss function of Mander et al. (Mander et al. 2012) is

$$L=w_{0}ESS\left(p_0\right)+w_{1}ESS\left(p_1\right)+\left(1-w_0-w_1\right)N,$$

where w₀, w₁ ∈ [0, 1] and w₀ + w₁ ≤ 1. In Mander et al., the admissible design was plotted on a grid of all possible combinations of weights. We extend this concept to allow the comparison of design realisations across differing design approaches: for each combination of weights, the design realisation with the lowest loss, L, across all design approaches (Simon, Mander and Thompson, SC, m-stage, etc.) is found, and this design realisation is termed the omni-admissible design realisation for that combination of weights. Across a grid of possible combinations of weights, the design approach to which each omni-admissible design belongs is plotted. In addition, the difference between the expected loss of admissible design realisations at each set of weights is quantified, for certain pairs of design types. From these plots, the number of admissible design realisations, and the range of weights for which each admissible design realisation has the lowest loss, can be seen. There are different ways of choosing a final design realisation using such a plot. One could choose a set of weights in advance; the design realisation with the lowest expected loss would then be shown at the corresponding point on the plot. Another way would be to select the design realisation that covers the largest area on the plot; this is the design realisation that has the lowest expected loss for most possible sets of weights.

2.9. Inference: estimation of response rate

The MLE of the response rate is the observed response rate, ${\widehat{p}}_{naive}=S\left(m\right)/m$, where m represents the number of participants after which the trial stopped. This naïve estimator is biased in trials that allow stopping at an interim analysis. However, there are a range of estimators available that aim to reduce this bias. In general, such estimators have only been previously presented for two-stage designs, with the notable exceptions of Girshick et al. (Girshick et al., 03 1946) and Jung and Kim (Jung and Kim 2004). We examine estimates of the response rate across existing and novel designs, for five estimators, extended here to the multi-stage case: the naïve estimator (MLE); the bias-adjusted estimator (Chang et al. 1989); the simplified bias subtraction estimator (Guo and Liu 2005); the median unbiased estimator (MUE) (Koyama and Chen 2008) and the uniformly minimum-variance unbiased estimator (UMVUE) (Jung and Kim 2004). We evaluate the bias and the root-mean-square error (RMSE).

The bias-subtracted and bias-adjusted estimators are described in terms of the expected value of the response rate and its bias. The expected estimate of the response rate, $\widehat{p}$, can be obtained by taking the product of the observed response rate for each possible terminal point and its probability given some true p, and summing across all possible terminal points:

$$\text{𝔼}\left(\widehat{p}|p,\mathbf{\text{e}},\mathbf{\text{f}},\mathbf{\text{n}}\right)={\displaystyle \sum _{j=1}^J{\displaystyle \sum _{S\left(n_j\right)=0}^{n_j}𝕀\left[\left\{S\left(n_j\right),n_j\right\}\in 𝒯\right]}}\widehat{p}\left(S\left(n_j\right),n_j\right)U\left(S\left(n_j\right),n_j|p,\mathbf{\text{e}},\mathbf{\text{f}},\mathbf{\text{n}}\right),$$

where $\widehat{p}\left(S\left(n_j\right),n_j\right)$ is the observed response rate $\widehat{p}$ at the point (S(n_j), n_j) and n = (n₁, n₂, …, n_J) is the vector of sample sizes in each stage. For continuous monitoring, n₁ = n₂ = … = n_J = 1. The bias, variance and RMSE are as follows:

$$\text{Bias}\left(\widehat{p}|p,\mathbf{\text{e}},\mathbf{\text{f}},\mathbf{\text{n}}\right)=\text{𝔼}\left(\widehat{p}|p,\mathbf{\text{e}},\mathbf{\text{f}},\mathbf{\text{n}}\right)-p$$

$$\text{Var}\left(\widehat{p}|p,\mathbf{\text{e}},\mathbf{\text{f}},\mathbf{\text{n}}\right)=\text{𝔼}\left({\widehat{p}}^2|p,\mathbf{\text{e}},\mathbf{\text{f}},\mathbf{\text{n}}\right)-\text{𝔼}{\left(\widehat{p}|p,\mathbf{\text{e}},\mathbf{\text{f}},\mathbf{\text{n}}\right)}^2$$

$$\text{RMSE}\left(\widehat{p}|p,\mathbf{\text{e}},\mathbf{\text{f}},\mathbf{\text{n}}\right)=\sqrt{\text{Bias}{\left(\widehat{p}|p,\mathbf{\text{e}},\mathbf{\text{f}},\mathbf{\text{n}}\right)}^2+\text{Var}\left(\widehat{p}|p,\mathbf{\text{e}},\mathbf{\text{f}},\mathbf{\text{n}}\right)}$$

The bias-subtracted estimator is

$${\widehat{p}}_{bias-sub}={\widehat{p}}_{naive}-\text{Bias}\left({\widehat{p}}_{naive}|{\widehat{p}}_{naive},\mathbf{\text{e}},\mathbf{\text{f}},\mathbf{\text{n}}\right).$$

The bias-adjusted estimator is the numerical solution to

$${\widehat{p}}_{bias-adj}={\widehat{p}}_{naive}-\text{Bias}\left({\widehat{p}}_{naive}|{\widehat{p}}_{bias-adj},\mathbf{\text{e}},\mathbf{\text{f}},\mathbf{\text{n}}\right).$$

The median unbiased estimator, ${\widehat{p}}_{MUE}$ is obtained by numerically searching for the value of p that would make the p-value equal to 0.5:

$$\text{p}-\text{val}\left(S\left(m\right),m|{\widehat{p}}_{MUE}\right)=0.5,$$

where the p-value is computed as the sum of the probability of possible outcomes with a larger value of the UMVUE. The UMVUE for a single-arm multi-stage binomial outcome trial was derived by Jung and Kim (Jung and Kim 2004). At some point (S(m), m), the UMVUE is

$${\widehat{p}}_{UMVUE}=\text{𝔼}({\widehat{p}}^{\left(m_1\right)}|S\left(m\right),m),$$

the expected value of the response rate after some m₁ participants, denoting the first point at which a decision may be made. The estimates for all estimators are obtained using the R package singlearm (Grayling 2019). The comparison of these likelihood-based estimators is similar to the comparisons made by Porcher and Desseaux in the context of the Simon design (Porcher and Desseaux 2012). However, we note that there are Bayesian approaches available that can give superior inference if one wishes to assume an informative prior. Specifically, there is an opportunity for early phase trials to benefit either from expert opinion or historical information that can be used dynamically such as the equivalence probability weighted power prior methodology (Bennett et al. 2021).

3. Results

3.1. Real data example

Kunz and Kieser (Kunz and Kieser 2012) present a real data example from Sharma et al. (Sharma et al. 2012). In this trial, the following design parameters were chosen: α = 0.05, β = 0.1, p₀ = 0.2, p₁ = 0.4. Kunz and Kieser compare the following combinations of designs to the (p₀-)optimal Simon design, with SC permitted for a no go decision only: NSC for no go only, SC in stage ∈ only (Simon + AR); NSC for no go only, SC in both stages (Simon + KK); NSC for go and no go, SC in stage ∈ only (CC + AR), and NSC for go and no go, SC in both stages (CC + KK), where AR is the design of Ayanlowo and Redden, KK is the design of Kunz and Kieser and CC is the design of Chi and Chen. The results for threshold θ_F = 0.4 from Kunz and Kieser (Kunz and Kieser 2012) are reported here, as the authors report ESS(p₀) for only θ_F = 0.4 and θ_F = 0.6 and state that trials using θ_F = 0.6 do not achieve adequate power. Table 1 contains the operating characteristics for these designs to as great an extent as possible, and also the following designs: p₀-optimal designs for Chi and Chen (CC), the SC design (SC ₂) and the m-stage design (m-stage ₂). Further designs included are SC and m-stage designs chosen for having a similar N to other design realisations (SC ₁, m-stage ₁) and also Wald’s SPRT (Wald 1947).

Table 1.

Comparison of designs, with design parameters (α, β, p₀, p₁) = (0.05, 0.10, 0.20, 0.40). CC: Chi and Chen. AR: Ayanloyo and Redden. KK: Kunz and Kieser. Blanks in a* and ESS(p₁) due to data not being included in Kunz and Kieser and not being reproducible using the Stata package simontwostage. * Median values, from simulation.

Design	r 1	n 1	r	N	a*	1 − β*	ESS(p0)	ESS(p1)	θF	θE
Simon	4	19	15	54	0.048	0.904	30.4	51.6	–	–
CC	4	19	15	54	0.048	0.904	28.2	37.6	0.000	1.000
Simon + AR	4	19	15	54	–	0.882*	26.6	–	0.400	1.000
Simon + KK	4	19	15	54	0.038	0.857*	21.2	–	0.400	1.000
CC + AR	4	19	15	54	–	0.882*	25.4	–	0.400	1.000
CC + KK	4	19	15	54	–	0.857*	21.0	–	0.400	1.000
SC 1	2	14	15	54	0.050	0.901	23.0	26.6	0.164	0.998
SC 2	4	21	16	58	0.050	0.900	22.6	25.5	0.199	0.998
m-stage 1	–	–	15	52	0.049	0.909	25.3	25.8	0.135	0.996
m-stage 2	–	–	26	94	0.049	0.902	22.1	23.3	0.228	0.998
Wald	–	–	–	–	0.050	0.900	21.8	22.7	–	–

The operating characteristics of Simon+AR, Simon+KK, CC+AR and CC+KK were obtained from Kunz and Kieser (Kunz and Kieser 2012) and from Stata using the simontwostage package (Kunz and Kieser 2011). The maximum N searched over for the SC and m-stage designs, respectively, is N = 58 and N = 94, due to limitations of computational power.

Table 1 shows that with the exception of Wald, the designs with the lowest ESS(p₀) are Simon+KK and CC+KK, which use a threshold of θ_F = 0.4 and allow stopping at any point. However, these designs both have power 1 − β* = 0.857 < 1 − β = 0.9. The designs Simon+AR and CC+AR also have power less than 1 − β = 0.9.

The four design realisations obtained using the proposed designs achieve a lower ESS(p₀) than all other feasible design realisations with the exception of Wald, while achieving the necessary type-I error-rate and power. Furthermore, the first m-stage design has a lower maximum sample size than all other designs.

The study by Sharma et al. (Sharma et al. 2012) ended at the first stage, with zero responses out of 19 participants. Using NSC only, the study would have ended after 15 participants. However, using m-stage ₁ or m-stage ₂ the study would have ended after 11 or 8 participants respectively.

3.2. Example trials

Three sets of design parameters, or scenarios, were used to compare five design approaches: Simon; Mander and Thompson’; NSC; SC and m -stage. For each scenario and design type, optimal design realisations were obtained that satisfy each of four single optimality criteria, and a set of admissible design realisations were obtained with regard to the loss function specified by Mander et al. (Mander et al. 2012) (Section 2.8). For Simon’s and Mander and Thompson’s design, the maximum sample size searched over was 20 % greater than the maximum sample size of the p₀-optimal design, as in Mander et al. (Mander et al. 2012). For the NSC and m-stage designs, the maximum sample size searched over was set to 80, approximately 2–3 times greater than the maximum sample size for the optimal Simon design’s under the p₀-optimal and p₀-minimax criteria. For the SC design, the maximum sample size was 43 to 47 depending on the scenario, due to computational intensity. Also reported is ESS(p₀) and ESS(p₁) from Wald’s SPRT. For the first scenario, the design parameters are α = 0.05, β = 0.15, p₀ = 0.1 and p₁ = 0.3, chosen to match Jung et al. and Mander et al. (Jung et al. 2004; Mander et al. 2012).

Table 2 shows the optimal design realisation for each design approach, for four optimality criteria: p₀-optimal, p₁-optimal, p₀-minimax and p₁-minimax. For all four optimality criteria, the optimal design realisations of the proposed designs outperform those of the existing designs, and use thresholds of θ_F < 0.23 and θ_E > 0.98 in each case. The ESSs using Wald’s SPRT are ESS(p₀) = 13.9, ESS(p₁) = 13.9, comparable to those of the m-stage design, ESS(p₀) = 14.1 under p₀-optimality, ESS(p₀) = 14.3 under p₁-optimality and ESS(p₁) = 14.4 under both p₀- and p₁-optimality.

Table 2.

Optimal design realisations for each design type, Scenario 1: (α, β, p₀, p₁) = (0.05, 0.15, 0.10, 0.30). For all designs, the requisite type-I error-rate and power is achieved. Columns ${\text{\%S}}_{p_0}$ and ${\text{\%S}}_{p_1}$ show ESS as a proportion of Simon's design under p = p₀ and p = p₁ respectively. MT: Mander and Thompson.

	r 1	e 1	n 1	r	N	ESS(p0)	${\text{\%S}}_{p_0}$	ESS(p1)	${\text{\%S}}_{p_1}$	θF	θE
p0-optimal
Simon	1	–	11	6	35	18.3	1.00	32.3	1.00	–	–
MT	1	4	11	6	35	18.2	1.00	27.2	0.84	–	–
NSC	1	–	13	5	28	17.6	0.97	18.5	0.57	0.000	1.000
SC	4	–	27	7	41	14.3	0.78	15.0	0.46	0.186	0.993
m-stage	–	–	–	13	80	14.1	0.77	14.4	0.45	0.226	0.997
p1 -optimal
Simon	2	–	18	5	27	20.4	1.00	26.5	1.00	–	–
MT	0	3	13	6	30	25.1	1.23	20.0	0.76	–	–
NSC	1	–	13	5	28	17.6	0.87	18.5	0.70	0.000	1.000
SC	4	–	24	8	43	15.5	0.76	14.6	0.55	0.126	0.984
m-stage	–	–	–	12	66	14.3	0.70	14.4	0.54	0.189	0.990
Wald’s SPRT	–	–	–	–	–	13.9	0.68	13.9	0.52	–	–
P0-minimax
Simon	2	–	18	5	27	20.4	1.00	26.5	1.00	–	–
MT	1	4	14	5	27	19.3	0.95	21.0	0.79	–	–
NSC	2	–	18	5	27	19.3	0.95	18.7	0.71	0.000	1.000
SC	0	–	10	5	27	17.1	0.84	16.3	0.62	0.070	0.990
m-stage	–	–	–	5	27	18.7	0.92	16.6	0.63	0.084	0.990
p1-minimax
Simon	2	–	18	5	27	20.4	1.00	26.5	1.00	–	–
MT	1	4	15	5	27	20.3	0.99	20.8	0.78	–	–
NSC	2	–	18	5	27	19.3	0.95	18.7	0.71	0.000	1.000
SC	4	–	24	5	27	18.8	0.92	15.8	0.60	0.050	0.986
m-stage	–	–	–	5	27	18.7	0.92	16.6	0.63	0.084	0.990

Figure 3 (left) shows the design approach to which the omni-admissible design realisation belongs, for each combination of weights. The omni-admissible design belongs to either the SC design or the m-stage design. The difference in expected loss between the SC and m-stage admissible design realisations for each combination of weights is shown in Figure 3 (right). It shows that the admissible m-stage design realisations have a slightly lower loss score than those of the SC design realisations near the triangle’s hypotenuse, that is, when there is low weight on maximum sample size N. For much of the surface of weight combinations, the difference in loss score is in favour of the SC design but negligible, including where both w₀ and w₁ are close to zero and the weight of N is close to one. Analogous results for two further scenarios are given in the Appendix.

Figure 3.

Type of design to which the omni-admissible design realisation belongs and difference in loss scores between the SC and m-stage admissible design realisations (positive favours m-stage), scenario 1 (α, β, p₀, p₁) = (0.05, 0.15, 0.10, 0.30).

3.2.1. Admissible design realisations by design type

In Figure 4, the scenario 1 admissible design realisations are shown for each design type and combination of weights. The plots of admissible design realisations for the Simon and Mander and Thompson’s designs, Figure 4 (top), match those obtained by Mander et al. The overall results are similar across all three scenarios (with scenarios 2 and 3 again in the Appendix): the proposed designs generally contain a greater number of admissible design realisations across the combinations of weights examined than the existing designs. This is expected as including SC thresholds necessarily results in an increased number of possible design realisations. For the proposed designs, the admissible design regions often contain slopes parallel to the hypotenuse, suggesting that the admissible design may be more dependent on the weight of N than ESS(p₀) or ESS(p₁) separately.

Figure 4.

Admissible design realisations for scenario 1 (α, β, p₀, p₁) = (0.05, 0.15, 0.10, 0.30). Format of design realisations: Simon, NSC: {r₁/n₁, r/N}; Mander and Thompson: {(r₁ e₁)/n₁, r/N}; SC: {r₁/n₁, r/N, θ_F/θ_E}; m-stage: {r/N, θ_F/θ_E}.

3.3. Effect of reduced monitoring frequency

Table 3 shows optimal design realisations using blocks of size four, that is, permitting SC after every four participants, for the first scenario (α, β, p₀, p₁) = (0.05, 0.15, 0.10, 0.30). These are shown alongside the optimal design realisations for Simon’s design and the m-stage design. The m-stage design may be considered to be equivalent to using blocks of size one. Under the p₀- and p₁-optimality criteria and p₀- and p₁-minimax criteria, which are combined in the table as the optimal (minimax) design realisations are identical in this instance, the design realisations with block size four produce considerable savings in ESS(p₁) and have comparable ESS(p₀) compared to Simon’s design.

Table 3.

Selection of optimal design realisations, including stochastically curtailed designs with stopping permitted after every four participants, for scenario 1: (α, β, p₀, p₁) = (0.05, 0.15, 0.10, 0.30). For all design realisations, requisite type-I error-rate and power is reached. Columns ${\text{\%S}}_{p_0}$ and ${\text{\%S}}_{p_1}$ show ESS as a proportion of Simon's design under p = p₀ and p = p₁ respectively.

	r 1	n 1	r	n	ESS(p0)	${\text{\%S}}_{p_0}$	ESS(p1)	${\text{\%S}}_{p_1}$	θF	θE
p0-optimal
Simon	1	11	6	35	18.3	1.00	32.3	1.00	–	–
m-stage	–	–	13	80	14.1	0.77	14.4	0.45	0.226	0.997
Block size 4	–	–	6	32	18.8	1.03	18.7	0.58	0.194	0.984
p1-optimal
Simon	2	18	5	27	20.4	1.00	26.5	1.00	–	–
m-stage	–	–	12	66	14.3	0.70	14.4	0.54	0.189	0.990
Block size 4	–	–	11	52	18.9	0.93	18.5	0.70	0.191	0.974
P0/1-minimax
Simon	2	18	5	27	20.4	1.00	26.5	1.00	–	–
m-stage	–	–	5	27	18.7	0.92	16.6	0.63	0.084	0.990
Block size 4	–	–	6	32	18.8	0.92	18.7	0.71	0.194	0.984

3.4. Estimation (scenario 1, selected)

Bias and RMSE in the response rate estimates are shown in Figures 5–6 for p₀-optimal design realisations for scenario 1 (α, β, p₀, p₁) = (0.05, 0.15, 0.10, 0.30), with the maximum absolute bias and RMSE shown in Table 4. In Simon’s design and the Mander and Thompson design, the bias is close to zero for all estimators (Figure 5, left). For designs that employ curtailment, the bias adjusted, bias subtracted MUE and UMVUE estimators have a bias consistently close to zero, while the naïve estimator gives more biased estimates (Figure 5, bottom left, Figure 6, left). Overall, bias and RMSE is only slightly poorer among the proposed designs than the existing designs when p < p₁. For greater p, the poorer estimates among the proposed designs are a result of the trial being curtailed with fewer participants compared to the existing designs. The maximum absolute bias is similar across designs, with the exception of somewhat greater bias among the proposed designs under the naïve estimator (Table 4).

Figure 5. Bias, RMSE for p₀-optimal designs, scenario 1 (α, β, p₀, p₁) = (0.05, 0.15, 0.10, 0.30).

Figure 6. Bias, RMSE for p₀-optimal designs, scenario 1 (α, β, p₀, p₁) = (0.05, 0.15, 0.10, 0.30), continued.

Table 4.

Maximum absolute bias and RMSE for various point estimators of p₀-optimal designs, scenario 1: (a = 0.05, β = 0.15, p₀ = 0.1, p = 0.3). MT: Mander and Thompson.

	Bias (absolute)					RMSE
	Bias adj.	Bias subt.	Naïve	MUE	UMVUE	Bias adj.	Bias subt.	Naïve	MUE	UMVUE
Simon	0.008	0.009	0.031	0.032	4.44 × 10−16	0.097	0.098	0.104	0.106	0.101
MT	0.010	0.010	0.029	0.049	2.22 × 10−16	0.147	0.147	0.138	0.146	0.150
NSC	0.009	0.010	0.041	0.030	3.33 × 10−16	0.165	0.165	0.161	0.159	0.166
SC	0.022	0.025	0.090	0.030	3.05 × 10−16	0.232	0.235	0.223	0.224	0.236
m-stage	0.025	0.024	0.094	0.024	3.33 × 10−16	0.232	0.236	0.231	0.232	0.246

The RMSE of the estimates gradually increases as the degree of permitted curtailment increases, with Simon’s design having the lowest RMSE and the proposed designs the greatest (Table 4). RMSE decreases sharply to zero as the response rate approaches one.

4. Discussion

Phase II binary outcome trials are a critical aspect of drug development. However, with high failure rates and high costs, it is valuable to find ways of making correct decisions more quickly. We have presented two single-arm designs created to improve binary outcome trials in this respect.

As part of these proposed designs, this work introduces five approaches to improving a search for an optimal or admissible design realisation that uses SC: firstly, the exact distribution of the trial outcomes is obtained, allowing the trial’s operating characteristics to be known without simulation error. Secondly, a new approach is proposed for finding relevant CP thresholds when using SC, based on the CP at each point in each possible set {r, N} or {r₁, n₁, r, N}, allowing more potential design realisations to be evaluated. Thirdly, the CP at each point in each potential design realisation is calculated taking the possibility of SC into account. Furthermore, in the design search, type-I error-rate and power are only calculated after taking curtailment into account. Finally, the design search is undertaken using wide ranges for maximum sample size and final rejection boundary, rather than being restricted to, say, a single realisation of Simon’s design.

Comparing our proposed designs to a number of existing designs, we found that the proposed designs were superior in almost all cases, whether considering either a single optimality criterion or a weighted combination of multiple optimality criteria. The effect of reducing the frequency of monitoring, was examined. It was shown that considerable savings in ESS can still be made even when employing designs with less frequent monitoring.

While all proposed methods performed well in the comparisons that have been made, there are limitations to their use. The two proposed single-arm designs find design realisations with similar operating characteristics. However, one design, the m-stage design, completes a design search more quickly than other design, the SC design, and this difference is approximately one order of magnitude. The proposed single-arm designs both use sequential monitoring, which may be seen as a logistical limitation. However, in our proposed designs, the frequency of monitoring can be specified at the trial design stage, to accommodate the practical needs of the investigators. A separate limitation of the sequential monitoring is that, depending on the design realisation, it is possible that a trial may end with a small number of participants, which may be undesirable in some circumstances. Conservative investigators may prefer to use a large block size.

Arguably, a limitation of this work in general is that it focuses solely on frequentist methods. Bayesian methods can be used for phase II clinical trial design (Dutton et al. 2018; Johnson and Cook 2009; Lee and Liu 2008; Lin and Lee 2020) and are becoming more widely used over time (Lee and Chu 2012; Lin and Lee 2020). However, some Bayesian and frequentist designs are closely related conceptually, for example, the frequentist CP-based approach and the Bayesian predictive power approach (Jennison and Turnbull 2000). Furthermore, in the context of binary outcome trials, Bayesian and frequentist designs can both be described using vectors of lower and upper stopping boundaries f and e. One main advantage of Bayesian trial design is the ability to incorporate prior information though not all Bayesian designs do so, and instead use an uninformative prior (or priors). In contrast, frequentist methods are deemed to discard such data, or at best use it in as a summary way. Bayesian designs may be created with Bayesian operating characteristics in mind, which can be more intuitive to non-statisticians. However, Bayesian designs may be required to satisfy certain frequentist operating characteristics. In the case that a Bayesian design both uses an uninformative prior and must satisfy some typical frequentist operating characteristics, the resulting design realisation may confer no advantage over an equivalent frequentist design. Another advantage of some Bayesian designs is that they are more flexible in terms of allowing interim analyses to occur at points that are not fixed in advance, with only minor negative consequences in terms of Bayesian operating characteristics (Lee and Liu 2008), though negative consequences persist in terms of frequentist operating characteristics. However, allowing such flexibility could result in unconscious bias, with investigators able to undertake an interim analysis after observing a succession of positive results or avoiding an interim analysis after observing a succession of negative results. Furthermore, in a two-stage design, it is possible to conduct an interim analysis at a different point to that planned, while still controlling the type-I error-rate (Englert and Kieser 2015). Finally, Bayesian methods often use simulation, though conjugate priors may be available in some circumstances, and in the area of very rare disease all posterior distributions can be found (Hampson et al. 2014). Consequences of using simulation may be simulation error and computational requirements, however, our proposed approach avoids simulation entirely.

While the m-stage design performed well in all circumstances examined, other designs performed similarly when sole importance was placed on minimising maximum sample size N. As such, existing designs should be preferred over the proposed designs when performance in similar for the optimality criterion of prime importance and the existing design uses fewer interim analyses. We acknowledge that stopping early for a go decision is not critical for every phase II study. However, a number of single- and two-arm approaches permit such early stopping (Carsten and Chen 2016; Chen et al. 2018; Chi and Chen 2008; Law et al. 2020; Mander and Thompson 2010). We have also provided examples of trials that stopped early for a go decision even without using a design that permitted such stopping (Mego et al. 2016; Necchi et al. 2014; Santana et al. 0000; Wagner et al. 2015), while an upcoming trial will use the m-stage design, including early stopping for a go decision (Positioning imatinib for pulmonary arterial hypertension (PIPAH) 2020). This suggests that early stopping for a go decision is both well-established methodologically and used in practice.

In practical terms, single-arm design realisations can be obtained by calling the appropriate functions in R (R Core Team 2019) after installing and loading the package curtailment (M. Law 2021). For each proposed design approach, a single function undertakes a single-arm design search. Admissible design realisations are returned, if they exist. A second function takes as its input any chosen design realisation and returns the corresponding stopping boundaries in a simple table and diagram. The final output is simply a collection of stopping boundaries.

The proposed single-arm designs could be extended in a number of ways in future work. It would be valuable to investigate the effects of delayed responses on ESS in curtailed designs. In case of a desire to collect a certain degree of information in a trial, a trial could be specified to end only after data is available for some minimum number of participants. With regard to estimation, estimates of confidence intervals and p values could be compared to those from existing design types. While the results of Wald’s SPRT (Wald 1947) are positive, the design has no upper limit for maximum sample size. This poses obvious problems in terms of approval and logistics. This is also why the designs are compared only to optimal designs and not minimax designs – the maximum sample size is infinite. Wald states that if a finite maximum sample size is desired, it can simply be chosen, and provides a “reasonable rule” for the final stopping boundary. Future work may compare this design to the proposed designs not only in terms of weighted combination of ESS(p₀), ESS(p₁) and N, but also in terms of minimising sup ESS(p) (Kiefer and Weiss 1957; Shuster 2002). p∈[0,1]

In summary, the clinical trial designs we propose offer considerably improved operating characteristics compared to existing designs, and widespread use of these designs would speed up drug development.