Point estimation following a two-stage group sequential trial

Abstract

Repeated testing in a group sequential trial can result in bias in the maximum likelihood estimate of the unknown parameter of interest. Many authors have therefore proposed adjusted point estimation procedures, which attempt to reduce such bias. Here, we describe nine possible point estimators within a common general framework for a two-stage group sequential trial. We then contrast their performance in five example trial settings, examining their conditional and marginal biases and residual mean square error. By focusing on the case of a trial with a single interim analysis, additional new results aiding the determination of the estimators are given. Our findings demonstrate that the uniform minimum variance unbiased estimator, whilst being marginally unbiased, often has large conditional bias and residual mean square error. If one is concerned solely about inference on progression to the second trial stage, the conditional uniform minimum variance unbiased estimator may be preferred. Two estimators, termed mean adjusted estimators, which attempt to reduce the marginal bias, arguably perform best in terms of the marginal residual mean square error. In all, one should choose an estimator accounting for its conditional and marginal biases and residual mean square error; the most suitable estimator will depend on relative desires to minimise each of these factors. If one cares solely about the conditional and marginal biases, the conditional maximum likelihood estimate may be preferred provided lower and upper stopping boundaries are included. If the conditional and marginal residual mean square error are also of concern, two mean adjusted estimators perform well.

1. Introduction

A desire to reduce the number of patients required in a clinical trial has seen a large number of trialists utilise a group sequential study design. In this approach, interim analyses after certain landmark amounts of data have been collected afford the possibility for a trial to terminate early. Approaches to design for fixed sample studies are not appropriate with repeated data analysis, due to inflation of error rates. Therefore, a large amount of methodology has been developed to facilitate attaining desired type I and type II error rates in a group sequential trial. For an overview of this methodology, see Jennison and Turnbull¹ or Whitehead.²

In addition, following a group sequential trial, special inferential techniques are required. In the case of confidence intervals, this is to ensure the desired coverage is attained. For p-values, it is to guarantee that they are consistent with the decision about the null hypothesis. Here, our focus is on point estimation, where specially developed methods are required because the standard maximum likelihood estimator (MLE) is no longer either unbiased or minimum variance.

Numerous authors have now proposed point estimation procedures for after a group sequential trial. Often, these have sought to reduce the bias in the estimate; an important aim given that the magnitude of a treatment effect is always of principal interest in a clinical trial. However, there are several important factors to consider when choosing a preferred estimator, including also the residual mean squared error (RMSE), and the division of the bias and RMSE into their marginal and conditional (on stage of termination) values.

Several authors have also sought to compare estimators.^3–6 However, each of these works has compared a limited number of estimators, and has been set within the context of a specific study setting (e.g., Shimura⁶ consider survival data). Recently, Robertson et al.⁷ provided an extended discussion of adjusted point estimation following adaptive design. Along with describing available methodologies, they provided guidance for researchers on best practice, and reviewed current use in published trials. They found, like Zhang et al.,⁸ that to date few studies have computed an adjusted estimate. Shimura et al.⁶ previously argued this is because of a lack of available comparisons of estimators and lack of software for their computation. Robertson et al.⁷ also noted the potential importance of conducting an extensive comparison of estimators to choose the preferred approach for a given trial. The purpose of this paper is therefore to describe a large number of estimators, and historical results pertaining to them, within a common notation. Through this, we make the principal evaluations that are required to compare estimators clear. We then examine the performance of these estimators in terms of their biases and RMSE through several informative examples. By focusing on a certain type of two-stage trial, we also derive a number of small new results that simplify the determination of several estimators in practice. Furthermore, we provide code for implementation such that modification for alternative settings can be readily achieved. We proceed by first describing the exact design setting considered.

2. Methods

2.1. Design setting

We consider point estimation following a two-stage group sequential trial that bases decisions on the standardised test statistics $\{Z_1,Z_2\}$, which follow the canonical joint distribution with information levels $\{I_1,I_2\}$ for the parameter $\theta$, as defined by Jennison and Turnbull.¹ That is

• $(Z_1,Z_2)$ is bivariate normal,

• $\mathbb{E}(Z_k)=\theta \sqrt{I_k},k=1,2$, and

• $Cov(Z_{k_1},Z_{k_2})=\sqrt{I_{k_1}/I_{k_2}}$, $1\le k_1\le k_2\le 2$.

At least approximately, this joint distribution holds for an extremely wide variety of trial designs and study endpoint types. We assume $H_0:\theta =0$, though this will be rarely used. We focus also on the case where the trial continues to stage 2 if $z_1\in (l,u)$. We make these restrictions for several reasons. Firstly, a large proportion of all group sequential trials include only a single interim analysis; see, for example, the review by Stevely et al.⁹ who found two-stage designs to be the most common, accounting for 44.4% (28/63) of included trials for which the number of interims was determinable. Secondly, assuming a continuous continuation region of the form $z_1\in (l,u)$ enables the arguably preferable stage-wise test statistic ordering to be employed when determining a median unbiased estimate (MUE) later. Finally, by making these restrictions we are able to develop new explicit solutions for certain point estimators that will aid their determination in practice. No additional assumptions are required for our specification of the point estimators that follow; additional assumptions, as required by specific study setting examples, are given in the Results as needed.

We let $K=1,2$ denote the number of analyses performed when the trial is terminated and $Z=Z_K$ be the corresponding terminal standardised test statistic. Then, the pair of statistics $(K,Z)$ is known to be a sufficient statistic for $\theta$.^3,10,11 We will make repeated use of the density of $(K,Z)$ given $\theta$, $f(k,z|\theta )$. Armitage et al.¹² described a generic recursive approach to calculating $f(k,z|\theta )$. In the Supplemental Materials, we demonstrate that in our two-stage case, it can be written as

$$\begin{array}{rl}f(1,z|\theta )& =\{\begin{array}{cc}\varphi (z,\theta \sqrt{I_1},1)& :z\notin (l,u)\\ 0& :z\in (l,u)\end{array}\\ f(2,z|\theta )& =\sqrt{I_2}{\int }_l^u\varphi (z_1,\theta \sqrt{I_1},1)\varphi \{z\sqrt{I_2}-z_1\sqrt{I_1},\theta (I_2-I_1),I_2-I_1\}\mathrm{d}{z}_1\\ & =\frac{e^{-(z-\theta \sqrt{I_2}{)}^2/2}}{\sqrt{2\pi }}[\mathrm{\Phi }\{u,z\sqrt{I_1/I_2},(I_2-I_1)/I_2\}\\ & -\mathrm{\Phi }\{l,z\sqrt{I_1/I_2},(I_2-I_1)/I_2\}]\end{array}$$ (1)

where $\varphi (x,\mu ,{\sigma }^2)$ and $\mathrm{\Phi }(x,\mu ,{\sigma }^2)$ are respectively the probability density and cumulative distribution functions of a $N(\mu ,{\sigma }^2)$ random variable evaluated at $x$. This final form for $f(2,z|\theta )$, which removes the need for integration, is particularly helpful for reducing computational run time when considering the performance of possible estimators in depth; it facilitates the large number of comparisons performed later and the code made available.

Note that $f(k,z|\theta )$ can be used to compute the probability of stopping at stage $k$, which we denote by $s(k|\theta )$, with

$$s(k|\theta )={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}f(k,z|\theta )\mathrm{d}z$$

We will then also utilise $f_{\text{cond}}(k,z|\theta )=f(k,z|\theta )/s(k|\theta )$, the conditional density of $Z$ given $K=k$. Note that in the two-stage case, we must have that $s(1|\theta )=1-s(2|\theta )$.

2.2. Point estimation

We compare the performance of nine-point estimators for $\theta$, which we denote by $\hat{\theta }=\hat{\theta }(K,Z)$, omitting explicit dependence on $(K,Z)$. Estimates, as defined by particular $(k,z)$, will always be denoted by $\hat{\theta }(k,z)$. Before we specify these estimators, we first define the measures used to compare them. We compute the bias and RMSE of point estimator $\hat{\theta }$, conditional on $K=k$, as

$$\begin{array}{rl}\mathrm{Bias}(\hat{\theta }|\theta ,k)& =\mathbb{E}(\hat{\theta }|\theta ,k)-\theta \\ \mathrm{RMSE}(\hat{\theta }|\theta ,k)& =\sqrt{Var(\hat{\theta }|\theta ,k)+\mathrm{Bias}(\hat{\theta }|\theta ,k{)}^2}\\ \mathbb{E}({\hat{\theta }}^x|\theta ,k)& ={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\hat{\theta }(k,z{)}^x{f}_{\text{cond}}(k,z|\theta )\mathrm{d}z,x=1,2\\ Var(\hat{\theta }|\theta ,k)& =\mathbb{E}({\hat{\theta }}^2|\theta ,k)-\mathbb{E}(\hat{\theta }|\theta ,k{)}^2\end{array}$$

From these, we can also compute the marginal bias and RMSE of $\hat{\theta }$ as

$$\begin{array}{rl}\mathrm{Bias}(\hat{\theta }|\theta )& =\sum _{k=1}^{2}s(k|\theta )\mathrm{Bias}(\hat{\theta }|\theta ,k)\\ \mathrm{RMSE}(\hat{\theta }|\theta )& =\sum _{k=1}^{2}s(k|\theta )\mathrm{RMSE}(\hat{\theta }|\theta ,k)\end{array}$$

That is, the marginal values are simply a weighted sum of the conditional values, with weights given by the probability of the trial stopping at each stage.¹³ Note that we then say $\hat{\theta }$ is unbiased if $\mathrm{Bias}(\hat{\theta }|\theta )=0$ for all $\theta$. Liu et al.¹³ defined $\hat{\theta }$ to be conditionally unbiased if $\mathrm{Bias}(\hat{\theta }|\theta ,k)=0$ for all $\theta$ and $k$. While the latter implies the former, vice versa is not true, and hence being conditionally unbiased is a stronger requirement. Previous examinations of possible point estimators have focused on the fact that low marginal bias may not result in low conditional bias.^13–15 Thus consideration of both conditional and marginal performance is important.

2.2.1. Maximum likelihood estimator

The MLE of $\theta$ is given by ${\hat{\theta }}_{\text{M}LE}(k,z)=z/\sqrt{I_k}$; see Chang¹⁶ for details. As has been remarked on extensively, ${\hat{\theta }}_{\text{M}LE}$ is a biased estimator for $\theta$. Typically, the MLE overestimates the magnitude of $\theta$, giving positive bias for $\theta >0$ and negative bias for $\theta <0$.^1,17 Emerson¹⁸ obtained an analytical expression for $\mathrm{Bias}({\hat{\theta }}_{\text{M}LE}|\theta )$ in our setting as

$$\mathrm{Bias}({\hat{\theta }}_{\mathrm{MLE}}|\theta )=\frac{I_2-I_1}{I_2\sqrt{I_1}}\{\varphi (u,\theta \sqrt{I_1},1)-\varphi (l,\theta \sqrt{I_1},1)\}$$ (2)

Fan et al.¹⁵ also provided several results on the conditional bias of the MLE.

2.2.2. Mean adjusted estimators

Whitehead¹⁹ suggested adjusting ${\hat{\theta }}_{\text{M}LE}(k,z)$ by subtracting an estimate of its bias; this has since been referred to as a mean adjusted estimator (MAE). With the true value of $\theta$ unknown, which determines the bias of ${\hat{\theta }}_{\text{M}LE}$, a proposed estimator which we refer to as ${\hat{\theta }}_{\text{M}AE1}$ is the solution to

$${\hat{\theta }}_{\text{M}AE1}(k,z)={\hat{\theta }}_{\text{M}LE}(k,z)-\mathrm{Bias}\{{\hat{\theta }}_{\text{M}LE}|{\hat{\theta }}_{\text{M}AE1}(k,z)\}$$

That is, an estimator which has the property $\mathbb{E}\{{\hat{\theta }}_{\text{M}LE}|{\hat{\theta }}_{\text{M}AE1}(k,z)\}={\hat{\theta }}_{\text{M}LE}(k,z)$. No closed-form solution for ${\hat{\theta }}_{\text{M}AE1}(k,z)$ exists and it must be determined using a numerical search.

A simpler alternative to ${\hat{\theta }}_{\text{M}AE1}$, which we refer to by ${\hat{\theta }}_{\text{M}AE2}$, is

$$\begin{array}{rl}{\hat{\theta }}_{\mathrm{MAE}2}(k,z)& ={\hat{\theta }}_{\text{M}LE}(k,z)-\mathrm{Bias}\{{\hat{\theta }}_{\text{M}LE}|{\hat{\theta }}_{\text{M}LE}(k,z)\}\\ & ={\hat{\theta }}_{\text{M}LE}(k,z)-\frac{I_2-I_1}{I_2\sqrt{I_1}}[\varphi \{u,{\theta }_{\text{M}LE}(k,z)\sqrt{I_1},1\}\\ & -\varphi \{l,{\theta }_{\text{M}LE}(k,z)\sqrt{I_1},1\}]\end{array}$$

The principal advantage of this estimator over ${\hat{\theta }}_{\text{M}AE1}$ is the ease of its determination, which has an explicit solution using equation (2). This estimator was considered by Guo and Liu²⁰ in the context of an exact two-stage group sequential single-arm trial for a Bernoulli-distributed primary outcome.

2.2.3. Median unbiased estimator

Next, we consider an MUE for $\theta$. MUEs are dependent on the definition of an ordering of the sample space. Following Jennison and Turnbull,¹ we write $(k^{\prime },z^{\prime })\succ (k,z)$ to denote that $(k^{\prime },z^{\prime })$ is above $(k,z)$ in a chosen ordering of the sample space, and denote by $\mathbb{P}\{(K,Z)\succ (k,z)|\theta \}$ the probability of observing a result above $(k,z)$ conditional on $\theta$. With this, a MUE ${\hat{\theta }}_{\text{M}UE}(k,z)$ can be defined as the solution to

$$\mathbb{P}\{(K,Z)\succ (k,z)|{\hat{\theta }}_{\text{MUE}}(k,z)\}=0.5$$

Unfortunately, there is no single most-logical way to order possible outcomes of a group sequential trial.¹ A choice must be made in practice between a number of orderings, all of which have their own appeal. Armitage²¹ and Emerson and Fleming³ have considered an MLE ordering. Chang¹⁶ considered a likelihood-ratio-based ordering. Rosner and Tsiatis²² considered a score test-based ordering. Respectively, these consider $(k^{\prime },z^{\prime })\succ (k,z)$ if $(k^{\prime },z^{\prime })$ results in a larger value of the MLE, likelihood-ratio, or score. For more details on these, see Jennison and Turnbull.¹

Here, we focus on the stage-wise ordering, first proposed by Armitage,²³ which has since been used by many authors (see, e.g., Siegmund²⁴, Fairbanks and Madsen²⁵, Tsiatis et al.²⁶). As noted earlier, this ordering requires the continuation region to be an interval, which was our reason for the restriction to designs such that stage 2 occurs when $z_1\in (l,u)$. In a two-stage trial, using the stage-wise ordering, $(k^{\prime },z^{\prime })\succ (k,z)$ if any of the following occurs

• $k^{\prime }=k$ and $z^{\prime }\ge z$

• $1=k^{\prime }<k=2$ and $z^{\prime }\ge u$, and

• $2=k^{\prime }>k=1$ and $z<l$

Using this, along with the distribution of the test statistics, we have

$$\mathbb{P}\{(K,Z)\succ (k,z)|\theta \}=\{\begin{array}{cc}1-\mathrm{\Phi }(z,\theta \sqrt{I_1},1)& :k=1\\ 1-\mathrm{\Phi }(u,\theta \sqrt{I_1},1)+{\int }_z^{\mathrm{\infty }}f(2,x|\theta )\mathrm{d}x& :k=2\end{array}$$

From observing the form when $k=1$, it is clear that ${\hat{\theta }}_{\text{M}UE}(1,z)={\hat{\theta }}_{\text{M}LE}(1,z)$. However, ${\hat{\theta }}_{\text{M}UE}(2,z)$ must be determined via a numerical search.

2.2.4. Uniform minimum variance unbiased estimator

A uniform minimum variance unbiased estimator (UMVUE) is an estimator of interest in most estimation problems, and the considered two-stage group sequential design framework is no exception. Emerson and Fleming³ proposed an unbiased estimator of $\theta$ calculated by applying the Rao-Blackwell technique to the unbiased estimate of $\theta$ formed from the stage 1 data, $Z_1/\sqrt{I_1}$. They considered $(K,Z)$ to be a complete sufficient statistic for $\theta$, which would mean their proposed estimator would be the UMVUE. Unfortunately, Liu and Hall²⁷ later proved that $(K,Z)$ is sufficient but not complete. They did, however, demonstrate that this estimator is uniformly minimum variance unbiased amongst a class of estimators termed truncation adaptable. Put simply, this means the class of estimators that do not require knowledge of the number of further analyses nor their associated information levels that would have occurred had the study continued past the terminal analysis $K$. We term this ${\hat{\theta }}_{\text{U}MVUE}$, with it’s formal definition being

$${\hat{\theta }}_{\text{U}MVUE}(k,z)=\mathbb{E}(\frac{Z_1}{\sqrt{I_1}}|k,z)$$

In the general case of a multi-stage group sequential trial, the computation of ${\hat{\theta }}_{\text{U}MVUE}$ can be computationally intensive; Emerson²⁸ and Emerson and Kittelson²⁹ discuss the necessary calculations at length. In our two-stage setting, we demonstrate in the Supplemental Materials that the UMVUE can be computed as

$${\hat{\theta }}_{\text{U}MVUE}(k,z)=\{\begin{array}{cc}{\hat{\theta }}_{\text{M}LE}(k,z)& :k=1\\ {\hat{\theta }}_{\text{M}LE}(k,z)-\frac{I_2-I_1}{I_2\sqrt{I_1}}\frac{\varphi \{u,z\sqrt{I_1/I_2},(I_2-I_1)/I_2\}-\varphi \{l,z\sqrt{I_1/I_2},(I_2-I_1)/I_2\}}{\mathrm{\Phi }\{u,z\sqrt{I_1/I_2},(I_2-I_1)/I_2\}-\mathrm{\Phi }\{l,z\sqrt{I_1/I_2},(I_2-I_1)/I_2\}}& :k=2\end{array}$$

By definition, $\mathrm{Bias}({\hat{\theta }}_{\text{U}MVUE}|\theta )=0$ for all $\theta$. Note that ${\hat{\theta }}_{\text{U}MVUE}(2,z)$ takes a functional form equivalent to an adjusted version of ${\hat{\theta }}_{\text{M}LE}(2,z)$.

2.2.5. Conditional maximum likelihood estimator

Highlighting the potential importance of reducing conditional bias, Liu et al.¹³ and Fan et al.¹⁵ considered an estimator, which we refer to as ${\hat{\theta }}_{\text{C}MLE}$, based on the conditional likelihood. Specifically, they examined

$${\hat{\theta }}_{\text{C}MLE}(k,z)=\underset{\theta }{argmax}{f}_{cond}(k,z|\theta )$$

Like several of the estimators above, no closed form solution exists for ${\hat{\theta }}_{\text{C}MLE}$, though it can be readily determined using, e.g., Newton–Raphson iteration. For this approach, Liu et al.¹³ provide a number of useful results including forms for the differential of the conditional log-likelihood and the conditional Fisher’s information. As highlighted by Liu et al.,¹³ ${\hat{\theta }}_{\text{C}MLE}$ is identical to the $K$th order bias adjusted estimator of Troendle and Yu.¹⁴ Similarly, Fan et al.¹⁵ demonstrated that ${\hat{\theta }}_{\text{C}MLE}$ is identical to an estimator defined as the solution to either of the following problems

$$\begin{array}{rl}\mathbb{E}\{(K,Z)|k\}& =(k,z)\\ \hat{\theta }(k,z)& ={\hat{\theta }}_{\text{M}LE}(k,z)-\mathrm{Bias}\{{\hat{\theta }}_{\text{M}LE}|\hat{\theta }(k,z),k\}\end{array}$$

i.e., as the solution to the conditional version of the problem that defines ${\hat{\theta }}_{\text{M}AE1}$.

2.2.6. Conditional weighted MAE

Shimura et al.³⁰ proposed a shrinkage estimator for use when a group sequential trial terminates early. Their estimator requires a prior guess at $\theta$; denoting this prior guess by ${\theta }_0$, Shimura et al.³⁰ suggested ${\theta }_0$ be specified as the difference under which the trial was powered. Then, their estimator, which we denote by ${\hat{\theta }}_{\text{C}WMAE}$, sets

$${\hat{\theta }}_{\text{C}WMAE}(1,z)={\hat{\theta }}_{\text{M}LE}(1,z)-Bias\{{\hat{\theta }}_{\text{M}LE}|{\hat{\theta }}_{*}(1,z),1\}$$

where

$${\hat{\theta }}_{*}(1,z)=\frac{\{{\hat{\theta }}_{\text{M}LE}(1,z)-{\theta }_0{\}}^2}{\{{\hat{\theta }}_{\text{M}LE}(1,z)-{\theta }_0{\}}^2+I_1^{-1}}{\hat{\theta }}_{\text{M}LE}(1,z)+[1-\frac{\{{\hat{\theta }}_{\text{M}LE}(1,z)-{\theta }_0{\}}^2}{\{{\hat{\theta }}_{\text{M}LE}(1,z)-{\theta }_0{\}}^2+I_1^{-1}}]{\theta }_0$$

The weighting factor in ${\hat{\theta }}_{*}(1,z)$ was chosen to minimise the RMSE of ${\hat{\theta }}_{\text{C}WMAE}(1,z)$.

Shimura et al.³⁰ proposed this estimator as a method of reducing conditional bias on early termination. They did not discuss a functional form for an estimate when a trial continues to stage 2. Here, to be able to compute the marginal performance of ${\hat{\theta }}_{\text{C}WMAE}$, we assume that ${\hat{\theta }}_{\text{C}WMAE}(2,z)={\hat{\theta }}_{\text{M}LE}(2,z)$. However, we note that alternative reasonable assumptions are possible and that ${\hat{\theta }}_{\text{C}WMAE}$ should be principally evaluated in terms of $Bias({\hat{\theta }}_{\text{C}WMAE}|\theta ,1)$ and $RMSE({\hat{\theta }}_{\text{C}WMAE}|\theta ,1)$.

2.2.7. Conditional median unbiased estimator

A conditional MUE (CMUE) was proposed by Zhong and Prentice³¹ and Koopmeiners et al.⁴ No explicit solution exists to its value in general, but it can be determined numerically as the solution to

$$0.5={\int }_{-\mathrm{\infty }}^z{f}_{\text{c}ond}\{k,x|{\hat{\theta }}_{\text{C}MUE}(k,z)\}\mathrm{d}x$$

In this instance, unlike for the MUE above, no ordering of the sample space is required.

2.2.8. Conditional UMVUE

The final estimator we consider is the conditional uniform minimum variance unbiased estimator (CUMVUE, sometimes called the uniform minimum variance conditionally unbiased estimator). That is, an estimator ${\hat{\theta }}_{\text{C}UMVUE}$ such that $\mathrm{Bias}({\hat{\theta }}_{\text{C}UMVUE}|\theta ,2)=0$ for all $\theta$. Such an estimator was originally considered by Cohen and Sackrowitz³² and has recently been extended by several authors (see, e.g., Kimani et al.³³ and Robertson et al.³⁴). It is calculated by applying Rao-Blackwell to the unbiased estimate of $\theta$ formed from the stage 2 data. Unfortunately, similar to the CWMAE, the CUMVUE does not come with an obvious implied definition of an estimate when the trial terminates at the interim analysis, which is required to compute the marginal bias and RMSE. Here, like Porcher and Desseaux,⁵ we assume that the MLE would be used, but note that other reasonable alternatives are possible. As we will see later, this is useful for comparing the CUMVUE to the CMLE in terms of the marginal bias and RMSE. With this, we prove in the Supplemental Materials that the CUMVUE can be written as

$${\hat{\theta }}_{\text{C}UMVUE}(k,z)=\{\begin{array}{cc}{\hat{\theta }}_{\text{M}LE}(k,z)& :k=1\\ \{z\sqrt{I_2}-I_1{\hat{\theta }}_{\text{U}MVUE}(k,z)\}/(I_2-I_1)& :k=2\end{array}$$

2.3. Code

Having described each of the nine estimators that will be compared, we next proceed to present evaluations of their performance in five indicative study examples. Code to reproduce these results exactly is available at https://github.com/mjg211/article_code. In addition, estimators can be compared within the setting of a two-stage group sequential trial with a parallel two-arm design assuming normally distributed outcome data via the GUI to the OptGS³⁵ package, available at https://mjgrayling.shinyapps.io/optgs/.

Note that in the majority of our examples, if the study is powered for $\theta =\delta >0$, then we evaluate estimator performance over the range $\theta \in [-2\delta ,2\delta ]$. Typically, $\delta$ corresponds to either a minimum clinically important difference or an a priori anticipated effect³⁶; either way, we view $\theta \in [-2\delta ,2\delta ]$ to then represent an extremely wide range of plausible effects, enabling patterns in estimator performance to become clear. In practice, when trying to choose a preferred estimator for a trial, clinical knowledge could be used to consider performance over a narrower range. We return to this point in the Discussion.

We highlight also that whilst each of our examples is given some clinical context to make them more tangible and practically useful, ultimately the performance of the estimators is only dependent on the clinical context through the specified values of $\theta$, $\delta$, $l$, $u$, $I_1$, and $I_2$.

3. Results

3.1. Log-rank test for survival data in a two-arm parallel-group trial

To begin, we consider an important context: A two-arm parallel-group individually randomised trial for time-to-event data under the proportional hazards assumption. This may correspond, for example, to an oncology trial in which the objective is to ascertain improved overall survival for some new treatment (indexed $E$, for experimental, in what follows) compared to the current standard of care (indexed $C$, for control, in what follows). We, therefore, phrase the language below as if it is such an oncology trial.

Firstly, it is assumed that the hazard rate at time $t$ for participants in arm $C$ is $h(t)$, and that it is $\lambda h(t)$ for participants in arm $E$. The log-rank test is a common approach to testing the null hypothesis that the hazard ratio is equal to one (i.e., $\lambda =1$). To explain this approach, following Jennison and Turnbull¹, let $d_k$ be the total number of uncensored deaths observed when analysis $k=1,2$ is conducted. Assuming no ties, denote the survival times of these participants by ${\tau }_{1k}<{\tau }_{2k}<\cdots <{\tau }_{d_{k}k}$, with the ${\tau }_{ik}$ denoting the elapsed time between entry to the study and death. Let also the number of participants at analysis $k$ known to have survived up to time ${\tau }_{ik}$ after treatment be $r_{iCk}$ and $r_{iEk}$ for arms $C$ and $E$, respectively. The log-rank score statistic at the stage $k$ analysis is then

$$S_k=\sum _{i=1}^{d_k}({\delta }_{iEk}-\frac{r_{iEk}}{r_{iCk}+r_{iEk}})$$

where ${\delta }_{iEk}=1$ if the death at time time ${\tau }_{ik}$ was on treatment $E$, and ${\delta }_{iEk}=0$ otherwise. It can be shown that

$$\widehat{Var}(S_k)=I_k=\sum _{i=1}^{d_k}\frac{r_{iCk}{r}_{iEk}}{(r_{iCk}+r_{iEk}{)}^2}$$

Next, set $\theta =-\log (\lambda )$, such that larger values of $\theta$ correspond to improved overall survival in arm $E$ compared to arm $C$. Then, for $\theta$ close to zero and sufficiently large $I_k$, the approximation $S_k\sim N(-\theta I_k,I_k)$ can be used. This means that if $Z_k=-S_k/\sqrt{I_k}$, $\{Z_1,Z_2\}$ has the canonical joint distribution with information levels $\{I_1,I_2\}$ for $\theta$.

To design such a trial, given the exact information levels $I_1$ and $I_2$ are not known pre-study, it is common to use the approximation $I_k\approx d_k/4$. We adopt this approach, and further suppose that the goal is to test the one-sided alternative $H_A:\theta >0$. We assume the desire is for a type-I error-rate of $\alpha =0.025$ and a type-II error-rate of $\beta =0.1$ when $\theta =\delta =-\log (0.8)\approx 0.182$ (i.e., to power for a hazard ratio of 0.8). We further suppose that the interim analysis will be conducted half-way through the maximal trial length in terms of the number of events observed, such that $I_2=2I_1$. We then focus on how the performance of the estimators changes when stopping for futility and efficacy is incorporated into the design ( $l,u\in \mathbb{R},u>l$), as opposed to stopping for futility only ( $l\in \mathbb{R},u=\mathrm{\infty }$). In both cases, we assume that $l=-0.674$. We choose this value because $\mathrm{\Phi }(l,0,1)\approx 0.25$ (i.e., to give a 25% chance of stopping for futility after stage 1 under $H_0$, which will in turn result in a low chance of incorrectly stopping for futility when $\theta =\delta$). In the case with efficacy stopping, $u=2.157$ is chosen, as this corresponds to the $\alpha$-spend when using Lan and DeMets error spending approach with their Pocock-like spending function.³⁷ The error spending approach can then be further used to show that to attain the overall desired error rates, the design with efficacy and futility stopping must have $I_1=117.5$ (i.e., the stage 1 analysis is performed after $d_1=470$ events). In contrast, the design with futility stopping only requires $I_1=105.75$ (i.e., $d_1=423$).

Figure 1 then shows the performance of the estimators for the design with efficacy and futility stopping, while Figure 2 gives the corresponding results for the design with futility stopping only. In Figure 1, the CMLE and CMUE arguably have the best performance conditional on termination after stage 1. They also perform well conditional on stage 2 termination, where the CUMVUE also provides effective performance. Though the marginal RMSE of the CMLE and CMUE is sometimes larger than the other estimators (e.g., for $\theta \notin [-\delta ,\delta ]$), we may believe that they represent the most effective options in this case (particularly since it may routinely be believed that a large negative treatment effect is unlikely, so performance for negative $\theta$ may be of little concern).

Figure 1.

The conditional and marginal biases and residual mean square error (RMSE) of the nine considered estimators is given for Example 1: two-arm survival data, with efficacy and futility stopping.

Figure 2.

The conditional and marginal biases and residual mean square error (RMSE) of the nine considered estimators is given for Example 1: two-arm survival data, with futility stopping only. The vertical axis limits have been constrained such that CMLE performance is not visible for all $\theta$; see Supplemental Figure S1 for a complete view.

However, in Figure 2, it can be seen that the marginal performance and performance conditional on termination after stage 1 of the CMLE and CMUE is extremely poor. We comment further on the CMLE’s performance in this setting in the subsequent example and in the Discussion. In the case of futility stopping only, the CUMVUE seems to be a substantially better option if one is willing to forgo a slightly larger absolute marginal bias. Alternatively, the MAEs, MAE1 and MAE2, may be preferred if the marginal bias of the CUMVUE is viewed to be too great. Contrasting the findings of Figures 1 and 2, it is thus clear that the performance of the estimators can be substantially impacted by the inclusion (or not) of efficacy stopping.

3.2. Single-arm phase II cancer trial with a normally distributed outcome

Wason et al.³⁸ discussed the use of continuous tumour shrinkage endpoints in two-stage phase II single-arm oncology trials, for the purposes of reducing requisite sample sizes. It was assumed that outcome $X_i$ from participant $i$ is distributed as $X_i\sim N(\mu ,{\sigma }^2)$, for known ${\sigma }^2$. Here, $X_i$ represents the percentage decrease in the sum of lesion diameters; positive values represent shrinkages in tumour size. The problem of testing $H_0:\theta =\mu =0$ against a one-sided alternative $H_A:\theta >0$ was then considered. Assuming data from $n_k$ patients is available at analysis $k$, setting

$$\begin{array}{rl}{Z}_k& =\frac{1}{\sqrt{n_k}{\sigma }^2}\sum _{i=1}^{n_k}{X}_i\\ I_k& =n_k/{\sigma }^2\end{array}$$

$k=1,2$, means $\{Z_1,Z_2\}$ has the canonical joint distribution with information levels $\{I_1,I_2\}$ for $\theta$. The restriction was made that early stopping is only permitted for futility (i.e., $u=\mathrm{\infty }$), to correspond to the typical approach in phase II single-arm oncology trials. Then, assuming a desired type-I error-rate of $\alpha =0.05$ and type-II error-rate of $\beta =0.1$ when $\theta =\delta =10$ (i.e., the target is 10% mean tumour shrinkage), Wason et al.³⁸ found the minimax design (i.e., the design that minimises $n_2$) to be $n_1=92$, $n_2=139$, and $l=0$, on the assumption that ${\sigma }^2=1600$.

Figure 3 displays the conditional and marginal biases and RMSE of the nine considered estimators in this design when $\theta \in [-2\delta ,2\delta ]=[-20,20]$. In this case, as in Figure 2, the CMLE is observed to have extremely large bias and RMSE, both marginally and conditionally, for certain values of $\theta$. The UMVUE has the worst bias and RMSE is conditional on stage 2 termination. The CMUE has strong conditional performance, but fares badly marginally. MAE1 and MAE2 reduce the marginal bias compared to the MLE, without compromising on conditional performance or marginal RMSE. Thus, for this example, we may prefer either MAE1 or MAE2 if we are unsatisfied with the UMVUE’s performance conditional on stage 2 termination.

Figure 3.

The conditional and marginal biases and residual mean square error (RMSE) of the nine considered estimators is given for Example 2: single-arm normally distributed data. The vertical axis limits have been constrained such that CMLE performance is not visible for all $\theta$; see Supplemental Figure S2 for a complete view.

3.3. Crossover trial

Jones and Kenward³⁹ considered sample size calculation for a $2\times 2\times 2$ placebo-controlled double-blind crossover trial to evaluate the efficacy of an inhaled drug given twice daily via an inhaler in patients with chronic obstructive pulmonary disease (see Sections 2.1 and 2.4 of Jones and Kenward³⁹). The primary outcome was the mean morning expiratory flow rate; outcome $X_{ijl}$ from patient $l=1,\dots ,n_i$, in period $j=1,2$, on sequence $i=1,2$, was assumed to be distributed as $X_{ijl}\sim N(\mu +{\pi }_j+{\tau }_{\text{d}(i,j)}+s_{il},{\sigma }_e^2)$. Here, ${\sigma }_e^2$ reflects the known within-patient variance, $\mu$ is an intercept term, ${\pi }_j$ is an effect associated with period $j$, ${\tau }_{\text{d}(i,j)}$ is a direct treatment effect associated with the treatment applied in period $j$ of sequence $i$, and $s_{il}$ is an effect associated with the $l$th subject on sequence $i$. Labelling the treatments such that $\text{d}(i,j)=A,B$, assuming that sequences $1$ and $2$ are $AB$ and $BA$ respectively, the problem of testing $H_0:\theta ={\tau }_B-{\tau }_A=0$ was considered. Suppose that at analysis $k$, $n_1=n_2=kn$. Then, setting

$$\begin{array}{rl}{Z}_k& =\frac{1}{2}\sqrt{\frac{kn}{{\sigma }_e^2}}\sum _{l=1}^{kn}(X_{1l}-X_{2l})\\ I_k& =kn/{\sigma }_e^2\\ X_{il}& =X_{i2l}-X_{i1l}\end{array}$$

it can be shown that $\{Z_1,Z_2\}$ has the canonical joint distribution with information levels $\{I_1,I_2\}$ for $\theta$^1,39,40. Here, we again assume a one-sided alternative $H_A:\theta >0$. Following Jones and Kenward³⁹, we assume that the desired type-I error-rate is $\alpha =0.05$ and type-II error-rate is $\beta =0.2$ when $\theta =\delta =10$, further supposing that ${\sigma }_e^2=326$. Then, if power-family stopping boundaries⁴¹ are used with shape parameter $-0.25$, the resulting two-stage design has $n=10.384$, $l=0.269$, and $u=2.730$.

Figure 4 displays the conditional and marginal biases and RMSE of the nine considered estimators in this design when $\theta \in [-2\delta ,2\delta ]=[-20,20]$. The results are overall similar to those given in Figure 1. The CMLE and CMUE perform well conditionally, but arguably not marginally, particularly in terms of their marginal RMSE. Here, it is clearer that the MUE seemingly trades an improvement in the conditional bias, conditional RMSE, and marginal RMSE, for a larger marginal bias. The CWMAE has particularly effective performance in terms of its marginal RMSE; whilst other estimators have lower marginal RMSE in the region around $\theta =9$, it maintains strong performance over a wider range of $\theta$. Given it also performs well in terms of its marginal bias and conditional bias/RMSE, it may be the best candidate for use in this case.

Figure 4.

The conditional and marginal biases and residual mean square error (RMSE) of the nine considered estimators is given for Example 3: crossover data.

3.4. Single-arm phase II cancer trial with a Bernoulli distributed outcome

Above, we have covered examples with time-to-event and normally distributed data. Here, we focus on a case with Bernoulli data. Schoffski et al.⁴² presented the results of a phase II single-arm oncology trial conducted to assess the activity of crizotinib in patients with advanced clear-cell sarcoma with MET alterations. Tumour response was used as the primary outcome and thus it was assumed that outcome $X_i$ from participant $i$ was distributed as $X_i\sim Bern(\pi )$, for response rate $\pi \in [0,1]$. Simon’s two-stage design was used, with the design specifying termination for futility after 12 patients if 1 or fewer participant exhibited a response (i.e., if $\sum _{i=1}^{12}{X}_i\le 1$). Otherwise, a further 23 participants were to be recruited. These parameters were chosen to detect an improvement from a 10% response rate to a 30% response rate, with type-I and type-II error-rates of 10%.

This design can be mapped to our setting as follows (see section 3.6 of Jennison and Turnbull¹ for further details). First, set $H_0:\theta =\pi -{\pi }_0=0$, where ${\pi }_0\in (0,1)$ is a specified clinically uninteresting response rate. Denote by $n_k$ the number of participant responses available at analysis $k$ and set

$$\begin{array}{rl}{Z}_k& =(\frac{1}{n_k}\sum _{i=1}^{n_k}{X}_i-{\pi }_0)\sqrt{I_k}\\ I_k& =\frac{n_k}{{\pi }_0(1-{\pi }_0)}\end{array}$$

Then, for $\pi$ close to ${\pi }_0$, $\{Z_1,Z_2\}$ approximately has the canonical joint distribution with information levels $\{I_1,I_2\}$ for $\theta$.

The design from Schoffski et al.⁴² corresponds to $u=\mathrm{\infty }$, with ${\pi }_0=0.1$, $n_1=12$, $n_2=35$, $H_A:\theta >0$, and

$$\begin{array}{rl}l& =(\frac{1}{12}-0.1)\sqrt{I_1}\approx -0.192\\ I_1& =\frac{12}{0.1\times 0.9}\approx 133.3\\ I_2& =\frac{35}{0.1\times 0.9}\approx 388.9\end{array}$$

Finally, the study was powered for $\theta =\delta =0.3-0.1=0.2$.

Figure 5 shows the nine estimators’ performance in this setting, for $\theta \in [-0.1,0.4]$, to correspond to $\pi \in [0,0.5]$, where the upper limit $0.5$ is chosen as a sufficiently large response rate that is unlikely to be observed in practice given the study was powered at $\pi =0.3$. Once more, we see that $u=\mathrm{\infty }$ results in extremely poor performance of the CMLE, and to a slightly lesser extent also the CMUE. Here, there is arguably only a direct choice to be made between either MAE1/MAE2 (if one is more concerned about marginal performance) or the MLE/CUMVUE (if one is more concerned about performance conditional on stage 2 termination). The UMVUE is, as always, optimal if one cares only about marginal bias. However, it fares particularly poorly conditionally in this case.

Figure 5.

The conditional and marginal biases and residual mean square error (RMSE) of the nine considered estimators is given for Example 4: single-arm Bernoulli distributed data. The vertical axis limits have been constrained such that CMLE performance is not visible for all $\theta$; see Supplemental Figure S3 for a complete view.

Note that we comment in the Discussion on the use of the canonical joint distribution framework for Bernoulli data.

3.5. Matched pairs trial

We conclude with an example with a two-sided alternative hypothesis. Jennison and Turnbull¹ discussed (see Sections 3.1.3 and 3.2.2 of their book) the sequential design of a matched pairs trial in which subjects are paired so that those in the same pair have similar values of important prognostic factors. Matched pairs designs are also commonly employed in paired-eye and paired-teeth trials, as well as twin studies. One subject in each pair is randomly allocated treatment A and the other receives treatment B; letting $X_{Ai}$ and $X_{Bi}$ denote the responses of the subject in pair $i$ who receive treatments A and B respectively, it was assumed that $X_{Ai}-X_{Bi}\sim N({\mu }_A-{\mu }_B,{\tilde{\sigma }}^2)$ with ${\tilde{\sigma }}^2$ known. The problem of testing $H_0:\theta ={\mu }_A-{\mu }_B=0$ against $H_A:\theta \ne 0$ was considered. If $n_k$ pairs of observations are available at analysis $k$, it was shown that

$$\begin{array}{rl}{Z}_k& =\frac{1}{\sqrt{n_k}{\tilde{\sigma }}^2}\sum _{i=1}^{n_k}(X_{Ai}-X_{Bi})\\ I_k& =n_k/{\tilde{\sigma }}^2\end{array}$$

$k=1,2$, results in $\{Z_1,Z_2\}$ having the canonical joint distribution with information levels $\{I_1,I_2\}$ for $\theta$. Assuming a desired type-I error-rate of $\alpha =0.05$ and type-II error-rate of $\beta =0.1$ when $\theta =\pm \delta =\pm 1$, the following information levels are required when utilising O’Brien-Fleming⁴³ stopping boundaries at equally spaced information levels $I_k=5.292k$. Thus, on the supposition that ${\tilde{\sigma }}^2=6$, $n_k=31.751k$, i.e., 31.751 pairs of observations are required per stage of the trial. Furthermore, the trial continues to stage 2 when $z_1\in (-2.796,2.796)$.

Figure 6 displays the conditional and marginal biases and RMSE of the nine considered estimators in this design when $\theta \in [-2\delta ,2\delta ]=[-2,2]$. It displays rotational symmetry about $\theta =0$ owing to the use of O’Brien-Fleming⁴³ boundaries. Observe that the UMVUE performs worst in terms of conditional bias and RMSE. The CMLE and CMUE arguably perform best in terms of $\mathrm{Bias}(\hat{\theta }|\theta ,1)$ and $\mathrm{RMSE}(\hat{\theta }|\theta ,1)$. Though, like in Figure 1, their additional strong performance conditional on stage 2 termination does not translate to effective marginal performance. The MAE estimators, as in all previous examples, have almost identical performance. The MAE and CWMAE estimators perform similarly conditional on interim termination, but the CWMAE performs slightly better conditional on progression to stage two, and slightly worse marginally. The CMLE and CUMVUE have almost identical values for $\mathrm{Bias}(\hat{\theta }|\theta ,2)$ and $\mathrm{RMSE}(\hat{\theta }|\theta ,2)$, but their different approaches after stage 1 termination lead to substantially different marginal bias and RMSE. In this example, it is likely either the CUMVUE should be preferred if one is primarily concerned about performance conditional on stage 2 termination, or one of the MAE estimators should be used if both marginal and conditional bias/RMSE are of sizeable concern.

Figure 6.

The conditional and marginal biases and residual mean square error (RMSE) of the nine considered estimators is given for Example 5: matched pairs data.

4. Discussion

In this paper, we have compared the performance of nine estimators for the principal parameter of interest after a two-stage group sequential trial, within the context of five example trials, evaluating their conditional and marginal biases and RMSE. Unfortunately, as is clear from Figures 1 to 6, there is no single estimator that performs best for the conditional and marginal biases and RMSE. However, a number of recommendations remain possible. Firstly, if one cares solely about the marginal bias, the UMVUE is naturally the optimal estimator. Secondly, if only bias on termination after the interim analysis is of concern, as it may be in the case where interim stopping is only allowed for futility, then the CUMVUE should likely be preferred. Both of these estimators are observed to perform poorly by some measures, though. In particular, as has been discussed previously, the UMVUE often has large conditional bias and RMSE.^13–15 In general, we would also caution against focusing solely on a single measure of estimator performance. For example, even on termination for futility after stage 1, effective estimation may still be important for decision-making about subsequent studies. Thirdly, the use of the CMLE (and to a lesser extent the CMUE) is not advisable in certain settings due to its large bias. This result about the CMLE was recently formally proved by Berckmoes et al.,⁴⁴ who demonstrated for the early stopping boundaries used in the second example ( $l=0$, $u=\mathrm{\infty }$) that the bias of the CMLE is unbounded. We elaborate on this further in the Supplemental Materials by giving an example of the conditional log-likelihood.

Where the conditional and marginal values of the bias and RMSE are all of importance, arguably the MAEs may be considered the best choice as they are typically amongst the better estimators on all six sub-panels of Figures 1 to 6. However, an additional potentially important consideration when choosing an estimator, beyond their bias and RMSE, is the predictability of the information levels. The MAE estimates of $\theta$ after stage 1 are dependent on the value of $I_2$, which may be undesirable if $I_2$ is unpredictable.¹ It was for this reason the stage-wise ordering was used to compute a MUE; this ensures that ${\hat{\theta }}_{\text{M}UE}(1,z)$ is independent of $I_2$. Such unpredictability in information levels may arise, for example, when the information levels are dependent on the unknown parameter of interest (e.g., for Bernoulli data in a randomised trial). In this setting, there is good reason to not utilise the MAEs. In contrast, in our opinion the computational complexity of the considered estimators is not a good justification for choosing a particular estimation procedure. Explicit formulae are given here for a number of estimators, and even those requiring numerical methods of determination are dependent only on a one-dimensional search which can be readily performed.

Overall, given the performance of the estimators is highly dependent on the underlying design, we would thus always recommend evaluations be performed to help choose the estimators for any proposed group sequential design. When this is not possible, our simple recommendation would be to utilise one of the MAE estimators in the case that information levels are predictable. When the information levels are not predictable, the CWMAE may be a fallback choice, owing to its effective performance across the sub-panels in the figures given here. Such recommendations, however, are based on surveying performance across a wide range of values of $\theta$. In practice, it would be logical to use available clinical knowledge to help better target the range of $\theta$ used to determine the preferred estimator. We focused on performance for $\theta \in [-2\delta ,2\delta ]$, where $\delta$ is the target difference at which the study is powered. In some trials, it may be the case that the treatments under investigation make a negative value of $\theta$ unlikely. In addition, $\theta$ close to twice the target difference would be a rare find. The region $\theta \in [0,\delta ]$ may be an effective narrower range to consider in general; when this is the case, our primary recommendations around the use of the MAE estimators or CWMAE above do not change. However, it may be more common to find the conditional performance of the UMVUE palatable; in Figure 2, for example, considering only $\theta \in [0,\delta ]=[0,10]$, the UMVUE may be far more easily argued to be the best choice.

We acknowledge some limitations to our work. Firstly, though we have considered a large number of possible estimators, others have been proposed in the literature. In particular, Wang and Leung⁴⁵ proposed the use of parametric-bootstrapping procedures to attain a bias-adjusted estimator. We omit consideration of it here due to the computational complexity involved in evaluating their approach across numerous values of $\theta$ in several trial examples. Additionally, Zhong and Prentice³¹ proposed an estimator that is a weighted combination of the stage-wise MLEs. We omit it here as the stage two estimate is dependent on both $z_1$ and $z$, not simply $z$ as for all the estimators considered in our work. We also considered only two-stage group sequential designs; it cannot be guaranteed that our recommendations above would hold true for designs with a larger number of stages. Furthermore, we are unavoidably only able to compare performance of the estimators across a small number of the infinite possible trial scenarios. We note again that the best estimator for a given trial will be (as discussed) design dependent; thus all conclusions come with the caveat that they should be investigated further for any specific parameters of interest. Our available R code, and implementation in a GUI for parallel arm trials with normal outcome data, may assist heavily with this task.

A reviewer raised an interesting comment in regard to the trial scenarios considered, about how point estimation may be affected by the choice of stopping boundaries (e.g., O’Brien-Fleming, Pocock). In general, it is reasonable to anticipate that more ‘aggressive’ stopping boundaries (i.e., boundaries that increase the probability of termination after stage 1) will result in reduced values of $\mathrm{Bias}(\hat{\theta }|\theta ,1)$ and $\mathrm{RMSE}(\hat{\theta }|\theta ,1)$, but increased values of $\mathrm{Bias}(\hat{\theta }|\theta ,2)$ and $\mathrm{RMSE}(\hat{\theta }|\theta ,2)$, everything else being equal. The effect on marginal performance is more challenging to predict; for some $\theta$ it will improve, for others it will be worse. We illustrate an example of this in the Supplemental Materials, providing results for the fifth example on matched pairs data, but where the stopping boundaries are Pocock rather than O’Brien-Fleming. In general, we believe it is likely that the impact of the stopping boundaries on the study’s maximal and expected sample sizes will be the principal driver of the choice of bounds, as opposed to their effect on inference. However, when a choice must be made between candidate group sequential designs that are otherwise considered to have similar advantages/disadvantages, the consequences for inference may be a useful additional consideration. Bowden and Wason⁴⁶ have considered this in the context of exact single-arm phase II oncology trial designs for Bernoulli data, determining optimised combinations of design and analysis procedure.

We also end with caution against the use of the canonical joint distribution and the estimators given here in some settings. In particular, though we included an example with Bernoulli data to illustrate how the methodology can be applied in that setting, for such studies designs leveraging exact binomial densities and associated estimators should likely be preferred for all but the largest trial sample sizes (see, e.g., Porcher and Desseaux⁵ for the single-arm case, while Bibbona and Rubba⁴⁷ provide relevant results for other designs). In addition, for time-to-event data in small-sample settings, direct simulation of study data and evaluation of estimator performance in this way (rather than using the canonical approximation) would be advisable.

In conclusion, the best estimator for a given trial is dependent on the estimator’s relative performance for, and the relative desire to minimise each of, the conditional and marginal biases and RMSE. Evaluating the performance of each of the estimators in a given trial design scenario can be efficiently completed. Undertaking this task in practice will enable investigators to make a more effective choice on how to estimate their parameter of interest.⁷ To date, few group sequential trials have computed adjusted estimates,^7,8 which may be particularly problematic for their subsequent inclusion in meta-analyses. It may also negatively impact decision-making around whether a treatment should be further developed. We encourage their increased use in future studies.