Point estimation for adaptive trial designs I: A methodological review

Abstract

Recent FDA guidance on adaptive clinical trial designs defines bias as “a systematic tendency for the estimate of treatment effect to deviate from its true value,” and states that it is desirable to obtain and report estimates of treatment effects that reduce or remove this bias. The conventional end‐of‐trial point estimates of the treatment effects are prone to bias in many adaptive designs, because they do not take into account the potential and realized trial adaptations. While much of the methodological developments on adaptive designs have tended to focus on control of type I error rates and power considerations, in contrast the question of biased estimation has received relatively less attention. This article is the first in a two‐part series that studies the issue of potential bias in point estimation for adaptive trials. Part I provides a comprehensive review of the methods to remove or reduce the potential bias in point estimation of treatment effects for adaptive designs, while part II illustrates how to implement these in practice and proposes a set of guidelines for trial statisticians. The methods reviewed in this article can be broadly classified into unbiased and bias‐reduced estimation, and we also provide a classification of estimators by the type of adaptive design. We compare the proposed methods, highlight available software and code, and discuss potential methodological gaps in the literature.

1. INTRODUCTION

Adaptive clinical trials allow for preplanned opportunities to alter the course of the trial on the basis of accruing information. ¹ , ² , ³ This may include changes such as increasing the recruitment target (in sample size re‐estimation designs), selecting the most promising treatment arms (in multi‐arm multi‐stage designs) or patient subpopulations (in population enrichment designs), shifting the randomization ratio toward more promising arms (in adaptive randomization designs), or terminating recruitment early for clear evidence of benefit or lack thereof (in group sequential designs). ⁴ Despite some additional complexities when implementing such trials, ⁵ they are increasingly being used in practice due to their attractive features, adding flexibility to a trial design while maintaining scientific rigour. ⁶ , ⁷ , ⁸ The challenges of the COVID‐19 pandemic have also recently accelerated their use. ⁹ , ¹⁰

To date most of the research undertaken has focused on the design of such studies and the associated question of maintaining desirable operating characteristics related to hypothesis testing (namely, type I error and power) rather than estimation. General methods on the basis of P‐value combination ¹¹ , ¹² and conditional error functions ¹³ , ¹⁴ have been proposed that are applicable to a wide range of adaptive designs as well as specialized methods for specific designs such as multi‐arm multi‐stage (MAMS) designs, ¹⁵ , ¹⁶ adaptive enrichment designs, ¹⁷ , ¹⁸ response‐adaptive randomization (RAR) designs, ¹⁹ , ²⁰ and sample size re‐estimation ²¹ , ²² (see Section 1.1 for definitions of the different adaptive designs considered in this article).

In contrast, the question of estimation of treatment effects in an adaptive clinical trial has received comparatively less attention, as reflected in the recent FDA guidance on adaptive designs ²³ which states “Biased estimation in adaptive design is currently a less well‐studied phenomenon than Type I error probability inflation.” This article is the first in a two‐part series that studies the issue of potential bias in point estimation for adaptive designs. In the current article (part I), we review methods for unbiased and bias‐reduced estimation of treatment effects after an adaptive clinical trial and critically discuss different approaches. In part II, we consider point estimation for adaptive designs from a practical perspective, including a set of guidelines for best practice. ²⁴

While equally important, the construction of related quantities for inference, such as confidence intervals or regions, is beyond the scope of this series so we signpost the interested reader to related literature. ²⁵ , ²⁶ Moreover, our focus is on bias in the statistical sense, and we do not consider issues such as operational bias in adaptive designs ² , ³ , ²⁷ or the impact of publication bias ²⁸ ; see also recent reference articles ²⁹ , ³⁰ for discussion of other types of biases in randomized controlled trials.

The structure of this article is as follows. After defining different types of adaptive designs in Section 1.1, we introduce key concepts and the different definitions of bias in Section 2. We describe the search strategy used for a systematic review of the methodological literature in Section 3. We present a summary of the results of the review grouped by the type of adaptive design in Section 4. Sections 5 and 6 then provide detailed descriptions and explanations of the different types of point estimators, divided into methods for unbiased estimation (which aims to completely remove bias) in Section 5 and bias‐reduced estimation (which aims to reduce the magnitude of the bias, but not necessarily to eliminate it) in Section 6. In the Supplementary Information, we provide an annotated bibliography giving further details of all the papers included in the systematic review. The overall conclusions are discussed in Section 7.

1.1. Glossary of adaptive designs

Table 1 gives definitions of the different types of adaptive designs considered in this article, including the different terminologies used for them. These definitions are based on those provided by Pallmann et al ¹ and Burnett et al, ⁴ who note that there can be ambiguous terminology used in the literature. Adaptive designs can combine multiple adaptive features within a single trial, see Dimairo et al ² for examples.

TABLE 1.

Definitions of the different types of adaptive designs considered in this article

Design	a.k.a.	Definition
Group sequential	Multi‐stage design	Allows for early stopping of the trial for safety, futility, or efficacy
		Group sequential designs can either be two‐arm or single‐arm trials, with Simon's two‐stage design being a common example of the latter. For the generalization to multiple treatment arms, see “Multi‐arm multi‐stage designs”
		When combined with the option of sample size re‐estimation and/or adjustment of the stopping boundaries during the trial, these designs are sometimes referred to as “adaptive group sequential”
Sample size re‐estimation	Sample size reassessment/recalculation/adaptations	Adjustment of the sample size, in either a blinded or unblinded fashion. This is typically done to help ensure the trial achieves the desired power
		Sample size re‐estimation is sometimes combined with other trial adaptations, such as group sequential testing
Multi‐arm multi‐stage (MAMS)	Pick‐the‐winner/drop‐the‐loser/treatment selection	Compares multiple treatments, doses, durations or combinations to a common control, allowing for early stopping of treatment arms for efficacy or futility. In addition, this can include interim treatment selection of the most promising treatment(s) for further investigation in the subsequent stages
	Variations of MAMS designs include “drop‐the‐loser” trials, where inferior treatment arms are dropped from the trial (with the control group typically retained). “Seamless” trials (such as phase II/III trials) are also closely related, where the selection and confirmatory phases of development are combined into one trial. “Platform” trials can also be viewed as an extension of MAMS designs, by allowing new treatment arms to be added to the trial over time
Response‐adaptive randomization (RAR)	Outcome‐adaptive randomization	Shifts allocation ratios toward more promising or informative treatment(s) based on the accumulating patient response data. RAR is sometimes combined with options for early stopping (eg, for futility or efficacy)
Adaptive enrichment	Population enrichment/patient enrichment/adaptive population enrichment	Allows for selection of a target population during the trial, typically using predefined patient subpopulations based on biomarker information. For example, one class of adaptive enrichment designs allows the eligibility criteria to be adaptively updated during the trial based on the estimated treatment‐biomarker interactions, restricting entry in the subsequent trial stage(s) to patients most likely to benefit from the new treatment

2. ESTIMATION BIAS IN ADAPTIVE DESIGNS

The issue with estimation after an adaptive trial is that traditional maximum likelihood (ML) estimators tend to be biased either because of some selection that took place following an interim analysis (see Bauer et al ³¹ for a detailed explanation of why selection results in bias) or other mechanisms utilized in an adaptive design, such as early stopping, which might affect the sampling distribution of the estimator (these depend on the nature of the design). For this reason, the usual ML estimator (MLE) is sometimes referred to as the “naive” estimator for the trial.

Before considering the issue of estimation further it is worth clarifying what we mean by a biased estimator. The FDA guidance on adaptive designs ²³ defines bias as “a systematic tendency for the estimate of treatment effect to deviate from its true value,” and states that “It is important that clinical trials produce sufficiently reliable treatment effect estimates to facilitate an evaluation of benefit‐risk and to appropriately label new drugs, enabling the practice of evidence‐based medicine.” It is clear that (all else being equal) it is desirable to obtain estimators of treatment effects that are unbiased in order to make reliable conclusions about the effects of study treatments.

While it is relatively easy to define statistical bias, different definitions of an unbiased estimator are relevant in our context. To introduce these, let us denote the population parameter of interest, the treatment effect, by θ and an estimator thereof by $\widehat{\theta }$.

2.1. Mean‐unbiased estimators

An estimator $\widehat{\theta }$ is called mean‐unbiased if its expected value is the same as the true value of the parameter of interest, that is, E($\widehat{\theta }$) = θ. This is the most commonly used definition of unbiasedness.

2.2. Median‐unbiased estimators

An estimator $\widehat{\theta }$ is called median‐unbiased if P($\widehat{\theta }$ < θ) = P($\widehat{\theta }$ > θ), that is if the probability of overestimation is the same as the probability of underestimation. Note that for symmetric sampling distributions of $\widehat{\theta }$, a median‐unbiased estimator is also mean‐unbiased.

2.3. Conditionally and unconditionally unbiased estimators

A further distinction of unbiasedness in estimators refers to whether they are conditionally or unconditionally unbiased. In our context, an estimator is unconditionally unbiased (also known as marginally unbiased), if it is unbiased when averaged across all possible realizations of an adaptive trial. In contrast, an estimator is conditionally unbiased if it is unbiased only conditional on the occurrence of a subset of trial realizations. For example, one might be interested in an estimator only conditional on a particular arm being selected at an interim analysis; as such, the focus becomes on a conditional unbiased estimator. We discuss the issue of conditional vs unconditional bias further throughout the rest of this article.

2.4. Bias and mean squared error

When considering estimation after an adaptive trial one is often faced with the following conundrum: precise estimators can potentially be biased while unbiased estimators tend to be less precise, which reflects the classical bias‐variance trade‐off. This means that in many instances the mean squared error (MSE), a measure of precision defined as E(${(\widehat{\theta }-\theta )}^2$), of a biased estimator is often smaller than the MSE of an unbiased estimator. This makes it challenging to find the “best” estimator that fits a particular trial design.

2.5. The focus of our review

Since bias as defined above is an expectation (or probability) taken over possible realizations of data, it is inherently a frequentist concept. However, we can still evaluate the frequentist bias of a Bayesian point estimator, such as the posterior mean. Similarly, it is possible to use frequentist point estimators in the analysis of Bayesian trial designs (ie, where the adaptations and decision rules are driven by Bayesian analyses). Hence we consider both Bayesian point estimators and Bayesian adaptive designs in this article. However, we restrict our attention to phase II and III trial designs, since phase III trial results have a direct impact on health policy and the adoption of new treatments for wider public use, while phase II trial results directly influence whether further research of a treatment in phase III is required.

3. METHODOLOGICAL REVIEW: SEARCH STRATEGY AND PAPER SELECTION

We conducted a database search of Scopus on July 13, 2022 of all available papers up to that date. We used a “title, abstract, keywords” search, with the predefined search terms for different categories of estimator given in Table 2.

TABLE 2.

Search strategy used for the initial database search of Scopus

Category of estimator	Search terms
Unbiased	((((unbiased OR ((conditionally OR mean OR median OR “uniformly minimum variance” OR “uniformly minimum variance conditionally”) AND unbiased)) AND estim*) OR “median adjusted” OR umvcue OR umvue) AND (“adaptive design” OR “adaptive trial” OR “adaptive clinical trial” OR “group‐sequential”))
Bias‐reduced	(((“bias‐reduced” OR “bias‐adjusted” OR “conditional moment” OR “empirical Bayes” OR shrinkage) AND estimat*) OR ((corrected OR adjusted OR conditional) AND (mle OR “maximum likelihood”))) AND (“adaptive design” OR “adaptive trial” OR “adaptive clinical trial” OR “group‐sequential”)
Resampling‐based methods	(((bootstrap OR nonparametric OR non‐parametric OR resampling OR “distribution free” OR distribution‐free OR jackknife OR “Monte Carlo”) AND estim*) AND (“adaptive design” OR “adaptive trial” OR “adaptive clinical trial” OR “group‐sequential”))

We excluded papers not involving adaptive clinical trial designs, as well as those that did not propose or evaluate unbiased or bias‐reduced estimators. Two authors (DSR and PP) separately reviewed the papers, extracted data and checked each other's work for accuracy.

Our search strategy retrieved a total of 257 papers, of which 148 were excluded immediately as they were not relevant based on the title and abstract. We then looked for additional relevant papers citing or cited by these remaining 109, which added 59 papers. After completing a full text review of these papers, a total of 145 were deemed relevant, and information about the trial contexts, advantages, limitations, code availability and case studies was extracted for qualitative synthesis (Figure 1). Full results giving a summary of each paper extracted for qualitative analysis are found in the supporting materials.

FIGURE 1.

Diagram showing the flow of information through the different phases of the systematic review. The Bayesian and empirical Bayes methods are included in the “bias‐reduced” category

4. COMPARISONS OF ESTIMATORS, TRIAL EXAMPLES AND SOFTWARE

In this section, we summarize the results of our systematic review in Table 3 by classifying them according to the broad class of adaptive design used. For each design class, we give references to the relevant literature for the different types of estimators that have been proposed. Where the literature focuses on designs that allow for multiple adaptive features within a single trial, we have used our judgment to place them into a class of adaptive designs, and refer the reader to Sections 5 and 6 for further clarifying details. In Table 3, we also give a summary of general pros and cons of the different estimators and the comparisons between them as given in the literature. Finally, we point out examples of the use of different estimators on trial data, as well as where software/code is available. Note that this list of software/code is by no means exhaustive, but only includes those that were specifically mentioned in the reviewed methodological literature or those of which the authors were already aware.

TABLE 3.

Summary of unbiased and bias‐reduced estimators, trial examples, and software/code for different classes of adaptive designs

Design	Method(s)	Pros and cons	Trial examples and software/code
Group sequential	Mean‐unbiased estimation
	Chang et al, 32 Kim, 33 Emerson and Fleming, 34 Emerson and Kittelson, 35 Liu and Hall, 36 Jung and Kim, 37 Liu et al, 38 , 39 Porcher and Desseaux, 40 Zhao et al 41 Secondary parameters: Gorfine, 42 Liu and Hall, 43 Liu et al, 44 Kunz and Kieser 45 Diagnostic tests: Shu et al, 46 Pepe et al 47	The UMVUE has zero bias, but tends to have a higher MSE than the naive or bias‐adjusted estimators Computation can be complex and extensive if the number of looks is relatively large (≥4), but otherwise can be simpler than for bias‐adjusted estimators UMVUE can be conditionally biased	Beta‐Blocker Heart Attack Trial, see Gorfine 42 Phase II trial in patients with adenocarcinoma, see Kunz and Kieser 45 GI06‐101 trial in hepatobiliary cancer, see Zhao et al 41 Software/code: OptGS R package
	Median‐unbiased estimation
	Kim, 33 , 48 Emerson and Fleming, 34 Todd et al, 49 Troendle and Yu, 50 Hall and Liu, 51 Koyama and Chen, 52 Wittes, 53 Porcher and Desseaux, 40 Shimura et al 54 Secondary parameters: Hall and Yakir 55 Adaptive group sequential trials: Wassmer, 56 Brannath et al, 57 Gao et al, 58 , 59 Levin et al, 60 Mehta et al, 61 Nelson et al 62	MUE reduces bias compared to the naive estimator, but can have an increased MSE. Bias‐adjusted estimators can have a lower MSE as well Calculation of the MUE can be complicated, and results can depend on the ordering of the sample space MUEs can be derived for adaptive group sequential designs, unlike for other estimation methods	Trials for acute bronchitis reported in Wassmer et al 63 Multicenter Automatic Defibrillator Implantation Trial (MADIT), see Hall and Liu, 51 Hall and Yakir 55 Non‐small cell lung cancer trial, see Wassmer 56 Randomized Aldactone Evaluation Study (RALES), see Wittes 53 Trial reported in Troger et al 64 Clinical Evaluation of Pertuzumab and Trastuzumab (CLEOPATRA) trial, see Shimura et al 54 Software/code: ADDPLAN59 Note: Also implemented in software such as SAS, East, as well as R packages such as rpact, RCTdesign, AGSDest, and OptGS. See https://panda.shef.ac.uk/techniques/group‐sequential‐design‐gsd/categories/27
	Resampling
	Parametric: Pinheiro and DeMets, 65 Wang and Leung, 66 Leung and Wang, 67 Cheng and Shen, 68 Magnusson and Turnbull 17 Nonparametric: Leblanc and Crowley 69	Essentially the same procedure can be used under different stopping rules and different study designs Bootstrap algorithms can be computationally intensive Bias is substantially reduced, with reasonable MSE Nonparametric approaches are robust to model misspecification	Trial for nasopharyngeal cancer, see Leblanc and Crowley 69
	Bias‐reduced
	Whitehead, 70 Chang et al, 32 Tan and Xiong, 71 Todd et al, 49 Li and DeMets, 72 Fan et al, 73 Liu et al, 74 Guo and Liu, 75 Porcher and Desseaux, 40 Shimura et al, 76 Li 77 Adaptive group sequential trials: Levin et al 60 Conditional: Troendle and Yu, 50 Coburger and Wassmer 78 Secondary parameters: Whitehead, 79 Liu et al, 44 , 74 Yu et al 80 Health economic outcomes: Flight 81	The MSE of the bias‐adjusted MLE is typically lower than that of the UMVUE, particularly for small sample sizes Shrinkage‐type estimators (which also have a Bayesian interpretation) can reduce both the conditional bias and MSE further	Three different phase II studies, see Tan and Xiong 71 Trial of immunosuppression for bone marrow transplantation, see Whitehead 79 Two cardiovascular trials (MERIT‐HF and COPERNICUS), see Fan et al 73 Trial in familial adenomatous polyposis, see Liu et al 74 Phase II trial in endometrial cancer, see Shimura et al 76 GUSTO trial for myocardial infarctions, see Marschner and Schou 82 Software/code: RCTdesign R package, OptGS R package 60
	Bayesian
	Hughes and Pocock, 83 Pocock and Hughes 84	Useful for producing shrinkage of unexpectedly large/imprecise observed treatment effects that arise in trials that stop early Bias reduction depends on the specification of the prior distribution.	None
Sample size re‐estimation	Mean‐unbiased estimation
	Unconditional: Liu et al, 85 , 86 Kunzmann and Kieser 87 Conditional (UMVCUE): Kunzmann and Kieser, 87 Broberg and Miller 88	Estimators have zero bias, either unconditionally or conditionally. However, the MSE tends to be greater than for the naive estimator and bias‐reduced estimators Does not guarantee compatibility with the test decision (see Kunzmann and Kieser 87 ) Estimators can have an explicit representation making computation easy	Schizophrenia trial, see Broberg and Miller 88
	Median‐unbiased estimation
	Lawrence and Hung, 89 Liu et al, 85 Wang et al, 90 Liu et al, 86 Kunzmann and Kieser, 87 Nhacolo and Brannath 91 Conditional perspective: Broberg and Miller 88 Flexible sample size adaptations: Bauer et al, 92 Liu and Chi, 93 Brannath et al, 94 Lawrence and Hung, 89 Proschan et al, 95 Brannath et al 96	MUE tends to have small mean bias, and can also have smaller MSE than the naive estimator Does not guarantee compatibility with the test decision MUE can be calculated for flexible adaptation rules that are not completely prespecified in advance, unlike for other estimation methods	Coronary artery disease trial, see Wang et al 90 Trial on reperfusion therapy for acute myocardial infarction, see Brannath et al 96 Software/code: R packages such as rpact and adpss
	Bias‐reduced
	Denne, 97 Coburger and Wassmer, 98 Cheng and Shen, 99 Shen and Cheng, 100 Liu et al, 85 Tremmel, 101 Broberg and Miller 88	Proposed bias‐reduced estimates are nearly unbiased with practical sample sizes, with similar variance to the naive estimator Numerical problems can occur when calculating adjusted estimators, and observations close to the critical boundaries can lead to unreasonably extreme adjusted estimators	Type II Coronary Intervention Study, see Denne 97 Colon cancer trial, see Shen and Cheng 100 Trial in chronic lymphocytic leukemia, see Tremmel 101 Schizophrenia trial, see Broberg and Miller 88
	Bayesian
	Kunzmann and Kieser, 87 Grayling and Mander 102	Guarantees compatibility with the test decision Reduces MSE of MLE except for very small or very large values of the success probability Reduces absolute bias compared with MLE except for small values of the success probability, where there can be a substantial positive bias Can reduce MSE substantially compared to the UMVUE for certain response rates	None
Multi‐arm multi‐stage designs (with treatment selection)	Mean‐unbiased estimation
	Two‐stage designs: Cohen and Sackrowitz, 103 Tappin, 104 Bowden and Glimm, 105 , 106 Pepe et al, 47 Koopmeiners et al, 107 Robertson et al, 108 , 109 Robertson and Glimm 110 Multi‐stage designs: Bowden and Glimm, 106 Stallard and Kimani 111 Seamless phase II/III trials: Kimani et al, 112 Robertson et al 109	UMVCUEs are conditionally unbiased. Compared to the MLE, the conditional MSE tends to be lower, but unconditionally the MSE can substantially increase UMVCUEs in the literature tend to have a closed‐form expression, allowing for easy computation In some settings, the UMVCUE can have comparable MSE to bias‐adjusted estimators	Trial for the treatment of anxiety disorder, see Kimani et al 112 and Robertson et al 109 INHANCE study, see Robertson and Glimm 110 ADVENT trial, see Stallard and Kimani 111 PROVE trial 113 implements approach of Stallard and Kimani 111 Software/code: Bowden and Glimm 106
	Resampling
	Pickard and Chang, 114 Whitehead et al 115	Provides a reasonable balance between bias and MSE across several scenarios Approach can be applied to endpoints coming from a variety of distributions (including normal and binomial) Approaches are robust to model misspecification	Software/code: Whitehead et al 115
	Bias‐reduced
	Two‐stage designs: Coad, 116 Shen, 117 Stallard et al, 118 Pepe et al, 47 Luo et al, 119 , 120 Bebu et al, 121 , 122 Koopmeiners et al, 107 Brückner et al 123 Multi‐stage designs: Coad, 116 Stallard and Todd, 124 Bowden and Glimm 106 Seamless phase II/III trials: Kimani et al 112	Bias‐adjusted MLE can have relatively low MSE and acceptably small bias in some scenarios Shrinkage methods can be the most effective in reducing the MSE Bias‐reduced estimators can run into computational/convergence problems Estimators can overcorrect for bias	Phase II study in colorectal cancer, see Luo et al 119 Software/code: Luo et al 119 FOCUS trial in advanced colorectal cancer, see Brückner et al 123 Phase III trial in Alzheimer's, see Stallard and Todd 124 Software/code: Bowden and Glimm 106 Software/code: Kimani et al 112
	Bayesian
	Two‐stage designs: Carreras and Brannath, 125 Bowden et al, 126 Brückner et al 123 Multi‐stage designs: Bunouf and Lecoutre 127	Shrinkage estimators can perform favorably compared with the MLE in terms of bias and MSE Approaches can run into practical and theoretical complications, necessitating further modifications such as estimating a between‐arm heterogeneity parameter	FOCUS trial in advanced colorectal cancer, see Brückner et al 123
Response‐adaptive randomization	Mean‐unbiased estimation
	Bowden and Trippa 128	Mean‐unbiased estimators can have a large MSE and can be very computationally intensive to calculate	Glioblastoma trial with multiple treatments
	Bias‐reduced
	Coad, 129 Morgan, 130 Marschner 131	The conditional MLE can be very effective at eliminating the conditional bias that is present in the unconditional MLE, but this comes at the cost of a loss of efficiency except for more extreme designs The penalized MLE exhibits very little conditional bias and is not subject to substantial efficiency loss (compared to the unconditional MLE) when the realized design is close to its average	None
Adaptive enrichment designs	Mean‐unbiased estimation
	Two‐stage designs: Kimani et al, 132 , 133 , 134 Kunzmann et al, 135 Di Stefano et al 136	UMVCUE is unbiased, but tends to have a higher MSE than the MLE UMVCUE can also have a larger MSE than shrinkage/bias‐adjusted estimators, but compensates by the corresponding eradication of bias UMVCUEs are easily computable, as they have a closed form expression	MILLY phase II study in asthma, see Kunzmann et al 135 Software/code: Kimani et al 134
	Bias‐reduced
	Two‐stage designs: Kunzmann et al, 135 Kimani et al, 133 Di Stefano et al 136	Can have lower MSE than UMVCUE that is comparable to the MSE of the MLE, but has residual bias	MILLY phase II study in asthma, see Kunzmann et al 135
	Resampling
	Magnusson and Turnbull, 17 Kunzmann et al, 135 Simon and Simon 137	Bootstrap estimator can have a higher bias than the MLE in the two‐stage setting of Kunzmann et al, 135 but Simon and Simon 137 found it was very effective at correcting for bias in their multimarker setting Procedures can be computationally intensive	MILLY phase II study in asthma, see Kunzmann et al 135
	Bayesian
	Two‐stage designs: Kunzmann et al, 135 Kimani et al, 133 Di Stefano et al 136	Estimators exhibit a higher bias than the MLE in many situations, with varying MSE properties	MILLY phase II study in asthma, see Kunzmann et al 135

Abbreviations: MLE, maximum likelihood estimator; MSE, mean squared error; MUE, median unbiased estimator; UMVCUE, uniformly minimum variance conditionally unbiased estimator; UMVUE, uniformly minimum variance unbiased estimator.

The systematic review identified mean‐unbiased estimators, median unbiased estimators, resampling‐based estimators and bias‐reduced estimators. The bias‐reduced estimators include those that estimate bias and subtract it from the MLE and those that use shrinkage methods. Detailed descriptions and explanations of the different types of point estimators referenced in Table 3 are given in the following two sections, with Section 5 focusing on unbiased estimation (both mean‐unbiased and median‐unbiased) and Section 6 focusing on bias‐reduced estimation (including Bayesian approaches and resampling‐based methods).

Looking at the results as a whole, we can see that the literature on unbiased and bias‐adjusted estimation for adaptive designs has grown rapidly in recent years, reflecting the increased use of adaptive designs in practice. However, only a few of these methodological papers present real‐life trial examples (such as reanalyzes of trials that have already been completed), and even fewer provide software/code. In terms of the different classes of adaptive designs, group sequential designs have received the most attention by far, and some of the estimation methods are implemented in widely available statistical software such as R or SAS. This reflects the relatively long history of group sequential designs and their more widespread use in practice. In contrast, we found very little literature on estimation for trials using RAR. With the increasing use of such trials (see, eg, the ISPY‐2 ¹³⁸ and BATTLE ¹³⁹ trials), this may be an important gap in the literature to fill.

Another potential gap is that some of the methods are restricted to two‐stage designs (especially for adaptive enrichment trials), and so a natural extension would be to develop these methods for multi‐stage trials. As well, most of the methodology has focused on binary and normally distributed outcomes, with comparatively few proposals tailored for trials with time to event outcomes. It is also unclear to what extent existing methodology can apply to longitudinal outcomes (which may be particularly common for interim analyses) or to cluster randomized trials.

In terms of different types of estimators, generally speaking there is a bias‐variance tradeoff (see the start of Section 6) unbiased estimators may pay the “price” for complete unbiasedness by having a high variance. Conversely, bias‐reduced estimators may have a lower variance than unbiased estimators, but may “overcorrect” and thereby introduce bias in the opposite direction. Another general observation is that mean‐unbiased estimators in the literature require prespecified boundaries and decision rules, and so are not applicable to flexible adaptive designs with arbitrary changes. Finally, resampling‐based methods have received comparatively little attention as opposed to other types of bias‐reduced estimation. For example, as far as we could tell, there has not been work looking explicitly at their use for sample size re‐estimation designs. Given that resampling‐based methods can be applied even to complex trial designs (see, eg, Whitehead et al ¹¹⁵ ), their use could potentially provide one solution to the problem of estimation for trials which combine different types of adaptations together and hence will not fit neatly into the classes of adaptive designs used above.

5. UNBIASED ESTIMATION

5.1. Mean‐unbiased estimation

In order to achieve exact mean‐unbiasedness, typically it is necessary to find an estimator whose distribution is independent of the trial adaptations. One key setting where this is possible is in classical group sequential trials. Given that there is a prespecified number of observations at the time of the first interim analysis, then the sample mean (which corresponds to the MLE) at the end of the first stage is (unconditionally) unbiased as no adaptations to the trial have occurred. However, this estimator is also clearly inefficient for estimating the overall treatment effect, as it does not use any information that may arise from later stages of the trial.

Another setting where unbiased estimation is possible is multi‐stage trials with ranking and selection from a subset of candidates, such as candidate treatments or patient subgroups. The sample mean (MLE) for the selected candidate(s) calculated using only the final stage data will be conditionally unbiased, conditional on the ranking and selection that has taken place in the previous stages. However, again this estimator is clearly inefficient as it ignores all the information about the selected candidate(s) from the previous stages of the trial. Moreover, unlike the first stage estimator described in the previous paragraph, it cannot be used to estimate the effect of a treatment or subpopulation that was dropped at an interim analysis.

5.1.1. Minimum variance unbiased estimation

One way to obtain a more efficient estimator that uses more of the information from the trial, while still maintaining exact mean‐unbiasedness, is to apply the Rao‐Blackwell theorem. This theorem implies that if U is an unbiased estimator of the unknown parameter of interest θ, and T is a sufficient statistic, then the estimator Û = E(U|T) is unbiased and var(Û) ≤ var(U). In certain cases, this “Rao‐Blackwellization” technique allows the derivation of the unbiased estimator that has the smallest possible variance, which is known as the uniformly minimum variance unbiased estimator (UMVUE). The Lehmann‐Scheffé theorem ¹⁴⁰ states that if 𝒯 is a sufficient and complete statistic, and U is an unbiased estimator, then the Rao‐Blackwellized estimator Û = E(U|𝒯) is the unique UMVUE.

The derivation of UMVUEs (from an unconditional perspective) has been a focus in the literature on group sequential trials, with work by Chang et al ³² and Jung and Kim ³⁷ for binary response data (see also Porcher and Desseaux ⁴⁰ and Zhao et al ⁴¹ for Simon's two‐stage designs), and Kim, ³³ Emerson and Fleming, ³⁴ and Emerson and Kittelson ³⁵ for normally distributed endpoints. Liu and Hall ³⁶ proved that the sufficient statistic for the latter setting is in fact not complete, but that the Rao‐Blackwellized estimator is still UMVUE among all “truncation‐adaptable” unbiased estimators—that is, where inference following stopping early does not require knowledge of the future analyses. For simplicity, in what follows we do not make this technical distinction when describing UMVUEs. Similar, more general theoretical results can be found in Liu et al ³⁸ for group sequential tests for distributions in a one‐parameter exponential family and Liu et al ³⁹ for multivariate normal group sequential tests. See also Liu et al ¹⁴¹ and Liu and Pledger ¹⁴² for further theoretical results for general two‐stage adaptive designs.

Methodology has also been developed to calculate the UMVUE for secondary parameters or endpoints in group sequential trials. Liu and Hall ⁴³ derived the UMVUE in the context of correlated Brownian motions, while Gorfine ⁴² considered normally distributed endpoints where the secondary parameter is the mean in a subgroup of subjects. Similarly, Liu et al ⁴⁴ derived the UMVUE for a secondary probability, such as the rate of toxicity, in group sequential trials with binary endpoints. Kunz and Kieser ⁴⁵ also derived UMVUEs for secondary endpoints in two‐stage trials with binary endpoints. Further settings include estimating the sensitivity and specificity of a diagnostic test using a group sequential design, where UMVUEs have been derived by Shu et al ⁴⁶ and Pepe et al. ⁴⁷

Moving away from classical group sequential designs, Liu et al ⁸⁶ derived the UMVUE for a two‐stage adaptive design (with sample size adjustment based on conditional power). Kunzmann and Kieser ⁸⁷ derived the UMVUE general two‐stage adaptive designs with binary endpoints, where the second stage sample size is determined by a function of the response rate observed in stage one. Meanwhile, Liu et al ⁸⁵ proposed Rao‐Blackwellized unbiased estimators for both primary and secondary endpoints in the context of a two‐stage adaptive trial with sample size re‐estimation. Bowden and Trippa ¹²⁸ showed how to calculate a Rao‐Blackwellized unbiased estimator for trials using RAR with binary endpoints.

In some trial contexts, it may be more appropriate to find unbiased estimators conditional on the adaptations that have taken place. For example, in multi‐stage trials with treatment selection, typically there would be greater interest in estimating the properties of the better‐performing treatments that are selected at the end of the trial, rather than those treatments that are dropped for futility. In addition, it may not be possible to find a complete statistic for the parameter of interest without additionally conditioning on the selection rule used (see, eg, Cohen and Sackrowitz ¹⁰³ ). For both these reasons, in settings such as multi‐stage trials with ranking and selection, there has been a focus on deriving uniformly minimum variance conditionally unbiased estimators (UMVCUEs, also known as CUMVUEs), where the conditioning is on the triggered adaptations.

One of the first papers to take this conditional perspective and calculate the UMVCUE was Cohen and Sackrowitz, ¹⁰³ in the context of a two‐stage drop‐the‐loser trial with treatment selection and normally distributed endpoints (see Tappin ¹⁰⁴ for the setting with binary endpoints), where only the best‐performing treatment is taken forward to the final stage. This was subsequently extended by Bowden and Glimm ¹⁰⁵ to allow the calculation of the UMVCUE for treatments that were not the best‐performing, Bowden and Glimm ¹⁰⁶ for multi‐stage drop‐the‐loser trials (this was only a Rao‐Blackwellized estimator) and Robertson and Glimm ¹¹⁰ for unknown variances. Meanwhile, Koopmeiners et al ¹⁰⁷ derived the UMVCUE for a two‐stage trial evaluating continuous biomarkers. A similar line of work focused on deriving UMVCUEs for binary data, with Pepe et al ⁴⁷ looking at two‐stage designs testing the sensitivity of a dichotomous biomarker (this UMVCUE can also be applied for Simon's two stage designs, see Porcher and Desseaux, ⁴⁰ which was extended by Robertson et al ¹⁰⁸ ).

For seamless phase II/III trials with a normally distributed endpoint, Kimani et al ¹¹² showed how to calculate a Rao‐Blackwellized conditionally unbiased estimator, which was subsequently extended by Robertson et al. ¹⁰⁹ More recently, Stallard and Kimani ¹¹¹ derived the UMVCUE for MAMS trials (again with normally distributed endpoints) with treatment selection and early stopping for futility, conditional on any prespecified selection or stopping rule.

A recent setting where UMVCUEs have been derived is two‐stage adaptive enrichment designs, where biomarkers are used to select a patient population to investigate in the second stage. Kimani et al ¹³² and Kunzmann et al ¹³⁵ derived the UMVCUE for such designs with normally distributed endpoints, which was extended by Kimani et al ¹³³ to allow the biomarker cut‐off to be determined by the first stage data, and Kimani et al ¹³⁴ and Di Stefano et al ¹³⁶ for time‐to‐event endpoints. Finally, in the context of two‐stage adaptive designs where the second‐stage sample size depends on the result of the first stage through a prespecified function, Kunzmann and Kieser ⁸⁷ and Broberg and Miller ⁸⁸ considered UMVCUEs for normally distributed and binary endpoints, respectively.

5.2. Median‐unbiased estimation

While exact mean‐unbiasedness is a common criterion to aim for, it is not always feasible or desirable to use mean‐unbiased estimators. The main potential disadvantage is an increase in the MSE when compared with the usual MLE, due to the bias‐variance trade‐off. As well, the calculation of Rao‐Blackwellized estimators may become difficult or infeasible for more flexible trial designs, where adaptations are not fully prespecified in advance. For both these reasons, median‐unbiased estimators (MUEs) have been proposed, which can have a lower MSE than mean‐unbiased estimators in some settings, and can be derived for a wider class of adaptive trial designs.

Consider testing a hypothesis about some parameter of interest θ. One common approach to calculating MUEs is as follows:

1. Define an ordering of the trial design space with respect to evidence against H ₀. For example, in group sequential trials the predominant option is stage‐wise ordering: this depends on the boundary crossed, the stage at which stopping occurs and the value of the standardized test statistic (in decreasing order of priority).

2. Using this ordering, define a P‐value function P(θ) which gives the probability that, at the stage the trial stopped, more extreme evidence against H ₀ could have been observed.

3. At the point the trial stops, find the MUE ${\widehat{\theta }}_{\mathrm{MU}}$, which satisfies P(${\widehat{\theta }}_{\mathrm{MU}}$) = 0.5.

The last step is essentially finding a 50% confidence bound for θ, or equivalently the midpoint of a symmetric 100(1 − α)% confidence interval (constructed using the P‐value function, which takes into account the previous adaptations or stopping rules).

A number of papers have derived MUEs for group sequential trials, including Kim, ³³ , ⁴⁸ Emerson and Fleming, ³⁴ Todd ⁴⁹ and Troendle and Yu. ⁵⁰ Hall and Liu ⁵¹ provided MUEs that account for overrunning, while Hall and Yakir ⁵⁵ derived MUEs for secondary parameters. Note also early work by Woodroofe ¹⁴³ showed how to calculate a MUE but in the context of sequential probability ratio tests. Meanwhile, MUEs for Simon's two‐stage design were derived by Koyama and Chen. ⁵²

For adaptive group sequential trials Wassmer ⁵⁶ and Nelson et al ⁶² showed how to calculate MUEs for normally distributed endpoints, while Brannath et al ⁵⁷ derived a MUE for survival endpoints. These results were subsequently extended by Gao et al, ⁵⁹ who presented general theory for point and interval estimation (see also Mehta et al ⁶¹ ). Gao et al ⁵⁸ also derived a MUE for an adaptive group sequential design that tests for noninferiority followed by testing for superiority.

Moving away from group sequential trials, Wang et al ⁹⁰ and Liu et al ⁸⁶ focused on two‐stage adaptive designs with sample size adjustments based on conditional power considerations, and showed how to calculate MUEs in these settings. Lawrence and Hung ⁸⁹ proposed a MUE for the setting where the maximum information of a trial is adjusted on the basis of unblinded data (see also Liu et al ⁸⁵ for an improved version of this estimator). More generally, MUEs for two‐stage trials with an adaptive sample size based on a prespecified function of the interim results are given by Kunzmann and Kieser ⁸⁷ and Nhacolo and Brannath. ⁹¹ Meanwhile, Li et al ¹⁴⁴ derived MUEs in the context of “2‐in‐1” adaptive designs, which allow for a seamless expansion of a phase II trial into a phase III trial or an expansion of biomarker positive populations, where the endpoints for the interim and final analyses can be different.

Like for mean‐unbiased estimation, the conditional perspective has also been used for median‐unbiased estimation. The previously mentioned work by Hall and Yakir ⁵⁵ for group sequential trials is an early example of this conditional perspective, while the approach of Koopmeiners et al ¹⁰⁷ can also be used to calculate a conditional MUE for two‐stage group sequential designs. In the specific context of two‐stage group sequential designs with survival endpoints, Shimura et al ⁵⁴ showed how to calculate conditional MUEs, for the cases where the study stops or does not stop early for either efficacy or futility. Meanwhile, Broberg and Miller ⁸⁸ considered the general setting of a two‐stage sample size adjustable design based on interim data, and derived the MUE conditional on continuing to the second stage.

MUEs can be derived for flexible adaptive designs, which allow arbitrary changes to the trial at each stage based on the interim results and/or external information. Bauer et al ⁹² first described the construction of MUEs for two‐stage designs based on combination tests and conditional error functions; see also Liu and Chi, ⁹³ Proschan et al, ⁹⁵ and Brannath et al. ⁹⁶ Finally, Brannath et al ⁹⁴ gave a general method to calculate MUEs when using recursive combination tests for flexible multi‐stage adaptive trials.

6. BIAS‐REDUCED ESTIMATION

While using an unbiased estimator whenever there is one available for a particular adaptive design may seem like a straightforward choice, this may not always be the best option, due to the inherent bias‐variance tradeoff: reducing the bias of an estimator can come at the cost of increasing its variance (thus decreasing its precision) substantially. For example, estimation procedures that involve conditioning on a statistic that is informative about the parameter of interest (eg, the stopping time) address the bias, but discard the information that is contained in that statistic and hence lead to a loss of efficiency. The bias‐variance tradeoff can lead to unbiased treatment effect estimates being accompanied by large standard errors and wide confidence intervals, which may be of little use practically. Therefore, reducing the bias but not completely eliminating it sometimes proves to be a better solution than aiming for complete unbiasedness.

6.1. Analytical and iterative procedures

Cox recognized as early as 1952 ¹⁴⁵ that in sequential test procedures the option to stop early introduces bias to the MLE. They proposed a modified MLE which can be used for repeated significance tests ¹⁴⁶ but not more general group sequential tests. Whitehead ⁷⁰ focused on the sequential probability ratio test ¹⁴⁷ and the triangular test, ¹⁴⁸ and derived an analytic expression to quantify the bias and proposed a bias‐adjusted estimator $\tilde{\theta }$ obtainable by subtracting the bias from the MLE:

$$\tilde{\theta }=\widehat{\theta }-\text{bias}(\tilde{\theta }).$$

Evaluating the bias at $\tilde{\theta }$ leads to an iterative procedure, and this method can be used for various types of endpoints including continuous, binary, and time‐to‐event. Guo and Liu ⁷⁵ argued that a simpler single‐iteration version could be used, where the estimated bias of the MLE is subtracted rather than the bias at the unknown $\tilde{\theta }$:

$$\widetilde{{\theta }_s}=\widehat{\theta }-\text{bias}(\widehat{\theta }).$$

They suggested that this approach could be extended from single‐arm phase II trials to more general estimation problems involving designs with early stopping. Note that these proposals aim to reduce the unconditional bias of the MLE, see below for the conditional perspective.

Chang et al ³² and applied Whitehead's idea to group sequential designs with binary response data, while Tan and Xiong ⁷¹ proposed using the bias‐adjusted MLE to estimate response rates in trials using fully sequential and group sequential conditional probability ratio tests. Li and DeMets ⁷² derived an exact analytical expression for the bias of the MLE for a group sequentially monitored Brownian motion process with a linear drift, and used it to construct a bias‐adjusted estimator based on the arguments of Whitehead. ⁷⁰ Todd ⁴⁹ studied the bias‐adjusted MLE further in simulations of group sequential designs with a normally distributed endpoint using the O'Brien‐Fleming or triangular test. Denne ⁹⁷ postulated that the same bias‐adjusted MLE could also be used to estimate the treatment effect following a two‐stage sample size re‐estimation design on the basis of conditional power, which was subsequently investigated by Liu et al ⁸⁵ (who also considered a bias‐adjusted MLE for secondary endpoints). Levin et al ⁶⁰ found that the bias‐adjusted MLE could be applied to a wider class of adaptive group sequential designs with predefined rules.

Whitehead ⁷⁹ extended the idea of the bias‐adjusted MLE to the estimation of secondary endpoints in sequential trials using two different approaches: conditional on the primary endpoint (in which case the MLE is unbiased) and unconditional. Liu et al ⁴⁴ further extended Whitehead's approach to estimate secondary probabilities after termination of a sequential phase II trial with response rate as the primary endpoint. The bias‐adjusted MLE has also been proposed for a group sequential strategy to monitor toxicity incidents in clinical trials by Yu et al, ⁸⁰ for which they noted the bias of the usual MLE was “considerably large.” Meanwhile, Flight ⁸¹ showed how to use the bias‐adjusted MLE for health economic outcomes following a group sequential design.

For trials with treatment selection, early work by Coad ¹¹⁶ in the setting with two treatments and either a two or three‐stage trial (where only one treatment is selected to be carried forward to the final stage), showed how to derive a bias‐adjusted MLE. In the context of adaptive enrichment designs with time‐to‐event endpoints, Di Stefano et al ¹³⁶ derived both single and multiple iteration versions of a bias‐adjusted MLE. Finally, for response‐adaptive trials, Coad ¹²⁹ and Morgan ¹³⁰ showed how to approximate the bias of the MLE, which then allows the construction of a bias‐adjusted estimator following the idea of Whitehead. ⁷⁰

6.1.1. Conditional perspective

Troendle and Yu ⁵⁰ developed methods for estimating treatment differences in group sequential designs with normally distributed endpoints that condition on the stopping time S of the trial, thereby reducing the conditional bias, that is, the discrepancy between the conditional expectation of the estimator and the parameter value:

$$\text{Conditional bias}(\theta )=E(\widehat{\theta }|S)-\theta .$$

They used a similar iterative approach as Whitehead ⁷⁰ to compute the estimator. Coburger and Wassmer ⁹⁸ took this one step further, by conditioning both on stopping time as well as the sample sizes of the stages up to that point; whereas Troendle and Yu had assumed equal sample sizes for all stages. They also considered adaptive group sequential designs, where the unconditional bias is no longer computable, but the conditional bias can be approximated. Tremmel ¹⁰¹ extended these considerations to stratified binomial and time‐to‐event endpoints. Coburger and Wassmer ⁷⁸ suggested using a similar conditional bias‐adjusted estimator not only when a trial stops but also during the course of an adaptive sample size re‐estimation design, to prevent an overestimation of the sample sizes for subsequent stages.

Fan et al ⁷³ showed that the conditional bias‐reduced estimator for group sequential designs as proposed by Troendle and Yu ⁵⁰ is equivalent to a conditional MLE as well as to a conditional moment estimator. They also proposed two modified estimators which are hybrids of the conditional and unconditional MLE that have smaller variance and MSE. The relationship between the conditional MLE and the conditional bias‐reduced estimator was also pointed out by Liu et al, ⁷⁴ who additionally derived the conditional MLE for secondary endpoints in a group sequential design and adaptive group sequential design. More general theoretical results for sequential trials focusing on the conditional MLE can be found in Molenberghs et al ¹⁴⁹ and Milanzi et al. ¹⁵⁰ , ¹⁵¹

Marschner ¹³¹ recently extended this notion of conditional MLEs to more general adaptive designs, as part of a unifying formulation of adaptive designs and a general approach to their analysis. More specifically, the unconditional likelihood can be expressed as the product of the design likelihood (ie, the information contained in the realized design) and the conditional likelihood (conditional on the realized design). Marschner also proposed a penalized MLE, which weights the design likelihood and can vary between the two extremes of the unconditional and conditional MLEs. Examples of the conditional and penalized MLEs were given in the context of a RAR design.

Shimura et al ⁷⁶ modified Troendle and Yu's conditional bias‐reduced estimator for two‐stage group sequential trials with normal endpoints by combining it with the shrinkage idea of Thompson ¹⁵² : in the calculation of the bias adjustment term, they replaced the MLE with a weighted average of the MLE at the interim analysis and a “prior” estimate of the effect size, ${\widehat{\theta }}_{*}=c\widehat{\theta }+(1-c){\theta }_0$, with ${\theta }_0$ the effect size assumed in the initial sample size calculation and shrinkage weights c determined as a function of the MLE as proposed by Thompson. ¹⁵² Although the authors presented their idea in a frequentist framework, it can also be interpreted as Bayesian, with ${\theta }_0$ the prior and ${\widehat{\theta }}_{*}$ the posterior expectation. Meanwhile, Li ⁷⁷ proposed using a weighted sum of the MLE and sample proportion for two‐stage group sequential designs with binary responses, where the weight is based on estimates of the MSEs of these estimators. Pepe et al ⁴⁷ and Koopmeiners et al ¹⁰⁷ considered conditional bias‐reduced estimation in the context of group sequential diagnostic biomarker studies with a single interim analysis for futility, conditional on not stopping early.

Cheng and Shen ⁹⁹ derived an easily computable bias‐adjusted estimator of the treatment effect in a “self‐designing” trial (where the sample size is sequentially determined to achieve a target power) with a normally distributed endpoint. ¹⁵³ They further proposed a modified estimator, involving a shrinkage factor, to reduce bias when block sizes are small. For “self‐designing” trials with censored time‐to‐event endpoints and group sequential sample size re‐estimation, ¹⁵⁴ Shen and Cheng ¹⁰⁰ developed bias‐adjusted estimators of the hazard ratio. Broberg and Miller ⁸⁸ considered two‐stage sample size adjustable designs with normally distributed endpoints and derived a conditional bias‐adjusted MLE. Meanwhile, in the context of “2‐in‐1” adaptive trial designs, Li et al ¹⁴⁴ derived a conditionally bias‐adjusted estimator.

Moving on to trial designs with treatment selection, when multiple treatments are compared to a control and the most promising treatment is selected at the first interim analysis (with or without further early stopping opportunities at subsequent interim analyses), Stallard and Todd ¹²⁴ developed a bias‐adjusted estimator for the treatment effect. Kimani et al ¹¹² adapted this estimator for the difference between the selected and control treatments in seamless phase II/III designs with normally distributed endpoints. Unlike Stallard and Todd's estimator, ¹²⁴ which aimed to be approximately unbiased conditional on the selected treatment, this new estimator aimed to be approximately unbiased conditional on both the selected treatment and continuation to the second stage. In the context of selecting the treatment dose with the highest response rate (using a normal approximation), Shen ¹¹⁷ also derived a stepwise bias overcorrection. When there are two treatments being compared with a control, Stallard et al ¹¹⁸ proposed a family of approximately conditionally unbiased designs.

Brückner et al ¹²³ extended Stallard and Todd's bias‐reduced estimator to two‐stage multi‐arm designs involving a common control and treatment selection at the interim analysis with time‐to‐event endpoints. Despite deriving analytical expressions for the selection bias of the MLE in the selected and stopped treatment arms, respectively, they experienced computational difficulties and convergence problems. They only defined the resulting bias‐reduced estimator for the interim analysis, and proposed using a two‐stage estimator for the final analysis, which is a weighted average of the bias‐reduced estimator at the interim analysis and the MLE.

Luo et al ¹¹⁹ used the method of conditional moments to derive a bias‐adjusted estimator of the response rate following a two‐stage drop‐the‐loser design with a binary endpoint. Adopting a stochastic process framework (as opposed to the conventional random variable viewpoint) for clinical trials with adaptive elements, Luo et al ¹²⁰ proposed approximating treatment effects based on a general estimating equation to match the first conditional moment. This generic approach can be used for a variety of adaptive designs with endpoints following any probability distribution. They illustrated it for two‐stage treatment selection designs comparing multiple treatments against a control.

Kunzmann et al ¹³⁵ applied the conditional moment estimator of Luo et al ¹¹⁹ to two‐stage adaptive enrichment designs with a normally distributed endpoint and binary biomarker with a prespecified cut‐off. Additionally, they studied two hybrid estimators, one between the UMVCUE and the conditional moment estimator, and the other one combining the UMVCUE and MLE. Kimani et al ¹³³ studied single‐ and multi‐iteration bias‐adjusted estimators for the treatment effect in the selected subpopulation of a two‐stage adaptive threshold enrichment design allowing the cut‐off of a predictive biomarker based on first stage data.

Finally, Bebu et al ¹²¹ devised a bias‐corrected conditional MLE for the mean of the selected treatment in two‐stage designs and normally distributed endpoints. They focused on the case of two experimental treatments, but this was extended by Bebu et al ¹²² to designs involving the selection of more than one treatment arm, selection rules based on both efficacy and safety, including a control arm, adjusting for covariates, and binomial endpoints. Bowden and Glimm ¹⁰⁶ further extended the method to multi‐stage drop‐the‐loser designs which must always proceed to the final stage (ie, no early stopping rules).

6.2. Bayesian and empirical Bayes approaches

Several Bayesian approaches to bias‐reduced estimation in adaptive designs have been developed, typically for trials using frequentist frameworks. These differ from fully Bayesian adaptive designs in that they only use the accumulating data, rather than the posterior, to make decisions about adaptations (eg, stopping or selecting arms) and rely primarily on classical frequentist inference such as hypothesis testing, but often require additional assumptions or information, such as the specification of a prior.

Hughes and Pocock ⁸³ and Pocock and Hughes ⁸⁴ proposed a Bayesian shrinkage method for estimating treatment effects in trials stopped at an interim analysis, whereby a prior quantifying the plausibility of different treatment effect sizes is specified at the outset of the trial and combined with the trial data using Bayes' rule, producing a posterior estimate of the treatment effect that is shrunk toward the median of the prior distribution. They acknowledged that this will be sensitive to the choice of prior and warned against using an overoptimistic prior as this would lead to little or no shrinkage, saying that their method “works best in the hands of prior specifiers who are realists by nature.” ⁸³

Hwang ¹⁵⁵ proposed using Lindley's ¹⁵⁶ shrinkage estimator for estimating the mean of the best treatment in single‐stage multi‐arm designs with 4 or more study treatments and a normally distributed endpoint. This was extended by Carreras and Brannath ¹²⁵ to multi‐arm two‐stage drop‐the‐losers designs. Their two‐stage version is a weighted mean of Lindley's estimator for the first stage and the MLE of the selected treatment for the second stage. It shrinks the MLE toward the mean across all arms, but only works with a minimum of 4 study treatment arms. For designs with 2 or 3 study treatments, they replaced Lindley's estimator with the best linear unbiased predictor of a random effects model for the first stage estimator. This method also allows for bias‐reduced estimation of the second best treatment mean.

Of note, Lindley's estimator can be viewed as an empirical Bayes estimator of the posterior mean of the posterior distribution of a specific treatment mean given its MLE. This fact was exploited by Bowden et al, ¹²⁶ who adapted Carreras and Brannath's method and replaced their two‐stage estimator with a single shrinkage equation for both stages. This, however, adds both practical and theoretical complications, necessitating further modifications such as estimating a between‐arm heterogeneity parameter. The authors suggest this approach could be applied to other MAMS designs.

Along the lines of Hwang's approach, ¹⁵⁵ Brückner et al ¹²³ derived two shrinkage estimators for two‐stage multi‐arm designs with a common control, treatment selection at the interim analysis, and time‐to‐event endpoints. They both shrink the MLE of the selected treatment toward the overall log‐hazard ratio. The authors also defined a two‐stage estimator similar to that of Carreras and Brannath, ¹²⁵ as well as a bias‐corrected Kaplan‐Meier estimator for the selected treatment.

Bunouf and Lecoutre ¹²⁷ derived two Bayesian estimators for multi‐stage designs with binary endpoints where after every stage a decision is made whether to stop or continue recruiting: one estimator is the posterior mean based on a “design‐corrected” prior, the other the posterior mode based on a similarly corrected uniform prior, and for the latter they also provided an easy‐to‐use approximation for two‐stage designs. Meanwhile, Kunzmann et al ¹³⁵ developed two empirical Bayes estimators for two‐stage adaptive enrichment designs with a single prespecified subgroup and prespecified decision rule based on a binary biomarker and a normally distributed endpoint with known variance. Kimani et al ¹³³ extended this work to adaptive threshold enrichment designs where the biomarker cut‐off is determined based on first stage data.

Finally, Kunzmann and Kieser ⁸⁷ developed an estimator for two‐stage single‐arm trials with a binary endpoint with sample size adjustment and early stopping for futility or efficacy. This estimator minimizes the expected MSE subject to compatibility with the test decision and monotonicity conditions, and can be viewed as a constrained posterior mean estimate. Similarly, in the same trial context Grayling and Mander ¹⁰² proposed an optimal estimator that minimizes a weighted sum of the expected MSE and bias.

6.3. Resampling‐based methods

The methods for unbiased and biased‐reduced estimation considered in the previous subsections require explicit formulas based on the population response model as well as the trial design and adaptations. For example, UMVUEs and UMVCUEs can often be given as closed‐form expressions, while for MUEs a P‐value function is often specified. Meanwhile, bias‐reduced estimators tend to have explicit expressions for an estimate of the bias, leading to equations that can be solved numerically.

An alternative class of methods is based on resampling procedures, where the trial data is resampled or generated via a parametric bootstrap a large number of times, and the resulting trial replicates can then be used to give empirical estimates of the bias. One advantage of these methods is that essentially the same procedure can be used under a variety of different stopping rules and trial designs, including complex designs as detailed below.

In the group sequential setting, Wang and Leung ⁶⁶ showed how to use a parametric bootstrap to calculate a bias‐corrected estimate for the mean of normally distributed endpoints with either known or unknown variance. This methodology was generalized by Leung and Wang, ⁶⁷ who proposed a generic stochastic approximation approach based on a parametric bootstrap, which can be used for non‐independent and identically distributed (iid) data and a large class of trial designs, including sequential ones. Cheng and Shen ⁶⁸ proposed a variant of the Wang and Leung ⁶⁶ procedure for complex group sequential monitoring rules that assess the predicted power and expected loss at each interim analysis. Magnusson and Turnbull ¹⁷ proposed a double bootstrap procedure for group sequential enrichment designs incorporating subgroup selection. Meanwhile, Pinheiro and DeMets ⁶⁵ described a simulation approach to estimate the bias of the MLE and then proposed constructing a bias‐reduced estimator for the treatment difference, although this is equivalent to Whitehead's bias‐adjusted MLE. ⁷⁰

Kunzmann et al ¹³⁵ carried out further work in the context of adaptive enrichment trials. In the two‐stage trial setting with a binary biomarker and normally distributed endpoints, they considered a number of alternatives to the MLE, including a parametric bootstrap procedure. More generally, Simon and Simon ¹³⁷ considered multi‐stage enrichment designs that develop model‐based predictive classifiers based on multiple markers, and showed how to use a parametric bootstrap method for bias correction for generic response distributions. Finally, in trials with treatment selection, Pickard and Chang ¹¹⁴ considered parametric bootstrap corrections for a two‐stage drop‐the‐losers design, for normally and binomially distributed endpoints.

Thus far, all the methods presented above are fully parametric, that is, assuming that the trial outcome data comes from a probability distribution with a fixed set of parameters. An alternative approach is to use nonparametric bootstrap procedures to correct for bias. An early example of this approach was given by Leblanc and Crowley, ⁶⁹ who proposed a nonparametric bootstrap in the context of group sequential designs with censored survival data using log rank testing. Subsequently, Rosenkranz ¹⁵⁷ described using nonparametric bootstrap to correct the bias of treatment effect estimates following selection (eg, selecting the treatment with the maximum effect). More recently, Whitehead et al ¹¹⁵ proposed an estimation approach based on the method of Rao‐Blackwellization (ie, targeting unbiased estimation), but used a nonparametric bootstrap procedure to recreate replicate observations in a complicated group sequential trial setting comparing multiple treatments with a control.

7. DISCUSSION

Motivated by recent FDA guidance on estimation after adaptive designs, this article provides a review of the potential methodological solutions to biased estimation that exist in the literature. We found that there is a growing body of work proposing and evaluating a range of unbiased and bias‐adjusted estimators for a wide variety of adaptive trial designs. Our ambition is that this article, combined with the annotated bibliography given in the Supplementary Information, provides an easily accessible and comprehensive resource for researchers in this rapidly evolving area. However, we note that statistical software and code to easily calculate adjusted estimators is relatively sparse, which is an obstacle to the uptake of methods in practice (see Grayling and Wheeler ¹⁵⁸ and part II of this article series). It also remains the case that for more complex or novel adaptive designs, adjusted estimators may not exist.

From a methodological perspective, there remain important open questions regarding estimation after adaptive trials, as already mentioned at the end of Section 6. More generally, an estimator for a parameter of interest should ideally have all of the following properties:

1. Adequately reflects the adaptive design used.

2. No or small bias.

3. Low MSE (reflecting a favorable bias‐variance trade‐off).

4. Is easily computable.

5. A procedure for calculating an associated confidence interval that:

   1. Has the correct coverage.
   2. Is consistent/compatible with the hypothesis test decisions (including early stopping).

Constructing an overarching framework for estimation after adaptive designs that has all these properties would be very useful, although challenging. Some initial steps in this direction have been taken by Kunzmann and Kieser ⁸⁷ , ¹⁵⁹ in the context of adaptive two‐stage single‐arm trials with a binary endpoint. Even if such a framework is not feasible more generally, the issue of constructing confidence intervals remains an important question that has so far received less attention in the literature than point estimation.

The property that an estimator should “adequately reflect the adaptive design used” also has implications when considering the Bayesian analysis of adaptive designs, as pointed out by an anonymous reviewer. The likelihood function will typically be insensitive to the adaptive design and sampling scheme (see, eg, Section 4.1 of Marschner ¹³¹ ). Thus, while the frequentist interpretation will take account of the adaptive design, the pure Bayesian approach following the likelihood principle will not. In practice this would mean that a pure Bayesian analysis would ignore the adaptive nature of the design and the issues discussed in this article would disappear. However, this (arguably) is intuitively unappealing and provides a rationale for evaluating Bayesian‐motivated approaches from a frequentist perspective.

In part II of this article series, we explore the practical considerations surrounding the use of unbiased and bias‐reduced estimators for adaptive designs. There, we review the use of such estimators in current practice and illustrate their application to a real trial design. We also provide a set of guidelines for best practice, considering their use in adaptive trials from the design stage through to the final reporting of results.

Supporting information

Appendix S1 Supporting Information

Robertson DS, Choodari‐Oskooei B, Dimairo M, Flight L, Pallmann P, Jaki T. Point estimation for adaptive trial designs I: A methodological review. Statistics in Medicine. 2023;42(2):122–145. doi: 10.1002/sim.9605

DATA AVAILABILITY STATEMENT

All of the data that support the findings of this study are available within the paper and supplementary information.