TABLE 3.

Summary of unbiased and bias‐reduced estimators, trial examples, and software/code for different classes of adaptive designs

Design	Method(s)	Pros and cons	Trial examples and software/code
Group sequential	Mean‐unbiased estimation
	Chang et al, 32 Kim, 33 Emerson and Fleming, 34 Emerson and Kittelson, 35 Liu and Hall, 36 Jung and Kim, 37 Liu et al, 38 , 39 Porcher and Desseaux, 40 Zhao et al 41 Secondary parameters: Gorfine, 42 Liu and Hall, 43 Liu et al, 44 Kunz and Kieser 45 Diagnostic tests: Shu et al, 46 Pepe et al 47	The UMVUE has zero bias, but tends to have a higher MSE than the naive or bias‐adjusted estimators Computation can be complex and extensive if the number of looks is relatively large (≥4), but otherwise can be simpler than for bias‐adjusted estimators UMVUE can be conditionally biased	Beta‐Blocker Heart Attack Trial, see Gorfine 42 Phase II trial in patients with adenocarcinoma, see Kunz and Kieser 45 GI06‐101 trial in hepatobiliary cancer, see Zhao et al 41 Software/code: OptGS R package
	Median‐unbiased estimation
	Kim, 33 , 48 Emerson and Fleming, 34 Todd et al, 49 Troendle and Yu, 50 Hall and Liu, 51 Koyama and Chen, 52 Wittes, 53 Porcher and Desseaux, 40 Shimura et al 54 Secondary parameters: Hall and Yakir 55 Adaptive group sequential trials: Wassmer, 56 Brannath et al, 57 Gao et al, 58 , 59 Levin et al, 60 Mehta et al, 61 Nelson et al 62	MUE reduces bias compared to the naive estimator, but can have an increased MSE. Bias‐adjusted estimators can have a lower MSE as well Calculation of the MUE can be complicated, and results can depend on the ordering of the sample space MUEs can be derived for adaptive group sequential designs, unlike for other estimation methods	Trials for acute bronchitis reported in Wassmer et al 63 Multicenter Automatic Defibrillator Implantation Trial (MADIT), see Hall and Liu, 51 Hall and Yakir 55 Non‐small cell lung cancer trial, see Wassmer 56 Randomized Aldactone Evaluation Study (RALES), see Wittes 53 Trial reported in Troger et al 64 Clinical Evaluation of Pertuzumab and Trastuzumab (CLEOPATRA) trial, see Shimura et al 54 Software/code: ADDPLAN59 Note: Also implemented in software such as SAS, East, as well as R packages such as rpact, RCTdesign, AGSDest, and OptGS. See https://panda.shef.ac.uk/techniques/group‐sequential‐design‐gsd/categories/27
	Resampling
	Parametric: Pinheiro and DeMets, 65 Wang and Leung, 66 Leung and Wang, 67 Cheng and Shen, 68 Magnusson and Turnbull 17 Nonparametric: Leblanc and Crowley 69	Essentially the same procedure can be used under different stopping rules and different study designs Bootstrap algorithms can be computationally intensive Bias is substantially reduced, with reasonable MSE Nonparametric approaches are robust to model misspecification	Trial for nasopharyngeal cancer, see Leblanc and Crowley 69
	Bias‐reduced
	Whitehead, 70 Chang et al, 32 Tan and Xiong, 71 Todd et al, 49 Li and DeMets, 72 Fan et al, 73 Liu et al, 74 Guo and Liu, 75 Porcher and Desseaux, 40 Shimura et al, 76 Li 77 Adaptive group sequential trials: Levin et al 60 Conditional: Troendle and Yu, 50 Coburger and Wassmer 78 Secondary parameters: Whitehead, 79 Liu et al, 44 , 74 Yu et al 80 Health economic outcomes: Flight 81	The MSE of the bias‐adjusted MLE is typically lower than that of the UMVUE, particularly for small sample sizes Shrinkage‐type estimators (which also have a Bayesian interpretation) can reduce both the conditional bias and MSE further	Three different phase II studies, see Tan and Xiong 71 Trial of immunosuppression for bone marrow transplantation, see Whitehead 79 Two cardiovascular trials (MERIT‐HF and COPERNICUS), see Fan et al 73 Trial in familial adenomatous polyposis, see Liu et al 74 Phase II trial in endometrial cancer, see Shimura et al 76 GUSTO trial for myocardial infarctions, see Marschner and Schou 82 Software/code: RCTdesign R package, OptGS R package 60
	Bayesian
	Hughes and Pocock, 83 Pocock and Hughes 84	Useful for producing shrinkage of unexpectedly large/imprecise observed treatment effects that arise in trials that stop early Bias reduction depends on the specification of the prior distribution.	None
Sample size re‐estimation	Mean‐unbiased estimation
	Unconditional: Liu et al, 85 , 86 Kunzmann and Kieser 87 Conditional (UMVCUE): Kunzmann and Kieser, 87 Broberg and Miller 88	Estimators have zero bias, either unconditionally or conditionally. However, the MSE tends to be greater than for the naive estimator and bias‐reduced estimators Does not guarantee compatibility with the test decision (see Kunzmann and Kieser 87 ) Estimators can have an explicit representation making computation easy	Schizophrenia trial, see Broberg and Miller 88
	Median‐unbiased estimation
	Lawrence and Hung, 89 Liu et al, 85 Wang et al, 90 Liu et al, 86 Kunzmann and Kieser, 87 Nhacolo and Brannath 91 Conditional perspective: Broberg and Miller 88 Flexible sample size adaptations: Bauer et al, 92 Liu and Chi, 93 Brannath et al, 94 Lawrence and Hung, 89 Proschan et al, 95 Brannath et al 96	MUE tends to have small mean bias, and can also have smaller MSE than the naive estimator Does not guarantee compatibility with the test decision MUE can be calculated for flexible adaptation rules that are not completely prespecified in advance, unlike for other estimation methods	Coronary artery disease trial, see Wang et al 90 Trial on reperfusion therapy for acute myocardial infarction, see Brannath et al 96 Software/code: R packages such as rpact and adpss
	Bias‐reduced
	Denne, 97 Coburger and Wassmer, 98 Cheng and Shen, 99 Shen and Cheng, 100 Liu et al, 85 Tremmel, 101 Broberg and Miller 88	Proposed bias‐reduced estimates are nearly unbiased with practical sample sizes, with similar variance to the naive estimator Numerical problems can occur when calculating adjusted estimators, and observations close to the critical boundaries can lead to unreasonably extreme adjusted estimators	Type II Coronary Intervention Study, see Denne 97 Colon cancer trial, see Shen and Cheng 100 Trial in chronic lymphocytic leukemia, see Tremmel 101 Schizophrenia trial, see Broberg and Miller 88
	Bayesian
	Kunzmann and Kieser, 87 Grayling and Mander 102	Guarantees compatibility with the test decision Reduces MSE of MLE except for very small or very large values of the success probability Reduces absolute bias compared with MLE except for small values of the success probability, where there can be a substantial positive bias Can reduce MSE substantially compared to the UMVUE for certain response rates	None
Multi‐arm multi‐stage designs (with treatment selection)	Mean‐unbiased estimation
	Two‐stage designs: Cohen and Sackrowitz, 103 Tappin, 104 Bowden and Glimm, 105 , 106 Pepe et al, 47 Koopmeiners et al, 107 Robertson et al, 108 , 109 Robertson and Glimm 110 Multi‐stage designs: Bowden and Glimm, 106 Stallard and Kimani 111 Seamless phase II/III trials: Kimani et al, 112 Robertson et al 109	UMVCUEs are conditionally unbiased. Compared to the MLE, the conditional MSE tends to be lower, but unconditionally the MSE can substantially increase UMVCUEs in the literature tend to have a closed‐form expression, allowing for easy computation In some settings, the UMVCUE can have comparable MSE to bias‐adjusted estimators	Trial for the treatment of anxiety disorder, see Kimani et al 112 and Robertson et al 109 INHANCE study, see Robertson and Glimm 110 ADVENT trial, see Stallard and Kimani 111 PROVE trial 113 implements approach of Stallard and Kimani 111 Software/code: Bowden and Glimm 106
	Resampling
	Pickard and Chang, 114 Whitehead et al 115	Provides a reasonable balance between bias and MSE across several scenarios Approach can be applied to endpoints coming from a variety of distributions (including normal and binomial) Approaches are robust to model misspecification	Software/code: Whitehead et al 115
	Bias‐reduced
	Two‐stage designs: Coad, 116 Shen, 117 Stallard et al, 118 Pepe et al, 47 Luo et al, 119 , 120 Bebu et al, 121 , 122 Koopmeiners et al, 107 Brückner et al 123 Multi‐stage designs: Coad, 116 Stallard and Todd, 124 Bowden and Glimm 106 Seamless phase II/III trials: Kimani et al 112	Bias‐adjusted MLE can have relatively low MSE and acceptably small bias in some scenarios Shrinkage methods can be the most effective in reducing the MSE Bias‐reduced estimators can run into computational/convergence problems Estimators can overcorrect for bias	Phase II study in colorectal cancer, see Luo et al 119 Software/code: Luo et al 119 FOCUS trial in advanced colorectal cancer, see Brückner et al 123 Phase III trial in Alzheimer's, see Stallard and Todd 124 Software/code: Bowden and Glimm 106 Software/code: Kimani et al 112
	Bayesian
	Two‐stage designs: Carreras and Brannath, 125 Bowden et al, 126 Brückner et al 123 Multi‐stage designs: Bunouf and Lecoutre 127	Shrinkage estimators can perform favorably compared with the MLE in terms of bias and MSE Approaches can run into practical and theoretical complications, necessitating further modifications such as estimating a between‐arm heterogeneity parameter	FOCUS trial in advanced colorectal cancer, see Brückner et al 123
Response‐adaptive randomization	Mean‐unbiased estimation
	Bowden and Trippa 128	Mean‐unbiased estimators can have a large MSE and can be very computationally intensive to calculate	Glioblastoma trial with multiple treatments
	Bias‐reduced
	Coad, 129 Morgan, 130 Marschner 131	The conditional MLE can be very effective at eliminating the conditional bias that is present in the unconditional MLE, but this comes at the cost of a loss of efficiency except for more extreme designs The penalized MLE exhibits very little conditional bias and is not subject to substantial efficiency loss (compared to the unconditional MLE) when the realized design is close to its average	None
Adaptive enrichment designs	Mean‐unbiased estimation
	Two‐stage designs: Kimani et al, 132 , 133 , 134 Kunzmann et al, 135 Di Stefano et al 136	UMVCUE is unbiased, but tends to have a higher MSE than the MLE UMVCUE can also have a larger MSE than shrinkage/bias‐adjusted estimators, but compensates by the corresponding eradication of bias UMVCUEs are easily computable, as they have a closed form expression	MILLY phase II study in asthma, see Kunzmann et al 135 Software/code: Kimani et al 134
	Bias‐reduced
	Two‐stage designs: Kunzmann et al, 135 Kimani et al, 133 Di Stefano et al 136	Can have lower MSE than UMVCUE that is comparable to the MSE of the MLE, but has residual bias	MILLY phase II study in asthma, see Kunzmann et al 135
	Resampling
	Magnusson and Turnbull, 17 Kunzmann et al, 135 Simon and Simon 137	Bootstrap estimator can have a higher bias than the MLE in the two‐stage setting of Kunzmann et al, 135 but Simon and Simon 137 found it was very effective at correcting for bias in their multimarker setting Procedures can be computationally intensive	MILLY phase II study in asthma, see Kunzmann et al 135
	Bayesian
	Two‐stage designs: Kunzmann et al, 135 Kimani et al, 133 Di Stefano et al 136	Estimators exhibit a higher bias than the MLE in many situations, with varying MSE properties	MILLY phase II study in asthma, see Kunzmann et al 135

Abbreviations: MLE, maximum likelihood estimator; MSE, mean squared error; MUE, median unbiased estimator; UMVCUE, uniformly minimum variance conditionally unbiased estimator; UMVUE, uniformly minimum variance unbiased estimator.