Estimation after blinded sample size reassessment

Abstract

Blinded sample size reassessment is a popular means to control the power in clinical trials if no reliable information on nuisance parameters is available in the planning phase. We investigate how sample size reassessment based on blinded interim data affects the properties of point estimates and confidence intervals for parallel group superiority trials comparing the means of a normal endpoint. We evaluate the properties of two standard reassessment rules that are based on the sample size formula of the z-test, derive the worst case reassessment rule that maximizes the absolute mean bias and obtain an upper bound for the mean bias of the treatment effect estimate.

1 Introduction

In clinical trials for the comparison of means of normally distributed observations, the sample size to achieve a specific target power depends on the true effect size and variance. For the purpose of sample size planning, the effect size is usually assumed to be equal to a minimally clinically relevant effect while the variance is often estimated from historical data. For situations where only little prior knowledge on the variance is available, clinical trial designs with sample size reassessment based on interim estimates of this nuisance parameter have been proposed. Stein¹ developed a two-stage procedure, where the second stage sample size is calculated based on a first stage variance estimate aiming to achieve a pre-specified target power. In the context of clinical trials, several extensions of Steins two-stage procedure have been considered^2–8 that all require an unblinding of the interim data for the computation of the variance estimate. However, regulatory agencies generally prefer blinded interim analyses as they entail less potential for bias.^9–13

Gould and Shih¹⁴ proposed to estimate the variance from the blinded interim data by computing the variance from the total sample (pooling the observations from both groups), instead. Although this interim estimate is not consistent¹⁵ and has a positive bias if the alternative hypothesis holds, the bias is negligible for effect sizes typically observed in clinical trials.¹⁶ Furthermore, sample size reassessment based on the total variance has no relevant impact on the type I error rate in parallel group superiority trials¹⁷ (see also the literature^18–21) and achieves the target power well. Similar results on sample size reassessment based on blinded nuisance parameter estimates were obtained for binary data,²² count data^23–25 longitudinal data,²⁶ and for fully sequential sample size reassessment.²⁷ For non-inferiority trials with normal endpoints, a minor inflation of the type I error rate for small sample sizes has been observed.^28,29 However, if the sample size reassessment rule is not only based on the primary endpoint but also on blinded secondary or safety endpoint data, the type I error rate for the test of the primary endpoint may be substantially inflated.³⁰

While for adaptive clinical trials with sample size reassessment based on the unblinded interim treatment effect estimate, it is well known that unadjusted point estimates of the effect size and confidence intervals may be biased;^31,32 the properties of point estimates and confidence intervals computed at the end of an adaptive clinical trial with blinded sample size reassessment have received less attention. In this paper, we investigate the bias and standard error of the treatment effect point estimate and compare the absolute mean bias due to blinded sample size reassessment to upper boundaries derived for adaptive designs with unblinded sample size reassessment.³³ We also investigate the bias of the final variance estimate under sample size reassessment rules that are based on the blinded interim variance estimate. We also investigate the bias of the final variance estimate under sample size reassessment rules that are based on the blinded interim variance estimate. Previously, this bias had been investigated for a corresponding unblinded reassessment rule (which had no upper sample size bound), sharp bounds for the bias had been derived and an additive bias correction had been suggested.⁸ Here, in this paper, we derive corresponding bounds for the bias of the final variance estimate in the blinded case. Furthermore, we assess the coverage probability of one- and two-sided confidence intervals.

In Section 2, we introduce adaptive designs with blinded sample size reassessment. Theoretical results on the bias of estimates are presented in Section 3. In Section 4, we report a simulation study quantifying the bias and coverage probabilities in a variety of scenarios. The impact of the results is discussed in the context of a case study in Section 5. Section 6 concludes the paper with a discussion and recommendations. Technical proofs are given in Appendix 1.

2 Blinded sample size reassessment

Consider a parallel group comparison of the means ${\mu }_a,{\mu }_b$ of a normally distributed endpoint with common unknown variance ${\sigma }^2$. The one-sided null hypothesis $H_0:\delta \le 0$, is tested against $H_1:\delta >0$ at level α, where $\delta ={\mu }_b-{\mu }_a$ denotes the true effect size. Let ${\delta }_0$ denotes the alternative for which the trial is powered and assume that in the planning phase a first stage per group sample size $n_1\ge 2$ is chosen based on an a priori variance estimate ${\sigma }_0^2$. Note, however, that all results below depend only on the chosen n₁ and not the way it has been determined. As a consequence, for given n₁, they also apply if it has been determined with other justifications. After the endpoints of the first stage subjects are observed, in a blinded interim analysis, the one-sample variance estimate is computed, given by

$$S_{1,\mathit{OS}}^2=\frac{1}{2n_1-1}[\sum _{i=a,b}\sum _{k=1}^{n_1}(X_{i1k}-{\overline{X}}_{·1·}{)}^2]$$

where X_ijk is the observation $k=1,\dots ,n_j$ in stage j = 1, 2 for treatment $i=a,b$ and ${\overline{X}}_{·1·}$ is the mean of the pooled first stage samples. Based on $S_{1,\mathit{OS}}^2$, the second stage sample size is chosen with a pre-specified sample size function $n_2(S_{1,\mathit{OS}}^2)$. Below we drop the argument of the function n₂ for convenience if the meaning is clear from the context. Then, the overall sample size is $n=n_1+n_2$ per group. After the study is completed and unblinded, we assume that the point estimates for the mean difference and the variance

$$\overline{\Delta }={\overline{X}}_{b··}-{\overline{X}}_{a··}{S}^2=\frac{1}{2n-2}[\sum _{i=a,b}\sum _{j=1}^2\sum _{k=1}^{n_j}(X_{\mathit{ijk}}-{\overline{X}}_{i··}{)}^2]$$

are computed, where ${\overline{X}}_{i··}=\sum _{j=1}^2\sum _{k=1}^{n_j}{X}_{\mathit{ijk}}/n,i=a,b$. Furthermore, (i) the standard fixed sample lower confidence bound corresponding to the one-sided null hypothesis H₀ given by $\overline{\Delta }-t_{2n-2,1-\alpha }S\sqrt{2/n}$, where $t_{2n-2,1-\alpha }$ denotes the $1-\alpha$ quantile of the central t-distribution with $2n-2$ degrees of freedom, (ii) the upper confidence bound—corresponding to the complementary null hypotheses $H_0^{-}:\delta \ge 0$—given by $\overline{\Delta }+t_{2n-2,1-\alpha }S\sqrt{2/n}$ and (iii) the two-sided confidence interval given by $\overline{\Delta }\pm t_{2n-2,1-\alpha /2}S\sqrt{2/n}$ is determined.

As example consider a sample size reassessment rule aiming to control the power under a pre-specified absolute treatment effect. It is derived from the standard sample size formula for the comparison of means of normally distributed observations with a one-sided z-test using the variance estimate $S_{1,\mathit{OS}}^2$

$$n_2^u(S_{1,\mathit{OS}}^2)=min\{n_{2max},max\{n_{2min},2(z_{1-\alpha }+z_{1-\beta }{)}^2\frac{S_{1,\mathit{OS}}^2}{{\delta }_0^2}-n_1+1\}\}$$ (1)

where $1-\beta$ is the desired power and $n_{2min}$ and $n_{2max}$ are pre-specified minimal and maximal second stage sample sizes with $0\le n_{2min}<n_{2max}$. The +1 in formula (1) was added for mathematical convenience to derive a lower bound for the variance estimate in Theorem 4 below. Given that we anyway have to round the sample size to an integer, this modification is of minor practical importance. While $S_{1,\mathit{OS}}^2$ is an unbiased variance estimator for ${\sigma }^2$ when the true effect size is zero, it is positively biased by ${\delta }^2{n}_1/(4n_1-2)$ for effect sizes $\delta \neg =0$.⁵ A sample size reassessment rule that is based on an adjusted variance estimate which is unbiased under the effect size δ₀ for which the trial is powered is obtained from equation (1) by replacing $S_{1,\mathit{OS}}^2$ by $S_{1,\mathit{OS}}^2-{\delta }_0^2{n}_1/(4n_1-2)$ and is given by

$$n_2^a(S_{1,\mathit{OS}}^2)=min\{n_{2max},max\{n_{2min},2(z_{1-\alpha }+z_{1-\beta }{)}^2(\frac{S_{1,\mathit{OS}}^2}{{\delta }_0^2}-\frac{n_1}{4n_1-2})-n_1+1\}\}$$ (2)

3 Theoretical results on the bias of the final mean and variance estimators

In this section, we consider adaptive two-stage designs with general sample size reassessment rules where the second stage sample size is given as some non-constant function $n_2:{ℝ}^{+}\to {\mathbb{N}}_0$ of the blinded interim variance estimate $S_{1,\mathit{OS}}^2$. We derive the bias of the final effect size and variance estimators $\overline{\Delta }$ and S².

Theorem 1 —

Consider an adaptive two-stage design with a general sample size reassessment rule $n_2(S_{1,\mathit{OS}}^2)$.

1. The bias $E(\overline{\Delta }-\delta )$ is given by

   $$\begin{array}{c}\multicolumn{1}{c}{{\int }_0^{\infty }{\int }_0^{\infty }\frac{t(2n_1-2)(r(q,\delta -t)-r(q,\delta +t))}{{\sigma }^2[1+r(q,\delta +t)][1+r(q,\delta -t)]}\times }\\ \multicolumn{1}{c}{{\varphi }_{0,\sqrt{2{\sigma }^2/n_1}}(t){\chi }_{2n_1-2}^2(\frac{q(2n_1-2)}{{\sigma }^2})\text{d}t\text{d}q}\end{array}$$

where

$$r(x,y)=n_2(\frac{2(n_1-1)}{2n_1-1}x+\frac{n_1}{2(2n_1-1)}{y}^2)/n_1$$

1. Under the null hypothesis δ = 0, the final mean estimator $\overline{\Delta }$ is unbiased.

2. If $n_2(S_{1,\mathit{OS}}^2)$ is increasing and $\delta >0$ ($\delta <0$) the estimator $\overline{\Delta }$ is negatively (positively) biased, respectively. Furthermore, the absolute bias is symmetric in δ around 0.

3. If the sample size reassessment rule is increasing and ${lim}_{S_{1,\mathit{OS}}^2\to \infty }{n}_2(S_{1,\mathit{OS}}^2)=\infty$, the bias of $\overline{\Delta }$ converges to 0 for $\delta \to \pm \infty$.

Note that the sample size functions $n_2^u,n_2^a$ are increasing such that the property (c) applies. If $n_{2max}=\infty$, $n_2^u,n_2^a$ tend to infinity for $S_{1,\mathit{OS}}^2\to \infty$ such that the property (d) applies.

In Theorem 2, we give an upper bound for the bias of the mean in adaptive two-stage designs with a general sample size reassessment rule $n_2(S_{1,\mathit{OS}}^2)$ such that $n_{2min}\le n_2(S_{1,\mathit{OS}}^2)\le n_{2max}$, for lower and upper bounds where $n_{2min},n_{2max}$, where $n_{2max}$ may be infinite (corresponding to the case of unrestricted sample size reassessment). For general unblinded sample size reassessment rules that may depend on the fully unblinded interim data, the upper bound for the bias is³³

$$(\frac{1}{n_1+n_{2min}}-\frac{1}{n_1+n_{2max}})0.4\sigma \sqrt{2n_1}$$ (3)

and is realized for the sample size reassessment rule that sets $n_2=n_{2min}$ if ${\overline{\Delta }}_1\le \delta$ and $n_2=n_{2max}$ otherwise. This upper bound obviously also applies for the case of blinded sample size reassessment based on the total variance estimate because the set of unblinded sample size reassessment rules for which n₂ may depend on the unblinded interim data in any way, includes all blinded sample size reassessment rules. This bound is not sharp if the sample size depends on $S_{1,\mathit{OS}}^2$, only. In the theorem below, we derive the sample size reassessment rule that maximizes the bias and compute the corresponding maximal bias which is a sharp upper bound for the bias that can occur under any sample reassessment based on sample size rules $n_2(S_{1,\mathit{OS}}^2)$. We show that for increasing effect sizes, the bias approaches the bias of unblinded sample size reassessment.

Theorem 2 —

Consider an adaptive two-stage design with a general sample size reassessment rule $n_2(S_{1,\mathit{OS}}^2)$, such that $0\le n_{2min}\le n_2(S_{1,\mathit{OS}}^2)\le n_{2max}$, where $n_{2max}$ may be infinite. Then,

1. the expected first stage treatment effect estimate conditional on the variance estimate $S_{1,\mathit{OS}}^2$ is given by

   $$\begin{array}{c}\multicolumn{1}{c}{E({\overline{\Delta }}_1|S_{1,\mathit{OS}}^2)={\int }_{-b}^{b}x{\varphi }_{\delta ,\sqrt{2{\sigma }^2/n_1}}(x)}\\ \times {\chi }_{2n_1-2}^2\{[(2n_1-1)S_{1,\mathit{OS}}^2-x^2{n}_1/2]/{\sigma }^2\}\mathit{dx}/K\hfill \end{array}$$

where ${\overline{\Delta }}_1$ denotes the unblinded first stage estimate of the mean treatment effect, and $b=\sqrt{2S_{1,\mathit{OS}}^2(2n_1-1)/n_1},K={\int }_{-b}^b{\varphi }_{\delta ,\sqrt{2{\sigma }^2/n_1}}(x){\chi }_{2n_1-2}^2\{[(2n_1-1)S_{1,\mathit{OS}}^2-x^2{n}_1/2]/{\sigma }^2\}\mathit{dx}$

1. the bias of $\overline{\Delta }$ is maximized for the sample size rule

   $$\begin{array}{c}\multicolumn{1}{c}{n_2(S_{1,\mathit{OS}}^2)=\{\begin{array}{cc}{n}_{2min}& \hfill \text{ if }E({\overline{\Delta }}_1|S_{1,\mathit{OS}}^2)>\delta \hfill \\ n_{2max}& \hfill \text{ otherwise }\hfill \end{array}}\end{array}$$ (4)
2. the maximum bias of $\overline{\Delta }$ depends on the true effect size δ and is given by

   $$(\frac{n_1}{n_1+n_{2min}}-\frac{n_1}{n_1+n_{2max}})E(E({\overline{\Delta }}_1-\delta |S_{1,\mathit{OS}}^2)1_{\{E({\overline{\Delta }}_1|S_{1,\mathit{OS}}^2)>\delta \}})$$ (5)

3. For $\delta \to \infty$ and σ fixed, the maximum bias of $\overline{\Delta }$ caused by blinded sample size reassessment rules based on $S_{1,\mathit{OS}}^2$ converges to the maximum bias caused by unblinded sample size reassessment rules given by equation (3).

Note that, by symmetry, the negative bias of $\overline{\Delta }$ is maximized by the sample size rule equation (4) and is given by equation (5) with the inequality signs reversed. If $n_{2min}=0$ and $n_{2max}=\infty$, the first factor in equation (5) reduces to 1.

In Theorem 3, we derive the bias of the variance estimator computed at the end of the trial for general sample size reassessment rules $n_2(S_{1,\mathit{OS}}^2)$ and show that it is symmetric in the true treatment effect δ and converges to 0 for increasing δ under suitable conditions (which are satisfied for $n_2^u,n_2^a$ if $n_{2max}=\infty$).

Theorem 3 —

Consider an adaptive two-stage design with a general sample size reassessment rule $n_2(S_{1,\mathit{OS}}^2)$. Then,

1. the bias of the variance estimator S² is given by

   $$\begin{array}{c}\multicolumn{1}{c}{{\int }_0^{\infty }{\int }_{-\infty }^{\infty }\frac{1}{(n_1-1+n_{1}r(q,t+\delta ))}((n_1-1)(q-{\sigma }^2)+\frac{t^2{n}_1/2-{\sigma }^2}{2+2r(q,t+\delta )})}\\ \times {\varphi }_{0,\sqrt{2{\sigma }^2/n_1}}(t){\chi }_{2n_1-2}^2(q\gamma )\gamma \text{d}t\text{d}q\hfill \end{array}$$

where $\gamma =(2n_1-2)/{\sigma }^2$ and r is defined in Theorem 1;

1. the bias of S² is symmetric in δ around 0;

2. if ${lim}_{S_{1,\mathit{OS}}^2\to \infty }{n}_2(S_{1,\mathit{OS}}^2)=\infty$, the bias of S² converges to 0 for $\delta \to \pm \infty$.

In Theorem 4, we derive a lower bound for the bias of the variance estimator S² (which is negative) in an adaptive two-stage design with sample size reassessment rule $n_2^u$.

Theorem 4 —

Consider a design with blinded sample size reassessment based on $S_{1,\mathit{OS}}^2$, where the second stage sample size is set to $n_2^u(S_{1,\mathit{OS}}^2)$. Under the null hypothesis δ = 0, the bias of the variance estimator S² at the end of the study has the following bounds

$$-\frac{2n_1-1}{2n_1-3}·\frac{1}{v}<{\mathit{ES}}^2-{\sigma }^2<0$$

where $v=2(z_{1-\alpha }+z_{1-\beta }{)}^2/{\delta }_0^2$. If $n_{2max}=\infty$ (no upper restriction for the sample size), the bias converges to the lower bound for large variances

$${\mathit{ES}}^2-{\sigma }^2\to -\frac{2n_1-1}{2n_1-3}·\frac{1}{v}\text{ for }{\sigma }^2\to +\infty$$

Note that this lower bound has a similar form as the bound derived for the bias in adaptive two-stage designs with sample size reassessment based on the unblinded variance estimate.⁸ In the latter setting, the bound is given by $-(n_1-1)/[(n_1-2)v]$. The bound derived in Theorem 4 does not apply to designs where the adjusted rule $n_2^a$ is applied. The numerical results below suggest that for the adjusted rule the absolute bias is maximized for a finite ${\sigma }^2$ and not for ${\sigma }^2\to \infty$ as for the unadjusted rule.

4 Simulation study of the properties of point estimates and confidence intervals

To quantify the bias of mean and variance estimates as well as the actual coverage of corresponding one- and two-sided confidence intervals in adaptive two-stage designs with blinded sample size reassessment, a simulation study was performed. We assumed that a trial is planned for an effect size ${\delta }_0=1$ and a priori variance estimate σ₀ at a one-sided significance level $\alpha =0.025$. The preplanned sample size of a two-armed parallel group trial was chosen to provide 80% power (based on the normal approximation) for ${\sigma }_0\in \{1,1.5,2\}$. The first stage sample size was set to half of the preplanned total sample size leading to first stage per group sample sizes of $n_1\in \{8,18,32\}$. For each first stage sample size, first stage data were simulated from normal distributions for effect sizes δ ranging from −2 to 2 (in steps of 0.1) and common standard deviations σ, ranging from 0.5 to 2 in steps of 0.5. The second stage sample sizes were then reassessed to $n_2^u,n_2^a$ (with $n_{2min}=0$). Below we refer to the latter reassessment rules as the unadjusted and adjusted sample size reassessment rules. For each scenario $5\times {10}^7$, trials were simulated with R.³⁴ The source code is available in the R-package blindConfidence which can be downloaded and installed from github. Details on the implementation and additional results are presented in the Supplementary Material. Figures 1 and 2 show the mean bias of the final estimates of the mean and variance based on the total sample. If the alternative holds, the estimates may be substantially biased for small first stage sample sizes.

Figure 1.

Bias of the mean under blinded sample size reassessment using the unadjusted (solid line) and adjusted (dashed line) interim variance estimate. The dotted lines show maximum negative and positive bias that can be attained under any blinded sample size reassessment rule according to Theorem 2. The dashed gray line shows the maximum bias that can be attained under any (unblinded) sample size reassessment rule. The treatment effect used for planning is set to ${\delta }_0=1$. Rows refer to the a priori assumed standard deviations σ₀ determining the first stage sample size n₁. The columns correspond to actual standard deviations. The x-axis in each graph denotes the true treatment effect δ, the y-axis shows the bias.

Figure 2.

Bias of the variance under blinded sample size reassessment using the unadjusted (solid line) and adjusted (dashed line) interim variance estimate. The gray line gives the lower bound from Theorem 4 for the bias under sample size reassessment based on the unadjusted variance estimate. The treatment effect used for planning is set to ${\delta }_0=1$. Rows refer to the a priori assumed standard deviations σ₀ determining the first stage sample size n₁. The columns correspond to actual standard deviations. The x-axis in each graph denotes the true treatment effect δ, the y-axis shows the bias.

For both sample size reassessment rules, the bias of the mean is zero under the null hypothesis and has the opposite sign as the true effect size, otherwise (in accordance with Theorem 1). For very large positive and negative effect sizes, the bias is close to zero. This is due to the fact that a large effect size results in a large positive bias of the blinded interim variance estimates which in turn results in very large second stage sample sizes. As a consequence, the overall estimates are essentially equal to the (unbiased) second stage estimates, and the bias becomes negligible. In the considered scenarios, the absolute bias is decreasing in the first stage sample size. This also holds for the upper bound of the bias for general sample size reassessment rules. However, while the bias for the adjusted and unadjusted reassessment rules is small for larger sample sizes, the upper bound for general sample size reassessment rules still exceeds 10% in the considered scenarios even in the case where $n_1=32$ subjects per group are recruited in the first stage. In the considered scenarios, we observe that the upper bound for the bias of the mean estimate increases in the true effect size δ and the true standard deviation σ. The numerical results confirm that for increasing $\left|\delta \right|$ the maximum bias under blinded sample size reassessment approaches to the maximum bias under unblinded sample size reassessment as shown in Theorem 2.

The variance estimate is negatively biased for both considered sample size reassessment rules. For increasing variances, the absolute bias under the null hypothesis δ = 0 for the rule $n_2^u$ approaches the lower bound derived in Theorem 4. For large positive and negative effect sizes, the bias approaches zero, again, because the overall estimate becomes essentially equal to the (unbiased) second stage estimates. In general, we observe that using the adjusted interim variance estimate for sample size reassessment results in a larger absolute bias of mean and variance estimates.

The difference of the coverage probabilities of the confidence intervals to the nominal coverage probabilities (i.e., 0.975 for the one-sided, 0.95 for the two-sided confidence interval) is shown in Figure 3 for the unadjusted sample size reassessment rule (see Figure 1 of the Supplementary Material for corresponding results for the adjusted sample size reassessment rule). For positive δ, the lower confidence bound is conservative (with coverage probability larger than 0.975) while the upper bound is anti-conservative, and vice-versa for negative δ. In fact, due to symmetry, the coverage probability for the upper confidence bound at a certain true effect size δ is the same as the coverage probability of the lower confidence bound at $-\delta$. If the true ${\sigma }^2$ and the true $\left|\delta \right|$ are small, the trial will stop with minimal sample size almost with probability 1. Then, sample size is essentially fixed, no bias occurs and coverage is as desired. This is, e.g., the case for $\sigma =1,n_1=32$ and $\left|\delta \right|<1.3$ and explains the exact coverage there. Similar cases occur for small $\left|\delta \right|$ in the three pictures for $\sigma =0.5$ as well.

Figure 3.

Difference between actual and nominal coverage probabilities in percentage points under blinded sample size reassessment using the unadjusted interim variance estimate $S_{1,\mathit{OS}}$: upper confidence bound (dashed line), lower confidence bound (dotted line) and two-sided interval (solid line). Rows refer to the a priori assumed standard deviations σ₀ determining the first stage sample size n₁. The columns correspond to actual standard deviations. The x-axis in each graph denotes the true treatment effect δ; the y-axis shows the difference of actual to nominal coverage probability such that negative values indicate settings where the confidence bound is not valid.

The two-sided coverage probability (which is given by one minus the sum of the non-coverage probabilities of the lower and upper bounds) is not controlled over a large range of δ. However, for δ = 0 (i.e., under the null hypothesis), the inflation of the non-coverage probability, which corresponds to the Type I error rate of the corresponding test, is minor for small sample sizes (0.5 to 0.01 percentage points $n_1<10$) and essentially controlled for larger samples sizes. (See Figure 15 of the Supplementary Material for a simulation study that aims to identify the σ where the Type I error rate of the original hypothesis test is maximized.) The good coverage under the null hypothesis is at first sight surprising, given that the variance estimate is negatively biased and the mean estimate is unbiased. We therefore computed the actual variance ${\sigma }_e^2$ of the mean estimate in the simulation study. To compare the actual variance of the mean to the estimated variance, we computed for each simulated trial the variance estimate ${\hat{S}}_e^2=\frac{2S^2}{n_1+n_2}$ as well as the actual variance ${\sigma }_f^2=\frac{2{\sigma }^2}{n_1+n_2}$ of a design with fixed sample sizes n₁, n₂ (thus, ignoring that n₂ is dependent on the first stage sample). Both quantities were then averaged over the Monte-Carlo samples. The results show that the true variance of the mean estimate of an adaptive design with blinded sample size reassessment is smaller than the average variance of a fixed sample design with the same sample sizes. The bias of the estimate of the variance of the mean estimate is close to zero around the null hypothesis. This holds both for designs using the adjusted and unadjusted interim variance estimates to reassess the sample size and gives some intuition why there is no relevant inflation of the coverage probability of confidence intervals under the null hypotheses, even though the variance estimate has a considerable negative bias (see Figures 5 and 6 of the Supplementary Material for results that show the variance of the mean estimate for designs that use the unadjusted and adjusted sample size rule).

While Figures 1 to 3 demonstrate the dependence of the bias and coverage probabilities on the true mean and standard deviation, in an actual trial these parameters are typically unknown. Therefore, we computed the maximum absolute bias of the mean, variance and the maximum difference between nominal and actual coverage probabilities for fixed first stage sample sizes over a range of values for the true mean and variance (see Figure 4). The optimization was performed in several steps based on simulations across an increasingly finer grid of true mean differences $\delta \in [0,4]$ and standard deviations $\sigma \in [.5,4]$ and using increasingly larger numbers of up to 10⁸ simulation runs (see Section 2 of the Supplementary Material for computational details of this simulation).

Figure 4.

Maximum absolute mean and variance bias as well as maximum negative difference between actual and nominal coverage probabilities (in percentage points) for given per group first stage sample sizes n₁ between 2 and 50; values of δ between 0 and 4; σ between 0.5 and 4. Left columns show the results for the unadjusted sample size reassessment rule, right columns for the adjusted sample size reassessment rule. The first row shows the value for bias and non-coverage, the second shows the effect size δ and the third shows the standard deviation σ at which the specific value is attained. The gray line denotes a Loess smoothed estimate. For the maximum absolute variance bias, based on the unadjusted rule, the theoretical bound is shown using the slightly thicker gray line.

Overall, the maximum bias of the mean and variance estimates as well as the maximum difference between nominal and actual coverage probabilities using the adjusted sample size reassessment rule exceed that of the unadjusted rule. The maximum absolute bias of the mean bias (Columns 1 and 2 of Figure 4) drops from 0.18 (0.08) for trials with a per group first stage sample size of $n_1=2$ to 0.02 (0.02) for trials with a per group first stage sample size of $n_1=50$ if the adjusted (unadjusted) interim variance estimate is used to reassess the sample size. For increasing values of the first stage sample size, the maximum of the absolute mean bias is found for larger values of the true mean and variance.

For the unadjusted rule, the simulations suggest that the absolute variance bias is maximized for δ = 0 and $\sigma \to \infty$ (see Figure 2 and the additional simulations in Section 2 of the Supplementary Material). Thus, Column 3 of Figure 4 shows the absolute bias for δ = 0 and σ = 4 (the largest value of σ considered in the parameter grid). For the adjusted rule, Column 4 of Figure 4 shows the maximum absolute bias for δ = 0 and σ maximized over the grid. For the adjusted rule, the maximum bias is attained for a finite σ. The maximum absolute bias of the variance estimate drops from 0.27 (0.19) for trials with a per group first stage sample size of $n_1=2$ to 0.07 (0.06) for trials with a per group first stage sample size of $n_1=50$ if the adjusted (unadjusted) interim variance estimate is used to reassess the sample size. In the simulation, the maxima of the absolute variance bias, using the unadjusted reassessment rule, is practically identical to the theoretical boundary derived in Theorem 4 (see the gray line in the first graph of Column 3 of Figure 4). If the adjusted rule is used, the maximum is attained for values of σ ranging from 1 to 2, increasing with the first stage sample size.

The maximum negative difference between actual and nominal coverage probabilities (“AC-NC,” in Columns 5 and 6 of Figure 4) of the one-sided confidence intervals ranges from about 2.8 (2.4) to 0.4 (0.3) percentage points if the adjusted (unadjusted) variance estimate is used to reassess the sample size. For the two-sided confidence intervals, differences between actual and nominal coverage are slightly lower ranging from 2.4 (2) to 0.2 (0.2) percentage points, respectively. For increasing first stage sample sizes, the maximum is attained for increasing values of both σ and δ where the former ranges between 1 and 3 and the latter between 0.5 and 1.5.

In Section 4 of the Supplementary Material, the corresponding results for restricted sample size rules that limit the maximum second stage sample size to twice the preplanned second stage sample size, are given. We observe that for scenarios where ${\sigma }_0<\sigma$, limiting the second stage sample size results in a reduction of the (absolute) bias and higher coverage probabilities. This, however, comes at the price of limited control in terms of power. When ${\sigma }_0>\sigma$ we observe that the (absolute) bias of the mean estimate (in line with the theoretical results in Theorem 2) is reduced by a factor $n_{2max}/(n_1+n_{2max})$. The bias of the variance estimate as well as the difference between actual and nominal coverage probabilities are less affected by restricting the second stage sample size. When σ is substantially smaller than σ₀ bias and coverage probabilities under the restricted and unrestricted sample size reassessment rules are nearly identical.

5 Case study

We illustrate the estimation of treatment effects with a randomized placebo controlled trial to demonstrate the efficacy of the kava-kava special extract “WS® 1490” for the treatment of anxiety³⁵ that was used to illustrate procedures for blinded sample size adjustment.¹⁷ The primary endpoint of the study was the change in the Hamilton Anxiety Scale (HAMA) between baseline and end of treatment. Assuming a mean difference between treatment and control of ${\delta }_0=5.5$ and a standard deviation of ${\sigma }_0=8$ (i.e., a variance of ${\sigma }_0^2=64$) results in a sample size of 34 patients per group to provide a power of 80%.

As in the previous case study,¹⁷ we assume that sample size reassessment based on blinded data is performed after 15 patients per group, that the sample size reassessment rules were applied with $n_{2min}=0$ and $n_{2max}=\infty$ and consider that the interim estimate of the standard deviation is $S_{1,\mathit{OS}}=6$. Consequently, we get $n_2^u=4.7$ and $n_2^a=0.6$, such that in the second stage five patients per group would have been recruited based on the unadjusted sample size reassessment rule $n_2^u$, and only one patient per group based the adjusted rule $n_2^a$.

To estimate the bias of the mean and variance estimate as well as the coverage probabilities of confidence intervals, we performed a simulation study for true effect sizes δ ranging from −11 $(=-2{\delta }_0)$ to 11 $(=2{\delta }_0)$ in steps of 0.05, and σ from 1 to 20 in steps of 1 (see Figure 5).

Figure 5.

Coverage probabilities and bias of mean and variance estimates for the case study. The first row of panels shows actual-nominal coverage probabilities (AC-NC) for the 97.5% upper (dashed line), lower (dotted line) and the 95% two-sided confidence intervals, for the unadjusted reassessment rule $n_2^u$. The second row shows the bias of the mean estimate if $n_2^u$ (solid line) or $n_2^a$ (dashed line) is used. The dotted line shows upper and lower bounds for the bias for general blinded sample size reassessment rules based on $S_{1,\mathit{OS}}^2$. The dashed gray line shows the bounds for the bias for a general unblinded sample size reassessment rule. The third panel shows the bias of the variance estimate if either $n_2^u$ (solid line) or $n_2^a$ (dashed line) is used. The red line shows the theoretical boundary for the bias give in Theorem 4.

5.1 Unadjusted sample size rule

For fixed δ, the absolute bias of the variance estimate is increasing in σ. For σ = 20, the bias of the variance becomes $-2.06$ which is within simulation error of the theoretical lower bound. The absolute mean bias takes its maximum (minimum) 0.2 (−0.2) for the effect sizes $\delta =-7.98$ $(7.98)$, respectively, and a standard deviation of $\sigma =5$. The maximum inflation of the non-coverage probabilities of one-sided 97.5% and two-sided 95% confidence intervals is 0.7 and 0.5 percentage points, respectively. The actual coverage probabilities are smallest for large absolute values of the true mean difference and standard deviations of around 5.

5.2 Adjusted sample size rule

If the sample size reassessment is based on the adjusted variance estimate, the absolute bias of the variance and mean estimate will be even larger taking values up to 2.49 for the variance and up to 0.25 for the mean, respectively. The inflation of the non-coverage probabilities goes up to 0.9 percentage points for the one-sided confidence intervals and 0.6 percentage points for the two-sided intervals.

6 Discussion

We investigated the properties of point estimates and confidence intervals in adaptive two-stage clinical trials where the sample size is reassessed based on blinded interim variance estimates. Such adaptive designs are in accordance with current regulatory guidance that proposes blinded sample size reassessment procedures based on aggregated interim data.^9–11 We showed that such blinded sample size reassessment may lead to biased effect size estimates, biased variance estimates and may have an impact on the coverage probability of confidence intervals. The extent of the biases depends on the specific sample size reassessment rule, the first stage sample size, the true effect size and the variance.

We showed that for the unadjusted and the adjusted sample size reassessment rules that aim to control the power at the pre-specified level, the bias of confidence intervals may be large for very small first stage sample sizes but is small otherwise. Under the null hypothesis, even for first stage sample sizes as low as 16, the confidence intervals do not exhibit a relevant inflation of the non-coverage probability. For positive effect sizes, inflations (even though minor) are observed also for somewhat larger sample sizes. This corresponds to previous findings that the type I error rate of superiority as well as non-inferiority tests is hardly affected by blinded sample size reassessment.^17,28 In addition, we show that for positive treatment effects the lower confidence interval (which is often the most relevant because it gives a lower bound of the treatment effect) is strictly conservative in all considered simulation scenarios. The upper bound in contrast shows an inflation of the non-coverage probability, which is however only of relevant size if sample sizes are small.

An approach to obtain conservative confidence intervals even for trials with small first stage sample sizes is to apply methods proposed for flexible designs with unblinded interim analyses^{32,33,36–39} which are based on combination tests or the conditional error rate principle. Such approaches allow to construct confidence intervals which control the coverage probabilities. However, they are based on test statistics that do not equally weigh outcomes from patients recruited in the first and second stage. A hybrid approach that strictly controls the coverage probability could be based on confidence intervals that are the union of the fixed sample confidence interval and the confidence interval obtained from the flexible design methodology. Similar approaches have been proposed for hypothesis testing in adaptive designs.^40,41

We demonstrated that the treatment effect estimates are negatively (positively) biased under the adjusted and unadjusted sample size reassessment rules for all positive (negative) effect sizes. The bias decreases with the first stage sample size but for worst case parameter constellations it remains noticeable also for larger sample sizes. In addition, for the worst case sample size reassessment rule based on blinded variance estimates, the bias may be substantial even for large first stage sample sizes: As the effect size increases the maximum bias of the blinded sample size reassessment rule approaches the bias of unblinded sample size reassessment. As a consequence to maintain the integrity of confirmatory clinical trials with blinded sample size reassessment, binding sample size adaptation rules must be pre-specified and it is important to verify on a case-by-case basis that the reassessment rules used do not have a substantial impact on the properties of estimators should be used.

Appendix 1: Proofs of the Theorems

Note that the blinded first stage variance estimate can be written as¹⁷

$$S_{1,\mathit{OS}}^2=\frac{2(n_1-1)}{2n_1-1}{S}_1^2+\frac{n_1}{2(2n_1-1)}{\overline{\Delta }}_1^2$$ (6)

where ${\overline{\Delta }}_1$ denotes the unblinded first stage estimates of the mean treatment effect and $S_1^2$ the (unblinded) pooled variance estimate. Thus, every second stage sample size rule $n_2(S_{1,\mathit{OS}}^2)$ can be written as function $n_2(S_1^2,{\overline{\Delta }}_1)$ of $S_1^2$ and ${\overline{\Delta }}_1$ which is symmetric in ${\overline{\Delta }}_1$ around 0. We can then write $r(x,y)=n_2(x,y)/n_1.$ Below we switch between both parameterizations of n₂.

Proof of Theorem 1

Proof —

(a) Let the true effect size δ be fixed. The bias of $\overline{\Delta }$ is given by $\mathit{Cov}(\frac{1}{1+r(S_1^2,{\overline{\Delta }}_1)},{\overline{\Delta }}_1)$, see Brannath et al.,³³ which can be computed by

$$\begin{array}{c}\multicolumn{1}{c}{E(\frac{{\overline{\Delta }}_1}{1+r(S_1^2,{\overline{\Delta }}_1)})-E(\frac{1}{1+r(S_1^2,{\overline{\Delta }}_1)})E({\overline{\Delta }}_1)=E(\frac{{\overline{\Delta }}_1-\delta }{1+r(S_1^2,{\overline{\Delta }}_1)})}\\ \multicolumn{1}{c}{={\int }_0^{\infty }{\int }_{-\infty }^{\infty }\frac{t}{1+r(q,\delta +t)}{\varphi }_{0,\sqrt{2{\sigma }^2/n_1}}(t){\chi }_{2n_1-2}^2(q\gamma )\gamma \text{d}t\text{d}q}\\ \multicolumn{1}{c}{={\int }_0^{\infty }{\int }_0^{\infty }t\frac{r(q,\delta -t)-r(q,\delta +t)}{[1+r(q,\delta +t)][1+r(q,\delta -t)]}{\varphi }_{0,\sqrt{2{\sigma }^2/n_1}}(t){\chi }_{2n_1-2}^2(q\gamma )\gamma \text{d}t\text{d}q}\end{array}$$ (7)

where $\gamma =(2n_1-2)/{\sigma }^2$.

(b) and (c) The sign of the integral is determined by the sign of $n_2(q,t+\delta )-n_2(q,-t+\delta )$. Note that r(q, y) is symmetric in y around 0 and increasing for y > 0 since this is true for $n_2(q,y)$. Thus, the integral is zero for δ = 0, negative for positive δ and positive for negative δ. Also the symmetry of the absolute bias follows by the symmetry of $n_2(q,y)$ in y around 0. (d) The first factor in equation (7) tends to 0, and the proof follows with the dominated convergence theorem. □

Proof of Theorem 2

Proof —

(a) By equation (6), the joint density of ${\overline{\Delta }}_1$ and $S_{1,\mathit{OS}}^2$ is given by

$$h(x,v)={\varphi }_{\delta ,\sqrt{2{\sigma }^2/n_1}}(x){\chi }_{2n_1-2}^2\{[(2n_1-1)v-x^2{n}_1/2]/{\sigma }^2\}·(2n_1-1)/{\sigma }^2$$

for $v\ge 0$ and $\left|x\right|\le b$ and $h(x,v)=0$ otherwise. Therefore, the conditional distribution of ${\overline{\Delta }}_1$ given $S_{1,\mathit{OS}}^2=v$ is $h(x,v)/{\int }_{-b}^{b}h(x,v)\mathit{dx}$ and (a) follows.

(b) Note that the bias is given by

$$\begin{array}{c}\multicolumn{1}{c}{E(\overline{\Delta }-\delta )=E[({\overline{\Delta }}_1-\delta )\frac{n_1}{n_1+n_2(S_{1,\mathit{OS}}^2)}]}\\ \multicolumn{1}{c}{=E\{E[({\overline{\Delta }}_1-\delta )\frac{n_1}{n_1+n_2(S_{1,\mathit{OS}}^2)}|S_{1,\mathit{OS}}^2]\}}\end{array}$$ (8)

$$=E\{E[E(({\overline{\Delta }}_1-\delta )\left|S_{1,\mathit{OS}}^2)\frac{n_1}{n_1+n_2(S_{1,\mathit{OS}}^2)}\right|S_{1,\mathit{OS}}^2]\}$$ (9)

For fixed $S_{1,\mathit{OS}}^2$, the term within the outer conditional expectation in equation (8) is deterministic and decreases in n₂ if $E({\overline{\Delta }}_1|S_{1,\mathit{OS}}^2)>\delta$ and increases if $E({\overline{\Delta }}_1|S_{1,\mathit{OS}}^2)<\delta$. Therefore, the sample size rule equation (4) maximizes the bias.

(c) We plug-in the worst case sample size rule equation (4) into equation (9) and obtain

$$\begin{array}{c}\multicolumn{1}{c}{E(\overline{\Delta }-\delta )=E[\frac{n_1}{n_1+n_{2min}}E({\overline{\Delta }}_1-\delta |S_{1,\mathit{OS}}^2)1_{\{E({\overline{\Delta }}_1|S_{1,\mathit{OS}}^2)>\delta \}}]}\\ \multicolumn{1}{c}{+E[\frac{n_1}{n_1+n_{2max}}E({\overline{\Delta }}_1-\delta |S_{1,\mathit{OS}}^2)1_{\{E({\overline{\Delta }}_1|S_{1,\mathit{OS}}^2)\le \delta \}}]}\end{array}$$

Because ${\overline{\Delta }}_1$ is unbiased such that $E({\overline{\Delta }}_1-\delta |S_{1,\mathit{OS}}^2)1_{\{E({\overline{\Delta }}_1|S_{1,\mathit{OS}}^2)>\delta \}}=-E({\overline{\Delta }}_1-\delta |S_{1,\mathit{OS}}^2)1_{\{E({\overline{\Delta }}_1|S_{1,\mathit{OS}}^2)\le \delta \}}$ we get equation (5).

(d) Let $\mathcal{N}$ denotes a set of sample size reassessment rules $n_{2,\delta }(S_{1,\mathit{OS}}^2,{\overline{\Delta }}_1)$ that may depend on $\delta ,S_{1,\mathit{OS}}^2,{\overline{\Delta }}_1$. Given a sample size rule $n_{2,\delta }(·)\in \mathcal{N}$, the bias can be written as

$$\begin{array}{c}\multicolumn{1}{c}{E_{\delta }(\overline{\Delta }-\delta )=E_{\delta }(({\overline{\Delta }}_1-\delta )\frac{n_1}{n_1+n_{2,\delta }(S_{1,\mathit{OS}}^2,{\overline{\Delta }}_1)})}\\ \multicolumn{1}{c}{={\int }_{-\infty }^{\infty }{\int }_0^{\infty }y\frac{n_1}{n_1+n_{2,\delta }(v,y+\delta )}{\varphi }_{0,\sigma /\sqrt{n_1}}(y)}\\ \times {\chi }_{2n_1-2}^2\{[(2n_1-1)v-(y+\delta {)}^2{n}_1/2]/{\sigma }^2\}\text{d}v\text{d}y\hfill \end{array}$$ (10)

The set of integrands of the outer integral in equation (10) for second stage sample size rules in $\mathcal{N}$ is uniformly integrable because the fraction $n_1/[n_1+n_{2,\delta }(y+\delta ,v)]\le 1$ and the ${\chi }^2$-density is bounded (since $n_1\ge 2$).

The unblinded sample size reassessment rule that maximizes the bias for given δ is³³

$$\begin{array}{c}\multicolumn{1}{c}{n_{2,\delta }({\overline{\Delta }}_1)=\{\begin{array}{cc}{n}_{2min}& \hfill \text{ if }{\overline{\Delta }}_1>\delta \hfill \\ n_{2max}& \hfill \text{ otherwise }\hfill \end{array}}\end{array}$$

Consider the blinded sample size rules

$$\begin{array}{c}\multicolumn{1}{c}{{\stackrel{\sim }{n}}_{2,\delta }(S_{1,\mathit{OS}}^2)=\{\begin{array}{cc}{n}_{2min}& \hfill \text{ if }{S}_{1,\mathit{OS}}^2>{\delta }^2\frac{n_1}{2(2n_1-1)}\hfill \\ n_{2max}& \hfill \text{ otherwise }\hfill \end{array}}\end{array}$$

We show that for fixed σ and $\delta \to \infty$ the second stage sample sizes of the blinded sample size rule ${\stackrel{\sim }{n}}_2(S_{1,\mathit{OS}}^2)\to n_2({\overline{\Delta }}_1)$, almost surely. Because, as shown above, the corresponding integrands in equation (10) are uniformly integrable, and it follows that also the respective biases converge and (d) follows.

Let $\epsilon >0$. Then, there exists an ${\epsilon }_1,1/3>{\epsilon }_1>0$, such that for all δ we have $P_{\delta }(|{\overline{\Delta }}_1-\delta |>3{\epsilon }_1)>1-\epsilon$. Furthermore, there exists a ${\delta }_0>2$ such that for all $\delta >{\delta }_0$ we have $P_{\delta }(S_1^2/\delta <{\epsilon }_1)>1-\epsilon$ and $P_{\delta }({\overline{\Delta }}_1>0)>1-\epsilon$. Thus, for the event $A=\{{\overline{\Delta }}_1>0\}\cap \{S_1^2/\delta <{\epsilon }_1\}\cap \{|{\overline{\Delta }}_1-\delta |>3{\epsilon }_1\}$ we have $P_{\delta }(A)\ge 1-3\epsilon$. However, on the set A we have $n_{2,\delta }({\overline{\Delta }}_1)={\stackrel{\sim }{n}}_{2,\delta }(S_{1,\mathit{OS}}^2)$: If $n_{2,\delta }({\overline{\Delta }}_1)=n_{2min}$ then ${\overline{\Delta }}_1>\delta$ and on A also ${\overline{\Delta }}_1>\delta +3{\epsilon }_1$. Therefore, by equation (6)

$$S_{1,\mathit{OS}}^2>{\delta }^2(1+6{\epsilon }_1/\delta +9{\epsilon }_1^2/{\delta }^2)\frac{n_1}{2(2n_1-1)}>{\delta }^2\frac{n_1}{2(2n_1-1)}$$

and ${\stackrel{\sim }{n}}_{2,\delta }(S_{1,\mathit{OS}}^2)=n_{2min}$. On the other hand, if $n_{2,\delta }({\overline{\Delta }}_1)=n_{2max}$ then ${\overline{\Delta }}_1<\delta$ and on A also $0\le {\overline{\Delta }}_1<\delta -3{\epsilon }_1$. Therefore, by equation (6), because $2(n_1-1)/(2n_1-1)\le 1$ and $1/3\ge n_1/2(2n_1-1)\ge 1/4$ and on A we have $S_1^2\le {\epsilon }_1\delta$

$$\begin{array}{c}\multicolumn{1}{c}{S_{1,\mathit{OS}}^2<{\epsilon }_1\delta +\frac{n_1}{2(2n_1-1)}({\delta }^2-6{\epsilon }_1\delta +9{\epsilon }_1^2)}\\ \multicolumn{1}{c}{\le {\delta }^2\frac{n_1}{2(2n_1-1)}+{\epsilon }_1\delta -\frac{3}{2}{\epsilon }_1\delta +3{\epsilon }_1^2}\\ \multicolumn{1}{c}{\le {\delta }^2\frac{n_1}{2(2n_1-1)}-\frac{{\epsilon }_1\delta }{2}+3{\epsilon }_1^2\le {\delta }^2\frac{n_1}{2(2n_1-1)}}\end{array}$$

where the last inequality follows because $\delta >2$ and ${\epsilon }_1<1/3$. □

Proof of Theorem 3

Proof —

Let ${\overline{\Delta }}_j={\overline{X}}_{\mathit{aj}·}-{\overline{X}}_{\mathit{bj}·}$ be the observed mean difference between the treatments in stage j = 1 and (if $n_2>0$) in stage j = 2.

We consider first the case that $n_2>0$; the following considerations are under the condition of $S_1^2$ and ${\overline{\Delta }}_1$ such that they lead to $n_2>0$. Let ${\Omega }_i={\overline{X}}_{i1·}-{\overline{X}}_{i2·}$ be the observed mean difference between the stages for treatment $i=a,b$. Let further $S_2^2$ be the (unblinded) pooled variance estimate of the second stage (for $n_2=1$, we define $S_2^2=0$), ${\Omega }_{+}={\Omega }_a+{\Omega }_b=2({\overline{X}}_{·1·}-{\overline{X}}_{·2·})$ be twice of the overall difference between the stages and ${\Omega }_{-}={\Omega }_a-{\Omega }_b={\overline{\Delta }}_1-{\overline{\Delta }}_2$. With this notation, we can split the total variance estimate S² into parts

$$S^2=\frac{2n_1-2}{2n-2}{S}_1^2+\frac{2n_2-2}{2n-2}{S}_2^2+\frac{n_1{n}_2}{2n(2n-2)}{\Omega }_{+}^2+\frac{n_1{n}_2}{2n(2n-2)}{\Omega }_{-}^2$$ (11)

Obviously, $E(S_2^2|n)=E(S_2^2)={\sigma }^2$ for all n. As ${\overline{X}}_{·1·},S_1^2$ and ${\overline{\Delta }}_1$ are mutually independent (and consequently n and ${\overline{X}}_{·1·}$), and as ${\overline{X}}_{·1·}$ and ${\overline{X}}_{·2·}$ are independent under the condition of n, ${\Omega }_{+}$ is conditionally on n distributed as $N(0,2{\sigma }^2(1/n_1+1/n_2))$.

Under the condition of n, ${\overline{\Delta }}_2$ is distributed as $N(\delta ,2{\sigma }^2/n_2)$. Further, under the condition of $S_1^2$ and ${\overline{\Delta }}_1,\frac{n_2}{2{\sigma }^2}({\overline{\Delta }}_1-{\overline{\Delta }}_2{)}^2=\frac{n_2}{2{\sigma }^2}{\Omega }_{-}^2$ has a non-central ${\chi }^2$ distribution with 1 degree of freedom and non-centrality parameter $\frac{n_2({\overline{\Delta }}_1-\delta {)}^2}{2{\sigma }^2}$. Therefore, it follows

$$\begin{array}{c}\multicolumn{1}{c}{E(S^2-{\sigma }^2|S_1^2,{\overline{\Delta }}_1)=E[\frac{2n_1-2}{2n-2}(S_1^2-{\sigma }^2)+\frac{2n_2-2}{2n-2}(S_2^2-{\sigma }^2)}\\ +\frac{{\sigma }^2}{2n-2}(\frac{n_1{n}_2}{2n{\sigma }^2}{\Omega }_{+}^2-1)+\frac{{\sigma }^2}{2n-2}(\frac{n_1}{n}\frac{n_2}{2{\sigma }^2}{\Omega }_{-}^2-1)|S_1^2,{\overline{\Delta }}_1]\hfill \\ \multicolumn{1}{c}{=E[\frac{n_1-1}{n-1}(S_1^2-{\sigma }^2)+\frac{{\sigma }^2}{2n-2}(\frac{n_1}{n}\frac{n_2}{2{\sigma }^2}{\Omega }_{-}^2-1)|S_1^2,{\overline{\Delta }}_1]}\\ \multicolumn{1}{c}{=E[\frac{n_1-1}{n-1}(S_1^2-{\sigma }^2)+\frac{{\sigma }^2}{2n-2}(\frac{n_1}{n}+\frac{n_1{n}_2}{2{\sigma }^{2}n}({\overline{\Delta }}_1-\delta {)}^2-1)|S_1^2,{\overline{\Delta }}_1]}\\ \multicolumn{1}{c}{=E[\frac{1}{n-1}((n_1-1)(S_1^2-{\sigma }^2)+\frac{n_2}{2n}(\frac{n_1}{2}({\overline{\Delta }}_1-\delta {)}^2-{\sigma }^2))|S_1^2,{\overline{\Delta }}_1]}\end{array}$$ (12)

For cases $S_1^2$ and ${\overline{\Delta }}_1$ leading to $n_2=0$, we can go directly from the first to the last row in the above derivation.

Using $n=(1+r(S_1^2,{\overline{\Delta }}_1))n_1$ and $n_2/n=(1+r(S_1^2,{\overline{\Delta }}_1){)}^{-1}$, the bias $E(S^2)-{\sigma }^2$ can therefore be written as

$$\begin{array}{c}\multicolumn{1}{c}{{\int }_0^{\infty }{\int }_{-\infty }^{\infty }\frac{1}{(n_1-1+n_{1}r(q,t+\delta ))}((n_1-1)(q-{\sigma }^2)+\frac{t^2{n}_1/2-{\sigma }^2}{2+2r(q,t+\delta )})}\\ \multicolumn{1}{c}{\times {\varphi }_{0,\sqrt{2{\sigma }^2/n_1}}(t){\chi }_{2n_1-2}^2(q\gamma )\gamma \text{d}t\text{d}q}\end{array}$$

1. The integral ${\int }_{-\infty }^{\infty }\dots \text{d}t$ can again as in Theorem 1 be divided into ${\int }_{-\infty }^0\dots \text{d}t$ and ${\int }_0^{\infty }\dots \text{d}t$. By replacing t with −t in the first integral, it can be written as one integral over $[0,\infty ]$ in the same way as in Theorem 1.

2. One can conclude that the bias is symmetric around 0 for δ since $r(q,y)=r(q,-y)$.

3. The asymptotic unbiasedness for $\delta \to \pm \infty$ follows from the integral expression with the dominated convergence theorem. □

Proof of Theorem 4

For the proof of Theorem 4, we show the following lemmas:

Lemma 1 —

Under the null hypothesis δ = 0

$$E(S_1^2|S_{1,\mathit{OS}}^2)=E[\frac{n_1}{2}{\overline{\Delta }}_1^2|S_{1,\mathit{OS}}^2]=S_{1,\mathit{OS}}^2$$

Proof —

$\frac{2n_1-1}{{\sigma }^2}{S}_{1,\mathit{OS}}^2$ is ${\chi }^2$-distributed with $2n_1-1$ degrees of freedom and by equation (6) can be written as sum of $2n_1-1$ independent ${\chi }_1^2$-distributed random variables $Q_j,j=1,\dots ,2n_1-1,$ such that $Q_1=\frac{n_1}{2{\sigma }^2}{\overline{\Delta }}_1^2$ and $\sum _{j=2}^{2n_1-1}{Q}_j=\frac{2n_1-2}{{\sigma }^2}{S}_1^2$. All $Q_1,\dots ,Q_{2n_1-1}$ are identically distributed, and all Q_j appear in the same way in the condition $S_{1,\mathit{OS}}^2=\mathit{const}·\sum _{j=1}^{2n_1-1}{Q}_j$. Therefore, the Q_j is exchangeable in the conditional distribution $E(Q_1|S_{1,\mathit{OS}}^2)$ without changing the value of it: we have $E(Q_1|S_{1,\mathit{OS}}^2)=E(Q_j|S_{1,\mathit{OS}}^2)$ for $j=2,\dots ,2n_1-1$ and

$$\begin{array}{c}\multicolumn{1}{c}{E(S_1^2|S_{1,\mathit{OS}}^2)=\frac{{\sigma }^2}{2n_1-2}\sum _{j=2}^{2n_1-1}E(Q_j|S_{1,\mathit{OS}}^2)={\sigma }^{2}E(Q_1|S_{1,\mathit{OS}}^2)}\\ \multicolumn{1}{c}{=\frac{{\sigma }^2}{2n_1-1}\sum _{j=1}^{2n_1-1}E(Q_j|S_{1,\mathit{OS}}^2)=S_{1,\mathit{OS}}^2}\end{array}$$

□

Lemma 2 —

Let c, d be arbitrary with $0<c<d\le \infty$. For $S_{1,\mathit{OS}}^2$, the following inequality is fulfilled

$$\text{Cov}\{S_{1,\mathit{OS}}^2,\frac{1}{min(max(S_{1,\mathit{OS}}^2,c),d)}\}>\text{Cov}(S_{1,\mathit{OS}}^2,\frac{1}{S_{1,\mathit{OS}}^2})$$

Proof —

Note that a proof for $d=\infty$ and for $S_1^2$ instead of $S_{1,\mathit{OS}}^2$ is in Miller.⁸ To include the possibility of $d<\infty$, we show

$$\begin{array}{c}\multicolumn{1}{c}{\text{Cov}\{S_{1,\mathit{OS}}^2,\frac{1}{min(max(S_{1,\mathit{OS}}^2,c),d)}\}}\\ \multicolumn{1}{c}{=\text{Cov}\{S_{1,\mathit{OS}}^2,max(min(1/S_{1,\mathit{OS}}^2,1/c),1/d)\}}\\ \multicolumn{1}{c}{=\text{Cov}(S_{1,\mathit{OS}}^2,\frac{1}{S_{1,\mathit{OS}}^2})-\text{Cov}\{S_{1,\mathit{OS}}^2,(1/S_{1,\mathit{OS}}^2-1/c)·1_{\{S_{1,\mathit{OS}}^2<c\}}\}}\\ \multicolumn{1}{c}{-\text{Cov}\{S_{1,\mathit{OS}}^2,(1/S_{1,\mathit{OS}}^2-1/d)·1_{\{S_{1,\mathit{OS}}^2>d\}}\}}\end{array}$$

As $(1/x-1/c)·1_{\{x<c\}}$ and $(1/x-1/d)·1_{\{x>d\}}$ are monotonically decreasing functions of x, the second and third covariances above are $\le 0$; moreover, as c > 0, the second covariance is negative.⁴² This completes the proof. □

Proof of Theorem 4 —

From equation (12) in the proof of Theorem 3 follows with Lemma 1 under the null hypothesis δ = 0

$$\begin{array}{c}\multicolumn{1}{c}{E(S^2-{\sigma }^2)=E[E(S^2-{\sigma }^2|S_{1,\mathit{OS}}^2)]}\\ \multicolumn{1}{c}{=E[\frac{n_1-\frac{1}{2}-n_1/(2n)}{n-1}(S_{1,\mathit{OS}}^2-{\sigma }^2)]}\end{array}$$

Since $E(S_{1,\mathit{OS}}^2)={\sigma }^2$ and $\frac{n_1-1/2-n_1/(2n)}{n-1}$ as function of $S_{1,\mathit{OS}}^2$ is a monotonically decreasing weight function, the bias of S² is non-positive. Since there is no constant c such that $P(\frac{n_1-1/2-n_1/(2n)}{n-1}=c)=1$, the bias is negative and the upper bound is shown. To show the lower bound, we calculate (writing $n_{min}=n_1+n_{2min}$ and $n_{max}=n_1+n_{2max}$)

$$E(S^2-{\sigma }^2)$$ (13)

$$=E[\frac{n_1-1/2}{n-1}(S_{1,\mathit{OS}}^2-{\sigma }^2)]-E[\frac{n_1}{2n(n-1)}(S_{1,\mathit{OS}}^2-{\sigma }^2)]$$ (14)

$$>\frac{n_1-1/2}{v}E[\frac{S_{1,\mathit{OS}}^2-{\sigma }^2}{min\{(n_{max}-1)/v,max\{(n_{min}-1)/v,S_{1,\mathit{OS}}^2\}\}}]$$ (15)

$$>\frac{n_1-1/2}{v}E(\frac{S_{1,\mathit{OS}}^2-{\sigma }^2}{S_{1,\mathit{OS}}^2})$$ (16)

$$=\frac{n_1-1/2}{v}\{1-E(\frac{{\sigma }^2}{S_{1,\mathit{OS}}^2})\}=-\frac{2n_1-1}{2n_1-3}·\frac{1}{v}$$ (17)

The last expected value in equation (14) is negative since $1/(n(n-1))$ as function of $S_{1,\mathit{OS}}^2$ is monotonically decreasing; therefore, the total expression becomes smaller when this term is removed in equation (15). Further, we used in equation (15) the sample size formula $n_2=min\{n_{2max},max\{n_{2min},{\mathit{vS}}_{1,\mathit{OS}}^2-n_1+1\}\}$. For the inequality in equation (16), we applied Lemma 2. For the equality in equation (17), we have used $E({\sigma }^2/S_{1,\mathit{OS}}^2)=E\{(2n_1-1)/X\}=(2n_1-1)E(X^{-1})=(2n_1-1)/(2n_1-3)$ where X is a chi-squared distributed random variable with $2n_1-1$ degrees of freedom. The expected value of $X^{-1}$ can be derived, for example, from Johnson et al.,⁴³ p. 421.

When ${\sigma }^2\to \infty$, the last expected value in equation (14) converges to 0 which implies that the difference between the expressions in equations (14) and (15) converges to 0. Further, as the probability for $S_{1,\mathit{OS}}^2>(n_1+n_{2min})/v$ converges to 1 in the case $n_{2max}=\infty$, also the difference between equations (15) and (16) converges to 0. Therefore, the lower bound becomes sharp when ${\sigma }^2\to \infty$. □

We note that in case of using the adjusted sample size rule $n_2^a$ instead of $n_2^u$, then we cannot go in a similar way from equations (15) to (16). Our simulation results in Figure 2 and in the lower panel of Figure 5 show that the bias for the adjusted sample size rule can be below the lower bound of Theorem 4.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Martin Posch and Franz Koenig were supported by Austrian Science Fund Grant P23167. Florian Klinglmueller was supported by Austrian Science Fund Grants Number P23167 and J3721.

Supplementary material

Supplementary figures and additional details on the simulation study are presented in an additional document, which is distributed with the paper as Supplementary Material.

The R code used to compute the results of the simulation, and the case study is available as an R-package named blindConfidence which is available on github or on request from the authors.