A flexible futility monitoring method with time-varying conditional power boundary

Abstract

Background

In an ongoing multi-center randomized control clinical trial, the Carotid Occlusion Surgery Study (COSS), the study protocol specifies multiple interim analyses whose results will be reviewed by an independent DSMB to determine if the trial needs to be stopped early due to either efficacy or futility. Conditional power is used as the decision rule for the DSMB to recommend stopping the trial for futility. An aggressive rule for futility stopping sets a relatively high threshold for the conditional power which may result in significant loss of overall power of the study. A conservative rule using a lower threshold may not be able to stop the trial early when there is indeed no treatment efficacy.

Purpose

The goal of this article is to develop a flexible futility monitoring plan with a time-varying conditional power boundary that maintains the overall power of the study well, but has a better chance to stop the trial earlier for futility compared to a futility stopping rule with a fixed value for the minimum conditional power to continue.

Methods

The conditional power boundary for futility is developed using the β-spending function method for sequential test statistics and assuming no interim analysis for efficacy. It is then modified to account for the repeated interim analyses for efficacy.

Results

Simulation studies that mirror the design of the COSS trial show that the proposed method with sample size calculated without considering interim analyses will maintain the designed size and power well when the designed effect size holds, but will have a better chance to exit the trial earlier if the true effect size is smaller than the designed size such that it is not clinically meaningful to conduct the trial.

Limitations

The method is valid for sequential test statistics that constitute of a stochastic process which approximates the Brownian motion. It is not applicable to the monitored process that behaves quire differently from the Brownian motion.

Conclusions

The proposed conditional power method facilitates a flexible futility monitoring plan that can be easily implemented in long-term clinical trials where multiple interim analyses are required. It provides the DSMB an objective guideline to use in considering early stopping for futility.

Introduction

This article is motivated by the ongoing Carotid Occlusion Surgery Study (COSS), a multi-center randomized controlled clinical trial for which the University of Iowa serves as the data coordinating center. The primary goal of this study is to provide strong scientific evidence that the procedure of extracranial/intracranial (ECIC) bypass by surgical anastomosis of the superficial temporal artery to the middle cerebral artery (STA-MCA) when added to best medical therapy significantly reduces subsequent ipsilateral ischemic stroke (fatal and nonfatal) at 2 years in patients with recently symptomatic internal carotid artery occlusion and Stage II hemodynamic failure. Study participants are randomly assigned to treatment (ECIC bypass surgery plus best medical therapy) and control (best medical therapy only). The primary endpoint of this trial is the 2-year ipsilateral ischemic stroke rate since randomization. The study is designed to detect a difference in the primary endpoint of 40% in the control group and 24% in the surgical group. This represents a clinically meaningful absolute risk reduction of 16% and a relative risk reduction of 40%. In the study design, 372 participants (including possibly 5% drop-out) need to be randomized for the two-sided 5% level test of proportions to have 90% power to detect the designed difference. The sample size calculation did not adjust for the effect of interim efficacy and futility analyses. However, the study protocol specifies that the study will be monitored by an independent DSMB at least yearly and that both efficacy and futility will be assessed. Considering the great cost associated with the ECIC bypass surgery and potential complications after the surgery, it is desirable to have a futility stopping rule, which maintains the overall power well when the designed effect size is true, but is also able to recognize the futility signal as early as possible.

Because the DSMB monitoring times are not fixed, the study proposes to monitor using the flexible error spending method [1] with the O’Brien–Fleming type spending function. This method is particularly useful in this COSS study in two ways. (1) The O’Brien–Fleming type spending function has minimal effect on the overall power of the study so it is not necessary to increase the sample size to account for the multiple interim analyses; (2) the crossing boundary can be determined using readily available packages with information accumulated over time. Although both the efficacy and futility boundaries can be similarly established using group sequential designs [2–5], an example [6] showed that the interim monitoring plans with the group sequential designs would result in a good chance of rejecting the null hypothesis at the final analysis even if the futility boundary is crossed at an interim analysis.

Conditional power (CP) is the probability that the final analysis will result in rejection of null hypothesis given accrued data at interim analysis and an effect size. It is widely accepted as a tool for futility monitoring as it is easily computed by the B-value using the Brownian motion techniques [7]. More importantly, it provides a nice interpretation of the projection to the end of study. Due to this consideration, the conditional power is adopted in the COSS protocol as the tool for futility monitoring.

Let R denote the rejection region, and Z_t the test statistic at information time t, $0<t\le 1$ where Z₁ is the test statistic at the end of study. The trial is stopped for futility (accepting H₀) if $C{P}_t=P\left\{Z_1\in R\mid Z_t,H_a\right\}<\gamma$ (H_a is the alternative hypothesis with the designed effect size) for a pre-specified $0<\gamma <0.5$ [8].

If interim result is monitored continuously, the use of a fixed CP futility stopping rule given by $C{P}_t<\gamma$ has the overall type II error bounded by $\beta /(1-\gamma )$ [9], where β is the type II error without interim futility monitoring. There is numerical evidence [10] that both type I and type II errors can deviate substantially from their nominal levels when an aggressive stopping boundary (bigger γ) for futility is adopted.

Selection of a futility stopping boundary based on the conditional power is frequently a subjective matter depending on the opinion of the members of DSMB and/or the study investigators. For example, investigators often decide to stop a trial for futility if the conditional power with the designed effect size becomes small, such as $C{P}_t<0.1$. This rule is attractive because it is simple and has little influence on the overall type I and type II errors. However, the rule is conservative as it is highly unlikely to recommend for futility early in the trial if the true effect size is smaller than the designed size such that it is not clinically meaningful to conduct the trial. To increase the chance of early stopping for futility, one may increase the value of γ with the trade-off of inflating type II error.

In this article, we propose a flexible CP method for futility monitoring in which the CP boundary is chosen to be a time-varying parameter instead of a fixed value, that is, $C{P}_t<{\gamma }_t$ with γ_t depending on information time t. This design allows the trial to correctly stop early for futility with larger probability than the conservative stopping rule based on the fixed CP method. It also has little effect on the overall type I error and maintains the overall power well. This futility monitoring method has been implemented in the COSS study. The proposed method will be assessed using simulation studies that mimic various possible outcomes in the COSS study. The advantage of this method will be established by comparing the results to those from the fixed CP method.

Methods

As the primary analysis in the COSS study is to compare the rate of the primary endpoint between the two groups, we illustrate the proposed method using the one-sided test of proportions: $H_0:p_1=p_2\text{vs.}{H}_a:p_1>p_2$. The method can be trivially extended to the two-sided test. Assume that a total of 2N subjects are randomly assigned to either control or study treatment with N subjects in each group. Let ${\widehat{p}}_1$ and ${\widehat{p}}_2$ be the sample proportions for the control and treatment groups, respectively. A Z-test statistic given by

$$Z_1=\frac{\sqrt{N}\left({\widehat{p}}_1-{\widehat{p}}_2\right)}{\sqrt{{\widehat{p}}_1\left(1-{\widehat{p}}_1\right)+{\widehat{p}}_2\left(1-{\widehat{p}}_2\right)}},$$ (1)

is asymptotically distributed as normal with mean zero and variance one. In order for the one-sided level test Z₁ to achieve 1 − β power to reject the null hypothesis at the designed alternative $H_a:p_1=p_2+\delta$, the sample size N for each group is given by

$$N=\frac{{\left(z_{1-\alpha }+z_{1-\beta }\right)}^2\left\{p_1\left(1-p_1\right)+p_2\left(1-p_2\right)\right\}}{{\delta }^2},$$

where z_p is the p-th percentile of the standard normal distribution. The drift parameter defined by $\theta =E\left\{Z_1\mid H_a\right\}$ [7] can be shown to be approximately equal to $Z_{1-\alpha }+Z_{1-\beta }$.

Suppose an interim analysis is conducted at information time $t=n/N$ where n is the number of subjects in each group who are available for accessing the primary endpoint and ${\widehat{p}}_{1,n}$ and ${\widehat{p}}_{2,n}$ are the sample proportions for the control and treatment groups, respectively. When n is large enough, the difference of the sample proportions ${\widehat{p}}_{1,n}-{\widehat{p}}_{2,n}$ is asymptotically distributed as normal with variance $n^{-1}\left\{p_1\left(1-p_1\right)+p_2\left(1-p_2\right)\right\}$. The inverse of the asymptotic variance $I_t=n{\left\{p_1\left(1-p_1\right)+p_2\left(1-p_2\right)\right\}}^{-1}$ is referred to as the information at interim analysis [8] and it can be consistently estimated by ${\widehat{I}}_t=n{\left\{{\widehat{p}}_{1,n}\left(1-{\widehat{p}}_{1,n}\right)+{\widehat{p}}_{2,n}\left(1-{\widehat{p}}_{2,n}\right)\right\}}^{-1}$. Hence the Z-test statistic at information time t can be written as

$$Z_t=\left({\widehat{p}}_{1,n}-{\widehat{p}}_{2,n}\right){\widehat{I}}_t^{1/2}.$$ (2)

If k sequential analyses are scheduled at discrete information times $t_i=n_i/N$ for $n_1<n_2<\dots <n_k$, it can be demonstrated that the sequential test statistics $\left\{Z_{t_1},Z_{t_2},\dots ,Z_{t_k}\right\}$ have the canonical joint distribution with information levels $\left\{{\widehat{I}}_{t_1},{\widehat{I}}_{t_2},\dots {\widehat{I}}_{t_k}\right\}$ asymptotically [8].

In practice, the B-value [7] is often used as a data monitoring tool. The B-value at information time t is given by $B(t)=Z_t{t}^{1/2}$ which has asymptotic expectation $E\left\{B(t)\mid H_a\right\}\approx \theta t$ under the designed effect size. In fact, $B(t)-\theta t$ can be shown an asymptotically standard Brownian motion process for $0\le t\le 1$ using the same arguments as given in [11]. If no interim analysis for efficacy is designed, the conditional power given the accrued data at information time t and the drift parameter θ can be approximated by

$$\begin{gathered} C{P}_t(\theta )=P\left\{B(1)=Z_1>Z_{1-\alpha }\mid B(t),\theta \right\} \\ \approx \text{Φ}\left(\frac{Z_t\sqrt{t}+\theta (1-t)-z_{1-\alpha }}{\sqrt{1-t}}\right), \end{gathered}$$ (3)

where $\text{Φ}(\cdot )$ is the cumulative distribution function of the standard normal random variable [7].

It is a common practice to calculate the conditional power $C{P}_t(\theta )$ under the designed effect size and decide to stop the trial by declaring futility if $C{P}_t(\theta )<\gamma$ for a fixed $0<\gamma \le 0.5$. Including stopping for futility can inflate type II error. As shown in [9], if data are monitored continuously using the futility stopping rule defined as above, the overall type II error β* will be bounded as ${\beta }^{\ast }\le \beta /(1-\gamma )$. For example, if γ is chosen to be 0.1, then the inflation of the overall type II error will be less than 11.1% of the designed error β used for determining the sample size. While this rule does not inflate type II error much, it is, nevertheless, unlikely to declare futility early when there is indeed no treatment efficacy. The rationale is that at an early time, say t < 0.5, the value of conditional power implicitly weights the future data (assumed to follow the designed effect size) more than the data accumulated up to time t and hence the conditional power would not be expected to drop from 0.9 to below 0.1 quickly.

We propose a futility stopping rule with a time-varying CP boundary. Instead of setting a fixed threshold γ, we allow the threshold γ_t to be a function of information time, which is larger when t is small and we declare for futility at an interim analysis with information time t if $C{P}_t(\theta )<{\gamma }_t$. The proposed method is motivated by the flexible error spending function monitoring method [1].

Suppose the trial will be monitored for futility k times at information times $\underset{\_}{t}=\left(t_1,t_2,\dots ,t_k\right):0<t_1<t_2<\dots <t_k<1$ before the final analysis, we determine ${\gamma }_{t_i}$ sequentially for $i=1,2,\dots ,k$ such that

$$\begin{gathered} P\left(C{P}_{t_1}(\theta )\ge {\gamma }_{t_1},\dots ,C{P}_{t_{l-1}}(\theta )\ge {\gamma }_{t_{l-1}},C{P}_{t_l}(\theta )<{\gamma }_{t_l}\right) \\ =f\left(t_l\right)-f\left(t_{l-1}\right), \end{gathered}$$ (4)

for $l=1,2,\cdots ,k$, where $f(\cdot )$ is an increasing function with $f(0)=0$ and $f(1)={\beta }^{\ast }$. In view of the Equation (3), if we let ${\gamma }_t=\text{Φ}\left({\eta }_t\right)$, the conditional power boundary at time t, $C{P}_t(\theta )<{\gamma }_t$ can be directly related to an inequality for the interim Z-test statistic,

$$\frac{Z_t\sqrt{t}+\theta (1-t)-Z_{1-\alpha }}{\sqrt{1-t}}<{\eta }_t.$$

Hence the critical value c_t for the standard normal variate $Z_t-\theta \sqrt{t}<c_t$ can be written as

$$C_t={\eta }_t\sqrt{\frac{1-t}{t}}-\frac{Z_{1-\beta }}{\sqrt{t}},$$

due to the fact $\theta \approx Z_{1-\alpha }+Z_{1-\beta }$. Therefore the value of the conditional power boundary γ_ti, at time $0<t_1<t_2<\dots <t_k<1$ can be computed by

$${\gamma }_{t_i}=\text{Φ}\left(c_{t_i}\sqrt{\frac{t_i}{1-t_i}}+\frac{Z_{1-\beta }}{\sqrt{1-t_i}}\right),i=1,2,\dots ,k.$$ (5)

Because the β-spending equations for the conditional powers (4) can be equivalently written for the Z-test statistics as

$$\begin{gathered} P(Z_{t_1}-\theta \sqrt{t_1}\ge c_{t_1},\dots ,Z_{t_{l-1}}-\theta \sqrt{t_{l-1}} \\ \ge c_{t_{l-1}},Z_{t_l}-\theta \sqrt{t_1}<c_{t_l})=f\left(t_1\right)-f\left(t_{l-1}\right), \end{gathered}$$ (6)

for $l=1,2,\dots ,k$ by the properties of the Brownian motion process of $\left\{Z_t\sqrt{t}-\theta t:0\le t\le 1\right\}$, the values of $\underset{\_}{c}=\left(c_{t_1},c_{t_2},\dots ,c_{t_k}\right)$ can be computed using the Lan and DeMets method implemented in the online software [12] (http://www.biostat.wisc.edu/landemets/).

The conditional power boundary that is easily determined by (6) and (5) assumes that no interim analysis for efficacy is conducted and hence the decision to declare for futility in the final analysis is given by $Z_1\le Z_{1-\alpha }$. However, when interim analyses for the efficacy (usually the primary purpose for monitoring the trial) are implemented, the critical value for the Z-test statistic at the final analysis, Z₁ to reject the null hypothesis is greater than $Z_{1-\alpha }$. Following the same lines as above, the conditional power boundary should be given by $\text{Φ}\left(c_{t_i}{\left[t_i/\left(1-t_i\right)\right]}^{1/2}+\epsilon (\underset{\_}{t})z_{1-\beta }/{\left(1-t_i\right)}^{1/2}\right)$, $i=1,2,\dots ,k$ for some unknown $0<\epsilon (\underset{\_}{t})<1$. Therefore, using the futility stopping boundary given by (5) obviously inflates the overall type II error. To reduce the overall type II error, we propose an ad hoc method by setting the conditional power stopping boundary as

$$C{P}_{t_i}(\theta )\le {\gamma }_{t_i}^{\ast }=\text{Φ}\left(c_{t_i}\sqrt{\frac{t_i}{1-t_i}}+z_{1-\beta }\right),i=1,2,\dots ,k$$ (7)

to offset the impact of the unknown $0<\epsilon (\underset{\_}{t})<1$. This futility monitoring plan is very flexible as the conditional power boundary ${\underset{\_}{\gamma }}^{\ast }=\left({\gamma }_{t_1}^{\ast },{\gamma }_{t_2}^{\ast },\dots ,{\gamma }_{t_k}^{\ast }\right)$ does not depend on when and how many future interim analyses would be conducted after t_k. This feature would be very attractive for monitoring a long-term clinical trial such as the COSS study.

As an illustration, we consider an artificial monitoring plan with four interim analyses (for both efficacy and futility) scheduled at information times t=0.25, 0.45, 0.65, and 0.80. Sample size is determined for the one-sided standard Z-test at level 0.05 with power 0.90 to detect the drift parameter θ without adjusting for multiple looks of data during the study. We compute the efficacy and futility stopping boundaries separately. For efficacy, the O’Brien–Fleming one-sided boundary of a total type I error 0.05 is adopted. The online program [12] produces the boundary for the Z-test statistic e = (3.7496,2.7016,2.1982,1.9815,1.7419). For futility, the O’Brien–Fleming type β-spending function of a total type II error 0.111 $\left({\beta }^{\ast }=\beta /(1-0.1)=0.111\right)$ is adopted. The same program produces the boundary for the Z-test statistic centered at the designed effect size $\underset{\_}{c}=(-2.9812,-2.1190,-1.7195,-1.5564)$ that maps to the conditional power boundary ${\underset{\_}{\gamma }}^{\ast }=\left({\gamma }_{t_1}^{\ast },{\gamma }_{t_2}^{\ast },{\gamma }_{t_3}^{\ast },{\gamma }_{t_4}^{\ast }\right)=(0.3301,0.2627,0.1442,0.0335)$ by (7). The decision rule with interim monitoring will be given as follows. at the i-th interim analysis, for $i=1,\dots ,4$,

1. if $Z_{t_i}\ge e_i$, stop for efficacy;

2. else compute the conditional power $C{P}_{t_i}(\theta )$ given accumulated data at information time t_i and the designed effect size, if $C{P}_{t_i}(\theta )<{\gamma }_{t_i}^{\ast }$, stop for futility;

3. else continue the trial and repeat steps 1 and 2 at information time $t_{i+1}$;

4. at the final analysis, if $Z_1\ge e_5=1.7419$, declare efficacy; otherwise accept the null hypothesis.

The larger conditional power boundary of the proposed time-varying CP method at the early times (t ≤ 0.5) intuitively allows the trial to pick up the futility signal more quickly than the fixed CP method of $C{P}_t(\theta )\le 0.1$ This method is easily implemented using the existed online software but it lacks of rigorous statistical justification in terms of controlling the overall size and power. The performance of the proposed method on maintaining adequate size and power is evaluated by the simulation studies described below.

The simulation studies are conducted to evaluate the proposed monitoring plan that is implemented in the ongoing COSS study. The data are generated by mimicking the COSS trial with various possible parameters that could occur in the study. For the simulation studies, we suppose the trial is designed to demonstrate the treatment efficacy in the 0.05 level one-sided two-sample proportion test powered at 90% to reject the null hypothesis with the designed alternative, p₁ = 40% and p₂ = 24%, a 40% reduction from the control. For the balanced trial as designed in the COSS study, it requires N = 142 subjects in each group. We illustrate the simulations with two candidate monitoring plans: (i) four interim analyses at information times t = (0.2,0.4, 0.6,0.8) and (ii) nine interim analyses at information times t = (0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9). The Lan–DeMets error spending method with the α –spending function $f(t)=1-\text{Φ}\left(Z_{0.95}/t^{1/2}\right)$ is used to compute the efficacy stopping boundary for the interim Z-test statistic, which produces the boundary similar to the O’Brien–Fleming method. for Plan (i) e = (4.229,2.888,2.298,1.962,1.740); for Plan (ii) e = (6.088,4.229,3.396,2.906,2.579, 2.342, 2.160,2.015,1.895,1.795). For the futility boundary, in addition to the fixed CP method of $C{P}_t(\theta )<\gamma =0.1$, we explore the proposed conditional power boundary (7) with four different β-spending functions. (i) $f_1(t)=1-\text{Φ}\left(Z_{1-{\beta }^{\ast }}/t^{1/2}\right)$; (ii) $f_2(t)={\beta }^{\ast }t$; (iii) $f_3(t)={\beta }^{\ast }{t}^{3/2}$; and (iv) $f_4(t)={\beta }^{\ast }{t}^2$. We set ${\beta }^{\ast }=0.1/(1-\gamma )=0.111$ to make the overall type II error comparable to the fixed CP boundary.

Table 1 presents the boundaries of the proposed time-varying CP method (7) with the four spending functions for the two plans. For each of these β-spending functions, the value of the stopping boundary decreases as information increases. It appears that the futility stopping rules based on the power family for the β-spending functions tend to be more aggressive than that based on the O’Brien–Fleming type β-spending function, because it is easier to accept the null hypothesis at the earlier times with the β-spending functions of power family. This fact can be also manifested graphically.

Table 1.

The conditional power boundaries for the two interim futility monitoring plans with β* = 0.111

Error functions	Information times (t)
	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
f1(t)
k=4	–	0.342	–	0.284	–	0.179	–	0.037	–
k=9	0.362	0.342	0.314	0.274	0.223	0.161	0.091	0.028	0.001
f2(t)
k=4	–	0.609	–	0.405	–	0.200	–	0.025	–
k=9	0.698	0.577	0.470	0.368	0.267	0.168	0.079	0.017	0.0001
f3(t)
k=4	–	0.547	–	0.361	–	0.186	–	0.030	–
k=9	0.649	0.527	0.425	0.333	0.246	0.161	0.082	0.021	0.0003
f4(t)
k=4	–	0.489	–	0.314	–	0.163	–	0.029	–
k=9	0.603	0.477	0.378	0.293	0.216	0.143	0.076	0.022	0.001

Equating (7) and (3), the futility stopping boundary of the conditional power (7) can be equivalently expressed as

$$Z_t<c_t-(\sqrt{1/t}-\sqrt{(1-t)/t})z_{1-\beta }+\theta \sqrt{t}.$$

Similarly, the fixed CP futility stopping boundary of $C{P}_t(\theta )<0.1$ can be equivalently expressed as

$$Z_t<{\text{Φ}}^{-1}(0.1)\sqrt{(1-t)/t}-Z_{1-\beta }/\sqrt{t}+\theta \sqrt{t}.$$

Figure 1 presents the monitoring boundaries of the Z-test statistic for nine interim looks at t = (0.1,0.2,...,0.9) with α = 0.05 and β* = 0.111 It shows that the fixed CP futility stopping boundary is substantially lower than the proposed time-varying CP futility monitoring boundaries early in the trial. Therefore, it is anticipated that the proposed time-varying CP futility monitoring has a better chance to stop the trial for futility earlier than the fixed CP monitoring.

Figure 1.

Comparison of the futility boundaries between the proposed time-varying CP method with the four different β-spending functions: (i) $f_1(t)=1-\left(Z_{1-{\beta }^{\ast }}/t^{1/2}\right)$, (ii) $f_2(t)={\beta }^{\ast }t$, (iii) $f_3(t)={\beta }^{\ast }{t}^{3/2}$ (iv) $f_4(t)={\beta }^{\ast }{t}^2$ and the fixed CP method for ${\beta }^{\ast }=0.111$.

In the simulation studies, the 2-year rate of the primary endpoint (ipsilateral ischemic stroke) for the control group is always set at p₁ = 40%. The rate for the surgical group p₂ is set at 24% (the designed effect size) and at 35% (a reduced effect size) for comparing the exit probabilities at interim analyses and the overall power. The value of p₂ is also set at 40% for comparing the exit probabilities and the overall size. The Monte Carlo method is used to estimate the exit probabilities due to either efficacy or futility at interim analyses.

For example, in a simulation study for the case with the designed effect size (p₁ = 40% and p₂ = 24%) under monitoring plan (i) with the O’Brien and Fleming boundary, the simulation proceeds as follows.

1. For the first interim analysis, we generate a sample of $142\times 0.2\approx 28$ dichotomous random observations from Bernoulli(0.4) and Bernoulli(0.24) for the control and study treatment groups, respectively. The Z-statistic at information time t₁ = 0.2, Z_0.2 is calculated using (2) and compares to 4.229. If Z_0.2>4.229, exit the trial for efficacy.

2. Otherwise compute the conditional power CP_0.2 given the samples and the designed effect size using (3) and compare to 0.342. If CP_0.2<0.342, exit the trial for futility.

3. If no stopping of the trial is advised, we further generate a new sample of 28 dichotomous random observations from Bernoulli(0.4) and Bernoulli(0.24) for the control and study treatment groups, respectively. Combine the data and recalculate the corresponding Z-statistic and the conditional power at information time t₂ = 0.4, Z_0.4 and CP_0.4, and compare them to the thresholds 2.888 and 0.284, respectively, in the same way as described in Steps 1 and 2.

4. If no stopping is recommended in any interim analyses, we come to the final analysis. Remaining data are generated from Bernoulli(0.4) and Bernoulli(0.24) for the control and study treatment groups, respectively. Calculate the Z-statistic with all the data, Z₁. If Z₁>1.740, declare efficacy; otherwise declare futility.

We repeat this procedure 100,000 times. The exit probabilities due to efficacy or futility at each interim analysis and final analysis are estimated by the proportions of corresponding exits. The overall power is estimated by the sum of empirical exit probabilities due to efficacy.

Results

Table 2 presents the Monte Carlo simulation results comparing the overall power and exit probabilities between the proposed time-varying CP method and the fixed CP method of $C{P}_t(\theta )<\gamma =0.1$ for data generated from the designed effect size. The results show that the loss of power is negligible. Increasing the number of interim analyses may increase type II error, but our results indicate that the inflation of type II error is not substantial. Using the proposed monitoring method, the chance of early stopping for futility, though larger than the fixed CP method, is again negligible. For example, if nine interim analyses are scheduled at the information times described in Table 2, the cumulative early exit probabilities due to futility for the monitoring plan (ii) at the middle of the trial (t=0.5) are only 0.007, 0.025, 0.015, and 0.01, respectively, for the proposed time-varying CP method with the four β–spending functions; while this probability is 0.001 for the fixed CP method. Inflations of type II error at early time can be virtually ignored, particularly for the proposed method with the O’Brien–Fleming type β-spending function and the fixed CP method.

Table 2.

Comparison of the exit probabilities between the proposed time-varying CP method with the four different β-spending functions and the fixed CP method at each interim analysis and the overall power based on 100,000 Monte Carlo samples for data generated with p₁ = 40% and p₂ = 24%

Information time (t)	f1(t)		f2(t)		f3(t)		f4(t)		Fixed CP method with γ = 0.1
	k = 4	k = 9	k = 4	k = 9	k = 4	k = 9	k = 4	k = 9	k = 4	k = 9
0.1
Efficacy	–	0.000	–	0.000	–	0.000	–	0.000	–	0.000
Futility	–	0.000	–	0.010	–	0.003	–	0.001	–	0.000
0.2
Efficacy	0.006	0.005	0.006	0.005	0.006	0.005	0.006	0.005	0.006	0.005
Futility	0.000	0.000	0.011	0.005	0.005	0.004	0.002	0.002	0.000	0.000
0.3
Efficacy	–	0.046	–	0.046	–	0.046	–	0.046	–	0.046
Futility	–	0.001	–	0.004	–	0.002	–	0.002	–	0.000
0.4
Efficacy	0.163	0.112	0.163	0.112	0.163	0.112	0.163	0.112	0.163	0.112
Futility	0.003	0.002	0.007	0.004	0.006	0.003	0.005	0.002	0.000	0.000
0.5
Efficacy	–	0.159	–	0.159	–	0.159	–	0.159	–	0.159
Futility	–	0.004	–	0.002	–	0.003	–	0.003	–	0.001
0.6
Efficacy	0.330	0.171	0.329	0.170	0.329	0.170	0.330	0.170	0.330	0.171
Futility	0.009	0.003	0.005	0.002	0.006	0.002	0.004	0.003	0.004	0.003
0.7
Efficacy	–	0.140	–	0.139	–	0.140	–	0.140	–	0.140
Futility	–	0.004	–	0.002	–	0.004	–	0.002	–	0.006
0.8
Efficacy	0.252	0.113	0.250	0.111	0.251	0.112	0.252	0.113	0.252	0.113
Futility	0.006	0.003	0.004	0.001	0.004	0.002	0.005	0.003	0.015	0.010
0.9
Efficacy	–	0.082	–	0.080	–	0.081	–	0.082	–	0.083
Futility	–	0.002	–	0.001	–	0.002	–	0.002	–	0.023
1.0
Efficacy	0.142	0.059	0.138	0.056	0.140	0.058	0.142	0.058	0.143	0.059
Futility	0.090	0.093	0.087	0.090	0.090	0.091	0.092	0.093	0.088	0.068
Power	0.893	0.887	0.886	0.878	0.889	0.883	0.893	0.885	0.894	0.888

Table 3 presents the results of the same Monte Carlo simulation study but with data in the surgical group generated from Bernoulli(0.35) that is, p₂ = 35%. This represents the case that the surgical treatment in COSS is much less effective than hypothesized and may not have a clinical significance leading to adoption of this treatment in practice considering the large cost and complication associated with the surgery. The results show that the proposed time-varying CP method has almost the same overall power as that based on the fixed CP method. However, the proposed method has a greater advantage in terms of being able to stop the trial earlier for futility. The cumulative exit probabilities due to futility for the monitoring plan (ii) at information time t=0.5 are 15.9%, 22.5%, 19.4%, and 14.9%, respectively, for the proposed time-varying CP method with the four β-spending functions and are much larger than 5.7% using the fixed CP method.

Table 3.

Comparison of the exit probabilities between the proposed time-varying CP method with the four different β-spending functions and the fixed CP method at each interim analysis and the overall power based on 100,000 Monte Carlo samples for data generated with p₁ = 40% and p₂ = 35%

Information time (t)	f1(t)		f2(t)		f3(t)		f4(t)		Fixed CP method with γ = 0.1
		k = 4	k = 9	k = 4	k = 9	k = 4	k = 9	k = 4	k = 9	k = 4	k = 9
0.1
Efficacy	–	0.000	–	0.000	–	0.000	–	0.000	–	0.000
Futility	–	0.000	–	0.047	–	0.022	–	0.010	–	0.000
0.2
Efficacy	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
Futility	0.004	0.004	0.088	0.036	0.052	0.027	0.028	0.017	0.000	0.000
0.3
Efficacy	–	0.003	–	0.003	–	0.003	–	0.003	–	0.003
Futility	–	0.023	–	0.056	–	0.041	–	0.029	–	0.001
0.4
Efficacy	0.012	0.009	0.012	0.009	0.012	0.009	0.012	0.009	0.012	0.009
Futility	0.075	0.054	0.096	0.044	0.081	0.051	0.083	0.041	0.012	0.011
0.5
Efficacy	–	0.017	–	0.017	–	0.017	–	0.017	–	0.017
Futility	–	0.078	–	0.042	–	0.053	–	0.052	–	0.045
0.6
Efficacy	0.045	0.027	0.045	0.027	0.045	0.027	0.045	0.027	0.045	0.027
Futility	0.158	0.057	0.099	0.044	0.125	0.049	0.100	0.068	0.132	0.089
0.7
Efficacy	–	0.034	–	0.034	–	0.034	–	0.034	–	0.034
Futility	–	0.080	–	0.043	–	0.073	–	0.051	–	0.122
0.8
Efficacy	0.071	0.038	0.070	0.038	0.071	0.038	0.071	0.038	0.071	0.038
Futility	0.132	0.051	0.084	0.038	0.090	0.048	0.115	0.070	0.254	0.136
0.9
Efficacy	–	0.041	–	0.039	–	0.040	–	0.041	–	0.041
Futility	–	0.061	–	0.035	–	0.041	–	0.065	–	0.167
1.0
Efficacy	0.085	0.046	0.082	0.044	0.084	0.045	0.085	0.046	0.086	0.046
Futility	0.418	0.376	0.424	0.403	0.440	0.380	0.460	0.381	0.387	0.211
Power	0.213	0.215	0.209	0.211	0.212	0.213	0.213	0.215	0.214	0.215

Table 4 summarizes the Monte Carlo simulation study for the case where the surgical treatment is no better than the control (p₁ = p₂ = 40%). The results show that the overall sizes are all close to the nominal value of 0.05, especially when only four interim analyses are conducted. The size increases to at most 0.056 when the number of interim analyses increases to nine. When the study treatment is not different from the control at all, the fixed CP method does not detect the futile trial quickly and the cumulative exit probability to stop the trial by the time the study is half completed Is only 16.1% under the monitoring plan (ii). However, the proposed time-varying CP method is able to boost this probability substantially: they are 34.3%, 42.0%, 38.8%, and 31.9%, respectively, for the four selected β-spending functions.

Table 4.

Comparison of the exit probabilities between the proposed time-varying CP method with the four different β-spending functions and the fixed CP method at each interim analysis and the overall size based on 100,000 Monte-Carlo samples for data generated with p1 =p2 = 40%

Information time (t)	f1(t)		f2(t)		f3(t)		f4(t)		Fixed CP method with γ = 0.1
	k = 4	k = 9	k = 4	k = 9	k = 4	k = 9	k = 4	k = 9	k = 4	k = 9
0.1
Efficacy	–	0.000	–	0.000	–	0.000	–	0.000	–	0.000
Futility	–	0.001	–	0.079	–	0.040	–	0.018	–	0.000
0.2
Efficacy	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
Futility	0.012	0.011	0.171	0.070	0.111	0.055	0.067	0.038	0.000	0.000
0.3
Efficacy	–	0.001	–	0.001	–	0.001	–	0.001	–	0.001
Futility	–	0.064	–	0.113	–	0.091	–	0.072	–	0.005
0.4
Efficacy	0.002	0.002	0.002	0.002	0.002	0.002	0.002	0.002	0.002	0.002
Futility	0.184	0.126	0.184	0.078	0.168	0.105	0.171	0.094	0.044	0.038
0.5
Efficacy	–	0.004	–	0.004	–	0.004	–	0.004	–	0.004
Futility	–	0.141	–	0.080	–	0.097	–	0.097	–	0.118
0.6
Efficacy	0.010	0.007	0.010	0.007	0.010	0.007	0.010	0.007	0.010	0.007
Futility	0.285	0.104	0.171	0.077	0.222	0.086	0.200	0.126	0.305	0.192
0.7
Efficacy	–	0.009	–	0.009	–	0.009	–	0.009	–	0.009
Futility	–	0.119	–	0.069	–	0.108	–	0.084	–	0.187
0.8
Efficacy	0.017	0.010	0.016	0.010	0.017	0.010	0.017	0.010	0.017	0.010
Futility	0.183	0.073	0.125	0.060	0.136	0.068	0.180	0.100	0.352	0.170
0.9
Efficacy	–	0.010	–	0.010	–	0.010	–	0.010	–	0.010
Futility	–	0.080	–	0.050	–	0.053	–	0.085	–	0.129
1.0
Efficacy	0.021	0.013	0.021	0.012	0.021	0.012	0.021	0.013	0.022	0.013
Futility	0.287	0.227	0.300	0.269	0.313	0.242	0.332	0.232	0.248	0.106
Size	0.050	0.056	0.049	0.055	0.050	0.055	0.050	0.056	0.051	0.056

Simulations with other values of p₂ were also conducted (results not shown here) and similar patterns were observed.

Discussion

For a long-term randomized clinical trial, a sequence of interim monitoring for both efficacy and futility may be required by the DSMB. A flexible CP futility monitoring plan with the time-varying boundary is developed that can be easily accommodated with any sequential interim monitoring for efficacy. Compared to the commonly used monitoring plan in practice that the efficacy boundary is derived using the Lan–DeMets α-spending method and the futility boundary is set at a fixed value for the conditional power calculated with the designed effect size, the proposed plan increases the chance of early stopping for futility substantially. This feature of the proposed time-varying CP method is very attractive to the trial that only the minimally clinically meaningful effect size is designed for the test, since it allows investigators to be able to stop the trial earlier when this effect size is unlikely to be realized and thus considerable resources can be saved for other promising trials. However, when the effect size less than the designed is still acceptable or investigators would like to accumulate as more data as possible for secondary analysis, the fixed CP method with small threshold for the conditional power may be more desirable.

The proposed method does not require pre-specifying the number and times of the interim analyses. The stopping boundaries for both efficacy and futility can be sequentially but independently obtained based on the principle of error spending method given the times of interim analysis. The efficacy boundary is determined based on α-spending function for type I error, the futility boundary is similarly obtained based on β-spending function for type II error, and these boundaries can be easily computed separately using the existed online software [12]. Because the number and times of interim monitoring may not be conveniently determined prior to the trial or may be frequently modified as exemplified by the ongoing COSS trial, this approach is obviously appealing to the investigators. Although the proposed method may not control for the overall size and power at the designed levels, our empirical results through the extensive simulation studies show that the proposed monitoring method maintains the designed overall size and power quite well for the two-sample proportion test.

In summary, the proposed time-varying CP method substantially increases the chance of early stopping for futility without substantially inflating the overall power compared to the fixed CP method. Among the four selected β-spending functions for the proposed method, using the β-spending function, $f(t)={\beta }^{\ast }t$ or ${\beta }^{\ast }{t}^{1.5}$ leads to a relatively aggressive plan in terms of stopping earlier for futility and it appears to inflate type II error more than other β-spending functions. The proposed time-varying CP futility monitoring method with O’Brien–Fleming type β-spending function agrees closely to the fixed CP futility monitoring method in terms of the overall size and power but has the advantage of having a reasonable chance to pick up the futility signal early in the trial and it is therefore recommended for use in a lengthy and costly randomized clinical trial.

If the number and times of the interim monitoring are exactly specified in the design, one can design a monitoring method to control the overall size and power exactly. For example, if the trial is scheduled to be monitored k times including the final analysis at times $0<t_1<t_2<\dots <t_k=1$ with the corresponding efficacy boundary $e_1,e_2,\dots ,e_k$, then the conditional power should be computed by $C{P}_t(\theta )=P\left\{B(1)>e_k\mid B(t),\theta \right\}$ and therefore the futility boundary will depend on the efficacy boundary. It is possible to compute both efficacy and futility boundaries jointly by the iterative integration method [8] but the computation can be quite involved and no software is currently available for the task. Moreover, there is no guarantee that this approach will yield a decreasing conditional power boundary that has the capability of exiting the trial for futility earlier than the fixed CP method.

For this proposed method, the Z-test statistic is defined as

$$Z_t=\sqrt{n}\left({\widehat{p}}_{1,n}-{\widehat{p}}_{2,n}\right)/\sqrt{{\widehat{p}}_{1,n}\left(1-{\widehat{p}}_{1,n}\right)+{\widehat{p}}_{2,n}\left(1-{\widehat{p}}_{2,n}\right)}.$$

This is different from the standard Z-test statistic for the two-sample proportion test. The reason for doing so is to make the test statistic asymptotically normally distributed with variance one for both the null and alternative hypotheses. If the standard Z-test statistic $Z_t=\sqrt{n}\left({\widehat{p}}_{1,n}-{\widehat{p}}_{2,n}\right)/\sqrt{2{\widehat{p}}_n\left(1-{\widehat{p}}_n\right)}$ for ${\widehat{p}}_n=\left({\widehat{p}}_{1,n}+{\widehat{p}}_{2,n}\right)/2$ is adopted, the sample size for the fixed design should be modified to

$$N={\left\{\frac{z_{1-\alpha }\sqrt{\overline{p}(1-\overline{p})}+z_{1-\beta }\sqrt{p_1\left(1-p_1\right)+p_2\left(1-p_2\right)}}{\delta }\right\}}^2,$$

with $\overline{p}=\left(p_1+p_2\right)/2$ the drift parameter θ is approximately equal to

$$\theta =Z_{1-\alpha }+Z_{1-\beta }\sqrt{\frac{p_1\left(1-p_1\right)+p_2\left(1-p_2\right)}{2\overline{p}(1-\overline{p})}},$$

and the conditional power given the designed effect size can be computed by

$$C{P}_t(\theta )=\text{Φ}\left\{\frac{Z_t\sqrt{t}+\theta (1-t)-z_{1-\alpha }}{\sqrt{1-t}R(\alpha ,\beta ,\theta )}\right\},$$

with $R(\alpha ,\beta ,\theta )=\left(\theta -Z_{1-\alpha }\right)/z_{1-\beta }$ Despite the slight difference in the formula for computing the conditional power, the futility stopping boundary turns out to be the same and our simulation experiments (not shown here) indicate that the two versions of the Z-test statistic yield very similar results unless the difference between p₁ and p₂ is very large.

Finally, though the proposed time-varying CP futility monitoring method is developed and illustrated for the two-sample test of proportions motivated by the ongoing COSS study. The method can be applied to other tests as long as the sequential test statistics constitute of the Brownian motion process asymptotically. However, this method may not be applicable to monitored process that behaves quite differently from the Brownian motion.