Implementation of statistical features of a Bayesian two-armed responsive adaptive randomization trial with post hoc analysis of time trend drift

Abstract

Bayesian adaptive designs with response adaptive randomization (RAR) have the potential to benefit more participants in a clinical trial. While there are many papers that describe RAR designs and results, there is a scarcity of works reporting the details of RAR implementation from a statistical point exclusively. In this paper, we introduce the statistical methodology and implementation of the trial Changing the Default (CTD). CTD is a single-center prospective RAR comparative effectiveness trial to compare opt-in to opt-out tobacco treatment approaches for hospitalized patients. The design assumed an uninformative prior, conservative initial allocation ratio, and a higher threshold for stopping for success to protect results from statistical bias. A particular emerging concern of RAR designs is the possibility that time trends will occur during the implementation of a trial. If there is a time trend and the analytic plan does not prespecify an appropriate model, this could lead to a biased trial. Adjustment for time trend was not pre-specified in CTD, but post hoc time-adjusted analysis showed no presence of influential drift. This trial was an example of a successful two-armed confirmatory trial with a Bayesian adaptive design using response adaptive randomization.

Background

The European Medicines Agency (EMA) issued a reflection paper in 2007 that recognized that adaptive designs have potential to increase a clinical trial’s efficiency in terms of time and required resources, without sacrificing scientific and regulatory standards (Committee for Medicinal Products for Human Use 2007). In 2019 the Food and Drug Administration (FDA) released guidance for the design, conduct, reporting and submission process of an adaptive clinical trial, including Bayesian adaptive trials (Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research 2019). The FDA recognizes four major potential advantages of an adaptive design: statistical efficiency, accommodation of ethical considerations, improved understanding of drug effects, and accessibility to stakeholders.

In the last two decades, numerous methodological papers were published on developments in Bayesian adaptive designs, for example, variants of the dose-finding methods, different randomization techniques, and solutions for master protocol trials (Berry S.M. 2010; Meyer et al. 2020; Robertson et al. 2020; Giovagnoli 2021). On par with methodological advances, there is an increase in a number of Bayesian adaptive trial designs adopted in clinical research (Barker et al. 2009; Kim et al. 2011; Carey and Winer 2016; Papadimitrakopoulou et al. 2016; Park et al. 2016; Rugo et al. 2016; Angus et al. 2020) such as response adaptive randomization, incorporating or removing treatment arms, early stopping and platform trials. Many scientific papers published in medical journals include only a brief section devoted to the conduct of Bayesian trials, but few published reports provide practical details exclusively on the specifics of the statistical conduct of Bayesian adaptive trials. Gajewski et al. (2023) provide specific execution of an RAR but only had a single endpoint and did not do a post-hoc time trend analysis.

In adaptive designs predefined rules and accrued data guide adaptations throughout a trial. In the Bayesian paradigm, the prior distribution gets updated at each predetermined step, and accrued information determines the next planning decision according to prespecified rules. Various aspects of a trial undergo adaptations, including sampling, stopping for futility or efficacy, treatment arm inclusion or exclusion, and randomization rules. In Bayesian response adaptive randomization designs, the randomization is usually driven by one primary endpoint. As more information is gathered at each interim analysis, more participants get randomized to the stronger performing arm. There is a debate in the literature on potential disadvantages of unequal allocation for participants (Hey and Kimmelman 2015; Thall 2015; Sim 2019). However, as argued by Robertson et al. (2020), Giovagnoli et al. (2021), Villar et al. (2020), and Berry (2015), Bayesian adaptive design is still advantageous if used appropriately in a suitable setting. It provides benefits to participants in comparative effectiveness clinical trials (Connor, Elm, and Broglio 2013) and is considered well-suited for Patient-Centered Outcomes Research Institute (PCORI) pragmatic clinical trials (Mullins et al. 2014). Another obstacle to the use of response-randomization designs is possible inflation of a false positive error rate caused by an unknown time trend over the course of the trial (Karrison, Huo, and Chappell 2003; Proschan and Evans 2020). Among other reasons the drift occurs when characteristics of subjects recruited at later stages diverge from those recruited at earlier stages of a trial. As explained by Lipsky et al. (2011), when background risk changes over time, combined with change in allocation ratio, this can lead to a shift in distributions in the arms. If confounding is left unadjusted, large enough time trends can lead to an increase in Type 1 error rate. “Bayesian Time Machine” models include a parameter adjustment for time interval using a hierarchical modelling (Saville et al. 2022). The variance of this parameter controls the amount of dynamic smoothing. With an appropriate level of flexibility, the estimated drift is driven by observed data. Although different methods have been developed to account for time trends, the lack of reporting on their magnitude in practice has only recently been recognized (Robertson et al. 2020).

In this paper we will discuss statistical aspects of the Changing the Default (CTD) a confirmatory single center prospective randomized comparative effectiveness trial, to determine the effectiveness of offering smoking cessation assistance to all patients who smoke unless they decline it (Opt-out) versus only those who say they are ready to quit (Opt-in, the standard approach to tobacco treatment) (Faseru et al. 2017). Tobacco cessation-oriented pharmacotherapy and counseling lead to higher rates of successful quitting (Gonzales et al. 2006; Fiore MC 2008). However, treatment utilization rate remains low (Browning et al. 2008; Freund et al. 2008; Jamal et al. 2012). CTD is a pioneer trial in its goal to test the impact of changing a treatment default. We chose an innovative Bayesian adaptive design with response adaptive randomization due to its greater benefit to participants in mind. We report statistical methodology, execution, and real analysis of CTD trial and perform post hoc drift-analysis to evaluate time trend influence over the course of the trial. The result of CTD is presented in Richter et al. (2023) but lacks details of the implementation of the Bayesian RAR and how it looked at each of the interim analyses; we fill the gap in this paper.

Methods

We here summarize the trial design, in its final form, as Bayesian two-armed responsive adaptive randomization study. We conducted a population-based adaptive randomization trial to compare the effects of opt-out care—presumptively providing medications/counseling to all—versus opt-in care among 1,000 smokers (Faseru et al. 2017). To maximize participation, the study randomized and treated all eligible patients. All the trial’s participants were offered cessation medication and 4 counselling sessions. Consent was obtained 1-month post-randomization during a phone call with a counselor. Delayed consent resulted in successful enrollment of most participants who were initially randomized into the trial, including those not motivated to quit (Faseru et al. 2022). Only consenting participants (74.1% of all randomized) were included in interim and outcome analyses. Outside of the project director and the data coordinating center, all study staff and investigators, including the principal investigator, were blinded to all interim analyses and changes in randomization.

Primary co-Endpoints

Initially the primary endpoint for the trial was the percentage of participants who quit (biochemically confirmed 7-day point prevalence abstinence) at short term (1-month). At the sixth interim analysis, 6-month co-endpoint was added to optimize the chance of seeing a signal at this important endpoint. This change took place before the trial had an opportunity to stop for success and the response adaptive randomization (RAR) formula was unchanged. For the remaining interim analyses, the endpoint was calculated as the percentage of participants who quit at each of the end points, both 1 month and 6 months post randomization. More details regarding the endpoints were prespecified in the published protocol (Faseru et al. 2017).

Subjects’ smoking status was verified based on cotinine cut-off point of no more than 10 ng/ml and carbon monoxide (CO) cutpoint of no more than 10 ppm. Cotinine was the preferred measure. CO testing was a method of verification for subjects who were taking nicotine replacement therapy or the ones who refused to provide a saliva sample.

Treatment arms

The patients were randomized into one of the two treatment arms. We labeled treatment arms as j, where j=1 is an indicator of opt-in arm and j=2 is an indicator for opt-out arm.

Statistical model

A Bayesian statistical model was used to make inferences in the trial. We assume biochemically confirmed 7-day point prevalence abstinence $\left(S_{Q_j}\right)$ follows a binomial distribution, i.e., $S_{Q_j}~\mathit{\text{Binom}}\left(n_j,{\theta }^{Q_j}\right)$, where $n_j$ denotes sample size of the $j^{th}$ arm and ${\theta }^{Q_j}$ denotes probability of a successful outcome (e.g. confirmed quit smoking) at 1-month with assumed prior knowledge in the form of a “uninformative” prior distribution $\text{log}\frac{{\theta }^{Q_j}}{1-{\theta }^{Q_j}}~N\left(0,{100}^2\right)$. An analogous separate model is used for the outcome at 6-months where ${\delta }^{Q_j}$ we denote probability of successful outcome.

Statistical Quantities

The posterior distribution of the model parameters was calculated using a Markov Chain Monte Carlo (MCMC) algorithm. For the k=1, …, K draws from the posterior distribution, the frequency each treatment arm has higher quit rates is the posterior probability that the treatment arm is better. The posterior probability was calculated for each arm at 1 or 6 months per formula (1) and (2) below:

$$P_j^{1m}=\frac{1}{K}\sum _{k=1}^{K}I\left({\theta }^{Q_j}>{\theta }^{Q_{j^{\prime }}}\right),$$ (1)

$$P_j^{6m}=\frac{1}{K}\sum _{k=1}^{K}I\left({\delta }^{Q_j}>{\delta }^{Q_{j^{\prime }}}\right).$$ (2)

Trial adaptation and stopping criteria

The first interim analysis was conducted after 400 patients were randomized—the study’s “burn-in” period. During the burn-in period, the randomization scheme had 1:1 ratio. The data set for the first interim analysis included participants consenting to be enrolled and 1-month survey windows that had closed. All analyses used follow the intent-to-treat (ITT) principle, in which all participants were included in the arm to which they were originally randomized. After the study’s “burn-in” period, RAR was used for the patient’s allocation (detailed in the following section). At each interim analysis posterior probability of having higher quit rates was calculated for both arms and the maximum out of two was picked (they are called $\text{max}\left(P_j^{1m}\right)$ and $\text{max}\left(P_j^{6m}\right)$). The trial’s stopping criteria was that the maximum posterior probability P_j at 1-month and 6-month endpoints was greater than 0.9925 (determined by simulations):

$$\left(\text{max}\left(P_j^{1m}\right)>0.9925\right)\cap \left(\text{max}\left(P_j^{6m}\right)>0.9925\right).$$

The trial team planned to continue recruitment until all 1000 patients were randomized unless the best arm was identified after 500 patients were randomized. Subsequent interims occurred every 13-weeks until randomization was stopped because of early stopping or 1000 patients randomized. A calendar-based interim schedule is advantageous so the study team can prepare for each interim analysis. If at any point of the trial after these first 500 patients the stopping criteria had been met, randomization would be stopped, and the remaining patients would be assigned into the more effective arm. Regardless of when the probability cutpoints had been reached, the finding would be confirmed with a subsequent analysis and evaluation (>.99 was determined by simulations), which could be at either 1-month or 6-month endpoints, after all data from participants are obtained:

$$\left(\text{max}\left(P_j^{1m}\right)>0.99\right)\cup \left(\text{max}\left(P_j^{6m}\right)>0.99\right).$$

Response adaptive randomization

RAR was firstly conducted after the 1^st interim analysis. The best utility probability (i.e., the posterior probability that the treatment arm is better) was evaluated at each interim and impacted the randomization of the newly enrolled patients after the 1^st interim analysis. The newly enrolled patients randomized to the j^th arm was proportional to

$$V_j={\left(\frac{P_j^{1m}Var\left({\theta }^{Q_j}\right)}{n_j+1}\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}$$ (3)

where ${\theta }^{Q_j}$ was the utility parameter (i.e., smoking quitting rate parameter) of the j^th arm at 1-month only, that took advantage of the information gained from our analyses up to that point. Simply weighting for the posterior probability carries a risk of unbalancing a sample in favour of the inferior treatment if initial data is poor (Thall and Wathen 2007). The adjusted weight (3) takes into account an estimate of the information added by one more subject assigned to an arm. This additional information is estimated by the current variance in the posterior distribution and the number of subjects already allocated to the arm. Allocation ratio in each arm was calculated as $V_j/(V_1+V_2)$, and it would randomize more newly enrolled patients to the more beneficial arm. Allocation probability was set to 0 for values less than 0.05. Such RAR process would be conducted at each following interim analysis.

Operating characteristics

It was projected that 6.7 patients would randomize each week. The interim analysis was planned to be conducted every 13 weeks. Three scenarios describing different patients’ prognosis were set to evaluate the power, sample size and duration of the trial. Under the assumed scenarios 100,000 trials were simulated using Fixed and Adaptive Clinical Trial Simulator (FACTS 6.4). Each trial had 15,000 Markov Chain Monte Carlo (MCMC) simulations with 5,000 simulations’ length of burn in period. Under “expected” scenario it was estimated that the response rate would be 15.7% in the opt-in arm and 25.2% in the opt-out arm at 1-month and 6-month endpoints. This scenario had 99% power with 75% of the early success trials, 24% late success. One percent of the trials were inconclusive. The average sample size of this trial scenario was 789 participants with more than half (546) in the better opt-out arm. The average length of these simulated trials was 145 weeks. Under “expected/smaller” scenario quit rates were projected to be 15.7% in the opt-in arm and 25.2% in the opt-out arm at 1 month and 15.7% in the opt-in arm and 20.0% in the opt-out arm at 6 months. Under this scenario 68% of trials had early success and 23% had late success, providing 91% power. The trial lasted 167 weeks on average and required 947 participants with 696 of them allocated to the opt-out arm. The “no difference” scenario proposed 15.7% quit rates in both arms at both endpoints. No trials had early success and 5% of the trials had late success. Mean number of participants enrolled was 1000 with a 1:1 allocation rate. The average trial duration was 175 weeks.

Re-calculated operating characteristics considering the observed dropout rate

Out of 1000 enrolled participants 261 dropped out at 1 month and 351 dropped out at 6 months. These participants were not included in the final analysis. The same three scenarios were re-run with the observed dropout rate. In the opt-in arm 21.7% of participants left the trial at 1 month and 11.9% at 6 months. In the opt-out arm 28.4% and 7.5% of participants dropped out at 1 month and 6 months respectively. Under the “expected” scenario the estimated power was 82% with 26% if the trials having early success and 56% having late success. It took 961 participants and 169 weeks on average. The “expected/smaller” scenario showed 6% of the early success trials and 54% of the late success trials, giving 60% power. Mean trial duration was 173 weeks with mean number of subjects 990. Under a “no difference” scenario 6% of the trial had late success and none of them had early success. The trial lasted 175 weeks on average and required 1000 participants. Simulations’ results summary is listed in Table 1.

Table 1.

Operating characteristics.

Description	Scenario	Percentage of successful trials (early success)	Participants (in opt-out arm)	Duration in weeks
No dropouts
Expected	15.7% vs 25.2% at month 1/ 15.7% vs 25.2% at month 6	99% (75%)	789 (546)	145
Expected/Smaller	15.7% vs 25.2% at month 1/ 15.7% vs 20.0% at month 6	91% (68%)	947 (696)	167
No difference	15.7% vs 15.7% at month 1/ 15.7% vs 15.7% at month 6	5% (0%)	1000 (500)	175
Accounting for dropouts
Expected	15.7% vs 25.2% at month 1/ 15.7% vs 25.2% at month 6	82% (26%)	961 (652)	169
Expected/Smaller	15.7% vs 25.2% at month 1/ 15.7% vs 20.0% at month 6	60% (6%)	990 (676)	173
No difference	15.7% vs 15.7% at month 1/ 15.7% vs 15.7% at month 6	6% (0%)	1000 (500)	175

Post-assessment of “drift” on the 1-month endpoint

Time trend drift are considered to be a major obstacle to the use of RAR. To estimate the degree to which time trend might affect trial results we modelled drift $l_i$ in ${\theta }^{Q_j}$ counting backwards from the final analysis (i=11) to the first interim (i=1).

$$\mathit{\text{logit }}\left({\theta }_i^{Q_j}\right)={\gamma }_j+l_i$$

$${\gamma }_j~N\left(0,{100}^2\right)$$

$$l_i~N\left(l_{i+1},{\tau }^2\right)$$

The following hyperprior distribution of ${\tau }^2$ is

$${\tau }^2~IG\left(0.1,0.001\right).$$

The model assumed the same drift in both arms. The “weakly informative” hyperprior of $\tau$ has weight of 0.2 and centred around 0.1 and puts emphasis on data while allowing dynamic smoothing. (Saville et al. 2022) Drift estimation follows the observed probability of the successful outcome in each arm from assumed zero drift at final analysis $(l_{11}=0)$ to the beginning of the trial, representing the most recent time of the trial life.

Results

Trial execution

Our standard operating procedure (SOP) for interim analyses is shown in Figure 1. For the duration of interim analysis randomization was paused for no more than two weeks. The CTD data management team (DM) from the data coordinating center provided the study project director (PD) with an analytic file with proposed cases for the interim analysis. The PD confirmed cases to be included, verified participants’ smoking status, excluded any patients who should not be included (for example, who had refused consent), and returned the revised interim analysis dataset to the DM. Based on the smoking status of consented participants a statistician checked stopping criteria and produced a randomization table. Future patients were randomized in blocks. Block size differed to accommodate allocation ratio.

Figure 1.

Standard operating procedure for each interim analysis.

Results at interim analyses and final analysis

Figure 2 details development of the trial through the 10 interim and final analyses. The study enrolled the first patient in September of 2016 and the last patient in June of 2020. The trial was conducted for 4 years and 2 months including time for the last tested sample. Starting at 4^th interim analysis, the posterior probability of 1-month higher quit rates in opt-out arm climbed above 0.9 and stayed above that level throughout the trial never reaching 0.9925 threshold. After 4^th interim a noticeably larger number of subjects were assigned to the “winner” arm. There was a concern that not enough data would be collected to evaluate the long-term effect of the intervention. The team decided to introduce 6-month co-endpoint into the protocol. It affected only the stopping criteria, conservatively, and was introduced prior to the 500-patient burn in period. This had the effect of making it harder to conclude the trial as the stopping criteria required both 1-month and 6-month posterior probabilities to be greater than 0.9925.

Figure 2.

Key clinical trial metrics across 10 interim analyses and the final analysis.

After the first 500 patients were randomized the stopping for success criteria could be initialized. Even though at the last two interims 6-month posterior probability climbed above 0.8, it suddenly dropped to 0.591 at the final analysis (Richter et al. 2023). The data analysis of the 10^th interim was performed during the first week of May 2020 and the final analysis was done at the last week of March 2021. Data from 208 and 325 participants in the opt-in and opt-out arms were available for the 10^th interim analysis. There were 15.9% and 19.1% verified 6-month quitters in the opt-in and opt-out arm. Using 10^th interim analysis data we calculated posterior predictive probabilities to predict the future consenting participants abstaining in the opt-in and opt-out arms. Posterior predictive distributions (Figure 3) suggest that 3 out of 21 participants would be abstinent in the opt-in arm and 18 out of 95 participants would be abstinent in the opt-out arm. Instead, 8 patients quitted in the opt-in arm bringing 6-months quit rate for the last cohort to 38.1%. There was only 4.5% probability of getting this or higher level of abstinence in the opt-in arm. The opt-out arm had slightly smaller than expected 16.8% 6-month quit rate which corresponds to 71.8% probability of having more quitters. Conversely, the 1-month final analysis didn’t show any dramatic changes and the final posterior probability was 0.971 (Richter et al. 2023).

Figure 3.

Posterior predictive distributions of 6-month abstinence rates for the last cohort using data from prior to 10^th interim analysis.

Impact of RAR on Allocation and Outcomes

Allocation was calculated based on 1-month quit rates only. After the first interim analysis the allocation ratio favoured the opt-out arm, and throughout the trial (Table 2). The variance of the probability of successful 1-month outcome decreased throughout the trial with an increased number of enrolled participants. The posterior probability was the major contributor to the increased proportion of participants allocated to the better performing arm, but its effect was reduced by an increased number of participants in this arm. The proportion of subjects assigned to opt-out arm was increasing from 60.1% after the 1^st interim to 83% after the 5^th interim. As the maximum posterior probability at the 6^th and the 7^th interim analyses dipped slightly, so did RAR in opt-out arm. At the 8^th interim analysis, the maximum posterior probability bounced back to the 5^th interim levels. Even though after the last interim only 16.5% participants were assigned to the opt-in arm, a larger initial allocation ratio kept the overall number of subjects in the opt-out arm at 65.5%.

Table 2.

Results of the RAR. The subscripts “1” and “2” represent opt-in and opt-out groups respectively. N is the number of subjects randomized and n is the number with observed outcomes (after dropout).

Interim	N1	N2	n1	n2	${\left({\mathit{P}}_1^{1\mathit{m}}\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}$	$({{\mathit{P}}_2^{1\mathit{m}})}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}$	${\mathit{V}\mathit{a}\mathit{r}\left({\mathit{\theta }}^{{\mathit{Q}}_1}\right)}^{1/2}$	${\mathit{V}\mathit{a}\mathit{r}\left({\mathit{\theta }}^{{\mathit{Q}}_2}\right)}^{1/2}$	${\mathit{V}}_1$	${\mathit{V}}_2$	${\mathit{R}\mathit{A}\mathit{R}}_1$	${\mathit{R}\mathit{A}\mathit{R}}_2$
1	201	199	143	148	0.558519	0.829492	0.030558	0.031574	0.001422	0.002146	0.399	0.601
2	225	237	175	174	0.384844	0.922982	0.026389	0.029257	0.000766	0.002041	0.273	0.727
3	245	281	198	207	0.381948	0.924184	0.024312	0.026226	0.000658	0.001681	0.281	0.719
4	266	332	217	244	0.240420	0.970669	0.022313	0.024284	0.000363	0.001506	0.194	0.806
5	282	396	231	277	0.195540	0.980696	0.021368	0.022846	0.000274	0.001344	0.170	0.830
6	295	455	220	273	0.232026	0.97271	0.023983	0.024322	0.000374	0.001429	0.208	0.792
7	309	506	229	314	0.294194	0.955746	0.023661	0.022432	0.000459	0.001208	0.275	0.725
8	325	555	238	351	0.195108	0.980782	0.023155	0.021621	0.000292	0.001130	0.205	0.795
9	335	598	253	390	0.132631	0.991165	0.022416	0.020780	0.000187	0.001042	0.152	0.848
10	342	639	259	422	0.137452	0.990508	0.022385	0.020053	0.000191	0.000966	0.165	0.835
Final	345	655	270	469

Comparison of trial results to operating characteristics

Simulation results showed that on average 67.8% of patients would be assigned to the better performing opt-out arm under “expected” scenario, and 68.3% patients would be in opt-out arm under “expected/smaller” scenario (Table 3). The difference is explained by the fact that a trial would run longer under “expected/smaller” scenario allowing more interims to happen. The final analysis resulted in 655 (65.5%) participants assigned to the opt-out arm (469 after dropouts). This ratio was lower than the “expected” description of simulations. The observed 1 month quit rates were slightly lower than the “expected” scenario, affecting the maximum posterior probability and consequently the allocation ratio. In terms of resulting posterior probabilities (Table 3), CTD had followed “expected/smaller” scenario more closely.

Table 3.

Allocation under different scenarios and trial enrollment rates. $P_2^{1m}$ and $P_2^{6m}$ denote the mean posterior probabilities of opt-out arm having higher quit rates at 1 and 6 months respectively.

Description	Scenario	n	Percentage of successful trials based on co-endpoint	${\mathit{P}}_2^{1\mathit{m}}$	${\mathit{P}}_2^{6\mathit{m}}$	Allocation to opt-out arm	More enrolled in opt-out arm	Dropout rate
Expected	15.7% vs 25.2% at month 1/ 15.7% vs 25.2% at month 6	961	82%	0.964	0.958	0.678	343	315
Expected/Smaller	15.7% vs 25.2% at month 1/ 15.7% vs 20.0% at month 6	990	60%	0.963	0.779	0.683	362	325
No difference	15.7% vs 15.7% at month 1/ 15.7% vs 15.7% at month 6	1000	6%	0.353	0.432	0.457	−86	322
Trial Results	N.A.	1000	N.A.	0.971	0.591	0.655	310	261

Post-hoc drift analysis

The model estimates that the drift was never large enough to be meaningfully influential (Figure 4, Figure 6). There is a slight divergence in the log-odds of the probability of observing a successful outcome at month 1. Appearing after the 5^th interim, negative drift indicates that there was a minor time trend present and that 1-month quit rates increased by 0.013 over the course of the trial. Almost all observed values were covered by credible intervals in the opt-out case (Figure 5). The opt-in arm observed quit rates were partially covered by the credible intervals. However, we observed values outside of both upper and lower limits. The fact that allocation started to favour the better performing opt-out arm after 2^nd interim might have influenced trial results for the 1-month outcome, but the estimated magnitude of the time trend doesn’t seem to be a concern in this particular case.

Figure 4.

Estimated drift $l_i$ with 95% credible intervals at 1-month analysis. The dashed grey lines indicate credible intervals. The dotted and dashed black lines indicate the estimated drift under alternative hyperpriors. The assumed zero $l_{11}$ at the final analysis is not included in the figure.

Figure 6.

Estimated drift $l_i$ with 95% credible intervals at 6-month analysis. The dashed grey lines indicate credible intervals. The dotted and dashed black lines indicate the estimated drift under alternative hyperpriors. The assumed zero $l_{11}$ at the final analysis is not included in the figure.

Figure 5.

Estimated 1-month ${\theta }^{Q_j}$ with 95% credible intervals. The final analysis is under the 11th interim. The dashed lines indicate credible intervals. The dashed-dotted lines indicate observed abstinence.

There is an indication that 6-month quit rates were fairly stable throughout the trial and were not affected by the time trend (Figure 6). Similar to the 1-month outcome model, we estimated that the log-odds increased over the course of the trial. The fitted quit rates increased for no more than 0.08 from the first interim to the last (Figure 7). The 95% credible intervals covered the observed rates well up to 9^th interim in both arms. After 9^th interim unexpectedly abstinence rate started to climb up in opt-in arm. However, this was at the late stage of the trial (6.7% of the participants to be randomized) and consequently the change in the trend shouldn’t have impacted the trial’s conclusion for the 6-month outcome drastically.

Figure 7.

Estimated 6-month ${\theta }^{Q_j}$ with 95% credible intervals. The final analysis is under the 11th interim. The dashed lines indicate credible intervals. The dashed-dotted lines indicate observed abstinence.

Sensitivity analysis

The sensitivity analysis explored effects of the choice of the prior on the smoothing parameter. We evaluated two assumptions of more aggressive $({\tau }^2~IG\left(1,0.001\right))$ and less aggressive $({\tau }^2~IG\left(1,0.1\right))$ smoothing. The dashed and dotted black lines on Figures 4 and 6 demonstrate the estimated drift for these two assumptions. At 1-month analysis the chosen model showed compromise between two alternative models. The original model $({\tau }^2~IG\left(0.1,0.001\right))$ was closer in level of smoothing to the more aggressive model, but 95% credible intervals still covered the estimated drift under the less aggressive model. At 6-month analysis the model with more smoothing showed drift levelling out the sharp drop in quit rates at the final analysis. The less aggressive model put more emphasis on the data and consequently we observed jump in the estimated drift from the final analysis. The posterior predictive distributions indicated the last drastic change in opt-in arm was unexpected. Otherwise, the estimated drift of the less aggressive model was fairly stable. After initial jump from 0 at the final analysis it followed the upper limit of the credible interval relatively close. This particular case demonstrated why the picked hyperprior provided good fit to the data without sacrificing assessment of the general trend.

Discussion

Helping people quit smoking is a powerful tool to improve public health. As a majority of smokers are interested in quitting (Babb 2017) even a small “nudge” could help them achieve their goal. CTD was essentially comparing two effective treatment strategies in a ‘real-world’ pragmatic setting. Bayesian adaptive design was suitable for this purpose. The uninformative prior, conservative initial allocation ratio, and higher threshold for stopping for success created a conservative setting to keep possible downsides of the design minimized. This conscious decision had been proven to be justified. Up to the 6^th interim analysis, the posterior probability of one arm showing abstinence rates higher than the other was climbing closer to predefined threshold of early stopping for success. Prior to 500 patients being enrolled, study investigators, who were blinded to interim analyses, feared the trial would stop. The month 6 outcome was introduced to the stopping criterion to make it more strict and to facilitate analyses of the long-term effects of opt-out care.

Initially, the protocol did not specify a dropout rate. During trial execution it became evident that dropout rates may have been affecting power. The post-hoc assessment of operating characteristics showed that the power came down to 82% from 99% initially under the “expected” scenario and from 91% to 60% under “expected/smaller” scenario. The trial resulted in 97.1% posterior probability of treatment arm being better at 1-month analysis and 59.1% at 6-month analysis. It is likely that the trial followed in general the “expected/smaller” scenario. The opt-out arm performed better than opt-in throughout all interims for the 1-month outcome. The 6-month posterior probability was around 80% for opt-out in the first 5 interims but dropped for opt-out participants in the last cohort.

The potential presence of a time trend is one of the major arguments against RAR designs. Bayesians now address this risk through a prespecified time trend model (e.g., “Bayesian time machine”). There was no reason to believe there was a drift in the presented case because no changes in trial features were introduced besides the co-endpoint. There was an increase in treatment effect over time as more patients were simultaneously being randomized to the opt-out arm. The drift model parsed out the treatment effect because the drift model assumed that the drift impacted both arms on the log-odds scale in the same magnitude and did not have separate drift for each treatment. This paper showed an example of a single trial where even though the analysis did not account for time trend, post hoc analysis showed that it did not occur at a magnitude to affect study outcomes. The fact that initial allocation accounted for 40% of patients minimized the possibility of overly skewed results and benefitted participants in the trial. As argued previously (Lipsky and Greenland 2011) time trends could be major concerns, but if managed properly the bias they bring could be minimized while at the same time maximizing benefit to participants.

While the short-term effect of the intervention was evident, the probability that opt-out was better than opt-in at 6-month post randomization was just over 0.5. As mentioned above participants had 4 counselling sessions during first months after hospitalization. Following unblinding of the study results, the CTD investigative team hypothesized that prolonged access to counselling could extend cessation. Future research could examine the impact of opt-out versus opt-in care by extended versus short-term treatment, converting CTD to essentially a multi-arm adaptive platform trial. Participants in “opt-out boost” arm could be offered counselling sessions for 6 months and quitting rates will be assessed at month 1 and month 6. As shown by Saville et al. (2022) “Bayesian Time Machine” model is helpful in a case of comparison of newer treatment arm to a former control or a treatment arm. A temporal drift is a major argument against pooled analysis in multi-arm platform trials. As we didn’t find presence of a time trend concerning in the original trial, this novel approach offers a solution for the next trial that builds of the obtained results.

Conclusion

Cautiously planned Bayesian response adaptive randomization design served well for the purpose of comparing two effective treatment strategies. The trial followed SOP closely and we encountered no operational difficulties that would prevent the team from adopting adaptive designs in future trials. The final analysis was not adjusted for time trend, but we found no evidence that patient drift was a concern for this trial. This wouldn’t be applicable for all situations, but we emphasize that the CTD trial was an example of successful two-armed confirmatory trial with Bayesian adaptive design.

Appendix 1

model<- function() {
 for (i in 1:11){
 for (j in first_pt_in[i]:last_pt_in[i]){
  dYY1in[j]~dbin(P_in[i],1)
 }
 P_in[i]<-exp(theta_in[i])/(1+exp(theta_in[i]))
 for (j in first_pt_out[i]:last_pt_out[i]){
  dYY1out[j]~dbin(P_out[i],1)
 }
 P_out[i]<-exp(theta_out[i])/(1+exp(theta_out[i]))
 }#end of i loop 
 for (j in 1:11){
 theta_in[j]<-Theta[1]+ltheta[j]
 theta_out[j]<-Theta[2]+ltheta[j]
 }
 ltheta[11]<−0
 for (j in 1:10){
 ltheta[11-j]~dnorm(ltheta[11-j+1],invT)
 }
 invT~dgamma(0.1,0.001)
 Theta[1]~dnorm(0,.0001)
 Theta[2]~dnorm(0,.0001)
 p<-step(Theta[2]-Theta[1])
 }
out <- bugs(data, inits=NULL,
  parameters.to.save = c(“ltheta”,”p”,”Theta”,”invT”,”theta_in”,”theta_out”), 
  model.file = model, n.chains = 1, n.iter = 102000,
  n.burnin = 60000, debug=T)