Bayesian Adaptive Cluster-Randomized Designs with Long Primary Endpoints for Pragmatic Weight Loss Studies Implemented in Rural Communities

Abstract

With the rise of obesity and other metabolic-related disorders since the early 2000s, there is a pressing need for innovative methods to reduce this trend. Many existing studies have long endpoints and protracted trial durations, often requiring substantial implementation costs. To address this, we propose the use of an adaptive trial design. This approach, which is at the forefront of research methodology, can significantly reduce the total duration of a trial, enhance participant outcomes by determining treatment efficacy earlier, and increase the number of participants randomized to the beneficial treatment groups. We suggest using a Bayesian Adaptive Cluster Randomized design to redesign the REPOWER trial, a weight loss study comparing behavioral interventions on weight loss in rural communities. Our approach incorporates an arm-dropping technique and will stop the trial early for futility if sufficient weight loss differences are not observed. We illustrate this by calculating operating characteristics under six scenarios to compare to the design used in the original trial.

1. Introduction

According to a systematic literature review conducted by Jones et al. 2023¹, only 27 of 211 papers with the keywords 'Bayes' and 'cluster-randomized control trials (CRCTs)' in their Title, abstract, or text contained the relevant content of both Bayesian and CRCT¹. Bayesian is a natural way to adopt adaptive trials as posterior and predictive distributions can easily be derived for early stopping such as futility.This scarcity underscores the urgent need for Bayesian methodological development in designing and analyzing cluster-randomized control trials, referred to as Bayesian cluster-randomized control trials (BCRCTs)¹.

In a CRCT, randomization occurs at the group or “cluster” level instead of the individual level, which is typical of traditional Randomized Controlled Trials (RCTs)^1–4. CRCTs are widespread in clinical trials due to the various naturally occurring clustering of participants. More participants are generally required for these trials due to an induced dependence between experimental units within clusters. As the correlation between participants within a cluster increases, the overall statistical power to detect effect sizes between the interventions decreases^2,3,4,7. To improve trial efficiency, incorporating adaptive features into the CRCT, referred to as adaptive cluster-randomized control trial (ACRCT), at the design phase may offset this reduction in statistical power⁷.

The natural fit of Bayesian methods into adaptive designs further enhances their potential, as Bayesian methods function by updating probabilities of certain population characteristics based on accumulated data from trial interims. While many advantages and limitations of adaptive designs are discussed in more detail by the FDA⁶, this paper aims to develop methods for Bayesian adaptive cluster randomized trials (BACRT) with extended endpoints. This will be accomplished by first introducing the motivating trial that will be redesigned under the BACRT framework. Following, we present details of the trial designs, the statistical model used to generate the data, a cluster-level and participant-level analysis comparing two software packages, and finally, the operating characteristics and results of redesigning the original trial as a BACRT with varying effect sizes based on the original trial outcomes.

1.1. Motivating Settings

With obesity and other metabolic-related disorders increasing in trend since the early 2000s^11–14, specifically in rural and other underserved communities^12,13, more clinical studies have attempted to address these growing concerns. One study, conducted at the University of Kansas Medical Center, called REPOWER¹³ compared the difference in weight loss between two novel behavioral intervention models and the current Medicare-approved Fee-for-Service (FFS) model. The population that they targeted were individuals who lived in rural communities in the United States. While REPOWER demonstrated effective intervention methods, the trial itself lasted about three years. Many similar weight loss studies tend to have long primary endpoints (> 12 months), causing several years to elapse before any conclusions regarding efficacy can be made. This can prove costly and require more participants to continue receiving the inferior or ineffective intervention methods for longer than necessary. As such, a redesign of REPOWER under similar constraints will be conducted to demonstrate the effectiveness of adaptive features on trial performance.

2. METHODS

2.1. Trial Designs

2.1.1. Original REPOWER Design

The original REPOWER trial was a fixed non-adaptive cluster randomized design aiming to compare three behavioral interventions’ effectiveness on weight loss in patients in rural communities. Thirty-six clinics were equally allocated to one of three intervention arms, two novel treatments (in-clinic group and telephone group), and one comparator. They were randomized at the clinic level in groups of twelve and set to be activated at three different time points within the first year of trial start. The clinics each enrolled about forty participants, and after dropouts and imputation, 1407 observations remained for the final analysis. The first 12 clinics were randomized at the start of the trial, 12 more were randomized six months into the trial, and the final 12 clinics were randomized about 10 months into the trial. Each participant had their weight measured at baseline, 6, 18, and 24 months. The primary outcome was a change in weight at 24 months post-randomization. The primary aim was to compare weight change in the novel treatment arms with weight change in the comparator arm. $\left({\mathrm{\Delta }}^{\prime }s\right)$. These ${\mathrm{\Delta }}^{\prime }s$ are specified in section 2.3, and they vary per scenario. The final analysis in the original trial was conducted using a linear mixed model to estimate the separate intervention effects and between intervention differences. The original trial had the relative timeline shown in Figure 1.

Figure 1.

Relative randomization and visit times

2.1.2. Bayesian Adaptive Cluster-Randomized Design

The adaptive design used the same maximum number of clinics, total observations, randomization dates, and measurement dates to create a realistic comparison to the original design. Under those constraints, a single interim analysis at 6-months into the trial was to be performed. This visit only included data for the first 12 clinics’ 6-month visits. The decision for a single interim at this time point is due to the length of the endpoints planned in the original trial design (6,18,24), thus limiting the number of adaptations that could be incorporated into the final design. An arm-dropping technique was implemented in the interim to stop the allocation of participants to the novel treatment arms performing as well as or worse than the comparator arm. This also included a futility rule that would stop the trial early for futility if both arms were dropped or deemed to be ineffective. If only a single arm is dropped, the subsequent participants would be allocated to the remaining treatment arm and the comparator arm in a 2:1 ratio. If both arms are dropped, the interventions are declared ineffective, and the trial is stopped early for futility. The flow chart for this new adaptive design is shown in Figure 2.

Figure 2.

Flow Chart for the Adaptive Design Process

2.1.3. Interim Analysis

An interim will be performed after the 6-month visit is completed for the first 12 clusters randomized in the trial. The model used at the interim will be a Bayesian independent arm model where the mean parameters ${\mu }_d$ are assumed to be independent.

$$Y_d{~}^{iid}N\left({\mu }_d,{\sigma }^2\right)$$

• $Y_d$ is the change in weight and varies between analysis types (described more in Section 2.8)

• ${\mu }_d$ is the mean for arm $d$, where $d=0,1,2$ for the comparator arm, in-person group, and telephone groups, respectively.

• ${\sigma }^2$ is the common variance across all arms.

Independent priors used for this analysis include: ${\mu }_d{~}^{iid}N\left(0,5^2\right)\&{\sigma }^2~IG(1,4)$, where IG is the Inverse-Gamma distribution. The mean change in weight for each of the intervention arms is as follows,

$$\text{Comparator}\text{Arm:}{\mu }_0$$

$$\text{In-Clinic}\text{Group}\text{arm:}{\mu }_1$$

$$\text{Telephone}\text{Group}\text{arm:}{\mu }_2$$

The priors for the mean parameter is weakly informative. We assume there was a 95% chance that the mean weight lost during the study would be between −10 and 10 kgs, for each arm prior to gathering any data. According to the study PI this is a reasonable assumption given that it was a behavioral intervention study. The hyperparameters for the variance prior were chosen as one and 4. This is because we parameterized the IG as a scaled-inverse-chi-square distribution and assumed a prior sample size of two and a prior standard deviation of two. As follows, the expected variance is about 4 kgs for each study arm with a small amount of “weight” being placed on that assumption so that if the data shows otherwise, it won’t weigh the posterior probabilities in favor of the prior over the likelihood.

2.3. Bayesian Quantities of Interest

Posterior distributions are used to estimate the primary quantity of interest, which is the probability that a novel treatment yields better weight loss outcomes than the comparator arm,

$$P\left({\mu }_d>{\mu }_0\right)$$

where ${\mu }_d$ for $\mathrm{d}=1,2$ represents the mean weight loss for the in-person and telephone groups, respectively, and ${\mu }_0$ is the mean weight loss for the comparator group. These posteriors are derived from the models in Section 2.9.

2.4. Success and Futility Decision Criteria

Frequentist operating characteristics (FOC) are used to compare the designs since the original trial design was constructed around frequentist methods. As such, a family-wise type-1 error rate of ≤ 5% and a hypothesis-wise type-1 error rate of < 2.5% were chosen, the same as in the original trial protocol.

A simulation was a success if the Bayesian Quantity of Interest exceeded the 0.975 threshold,

$$P\left({\mu }_d>{\mu }_0\right)>0.975$$

The type-1 error and success criteria were the same for both designs and did not need to be adjusted in the adaptive design due to no efficacy decisions made at the interim.

For the futility criteria for the first set of simulation scenarios are,

$$P\left({\mu }_d>{\mu }_0\right)\le 0.5$$

The futility criteria for the second set of simulation scenarios are,

$$P\left({\mu }_d>{\mu }_0\right)\le 0.65$$

The decision to choose these exact criteria was made after comparing the operating characteristics across a range of potential futility thresholds (0.10 – 0.90). Of these 0.5 and 0.65 maintained the nominal type-1 error rate of 0.05 for all scenarios while giving a modest increase in power and reducing trial length when needed. Details around the operating characteristics are given in the supplementary materials, Supplementary Table 1.

2.5. Statistical Model Used to Generate the Trial Data and Simulation Scenarios

2.5.1. Hierarchical Model

The model used to generate the trial data for calculating operating characteristics is based on Model 1 in Tang¹⁸. It is a three-level hierarchical model:

$$Y_{ijk}={\alpha }_{0ij}+{\beta }_1{T}_1+{\beta }_2{T}_2+{\beta }_3{t}_{18}+{\beta }_4{t}_{24}+{\beta }_5\left(T_1{t}_{18}\right)+{\beta }_6\left(T_2{t}_{18}\right)+{\beta }_7\left(T_1{t}_{24}\right)+{\beta }_8\left(T_2{t}_{24}\right)+{ϵ}_{ijk}$$

• ${\alpha }_{0ij}={\alpha }_{0j}+{\gamma }_i$, where ${\gamma }_i{~}^{iid}N(0,{\sigma }_{\gamma }^2)$ is the patient-level variation.

• ${\alpha }_{0j}={\beta }_0+{\eta }_j$, where ${\eta }_j{~}^{iid}N\left(0,{\sigma }_{\eta }^2\right)$ is the clinic-level variation.

• ${ϵ}_{ijk}{~}^{iid}N\left(0,{{\sigma }^2}_{ϵ}\right)$ is the within-patient residual error.

• $\gamma ,\eta ,ϵ$ are assumed to be mutually independent.

Where, $k$ indexes visits, $\mathrm{j}$ indexes clinics, and $i$ indexes patients such that $Y_{ijk}$ is the change in weight for a single subject within a single clinic at a single visit. The variables $T_1$ and $T_2$ are intervention arm indicators that take values of (0,0) if the patient is receiving the in-clinic individual intervention (comparator arm), (1,0) for the in-clinic group treatment, and (0,1) for the telephone group treatment. The time indicators, $t_{18}$ and $t_{24}$, represent visits 18 and 24, respectively. They take values of (0,0) for the 6-month visit, (1,0) for the 18-month visit, and (0,1) for the 24-month visit.

In this model, ${\beta }_0$ represents the mean weight loss for the comparator arm at the 6-month visit. ${\beta }_1$ and ${\beta }_2$ represent the difference in weight loss between the comparator arm and the IC Group and Telephone Group at the 6-month visit, respectively. ${\beta }_3$ and ${\beta }_4$ represent the difference in weight loss for a given arm at the 18-month and 24-month visits compared to the 6-month visit. ${\beta }_5,{\beta }_6,{\beta }_7$, and ${\beta }_8$ are the interactions between time and treatment group when compared to the comparator arm. The mean weight loss at the 24-month visit by group can be written as follows,

$$\text{Comparator}\text{Arm:}{\mu }_0={\beta }_0+{\beta }_4$$

$$\text{In-Clinic}\text{(IC)}\text{Group}\text{arm:}{\mu }_1={\beta }_0+{\beta }_1{+\beta }_4+{\beta }_7$$

$$\text{Telephone}\text{Group}\text{arm:}{\mu }_2={\beta }_0+{\beta }_2+{\beta }_4+{\beta }_8$$

2.5.2. Effect Sizes

Six sets of effect sizes are used for simulation. The values for the variance parameters are derived from the secondary analysis conducted by Tang¹⁸. For simplicity they are assumed to be the same across scenarios. The parameters, ${\Delta }_{0d}$, represent the difference in means between the novel interventions and the comparator arm at the 24-month visit, for $\mathrm{d}=1,2$ (1 = in-clinic group, 2 = telephone group). They’re calculated for each of the arms as, ${\Delta }_{01}={\mu }_1-{\mu }_0$ and ${\Delta }_{02}={\mu }_2-{\mu }_0$.

The null scenario is where the difference in means is zero for all treatment comparisons. The protocol scenarios are based on the original trial protocol. They consist of one best (only one treatment is superior to comparator), two best (two treatments are superior), and half best (both are superior but one of them is only half as effective as the other). The non-protocol one best and two best scenarios are based on the actual effect sizes estimated from the study data given in Teng¹⁸ but follow the same logic as the protocol versions.

2.7. Software

Two software packages were used in this study. The first is Fixed and Adaptive Clinical Trial Designs Simulator (FACTS), a proprietary software of Berry Consultants, LLC. The second is Statistical Analysis Software (SAS).

2.8. Aggregated (Clinic-Level) vs Unaggregated (Patient-Level) Analyses

There are two “levels” of analyses compared in this study. This differs from the “levels” used to refer to hierarchical models in most contexts. The “levels” in this paper will refer to aggregated vs unaggregated data. The first of these levels is the patient-level analysis conducted in the original trial and subsequent secondary analyses¹⁷. Each patient is treated as a single observational unit for the final analysis, which usually better accounts for more sources of variation observed in the data such as between visit variation.

The second level is clinic-level analysis. This is where the patient-level data is aggregated within the clinic, making the clinic average the observational units for the final analysis. As the number of participants per clinic increases, the between-subject variation shrinks towards zero. Under the assumption of conditionally normally distributed responses, averaging over patients necessarily shrinks the total variation observed in this newly aggregated data while also reducing the sample size with which to work. As will be discussed in a later section, working with this level of data, based on the assumptions used to produce the original data in the section 2.5.1, holds several advantages from a simulation perspective.

2.9. Models for Comparison

For the patient-level data analysis, a Bayesian linear mixed model (BLMM) was fit using the BGLIMM procedure in SAS, and the study arm means and standard deviations were estimated. The BLMM is as follows,

$$y_{ijk}={\alpha }_{0ij}+{\beta }_1{T}_1+{\beta }_2{T}_2+{\beta }_3{t}_{18}+{\beta }_4{t}_{24}+{\beta }_5\left(T_1{t}_{18}\right)+{\beta }_6\left(T_2{t}_{18}\right)+{\beta }_7\left(T_1{t}_{24}\right)+{\beta }_8\left(T_2{t}_{24}\right)+{ϵ}_{ijk}$$

• ${\alpha }_{0ij}={\alpha }_{0j}+{\gamma }_i$, where ${\gamma }_i{~}^{iid}N(0,{\sigma }_{\gamma }^2)$ is the patient-level variation.

• ${\alpha }_{0j}={\beta }_0+{\eta }_j$, where ${\eta }_j{~}^{iid}N\left(0,{\sigma }_{\eta }^2\right)$ is the clinic-level variation.

• ${ϵ}_{ijk}{~}^{iid}N\left(0,{{\sigma }^2}_{ϵ}\right)$ is the within-patient residual error.

• $\gamma ,\eta ,ϵ$ are assumed to be mutually independent.

The priors and hyperparameters used for this analysis include: a noninformative prior for the patient-level variation, ${\sigma }_{\gamma }^2~IG\left(0.05,0.05\right)$, weakly informative priors for both the clinic-level variation and residual variation, ${\sigma }_{\eta }^2~IG\left(2,2\right)$ and ${\sigma }_{ϵ}^2~IG(1,4)$, respectively, and independent flat priors for the fixed-effect coefficients, $\pi \left(\beta \right)\propto 1$. IG is the inverse gamma distribution. Using this model, the intervention effects for each of the study arms at the 24-month visit can be written as follows:

$$\text{Comparator}\text{Arm:}{\mu }_0={\beta }_0+{\beta }_4$$

$$\text{In-Clinic}\text{Group}\text{arm:}{\mu }_1={\beta }_0+{\beta }_1{+\beta }_4+{\beta }_7$$

$$\text{Telephone}\text{Group}\text{arm:}{\mu }_2={\beta }_0+{\beta }_2+{\beta }_4+{\beta }_8$$

Two separate models were used in the comparison for the aggregated clinic-level data. The first is a Bayesian linear mixed model, while the second is an independent arm model. The aggregated Bayesian linear mixed model, like the one used in the patient-level analysis, is as follows,

$${\underset{\_}{y}.}_{jk}={\alpha }_{0j}+{\beta }_1{T}_1+{\beta }_2{T}_2+{\beta }_3{t}_{18}+{\beta }_4{t}_{24}+{\beta }_5\left(T_1{t}_{18}\right)+{\beta }_6\left(T_2{t}_{18}\right)+{\beta }_7\left(T_1{t}_{24}\right)+{\beta }_8\left(T_2{t}_{24}\right)+{\underset{\_}{ϵ}}_{jk}$$

• ${\alpha }_{0j}={\beta }_0+{\eta }_j$, where ${\eta }_j{~}^{iid}N\left(0,{\sigma }_{\eta }^2\right)$ is the clinic-level variation.

• ${\underset{\_}{ϵ}}_{.jk}{~}^{iid}N\left(0,{\sigma }_{\underset{\_}{ϵ}}^2\right)$ is the averaged within clinic residual variation post aggregation.

• ${\underset{\_}{y}.}_{jk}={\sum }_i^{n_j}\frac{y_{ijk}}{n_j}$, is the average response for the j^th clinic at the k^th visit.

• ${\underset{\_}{ϵ}}_{.jk}={\sum }_i^{n_j}\frac{{ϵ}_{ijk}}{n_j}$, is the average residual error for the j^th clinic at the k^th visit.

• $\underset{\_}{ϵ},\eta$ are assumed to be mutually independent.

The priors and hyperparameters used for this analysis are very similar to the ones used for the patient-level model. The priors and hyperparameters include weakly informative priors for both the clinic-level and residual variations, ${\sigma }_{\eta }^2~IG\left(2,2\right),{\sigma }_{\underset{\_}{ϵ}}^2~IG(1,4)$, and independent flat priors for the fixed-effect coefficients, $\pi \left(\beta \right)\propto 1$, same as in the previous model. The 24-month intervention effects are computed in the same way that as the previous model.

$$\text{Comparator}\text{Arm:}{\mu }_0={\beta }_0+{\beta }_4$$

$$\text{In-Clinic}\text{Group}\text{arm:}{\mu }_1={\beta }_0+{\beta }_1{+\beta }_4+{\beta }_7$$

$$\text{Telephone}\text{Group}\text{arm:}{\mu }_2={\beta }_0+{\beta }_2+{\beta }_4+{\beta }_8$$

The second model used for the aggregated data analysis is an independent arm model where only the 24-month visit is analyzed. This analysis was conducted in FACTS and is as follows,

$$Y_d{~}^{iid}N\left({\mu }_d,{\sigma }^2\right)$$

• $y_d={\stackrel{-}{y}}_{.\mathrm{j}\mathrm{k}}={\sum }_i^{n_j}\frac{y_{ijk}}{n_j}$, is the average response for the j^th clinic at the k^th visit.

• ${\mu }_d$ is the mean for the $d^{th}$ arm, $d=0,1,2$ for the comparator arm, in-person group, and telephone groups, respectively.

• ${\sigma }^2$ is the common variance across all arms.

Independent priors used for this analysis include: ${\mu }_d{~}^{iid}N\left(0,5^2\right)\&{\sigma }^2~IG(1,4$). Since this model is only concerned with the 24-month visit, the effects for each of the intervention arms are as follows,

$$\text{Comparator}\text{Arm:}{\mu }_0$$

$$\text{In-Clinic}\text{Group}\text{arm:}{\mu }_1$$

$$\text{Telephone}\text{Group}\text{arm:}{\mu }_2$$

3. Results

3.1. Comparison of Estimates and Posterior probabilities for FACTS vs. SAS on a Single Simulated Dataset

This section aims to show that the clinic-level analysis conducted in FACTS, which relies only on the 24-month visit, is sufficient for the analysis purposes compared to the patient-level and clinic-level analyses using Bayesian Linear Mixed Models (BLMM) conducted in SAS. To demonstrate, we generated a single dataset for each of the first three scenarios in the “Effect Sizes” section. Then, we compare the posterior estimates after fitting the three models specified in the “Models used for Comparison” section.

For each of the scenarios, the posterior means for the intervention arms, standard deviations of the posterior means, and the quantities of interest $P\left({\mu }_d>{\mu }_0\right)$ were estimated using the three different models. The results of the model comparisons are summarized in Table 2.

Table 2.

Comparison of three models on a single set of simulated data. ( IC Grp = “In-clinic Group”, Tel Grp = “Telephone Group”, Comparator = “In-clinic Individual”)

Scenario 1. Null Scenario	N = 5000	Patient BLMM	Clinic BLMM	Clinic Indep. Arm
	Mean Comparator	3.3	3.2	3.1
	Mean IC Grp	3.4	3.4	3.4
	Mean Tel Grp	4.0	4.0	3.9
	SD Comparator	0.77	0.80	0.80
	SD IC Grp	0.78	0.80	0.82
	SD Tel Grp	0.76	0.80	0.78
	P(IC Grp > Comparator)	0.55	0.56	0.58
	Pr(Tel > Comparator)	0.75	0.76	0.76
Scenario 2. One Best	N = 5000	Patient BLMM	Clinic BLMM	Clinic Indep. Arm
	Mean Comparator	3.8	3.8	3.7
	Mean IC Grp	6.1	6.1	6.0
	Mean Tel Grp	2.6	2.6	2.5
	SD Comparator	0.58	0.58	0.59
	SD IC Grp	0.58	0.58	0.60
	SD Tel Grp	0.58	0.57	0.57
	P(IC Grp > Comparator)	1.0	1.0	1.0
	Pr(Tel > Comparator)	0.07	0.07	0.07
Scenario 3. Two Best	N = 5000	Patient BLMM	Clinic BLMM	Clinic Indep. Arm
	Mean Comparator	3.1	3.0	3.0
	Mean IC Grp	5.7	5.7	5.6
	Mean Tel Grp	4.8	4.8	4.7
	SD Comparator	0.65	0.69	0.71
	SD IC Grp	0.67	0.69	0.72
	SD Tel Grp	0.66	0.69	0.69
	P(IC Grp > Comparator)	1.0	1.0	1.0
	Pr(Tel > Comparator)	0.97	0.97	0.96

The results show that while the patient-level BLMM does produce slightly more reliable results due to incorporating more information, the independent arm model in FACTS performs sufficiently well and yields similar results to the other two models. This is consistent with what we would expect since the independent arm model only incorporates information about each clinic through the 24-month visit and fails to explain the additional sources of variation due to the trial design. The decisions reached for each scenario are also consistent based on the success criteria in the Section 2.4.

Since the comparison above shows that the model in FACTS produces sufficient results, the comparison of trial designs will only use the FACTS model. This is due primarily to the speed at which the simulations in FACTS occur. For example, running 1,000 iterations for a single scenario for the patient-level analysis in SAS took about 8 hours. A large part of this is due to the nature of BLMMs, where, typically, a much larger number of posterior distributions are being simulated. The clinic-level analysis using BLMM in SAS for the same scenario and the same number of iterations took about an hour, which was much better but still nowhere close to the speed that FACTS achieved. When all conditions outside the model type are the same, the simulation in FACTS took about 1 minute for 1000 iterations. This decrease in computation time and ease of use were the primary reasons FACTS was used to compare trial designs in section 3.2.

3.2. Fixed vs Adaptive Design Simulations

Using FACTS for the simulations, frequentist operating characteristics for each design and futility criteria were calculated and listed in Table 1. Frequentist operating characteristics were used to evaluate simulation scenarios due to being required by the FDA for design planning.

Table 1.

Scenarios

Parameter	Null Scenario	One Best Scenario	Two Best Scenario	Protocol One Best Scenario	Protocol Half Best Scenario	Protocol Two Best Scenario
${\beta }_0$	6.24	6.24	6.24	6.24	6.24	6.24
${\beta }_1$	0	2.67	2.67	3.25	3.25	3.25
${\beta }_2$	0	0	1.98	0	1.45	2.90
${\beta }_3$	−2.26	−2.26	−2.26	−2.26	−2.26	−2.26
${\beta }_4$	−3.05	−3.05	−3.05	−3.05	−3.05	−3.05
${\beta }_5$	0	0.06	0.06	0.06	0.06	0.06
${\beta }_6$	0	0	−2.16	−2.16	−2.16	−2.16
${\beta }_7$	0	−0.41	−0.41	−0.50	−0.50	−0.41
${\beta }_8$	0	0	−0.1	0	−0.07	−0.1
${\sigma }_{\gamma }$	6.66	6.66	6.66	6.66	6.66	6.66
${\sigma }_{\eta }$	0.98	0.98	0.98	0.98	0.98	0.98
${\sigma }_{ϵ}$	3.92	3.92	3.92	3.92	3.92	3.92
${\Delta }_{01}={\mu }_1-{\mu }_0$	0	2.26	2.26	2.75	2.75	2.75
${\Delta }_{02}={\mu }_2-{\mu }_0$	0	0	1.88	0	1.375	2.75

Overall, the adaptive design tended to perform as well as or better than the fixed design for all scenarios and futility criteria. This included having similar Bayesian type 1 errors and power. For the null scenario where the Futility Rule was set to 0.5, the adaptive design stopped the trial early about 33% of the time. While in the scenario where the futility rule was set to 0.65, the null scenario stopped the trial early about 50% of the time. The null scenarios also maintained the family-wise type-1 error rate described insection 2.1, $P({P(\theta }_d>{\theta }_0)>0.975|{\theta }_d\le {\theta }_0)<0.05$, for all the designs and futility criteria (Table 3). This conditional probability is evaluated by averaging the number of simulations in the null scenario where the BQI exceeds the decision threshold of 0.975 (i.e. leads to the incorrect decision).

Table 3.

Fixed vs Adaptive Operating Characteristics for 10,000 simulations. ( IC Grp = “In-clinic Group”, Tel Grp = “Telephone Group”, Comparator = “In-clinic Individual”)

Design and Futility Criteria	Scenario	Prop Stop Early	Average Units Used		Average Clinics Per Arm			Prop Successful Simulations
			Clin	Patient	Comp Grp	IC Grp	Tel Grp	IC Grp	Tel Grp	Either Group
Fixed-No Futility Rule	Null	0.00	36	1407	12	12	12	0.03	0.03	0.05
	One Best	0.00	36	1407	12	12	12	0.63	0.02	0.63
	Two Best	0.00	36	1407	12	12	12	0.63	0.49	0.74
	Protocol One Best	0.00	36	1407	12	12	12	0.80	0.02	0.80
	Protocol Half Best	0.00	36	1407	12	12	12	0.80	0.29	0.82
	Protocol Two Best	0.00	36	1407	12	12	12	0.81	0.81	0.92
Adaptive with Futility Rule = 0.50	Null	0.33	27	1055	9	9	9	0.02	0.02	0.04
	One Best	0.05	35	1368	12	15	8	0.68	0.02	0.68
	Two Best	0.02	35	1368	12	12	11	0.64	0.48	0.74
	Protocol Two Best	0.00	36	1402	12	12	12	0.80	0.81	0.92
Adaptive with Futility Rule = 0.65	Null	0.50	24	937	8	8	8	0.02	0.02	0.04
	Protocol One Best	0.06	35	1352	12	16	7	0.83	0.02	0.83
	Protocol Half Best	0.04	36	1368	12	14	10	0.8	0.26	0.81
	Protocol Two Best	0.02	35	1388	12	12	12	0.79	0.78	0.91

For the “One Best” scenario, the fixed design had a slightly smaller proportion of simulations that detected a significant difference between the novel treatment and comparator arm compared to the adaptive design (0.63 for fixed vs. 0.68 for Adaptive with Futility Rule = 0.5). This is due to the more consistent dropping of the Telephone group and reallocation of participants to the IC group. A similar result is seen for the “Protocol One Best” scenario when comparing the Adaptive with Futility Rule = 0.65 and Fixed Designs. The proportion of successful simulations slightly increased from 0.8 in the Fixed Design to 0.83 in the Adaptive Design with Futility Rule = 0.65.

For the “Half Best” scenario, the number of times an arm is dropped is less than that of the “Protocol One Best” and the “One Best” scenarios. This is due to the larger effect size for the Telephone group in those scenarios, which reduces the number of times an arm is dropped.

For the scenarios “Two Best” and “Protocol Two Best,” where both the novel treatments perform well, their respective fixed and adaptive designs perform the same. Since the actual treatment effects (${\Delta }_{10}$ and ${\Delta }_{20}$) are very similar for the “Two Best” scenario and equivalent for the “Protocol Two Best” scenario, the probability of dropping one of the arms shrinks considerably. This can be seen in the Average Number of Clinics Randomized Per Arm column, where the clinics for each arm are very similar across both designs and futility rules. It can also be seen for both scenarios in the proportion of simulations stopped early (Prop Stop Early) column.

4. Discussion and Limitations

The adaptive design was limited in the number of adaptations that could be made since it was based on the original trial design under similar constraints. If any adaptations could have been implemented (altering subject accrual dates, randomization schedules, etc.), then detecting the smaller effect sizes with greater power would have been more plausible. Instead, for the scenarios based on the actual trial results (Scenarios 2 and 3), the individual power was always less than 0.70. This would usually not be an acceptable power to design a study around. However, when considering the effect sizes specified in the original trial protocol (scenarios 4–6), we did show that the desired power of ≥0.80 was achieved, even though the adaptive approach would not have stopped the trial early for success.

In the original trial, the adaptive design would have stopped the trial early if no differences between treatments and comparator were detected, thus saving unnecessary resources. When the futility rule was increased from 0.5 to 0.65, the proportion of times the trial would stop early increased even further without too much loss of power.

As was mentioned in the “Missing Data” section, using the last observation carried forward (LOCF) is a domain-specific way of handling the missing data problem, assuming a monotonic trend in the differences between the comparator and novel interventions. Such an assumption is not always valid; generally, these relationships tend to be nonlinear, complicated, and difficult to predict. There are many cases and scenarios that do not always exhibit a monotonic trend across time, as in the studies cited here. Some studies have shown that the difference in weight loss between interventions switches directions instead of gradually shrinking over time. As such, one or more treatments exhibit better weight loss outcomes later in the trial compared to earlier in the trial vs the comparator treatment, which shows the opposite trend. Implementing redesigns for these relationships is an area of future research, along with using hierarchical models to control the between participant dependence better.

5. Conclusion

In conclusion, although the adaptive features that could be implemented were limited due to the imposed study constraints, the Bayesian Adaptive Cluster Randomized design performed as well as or better than the original trial design for similar effect sizes. The arm-dropping adaptation improved the number of trials that stopped early when no novel treatment effects were detected. In an actual trial, this would mean getting people off treatments that don’t work and onto better options. In instances where only one treatment worked, the power of the trial was increased slightly due to the reallocation of participants to the better-performing arm. Again, in an actual trial, this would improve the probability of detecting smaller effect sizes when more participants are allocated to the better treatments, as was demonstrated in the “One Best” and “Protocol One Best” scenarios. Overall, this study shows the benefits of minimal adaptations to longitudinal designs with long endpoints relative to accrual.