A Group-Sequential Randomized Trial Design Utilizing Supplemental Trial Data

Abstract

Definitive clinical trials are resource intensive, often requiring a large number of participants over several years. One approach to improving the efficiency of clinical trials is to incorporate historical information into the primary trial analysis. This approach has tremendous potential in the areas of pediatric or rare disease trials, where achieving reasonable power is difficult. In this manuscript, we introduce a novel Bayesian group-sequential trial design based on Multisource Exchangeability Models, which allows for dynamic borrowing of historical information at the interim analyses. Our approach achieves synergy between group sequential and adaptive borrowing methodology to attain improved power and reduced sample size. We explore the frequentist operating characteristics of our design through simulation and compare our method to a traditional group-sequential design. Our method achieves earlier stopping of the primary study while increasing power under the alternative hypothesis but has a potential for type I error inflation under some null scenarios. We discuss the issues of decision boundary determination, power and sample size calculations, and the issue of information accrual. We present our method for a continuous and binary outcome, as well as in a linear regression setting.

1 |. INTRODUCTION

Implementing an adaptive clinical trial can reduce the required sample size and the duration of a trial^1;2;3. The reductions in sample size and duration can also reduce the time to approval of new therapies and provide faster access to the new treatment⁴. These benefits are one of the main goals of the 21st Century Cures Act⁵. A closely related problem in biomedical research is the rising cost of clinical trials^6;7;8. It has been noted that the costs of clinical trials rise faster than the inflation rate^7;9, making the real cost of drug development higher over time. Some argue that rising costs could stifle innovation as well as increase the out-of-pocket costs to patients^8;6. In addition, shorter trials conducted in a rapidly changing environment are less likely to be outdated by the time the results are published. Lastly, there are compelling ethical reasons for using adaptive designs. For example, a clinical trial should be terminated if one arm is clearly superior to the other as it is no longer ethical to randomize participant to the inferior arm(s)¹⁰.

Group-sequential designs are a widely used adaptive design method that can address many of these issues. Such designs allow us to perform interim analyses to compare the treatment arms and to stop the trial early if there is strong evidence for or against the alternative hypothesis. Allowing for early stopping can reduce the expected sample size, reducing cost and trial duration.

An alternative approach to reducing required sample size is to incorporate supplemental or historical information. There is existing literature discussing borrowing of supplemental data^{11;12;13;14;15;16}. Static borrowing methods are the simplest approach, but they are sub-optimal as they do not allow for any re-evaluation of the initial assumption about exchangeability¹³. Dynamic approaches are data-driven and do not require that the extent of borrowing be pre-specified. Common dynamic approaches use a hierarchical structure^{17;18;19;20;15;13;11;21}, but are sensitive to the choice of hyperpriors that control the amount of borrowing^14;22;23. Multisource Exchangeability Models (MEMs)^24;25;26 are an approach to dynamic borrowing that enumerate all the possible exchangeability patterns and aggregate the posteriors for each exchangeability pattern using Bayesian Model Averaging (BMA)²⁷. Unlike other existing methods, MEMs can incorporate multiple supplemental sources to a different degree into the primary study analysis.

Supplemental data may take many forms. For example, the same intervention could have been studied previously in another part of the world, in different population, or even in the earlier phases of the development process (i.e., using phase II data to augment phase III data). We consider one such situation where we supplement a clinical trial with data from the pilot study. We will show that under the alternative hypothesis, using our method would allow the ongoing trial to both stop earlier (and save costs by reducing necessary sample size) and increase the power to detect a treatment effect.

In this manuscript, we propose a group sequential design utilizing dynamic borrowing at the interim analyses of a randomized controlled trial (RCT) to increase efficiency. Our design utilizes the MEM framework^24;28. It combines the group-sequential and dynamic borrowing methods to achieve synergy and attain even greater savings in terms of sample size. We consider the case where the supplemental trial is already concluded, as well as the case where the two trials are running concurrently. We investigate the operating characteristics of our design assuming a continuous and binary outcome, as well as an extension to a regression setting. We show that our method yields greater power to detect a treatment difference in the current trial as compared to traditional group-sequential design, albeit with minimal type I error probability inflation under the null hypothesis. Our method also tends to achieve earlier stopping and smaller sample size in the primary study. Methods for the planning of the primary trial (power / sample size calculations) are discussed. We also address the issue of potentially non-monotone information accrual.

As a motivating example, we consider a sequence of randomized trials in tobacco regulatory science, which focuses on evaluating potential strategies for regulating the sale, marketing, and content of tobacco products. This field is currently undergoing major and rapid changes in light of the emerging research on the dangers of vaping and e-cigarettes^29;30;31.

The Impact of Very Low Nicotine Content Cigarettes in a Complex Marketplace (VLNC-CM) study (ClinicalTrials.gov Identifier NCT03272685) is an ongoing randomized trial which compares measures of smoking behavior when smokers have access to a tobacco marketplace with normal nicotine content (NNC; 16 mg nicotine per gram of tobacco) cigarettes to outcomes when smokers have access to a tobacco marketplace where NNC cigarettes are replaced by very low nicotine content (VLNC; 0.4 mg nicotine per gram of tobacco) cigarettes. Participants, who are smokers not currently interested in quitting, go through 3 stages of the trial: In the first stage they are followed for a 2 week baseline period, during which they continue their usual smoking behaviors. In the second stage, they are introduced for 2 weeks to an experimental marketplace where they can exchange vouchers for their usual brand of cigarettes as well as non-combustible tobacco and nicotine products (smokeless tobacco, snus, electronic cigarettes, medicinal nicotine replacement). After the 2-week second stage, participants are randomized to a marketplace that includes either VLNC or NNC cigarettes along with the other tobacco and nicotine products, as in stage 2 of the trial. At the end of the study, they are reimbursed for unspent vouchers. The primary outcomes are the number of cigarettes smoked per day (CPD) and cigarette-free days. As such, this study examines the effect of VLNC cigarettes in a complex marketplace where participants randomized to VLNC can supplement nicotine from alternative sources. This scenario mimics a hypothetical regulatory environment in which the FDA lowers the nicotine content of cigarettes sold in the US to very low levels, under the authority granted to the FDA by the 2009 Family Smoking Prevention and Tobacco Control Act. The ongoing VLNC-CM study will be considered the primary data source.

A pilot study titled Reduced Nicotine Content Cigarettes and Use of Alternative Nicotine Products³² compared 3 experimental conditions: NNC with access to non-cigarette combusted and non-combusted tobacco/nicotine products, VLNC 1: VLNC cigarettes with access to non-cigarette combusted and non-combusted tobacco/nicotine products, or VLNC 2: with access to only non-combusted products. To consider only interventions consistent with the primary study, we will focus only on the NNC and VLNC 1 groups. The inclusion-exclusion criteria and study conduct were similar to those described above for the primary study, with the exception that there was no second phase (introduction to the marketplace). We will consider trial designs that integrate data from the pilot study to improve efficiency.

2 |. METHODS

2.1 |. Notation

We first present MEMs in generality and discuss MEMs for normal and binary outcomes, as well as for linear regression. Assume that the primary study is a two-arm randomized, controlled trial and the secondary sources contain data on either one or both of the trial arms, with the same endpoint as the primary study.

Assume that data are available from the current primary study P and 1, …, H supplemental sources with sample sizes n_P, n₁, …, n_H. Let g ∈ 1, 2 denote the experimental and control arms respectively. Similarly, let θ¹ denote the parameter of interest in the experimental trial arm, and θ² the parameter of interest in the control arm (mean, binomial proportion, etc.). Throughout, we assume that the summary measure of the treatment effect can be expressed as the difference in θ¹ and θ² (i.e., θ = θ¹−θ²), but other contrasts could be considered. Following the notation from previous literature²⁴, let Y denote the outcome of interest, T the treatment indicator and S the study to which the patient belongs, S ∈ (P, 1, 2 …, H ) and let D^g denote all the data collected in trial arm g and let D denote all the data collected. Finally, let Z_h, h = 1, …, H denote an indicator for whether or not the data observed in supplementary study h are exchangeable with the primary study. MEMs enumerate all possible exchangeability patterns, which can be expressed in terms of Z_h, h =1, …, H. Each exchangeability pattern is denoted as Ω_m = (Z₁ = z_1,m, Z₂ = z_2,m, …, Z_H = z_H,m ), where m =1, …, 2^H and where z_h,m indicates whether source h is assumed exchangeable with the primary source in pattern Ω_m.

2.2 |. Multisource Exchangeability Models

2.2.1 |. MEMs for normally distributed outcomes

We begin with a review of MEMs for a normally distributed response and known variance²⁴. The parameter of interest is the mean of Y in a given arm g, g ∈ 1, 2 of the primary study. In the primary source study arm g, $y_{1,P}^g,y_{2,P}^g,\dots ,y_{n_P,P}^g$ are independently, identically distributed (i.i.d.) $N\left({\mu }_P^g,{\sigma }_P^g\right)$. Similarly, in arm g of the secondary source h, $y_{1,h}^g,y_{2,h}^g,\dots ,y_{n_h,h}^g$ are i.i.d. $N\left({\mu }_h^g,{\sigma }_h^g\right)$. The parameter of interest θ^g is the mean response ${\mu }_P^g$ for a given trial arm in the primary study. The exchangeability patterns are defined as: if ${\mu }_P^g\ne {\mu }_h^g$ then the arms g of sources P and h are not exchangeable, but if ${\mu }_P^g={\mu }_h^g$ then the arms g of the sources are exchangeable (two possibilities per source, 2^H permutations in total). To model the treatment effect, we fit independent MEMs for the two treatment arms and base our inference on the posterior distribution of $\theta ={\mu }_P^1-{\mu }_P^2$, P (θ |D). In other words, the MEMs are applied twice, once in the active and once in the control arm. Each yields a posterior for the parameter of interest within the trial arm, from which we derive the posterior distribution for the treatment effect via MCMC.

Under each assumed exchangeability pattern Ω_k, let $P\left({\mu }_P^g|{\mathrm{\Omega }}_k,D^g\right)$ be the posterior for the parameter of interest in arm g. The posterior distribution for ${\mu }_P^g$ is a mixture, obtained by Bayesian Model Averaging of the sub-distributions $P\left({\mu }_P^g|{\mathrm{\Omega }}_k,D^g\right)$:

$$P\left({\mu }_P^g|D^g\right)=\sum _{k=1}^{2^H}P\left({\mu }_P^g|{\mathrm{\Omega }}_k,D^g\right)P\left({\mathrm{\Omega }}_k|D^g\right)=\sum _{k=1}^{2^H}{w}_k^{g}P\left({\mu }_P^g|{\mathrm{\Omega }}_k,D^g\right)$$ (1)

where P (Ω_k |D ^g ) are the posterior model weights defined below. For a normally distributed response with known variances and flat prior on ${\mu }_P^g$, $P\left({\mu }_P^g|D^g\right)$ is:

$$\begin{gathered} p\left({\mu }_P^g|D^g\right)=\sum _{k=1}^{2^H}{w}_k^{g}N({\mu }_P^g|\frac{{\overline{y}}_P^g{\prod }_{h=1}^H\left\{{\left(v_h^g\right)}^{z_{h,k}}\right\}+{\sum }_{i=1}^H\left\{\frac{v_P^g{\overline{y}}_i^g}{v_i^g}\right\}{\prod }_{j=1}^K\left\{{\left(v_j^g\right)}^{z_{j,k}}\right\}}{v_P^g\left[\sum _{l=1}^H\left\{\frac{z_{l,k}}{v_l^g}\right\}\prod _{m=1}^H\left\{{\left(v_m^g\right)}^{z_{m,k}}\right\}\right]+\prod _{r=1}^H\{{\left(v_r^g\right)}^{z_{r,k}}]}, \\ {\left(\frac{1}{v_P^g}+\sum _{e=1}^H\frac{z_{e,k}}{v_e^g}\right)}^{-1}) \end{gathered}$$ (2)

where ${\overline{y}}_h^g$ is the source h trial arm g sample mean, z_h,k is the indicator of whether source h is assumed exchangeable with the primary source in pattern k and $v_h^g={\left({\sigma }_h^g\right)}^2/n_h$.

The posterior weight of pattern Ω_k in arm g is $w_k^g=P\left({\mathrm{\Omega }}_k|D^g\right)=\frac{\pi \left({\mathrm{\Omega }}_k\right)P\left(D^g|{\mathrm{\Omega }}_k\right)}{{\sum }_{j=1}^{2^H}\pi \left({\mathrm{\Omega }}_j\right)P\left(D^g|{\mathrm{\Omega }}_j\right)}$, where π (Ω_k ) is the prior probability of pattern Ω_k. The prior π (Ω_k ) can be expressed as π (Ω_k ) = P (Z1 = z_1,k ) × · · · × P (Z_H = z_H,k ). In our manuscript, we let π_e = P (Z_h = 1) for all h, i.e., we use equal prior probabilities of exchangeability for all sources h = 1, …, H. In the calculation of weights we also require P (D^g |Ω_k ) which is given by: $P\left(D^g|{\mathrm{\Omega }}_k\right)=\int P\left(D^g|{\mu }_P^g,{\mathrm{\Omega }}_k\right)\pi \left({\mu }_P^g|{\mathrm{\Omega }}_k\right)d{\mu }_P^g$. Assuming a flat prior for ${\mu }_P^g$, $\pi \left({\mu }_P^g\right)\propto 1$, the marginal distribution p (D^g |Ω_k ) is²⁴:

$$\begin{gathered} p\left(D^g|{\mathrm{\Omega }}_k\right)=\frac{{(\sqrt{2\pi })}^{(H+1)-{\sum }_{h=1}^H\left\{z_{h,k}\right\}}}{\sqrt{\left(\frac{1}{v_P^g}+{\sum }_{i=1}^H\left\{\frac{z_{i,k}}{v_i^g}\right\}\right)\left({\prod }_{j=1}^H\left\{{\left[\frac{1}{v_j^g}\right]}^{1-z_{j,k}}\right\}\right)}} \\ \times \exp \left(-\frac{1}{2}\left[\sum _{l=1}^H\left\{\frac{z_{l,k}{\left({\overline{y}}_P^g-{\overline{y}}_l^g\right)}^2}{v_P^g+v_l^g+v_P^g{v}_l^g\left({\sum }_{m\ne l}\left\{z_{m,k}{\left(v_m^g\right)}^{-1}\right\}\right)}+\sum _{l<r}^H\left\{\frac{z_{l,k}{z}_{r,k}{\left({\overline{y}}_l^g-{\overline{y}}_r^g\right)}^2}{v_l^g+v_r^g+v_l^g{v}_r^g\left({\left(v_P^g\right)}^{-1}+{\sum }_{e\ne l,r}\left\{z_{e,k}{\left(v_e^g\right)}^{-1}\right\}\right)}\right\}\right\}\right]\right) \end{gathered}$$ (3)

Previous literature²⁴ took an empirical Bayesian approach and plugged the maximum likelihood estimate of $v_h^g$ into Equations 3 and 2. Below we will introduce an extension that treats ${\sigma }_h^g$ as unknown. Similar results can be derived when the outcome is binary and are shown elsewhere³³.

MEMs have the desirable property of the consistency of MEM weights, as described in previous literature²⁴, where it is shown in Theorem 3.1 that under normally distributed data with known variance, as n_P, …, n_H → ∞, w_k∗ → 1 for some model k^∗ defined by $\left(Z_1=z_{1,k^{*}},Z_2=z_{2,k^{*}},\dots ,Z_H=z_{H,k^{*}}\right)$ and w_k → 0 for all k ≠ k^∗. In other words, as the primary and supplemental sample sizes approach infinity, the MEMs model weights converge to 1 for a single model and to 0 for all other models.

The parameter π_e is a crucial one when using any MEM method, as it can have a substantial impact on the operating characteristics, which can be seen from simulation results presented below. In general, a researcher would select π_e using simulations that achieves preferred trial operating characteristics. The discussion of a selection procedure for this parameter is beyond the scope of this paper, but an exploration of this issue is presented in other literature^33;28, where optimization criteria for calibrating and evaluating different hyperprior choices are discussed.

2.2.2 |. Extension to Regression

Using the model described above, we can estimate the difference in means (or proportions) between trial arms, but cannot adjust for covariates. Additionally, we assumed known variances. In this section, we describe MEMs for a linear model, which allows us to both control for covariates and to relax the known variances assumption.

Assume we only have one supplemental source, with the same two arms as the primary source. The extension to more than one supplemental source is straightforward. Let X be a collection of covariates. In the primary source, assume $y_{1,P},y_{2,P},\dots ,y_{n_P,P}$ are independently distributed as $N\left({\beta }_{0,P}+X\gamma +T{\beta }_{T,P},{\sigma }_P^{2}I\right)$ where T is again the treatment indicator. Similarly, in the secondary source, $y_{1,1},y_{2,1},\dots ,y_{n_1,1}$ are independently distributed as $N({\beta }_{0,1}+X\gamma +T{\beta }_{T,1},{\sigma }_1^{2}I)$. In this case, the parameter of interest is the treatment effect in the primary study, i.e., θ = β_T,P.

Previously, we evaluated exchangeability separately for each trial arm and calculated the difference between the estimated means of treatment groups in the primary data source. In this case, we borrow directly on the treatment effect. That is, if β_T,P = β_T,1 we consider the two sources exchangeable. As such, there are only two exchangeability patterns: either the sources are exchangeable or not, which correspond to the following regression equations:

$$\mathbf{1}.Y={\left({\beta }_{0,P}+T{\beta }_{T,P}+X\gamma \right)}^{I(S=P)}+{\left({\beta }_{0,1}+T{\beta }_{T,1}+X{\gamma }_1\right)}^{I(S=1)}+ϵ$$

$$\mathbf{2}.Y={\left({\beta }_{0,P}+T{\beta }_{T,P}+X\gamma \right)}^{I(S=P)}+{\left({\beta }_{0,1}+T{\beta }_{T,P}+X{\gamma }_1\right)}^{I(S=1)}+ϵ$$

where ϵ ∼ N (0, E ). The two regression equations correspond to two linear models. In the first regression model, corresponding to the first exchangeability pattern, the sources are assumed to be non-exchangeable. As such, there is a regression model with an intercept, treatment effect and a covariate X used independently for each of the two data sources. Therefore, the treatment effect β_T,P is estimated only using primary data. Notice how the only difference introduced in the second regression model is that the main effect of treatment β_T,P is forced equal across the primary and secondary source. As such, in model 2, β_T,P is estimated using both primary and supplemental data. All other nuisance parameters are allowed to vary between studies. We assume equal variance of the response Y within a study, but unequal across studies, therefore we assume E to be block diagonal where each block is ${\sigma }_l^{2}I,l=P,1$.

In this manuscript, we use vague independent normal priors N (0, 100²) on each regression coefficient and vague independent inverse gamma priors on the variance parameters ${\sigma }_l^2$, $\pi \left({\sigma }_l^2\right)\sim IG(0.001,0.001),l=P,1$. The two models yield different posterior distributions, samples from which can be obtained using standard Markov Chain Monte Carlo (MCMC) software. The final posterior is $P\left({\beta }_{t,P}|D\right)={\sum }_{k=1}^2{w}^{k}P\left({\beta }_{t,P}|{\mathrm{\Omega }}_k,D\right)$ where again $w_k=P\left({\mathrm{\Omega }}_k|D\right)=\frac{\pi \left({\mathrm{\Omega }}_k\right)P\left(D|{\mathrm{\Omega }}_k\right)}{{\sum }_{j=1}^2\pi \left({\mathrm{\Omega }}_j\right)P\left(D|{\mathrm{\Omega }}_j\right)}$. An additional challenge that arises in this setting is that there is no analytical solution for P (D |Ω_j ). We approximate the posterior weight for model k using the BIC approximation, as described in previous literature³⁴, as $w_k=\frac{\pi \left({\mathrm{\Omega }}_k\right)exp\left(-0.5{\mathrm{\Delta }}_k\right)}{{\sum }_{j=1}^2\pi \left({\mathrm{\Omega }}_j\right)exp\left(-0.5{\mathrm{\Delta }}_j\right)}$, where Δ_k = BI C_k −min (BI C₁, BI C₂). An alternative linear model-based MEMs, which can utilize supplemental sources containing information on only one of the trial arms, is outlined in the Supporting Web Materials.

2.3 |. Group-Sequential Design Based on MEMs

In this section, we discuss design considerations for a group sequential trial using MEMs. In a group-sequential design, we complete a hypothesis test at a set of pre-defined time points and terminate the trial early based on a pre-defined stopping rule. In this paper we utilize a flat decision boundary, similar to Pocock boundaries, and evaluate their frequentist operating characteristics. We note, though, that other boundary shapes could also be considered.

2.3.1 |. Decision Boundary Selection

Assume a equally spaced analyses in the primary study, with the final analysis being performed when the maximum sample size in the primary study $n_P^{max}$ is reached. We use the overall type I error level at α =2.5%. We compare our design that allows borrowing to a standard, frequentist design with Pocock stopping boundaries, which assumes a constant critical value c_i = c across all analyses i =1, …, a, performs a statistical test for superiority at each analysis, yields a p-value p_i, and stops and declares superiority of the experimental arm if p_i < c.

MEMs are fit in the Bayesian paradigm; as with any Bayesian method, all inference is based on the posterior distribution (in our case, the posterior distribution of θ). We consider a stopping rule based on the posterior probability of the treatment effect being greater than 0 at analysis i

$$p_i^{*}=P\left(\theta >0|D_i\right)={\int }_0^{\infty }P\left(\theta |D_i\right)d\theta ,$$ (4)

where D_i denotes the data collected both in primary and supplementary studies up to the analysis i and P (θ |D_i ) denotes the posterior distribution of the treatment effect after observing data D_i. Our Pocock-style boundary is: if $p_i^{*}>c^{*}$ at any analysis i =1, …, a, stop and declare superiority of the treatment arm. Otherwise, continue to the next analysis or (if final analysis was reached) terminate the trial and fail to conclude superiority for the treatment arm.

Although MEMs are implemented in the Bayesian paradigm, we wish to select the boundary value c^∗ which controls the frequentist overall type I error rate at the desired α =2.5% level. We define overall type I error rate α as:

$$\begin{gathered} \alpha =P\left(p_1^{*}>c^{*}|{\theta }^1={\theta }^2\right)+P\left(p_2^{*}>c^{*}|p_1^{*}\le c^{*},{\theta }^1={\theta }^2\right)+\dots \\ +P\left(p_a^{*}>c^{*}|p_1^{*}\le c^{*},\dots ,p_{a-1}^{*}\le c^{*},{\theta }^1={\theta }^2\right) \end{gathered}$$

The selection of c^∗ is not straightforward in the Bayesian setting. Bayesian designs suffer from the same issue of testing multiplicity when frequentist operating characteristics are considered^35;36. The multiplicity arises due to the fact that the posterior distribution is paired with a decision rule and the decision rule is evaluated multiple times.

The boundary value c^∗ that controls the overall type I error rate is selected using a modification of the theoretical approach of Shi and Yin³⁵. As described in Shi and Yin³⁵, there is an asymptotic connection between c^∗ and c. First, notice that $p_i^{*}$ is the posterior probability of a half space. A half space $\mathcal{H}$ is a set satisfying $\mathcal{H}=\left\{\delta :a^T\delta \ge b\right\}$, where δ ∈ Δ, Δ is an open subset of ${ℝ}^d$, $a\in {ℝ}^d$, and b is a scalar. As such, $p_i^{*}$ is the posterior probability of a half space defined by {(θ¹, θ²) : θ¹ > θ²}. Dudley and Haughton³⁷ studied the asymptotic normality of half spaces. Using Theorem 1 from Shi and Yin³⁵, under the 7 regularity assumptions described in Dudley and Haughton³⁷, we have:

Theorem 1

When no borrowing is allowed, for cumulative sample sizes $\frac{n_P^{max}}{a},\dots n_P^{max}$, as $n_P^{max}\to \infty$ the joint statistics $\left\{{\mathrm{\Phi }}^{-1}\left(p_1^{*}\right),\dots ,{\mathrm{\Phi }}^{-1}\left(p_a^{*}\right)\right\}$ converge in distribution to $\left(S_{n_P^{max}}/a,\dots ,S_{n_P^{max}}\right)$, which converges in distribution to a multivariate normal distribution of the form described in Jennison and Turnbull³⁶, Chapter 11.2.

where Φ() is the standard normal cumulative distribution function and S_n,P is the signed root likelihood ratio statistic based on sample size of n_P. In other words, the posterior probability of $\mathcal{H}$ at each interim analysis converges in distribution to a normal distribution as $n_P^{max}\to \infty$ when no borrowing from supplemental sources takes place. The proof of Theorem 1 follows from Shi and Yin³⁵. This theorem only concerns the selection of the decision boundary c^∗ under no borrowing. As shown in later results and in previous literature^24;38, if borrowing takes place under non-exchangeability, this can lead to bias an inflated type I error rate. However, we believe that c^∗ should be determined assuming no borrowing. This approach may actually lead to conservative analyses, as borrowing methods often reduce type I error rates below α in a “global” null scenario, i.e., where both the primary and supplemental studies are generated under the null hypothesis³⁹. The “global” null setting is usually only relevant in a concurrent studies setting. When using historical data, borrowing often only takes place if the historical data yielded positive results. Calculating c^∗ assuming no borrowing controls type I error at α in a “local null” situation, where historical data are generated under the alternative hypothesis, primary data are generated under the null hypothesis, and no borrowing takes place. However, calibration of c^∗ across all null scenarios may be desirable, and may yield more attractive type I error / power tradeoff.

Using Theorem 1, frequentist group sequential boundaries can be translated to the posterior probability scale to approximately control the overall type I error rate for a Bayesian design. To take advantage of asymptotic normality, we can set c^∗ =1 − c as seen in previous literature³⁵. Such an approach would yield decision boundaries that control type I error asymptotically. Otherwise, one could use simulations to select decision boundaries that control type I error in finite samples. In our manuscript, we use the simulation based approach to select our decision boundaries. However, when constructing the simulations to investigate which decision boundary achieves the desired type I error, we construct a grid search over values near the asymptotic boundary. Such an approach can cut down on computation time.

2.3.2 |. Power and Sample Size Calculations

We consider a power and sample size calculation to facilitate the planning of the primary trial utilizing our method. The researcher has two options for planning the primary trial. Either she can assume the worst-case scenario (where the secondary data will be clearly non-exchangeable with the primary data and no borrowing will take place) and simply perform a traditional power / sample size calculation to obtain the sample size required to achieve the desired power. Such an approach is conservative and may result in over-powering, but early stopping due to group sequential testing would likely partially offset this disadvantage. Alternatively, the researcher can assume a certain amount of similarity between sources and perform a power / sample size calculation specific to our method. This would be appropriate in settings where the sample size required to achieve the desired power with no borrowing is unrealistic (i.e. rare diseases), in which case borrowing is required to achieve adequate power regardless of sequential testing.

For a fixed-sample trial, we can perform power and sample size calculation for normal outcomes analytically by using the results in Equations 2 and 3. Given sample sizes n_P and n₁, let ${\overline{y}}_1^g,g\in 1,2$ be the secondary study response means in the treatment and control group, with response standard deviation σ₁. Next, assume control group mean in the primary study ${\overline{y}}_P^2$ and response standard deviation σ_P. Then, use a numerical search to find the critical value (denoted as c_r ) of the decision rule based on Equations 2 and 3 in terms of ${\overline{y}}_P^1$ (there is no closed form solution for c_r ). To calculate power to detect treatment effect ω in the primary study, we note that under the alternative hypothesis, the treatment effect in the primary study is distributed as $N\left(\omega ,2{\sigma }_p^2/n_p\right)$. From that distribution, we can calculate the probability of concluding superiority as $1-\mathrm{\Phi }(\frac{\omega -\left(c_r-{\overline{y}}_P^2\right)}{\sqrt{2}{\sigma }_p/\sqrt{n_p}})$.

The maximum sample size of a group sequential design must be inflated relative to a fixed-sample design to preserve power (due to multiple testing). Typically, we would determine the maximum sample size by simulations, but below we present a quick approximation of the inflation factor, which can be computed by using the asymptotic connection between frequentist and Bayesian methods in Theorem 1.

The calculation of some inflation factor s that preserves power in a frequentist group-sequential design is wellknown³⁶. An implication of Theorem 1 is that assuming a sample size n achieves desired power in a fixed-sample setting for a Bayesian method which satisfies the conditions in Theorem 1, a maximum group-sequential sample size of n^max = s × n achieves approximately the same overall power in a group-sequential setting. However, Theorem 1 only applies when no borrowing is allowed.

To consider a power calculation for a group sequential design using MEMs, combine Theorems 1 and Theorem 3.1 in Kaizer et al.²⁴, assuming supplemental studies are already concluded, to get:

Theorem 2

For cumulative sample sizes $\frac{n_P^{max}}{a}+n_1\cdots +n_H,\dots ,n_P^{max}+\cdots +n_H$, as $n_P^{max},\dots ,n_H\to \infty$, the joint statistics $\left\{{\mathrm{\Phi }}^{-1}\left(p_1^{*}\right),\dots ,{\mathrm{\Phi }}^{-1}\left(p_a^{*}\right)\right\}$ from MEMs converge in distribution to $\left(S_{n_{s,1}},\dots ,S_{n_s}\right)$, which converge in distribution to a multivariate normal distribution of the form described in Jennison and Turnbull³⁶, Chapter 11.2.

where the quantities $p_i^{*}$ are posterior probabilities of $\mathcal{H}=\left\{\left({\theta }^1,{\theta }^2\right):{\theta }^1>{\theta }^2\right\}$, $n_{s,1}=\frac{n_P^{max}}{a}+{\sum }_{h=1}^H{n}_h{z}_{h,k^{*}}$ and $n_s=n_P^{max}+{\sum }_{h=1}^H{n}_h{z}_{h,k^{*}}$. Theorem 2 follows from Theorem 1, Theorem 3.1 in Kaizer et al.²⁴ and Slutsky’s theorem. Theorem 3.1 in in Kaizer et al.²⁴ shows the MEM weights converge to 1 for a particular model k^∗ and to 0 for all other models. As such, MEMs asymptotically pool source h with the primary study if $z_{h,k^{*}}=1$ and ignore source h if $z_{h,k^{*}}=0$. Combining this fact with Theorem 1, it follows that the joint statistics $\left\{{\mathrm{\Phi }}^{-1}\left(p_1^{*}\right),\dots ,{\mathrm{\Phi }}^{-1}\left(p_a^{*}\right)\right\}$ converge in distribution to multivariate normal distribution.

The power to detect treatment effect ω is:

$$\begin{gathered} 1-\beta =E\left({\cup }_{i=1}^{a}I\left(p_i^{*}>c^{*}\right)|\theta =\omega \right)=E\left(I\left(p_1^{*}>c^{*}\right)|\theta =\omega \right) \\ +\sum _{i=2}^{a}E\left({\cap }_{i^{\prime }=1}^{i-1}I\left(p_{i^{\prime }}^{*}\le c^{*}\right)\cap I\left(p_i^{*}>c^{*}\right)|\theta =\omega \right)) \\ =P\left(p_1^{*}>c^{*}|\theta =\omega \right)+P\left(p_1^{*}\le c^{*}|\theta =\omega \right)P\left(p_2^{*}>c^{*}|p_1^{*}\le c^{*},\theta =\omega \right)+\dots \\ +P\left(p_1^{*}\le c^{*}|\theta =\omega \right)\times \cdots \times P\left(p_{a-1}^{*}\le c^{*}|p_1^{*}\le c^{*},\dots ,p_{a-2}^{*}\le c^{*},\theta =\omega \right) \\ \times P\left(p_a^{*}>c^{*}|p_1^{*}\le c^{*},\dots ,p_{a-1}^{*}\le c^{*},\theta =\omega \right) \end{gathered}$$ (5)

where I () is the indicator function. Each of the quantities in Equation 5 can be calculated by integrating a multivariate normal distribution³⁶. For more detail on group sequential designs and boundary calculations, see also the work by Yin⁴⁰. To take advantage of Theorem 2 and the implication of Theorem 1 discussed above, one can calculate the sample sizes required to achieve the desired power using MEMs for a fixed-sample design, and then use an inflation factor s from standard frequentist software, for example the R package gsDesign⁴¹. However, this approach only preserves power approximately.

2.3.3 |. Truncated MEMs

Information (or information fraction)³⁶ is an important quantity in the group-sequential design literature. In a standard design, there is a direct relationship between information growth and the proportion of data observed: if for example 50% of the data are collected, the information fraction is 0.5. When borrowing is allowed, information growth is more complicated because some information comes from supplemental sources and the extent of borrowing changes throughout the course of the trial. Above we made the common assumption of equally spaced analyses. This assumption is sufficient in the absence of borrowing, but linear information accrual can not be assumed when implementing dynamic borrowing (or indeed even monotone information accrual). For example, if our method borrows heavily at the first analysis but borrows substantially less at the second analysis, the amount of information available at second interim analysis could actually be lower than the information at first analysis even though the total number of participants in the primary trial increases from first to second interim analysis.

In this section, we introduce a modified approach that constrains the maximum amount of borrowing allowed during interim analyses. While the prior source inclusion parameter π (Ω_k ) can be thought of as a tuning parameter that can be adjusted to achieve a desired tradeoff between efficiency and bias, it affects all interim analyses equally and changing π (Ω_k ) is not sufficient to address the aforementioned information issue.

To address this issue, one can utilize a constraint based on effective supplementary sample size (E SSS), which extends the concept of effective sample size⁴² to the context of dynamic borrowing. Consistent with previous literature^14;24, define $ESS{S}_{i,k}^g$ at analysis i for a treatment arm g and some model k as $ESS{S}_{i,k}^g=n_{P,i}^g(\frac{{\mathcal{P}}_k^g}{{\mathcal{P}}_1^g}-1)$ where ${\mathcal{P}}_1^g$ is the precision of posterior for θ^g in the no-borrowing model, ${\mathcal{P}}_k^g$ is the equivalent quantity for a model k that is the precision of posterior for θ^g allows some borrowing at analysis i, and $n_{P,i}^g$ is the sample size in the primary study arm g at analysis i. E SSS can be interpreted as additional primary study sample size that would be needed to achieve the same precision achieved by borrowing.

At analysis i, $ESS{S}_i^g=n_{p,i}^g{\sum }_k{w}_k^g(\frac{{\mathcal{P}}_k^g}{{\mathcal{P}}_1^g}-1))$, k =1, …, 2^{H 24} where $w_k^g$ are again model weights as above, ${\mathcal{P}}_1^g$ is the posterior precision of the no-borrowing model, i.e., one of the 2^H models which utilizes primary data only.

An extension to MEMs, which we refer to as truncated MEMs, constrains the maximum amount of borrowing at each interim analyses, is implemented as follows:

1. If $ESS{S}_i^g\le {\zeta }_i$, where ζ_i is some maximum amount of borrowing at analysis i, continue to next stage as planned and leave the method unmodified. Otherwise continue to the next steps.

2. Let $s={\zeta }_i/ESS{S}_i^g$ be a shrinkage factor.

3. Update the posterior weight of the no-borrowing model, $w_1^g$ to $w_{1^{\prime }}^g=w_1^g+(1-s)\left(1-w_1^g\right)$ where $w_{1^{\prime }}^g$ is the new weight of no-borrowing model in arm g, $w_1^g$ is the original posterior weight as calculated by MEMs.

4. For all models k =2, …, 2^H, all of which utilize some amount of borrowing, let $w_{k^{\prime }}^g=s\ast w_k^g$ where $w_{k^{\prime }}^g$ and $w_k^g$ are as defined in previous step.

As such, we shrink the weights of all models that allow for borrowing and inflate the weight of the no-borrowing model. This retains the relative weight among the models that allow borrowing while increasing the weight given to the model with no borrowing to cap the E SSS. Truncated MEMs introduce a second tuning parameter ζ_i. This new tuning parameter is useful in the sense that it allows the trial operating characteristics over time to be more detached from the global tuning parameter π_e (which does not change over time). One would traditionally calibrate π_e to achieve desired trial operating characteristics, which may be difficult if the aim is to modify the operating characteristics over time. When implementing truncated MEMs to control the information behavior, a natural choice is ${\zeta }_i=n_{p,i+1}^g$ for all i =1, …, a − 1 where $n_{p,i+1}^g$ is the number of patients who will be enrolled in the next stage of the primary study in a given study arm g. In other words, if in the next stage of the trial we are about to enroll 25 participants, but the ESSS would have been 50 without a constraint, the proposed method would perform less aggressive borrowing and scale ESSS down to 25. This is the constraint we will use throughout this manuscript. No constraint will be imposed at the final analysis as there is no concern about decreasing information in the future. The parameter ζ_i could also be set to other values, or it could be set to vary over time. Our choice was motivated by controlling the behavior of the information fraction, but ζ_i could be used for other purposes. Briefly, the choice of ζ_i could be impacted by the overall size of the trial, frequency and timing of interim analyses or even the amount of information collected in the trial to date. Operating characteristics of those different choices would then be explored using simulations. A thorough investigation of this tuning parameter is outside the scope of this manuscript and would be a great topic for future work.

Another reason to utilize the truncated MEMs method is that especially if a is large, at early analyses there is very limited amount of data, which can lead to considerable uncertainty as to whether sources are exchangeable. A possible consequence may be that MEMs borrow too much data too early. Tuning π_e again does not address that issue, but truncated MEMs can be used to achieve the desired trial operating characteristics. However, it is important to note that when utilizing the truncated MEMs method in a group-sequential setting, Theorem 2 no longer applies, and the asymptotic power and sample size calculations introduced in Section 2.3.2 cannot be used. In this setting, power must be explored using simulations.

3 |. APPLICATION

In this section, we will consider various possible scenarios for the VLNC-CM trial via simulations and explore the operating characteristics of the proposed design. As described above, we will use the pilot study as the supplemental source. The pilot study is already concluded, and therefore data from the pilot study are fixed. The primary outcome CPD is assumed to be normally distributed, and the linear-model based MEM described in Section 2.2.2 will be used. In the pilot study, the mean CPD in the treatment and control groups were 11.9 and 19.2 (treatment effect of 7.3), with standard deviations of 7.4 and 8 respectively. The sample sizes are 53 in the treatment group and 27 in the control. The primary study will be simulated under 6 different data generating scenarios, where the primary study treatment effect is varied.

There are a = 4 equally spaced analyses assumed in the primary study. For a = 4 and a target overall type I error rate of 0.025, the frequentist Pocock boundary uses a critical value c =0.0091 and our method uses boundary c^∗ =0.9909, which was selected using the approach outlined above. We set the primary study maximum sample size to 320, 4 times larger than the secondary study. The primary study has a randomization ratio of 1:1. We present results for a group sequential design that allows dynamic borrowing with a prior probability of exchangeability of π_e =0.05 and another result using π_e =0.1. Although these choices of π_e seem overly conservative, they provide an attractive tradeoff between efficiency due to borrowing and bias. The results for constrained MEMs are presented for higher values of π_e of 0.2 and 0.5, as for low π_e the constrained and unconstrained results are very similar. A choice of higher π_e coupled with the MEM constraint introduced above could offer a different benefit-cost tradeoff: on one hand the method is a priori more optimistic towards borrowing, but this is counteracted by the truncation.

Table 1 shows the power, type I error rate, expected sample size and the distribution of stopping times in the primary study for the 5 methods considered over 10,000 simulations.

TABLE 1.

Simulation of VLNC-CM trial using pilot study as supplemental source.

	Method	Primary study treatment effect
		7.3	5	4	3	2	0
Power or type I error	Pocock	>0.99	0.984	0.909	0.677	0.352	0.025
	MEMs(0.05)	>0.99	0.988	0.914	0.682	0.371	0.027
	MEMs(0.1)	>0.99	0.988	0.917	0.703	0.380	0.034
	MEMs*(0.2)	>0.99	0.989	0.927	0.721	0.406	0.039
	MEMs*(0.5)	>0.99	>0.99	0.943	0.777	0.466	0.060
Expected sample size	Pocock	97.6	148.0	188.0	237.6	280.8	316.0
	MEMs(0.05)	93.6	139.2	179.2	232.0	275.2	315.2
	MEMs(0.1)	92.0	133.6	172.8	225.6	272.0	314.4
	MEMs*(0.2)	90.4	128.8	167.2	217.6	265.6	313.6
	MEMs*(0.5)	89.6	124.0	158.4	204.0	255.2	309.6
P(Stop at 1st analysis)	Pocock	0.79	0.43	0.26	0.15	0.07	0.01
	MEMs(0.05)	0.84	0.49	0.32	0.18	0.09	0.01
	MEMs(0.1)	0.86	0.53	0.36	0.21	0.11	0.02
	MEMs*(0.2)	0.87	0.57	0.39	0.24	0.13	0.02
	MEMs*(0.5)	0.88	0.57	0.40	0.26	0.14	0.03
P(Stop at 2nd analysis)	Pocock	0.20	0.36	0.33	0.20	0.09	0.01
	MEMs(0.05)	0.15	0.33	0.30	0.20	0.09	0.01
	MEMs(0.1)	0.13	0.30	0.29	0.20	0.10	0.01
	MEMs*(0.2)	0.12	0.29	0.29	0.21	0.11	0.01
	MEMs*(0.5)	0.12	0.33	0.34	0.27	0.15	0.02
P(Stop at 3rd analysis)	Pocock	0.01	0.15	0.21	0.18	0.09	0.01
	MEMs(0.05)	0.01	0.13	0.19	0.17	0.10	0.01
	MEMs(0.1)	0.01	0.12	0.18	0.17	0.09	0.01
	MEMs*(0.2)	0.01	0.10	0.16	0.16	0.09	0.01
	MEMs*(0.5)	<0.01	0.07	0.13	0.15	0.10	0.01
P(Stop at 4th analysis)	Pocock	<0.01	0.06	0.20	0.47	0.75	0.97
	MEMs(0.05)	<0.01	0.05	0.19	0.45	0.72	0.97
	MEMs(0.1)	<0.01	0.05	0.17	0.42	0.70	0.96
	MEMs*(0.2)	<0.01	0.04	0.16	0.39	0.67	0.96
	MEMs*(0.5)	<0.01	0.03	0.13	0.32	0.61	0.94

MEMs(x) denotes MEMs with x prior probability of exchangeability. Asterisk denotes truncated MEMs, ζ_i = 40 for both arms at each interim.

We see that with a prior probability of exchangeability of 0.05, we can control the overall type I error rate very close to the desired level of 2.5% while still realizing improvements in terms of power and expected sample size under the alternative hypothesis. Our method has a higher chance to stop the trial early and also attains improved power to detect a treatment effect. The difference is clearest when considering the probability of stopping the trial at the first interim analysis, where the trial has a considerably higher chance of stopping when allowing for dynamic borrowing (for example, 79% vs. 84% probability of declaring superiority at first analysis under treatment effect of 7.3). The earlier stopping also results in smaller expected sample size. The design that allows dynamic borrowing is able to realize those improvements under wide variety of treatment effects, even when the treatment effect is smaller than the expected treatment effect of approximately 7. For example, under a treatment effect of only 2, our method improves power by approximately 2%, while also stopping the trial earlier on average. Larger gains are observed if we use a less conservative prior probability of exchangeability, but this results in a inflated type I error rate. The constrained MEM results also demonstrate this trend.

A natural question to ask is which of the benefits (increase in power, decrease in expected sample size) of borrowing presented in Table 1 are due to type I error inflation and what benefits there are due to increased efficiency and sample size. We explore this distinction in Figure 1 where we use the same 4 MEMs-based methods as above, but also introduce a calibrated traditional no-borrowing analysis, which is calibrated to achieve the type I error presented in Table 1 corresponding to a matched MEMs method. This allows us to separate which benefits are due to type I error inflation and which benefits are achieved through other ways. In Figure 1 we present power,expected sample size and mean square error (MSE) as a function of the treatment effect size. It is important to note that the MSE is simply the MSE of a final point estimate yielded at whichever analysis the study stops. In each plot we present a given MEMs method, a corresponding calibrated traditional method and also an uncalibrated traditional method, which achieves the nominal type I error rate of 0.025. As we can see in Figure 1, the MEMs methods generally achieve virtually identical power and expected sample size curves as the corresponding calibrated traditional method. As such, improvements in these metrics seen in Table 1 were mainly due to type I error inflation. We believe this is due to the fact that the supplemental source consists only of 80 patients randomized in a 2:1 ratio with a large treatment effect of 7.3, while the simulated primary study is much larger. The studies are truly exchangeable only when treatment effect in the primary study is also at or near 7.3, which is where data borrowing is most appropriate, but at this point the study is already over-powered and it is difficult to yield improvements. Traditionally, the primary study would be smaller than the secondary study and not over-powered at the point of exchangeability; this setting is explored in simulations below.

FIGURE 1.

Simulation of VLNC-CM trial using pilot study as supplemental source. MEMs(x) denotes MEMs method with x prior probability of exchangeability. Asterisk denotes truncated MEMs with ζ_i =25 for both arms at each interim analysis. Traditional (calib.) denotes a traditional no-borrowing analysis that is calibrated to achieve the same type I error rate as the corresponding MEM-based method. MSE=mean square error. Expected sample size denotes the expected sample size in the primary trial. In all plots the grey dashed line denotes the traditional method that controls type I error at the nominal rate of 0.025. Solid black vertical line denotes the point where sources are exchangeable

As discussed above, the choice of π_e is quite important and throughout our manuscript, we considered π_e =0.05 as our preferred choice, as it often provided a good trade-off between the benefits and drawbacks of the proposed method. However, the optimal choice can be quite subjective, which is why we present a variety of values for π_e.

4 |. CONCURRENT SUPPLEMENTAL SOURCES SIMULATIONS

In Section 3, we considered the case where the supplemental trial was completed before the primary study began. Alternately, we could consider a scenario where the two trials are run concurrently. It is common practice to incorporate smaller substudies into larger trials as a means to answer additional questions⁴, for instance in pediatric trial settings, which can be built-into an adult trial. As an example, consider the Phase III trials of Besivance, a drug aimed at treatment of bacterial conjunctivitis^43;44. In both trials, patients 1 year or older were enrolled, which resulted in a very wide patient age range (approximately 1 to 90 years old). Consider the problem of seeking regulatory approval for the pediatric population. Traditionally, this approval would be based on pediatric data, but the sample size may be small. In addition, it seems reasonable to assume that the drug may work similarly in adults. One could also consider different age groups as separate sources (e.g., adolescents, adults, seniors), which naturally calls for a method like MEMs, as it can account for the fact that adolescents are more likely to be exchangeable with children than adults or seniors. In a trial such as this, it would be possible to implement a group-sequential design where the pediatric population is considered the primary source for the purposes of pediatric regulatory approval while other age groups are supplemental. In that case, the sources would enroll patients concurrently. Additionally, the primary study will often be smaller than the secondary source. In the following simulations, we explore this scenario.

There is another compelling reason to share infrastructure between the primary and secondary studies. Whenever we incorporate supplemental data, we should be concerned about inter-trial effects^45;24 which could manifest themselves as differences in standards of care, event definitions, protocols etc. Many of these concerns are resolved when we allow the primary and secondary sources to be a part of one larger study, as recruitment happens in the same places and at the same time, the exclusion and inclusion criteria, study protocols, event definitions are all likely very similar. Furthermore, a similar approach has recently been proposed in order to expedite the generation of child-specific evidence⁴⁶.

We again assume a = 4 equally spaced analyses and only one supplemental two-arm trial with concurrent enrollment with the primary study. We use the same critical values for early termination as were used in Section 3. The maximum sample sizes are 200 in the primary study and 400 in the secondary study. The simulations explore type I error, power and distribution of stopping times. Our simulations assume that the mean for the control group is the same for the two studies and we vary the treatment effect. We consider 4 scenarios: 1) one where there is an equal treatment effect in the primary and secondary data sources (where the MEMs method stands to gain the most, as sources truly are exchangeable), 2) a scenario where there is no treatment effect in the primary study, but the treatment effect in the secondary study is the same as in scenario 1, 3) a scenario where there is a small effect in the supplemental source, but no effect in the primary source, and 4) a scenario where there is no treatment effect in either study (global null setting). In scenario 2 the sources should be noticeably different, with low amount of data borrowing happening, but also there is a considerable potential for bias when data borrowing does happen. In scenario 3 there is no treatment effect in the primary study, but a small treatment effect in the secondary study, which could make it more difficult for MEMs to recognize that the sources are different. As such, we would expect more data borrowing to happen between sources, but the potential for bias is lower than in scenario 2. In scenario 4 the sources are again in truth exchangeable. Therefore, scenario 1 is investigating power, while scenarios 2, 3 and 4 are investigating the type I error rate, with scenarios 2 and 3 exploring type I error under a “local null” setting, while scenario 4 is under a “global null” setting. All the simulations are done with 10,000 replications. We consider the same 5 methods as above (no borrowing, two unconstrained MEMs, two truncated MEMs).

4.1 |. Normal Response

In this simulation, the response Y in each trial arm is normally distributed. The treatment effect is the difference in means of Y between treatment and control arms. In the first scenario, the control mean of Y is 5 and the treatment mean is 6 in the primary study, therefore there is a true treatment effect of 1. The parameters are the same in the secondary study. In the second scenario, the mean of Y is 5 in both arms of the primary study (i.e., there is no treatment effect), but the treatment effect equals 1 in the secondary study. The third scenario is the same as the second, except that the treatment effect in the secondary study is 0.5 instead of 1. In all scenarios, the standard deviations of the outcome Y are 3 and 4 in the primary and secondary studies respectively.

In scenario 1 in Table 2, the group sequential design using MEMs with π_e =0.05 achieves a considerable gain in power over a frequentist design with Pocock boundaries (approximately 56% vs. 63%) along with an expected sample size that is slightly lower than the frequentist method. Our design stops more often for superiority at all the interim analyses and reaches the maximum sample size less often than the model that does not allow borrowing. There is a slight type I error rate inflation in scenario 2 which gives us an idea of the price one pays for the considerable benefits demonstrated in scenario 1. In scenario 3, the methods perform similarly. In scenario 4 we see the borrowing methods can lead to a rather considerable reduction in type I error rate under the “global null” setting. The constrained MEMs demonstrate a different yet interesting benefit-cost tradeoff, with much larger power gain and smaller savings in terms of expected sample size under the alternative. Under the “local” null, type I error inflation is now slightly larger using the constrained MEMs, but lower under the “global” null.

TABLE 2.

Simulation results: Normal response.

	Method	Scenario 1 (1,1)	Scenario 2 (0,1)	Scenario 3 (0,.5)	Scenario 4 (0,0)
Power or type I error	Pocock	0.563	0.027	0.027	0.026
	MEMs(0.05)	0.635	0.030	0.027	0.023
	MEMs(0.1)	0.675	0.036	0.029	0.022
	MEMs*(0.2)	0.717	0.040	0.030	0.019
	MEMs*(0.5)	0.793	0.058	0.040	0.017
Expected sample size	Pocock	159.5	197.5	197.5	197.5
	MEMs(0.05)	154.5	197.5	197.5	198.0
	MEMs(0.1)	152.0	197.0	197.5	198.0
	MEMs*(0.2)	154.0	197.5	197.5	198.0
	MEMs*(0.5)	153.5	197.0	197.5	198.5
P(Stop at 1st analysis)	Pocock	0.11	0.01	0.01	0.01
	MEMs(0.05)	0.13	0.01	0.01	0.01
	MEMs(0.1)	0.13	0.01	0.01	0.01
	MEMs*(0.2)	0.14	0.01	0.01	0.01
	MEMs*(0.5)	0.15	0.01	0.01	0.01
P(Stop at 2nd analysis)	Pocock	0.16	0.01	0.01	0.01
	MEMs(0.05)	0.17	0.01	0.01	0.01
	MEMs(0.1)	0.19	0.01	0.01	0.01
	MEMs*(0.2)	0.18	0.01	0.01	0.01
	MEMs*(0.5)	0.16	0.01	0.01	0.00
P(Stop at 3rd analysis)	Pocock	0.16	0.01	0.01	0.01
	MEMs(0.05)	0.18	0.01	0.01	0.00
	MEMs(0.1)	0.19	0.01	0.01	0.00
	MEMs*(0.2)	0.16	0.01	0.01	0.00
	MEMs*(0.5)	0.16	0.01	0.01	0.00
P(Stop at 4th analysis)	Pocock	0.57	0.97	0.97	0.97
	MEMs(0.05)	0.52	0.97	0.97	0.98
	MEMs(0.1)	0.49	0.97	0.97	0.98
	MEMs*(0.2)	0.52	0.97	0.97	0.98
	MEMs*(0.5)	0.53	0.97	0.97	0.99

True treatment effects for primary, secondary study are in parentheses for each scenario. Asterisk denotes truncated MEMs with ζ_i = 25 for both arms at each interim analysis.

We again produce a figure comparing the MEMs methods, a matched calibrated traditional method and an uncalibrated traditional method in terms of power, expected sample size and MSE in Figure 2. In this Figure, we see that the MEMs methods yield some improvement in power even over the traditional methods calibrated to achieve the same type I error. The MEMs methods also achieve generally the same or higher expected sample size, along with a generally lower MSE of the point estimate. Therefore, unlike the above results in Figure 1, in this more common setting we now demonstrate advantages even over the calibrated traditional method.

FIGURE 2.

Simulation results: Normal response. MEMs(x) denotes MEMs method with x prior probability of exchangeability. Asterisk denotes truncated MEMs with ζ_i =25 for both arms at each interim analysis. Traditional (calib.) denotes a traditional no-borrowing analysis that is calibrated to achieve the same type I error rate as the corresponding MEM-based method. MSE=mean square error. In all plots the grey dashed line denotes the traditional method that controls type I error at the nominal rate of 0.025. Solid black vertical line denotes the point where sources are exchangeable

4.2 |. Binary Response

In this simulation, the response Y in each trial arm is Bernoulli distributed. The treatment effect is the difference in probabilities of Y =1 between treatment and control arms. In the first scenario, the control probability of success is 0.4 and the treatment mean is 0.6 in the primary study, therefore there is a true treatment effect of 0.2. The parameters are the same in the secondary study. In the second scenario, the probability of success is 0.4 in both treatment and control arms in the primary study (i.e., there is no treatment effect), but in the secondary study the treatment effect equals 0.2. Lastly, the third scenario is the same as the second one, except there is a smaller treatment effect in the secondary study of 0.1. The prior used for the proportion of successes is beta(1, 1), i.e., a uniform prior.

In scenario 1 in Table 3, the group-sequential design using MEMs has a noticeable gain in power over a frequentist design with Pocock boundaries. Furthermore, the expected sample size is lower under dynamic borrowing. Our design again stops more often for superiority at all the interim analyses and reaches the maximum sample size much less often than a model that does not allow borrowing. In these simulation results, the MEMs with π_e =0.05 and π_e =0.1 seem to have an advantageous benefit/cost tradeoff. Using this method there would likely be a considerable saving in terms of the length of the trial and power. The constrained MEMs are also an appealing option with a similar tradeoff vs unconstrained MEMs as seen above (smaller gain in expected sample size, but larger gain in power). The results in scenarios 2 and 3 are similar and mostly suggest little type I error rate inflation. The results in scenario 4 show some reduction in type I error rate under the “global” null scenario.

TABLE 3.

Simulation results: Binary response.

	Method	Scenario 1 (.2,.2)	Scenario 2 (0,.2)	Scenario 3 (0,.1)	Scenario 4 (0,0)
Power or type I error	Pocock	0.738	0.025	0.025	0.025
	MEMs(0.05)	0.766	0.027	0.025	0.023
	MEMs(0.1)	0.800	0.028	0.026	0.022
	MEMs*(0.2)	0.829	0.027	0.028	0.019
	MEMs*(0.5)	0.899	0.032	0.034	0.017
Expected sample size	Pocock	143.0	197.5	197.5	197.5
	MEMs(0.05)	141.5	197.5	198.0	198.0
	MEMs(0.1)	136.5	197.5	198.0	198.0
	MEMs*(0.2)	139.0	197.5	198.0	198.0
	MEMs*(0.5)	139.0	197.5	198.0	198.5
P(Stop at 1st analysis)	Pocock	0.15	0.01	0.01	0.01
	MEMs(0.05)	0.16	0.01	0.01	0.01
	MEMs(0.1)	0.18	0.01	0.01	0.01
	MEMs*(0.2)	0.19	0.01	0.01	0.01
	MEMs*(0.5)	0.18	0.01	0.01	0.01
P(Stop at 2nd analysis)	Pocock	0.25	0.01	0.01	0.01
	MEMs(0.05)	0.24	0.01	0.01	0.01
	MEMs(0.1)	0.27	0.01	0.01	0.01
	MEMs*(0.2)	0.23	0.01	0.01	0.01
	MEMs*(0.5)	0.23	0.01	0.01	0.00
P(Stop at 3rd analysis)	Pocock	0.18	0.01	0.01	0.01
	MEMs(0.05)	0.21	0.01	0.01	0.00
	MEMs(0.1)	0.21	0.01	0.01	0.00
	MEMs*(0.2)	0.20	0.01	0.01	0.00
	MEMs*(0.5)	0.21	0.01	0.01	0.00
P(Stop at 4th analysis)	Pocock	0.42	0.97	0.97	0.97
	MEMs(0.05)	0.39	0.97	0.97	0.98
	MEMs(0.1)	0.34	0.97	0.97	0.98
	MEMs*(0.2)	0.38	0.97	0.97	0.98
	MEMs*(0.5)	0.38	0.97	0.97	0.99

True treatment effects for primary, secondary study are in parentheses for each scenario. Asterisk denotes truncated MEMs with ζ_i = 25 for both arms at each interim analysis.

We also produce a figure comparing the MEMs methods, a matched calibrated traditional method and an uncalibrated traditional method in terms of power, expected sample size and MSE in Figure 3. In this Figure, we see that the MEMs methods again yield encouraging results with possible large improvement in power even over the traditional methods calibrated to achieve the same type I error, along with generally the same or higher expected sample size and considerably lower MSE of the point estimate close to the point of exchangeability. We also note the impact of borrowing on MSE can vary depending on how close to the point of exchangeability the two studies are, which is a behavior consistent with results in previous literature^24;38.

FIGURE 3.

Simulation results: binary response. MEMs(x) denotes MEMs method with x prior probability of exchangeability. Asterisk denotes truncated MEMs with ζ_i =25 for both arms at each interim analysis. Traditional (calib.) denotes a traditional no-borrowing analysis that is calibrated to achieve the same type I error rate as the corresponding MEM-based method. MSE=mean square error. In all plots the grey dashed line denotes the traditional method that controls type I error at the nominal rate of 0.025. Solid black vertical line denotes the point where sources are exchangeable

4.3 |. Linear Model Based MEMs

In this simulation, the response Y in each trial arm is again normally distributed, but we are interested in estimating a treatment effect as difference in means of Y between treatment and control groups after controlling for one important (continuous) predictor. We present 4 scenarios similar to those used above.

Simulation parameters described in Section 2.2.2 are set to: β_0,P = β_0,1 =2, γ = (0.8), ${\sigma }_P^2=5^2$, ${\sigma }_1^2=4^2$ and the covariate X is distributed as N (14, 2²) in the primary source and N (12, 3²) in the supplemental source. The regression parameter β_T,P is varied based on simulation scenario. The proportion of the variance in Y explained by the covariates is approximately 26%. In scenario 1, there is a treatment effect of 3 in both sources, in scenario 2, the treatment effects are 0 and 3 in the primary and secondary sources respectively, and in scenario 3, the treatment effects are 0 and 2 in the primary and secondary sources respectively.

As we can see from Table 4, we gain a large amount of power in scenario 1, with considerably lower expected sample size for all MEMs. In scenario 2, the type I error inflation of our method is more considerable for almost all MEMs methods presented, which means that the large benefits in scenario 1 come at a considerable price. This price could be diminished by tuning the MEM hyperparameters, but the benefits would be diminished as well. In scenario 3 we also see considerable type I error inflation. The unconstrained MEMs with π_e = 0.05 seem to offer the most advantageous benefit-cost tradeoff overall. Truncated MEMs could again be an interesting option, with a different benefit-cost tradeoff. However, we can see that the rather large increases in “local” type I error rates in scenarios 2 and 3 go hand in hand with similarly noticeable reductions in type I error rates under the “global” null setting in scenario 4 when data borrowing is used.

TABLE 4.

Simulation results: regression-based MEMs.

	Method	Scenario 1 (2, 2)	Scenario 2 (0, 2)	Scenario 3 (0, 1)	Scenario 4 (0, 0)
Power or type I error	Pocock	0.708	0.026	0.026	0.026
	MEMs(0.05)	0.810	0.043	0.038	0.017
	MEMs(0.1)	0.852	0.065	0.050	0.015
	MEMs*(0.2)	0.875	0.060	0.056	0.010
	MEMs*(0.5)	0.937	0.109	0.115	0.010
Expected sample size	Pocock	146.0	197.5	197.5	197.5
	MEMs(0.05)	134.5	196.0	197.0	198.5
	MEMs(0.1)	126.5	194.5	196.5	198.5
	MEMs*(0.2)	138.0	196.5	197.5	199.0
	MEMs*(0.5)	138.0	196.5	197.5	199.0
P(Stop at 1st analysis)	Pocock	0.15	0.01	0.01	0.01
	MEMs(0.05)	0.19	0.01	0.01	0.01
	MEMs(0.1)	0.21	0.02	0.01	0.01
	MEMs*(0.2)	0.18	0.01	0.01	0.01
	MEMs*(0.5)	0.18	0.01	0.01	0.01
P(Stop at 2nd analysis)	Pocock	0.22	0.01	0.01	0.01
	MEMs(0.05)	0.27	0.01	0.01	0.01
	MEMs(0.1)	0.31	0.02	0.01	0.01
	MEMs*(0.2)	0.25	0.01	0.01	0.00
	MEMs*(0.5)	0.25	0.02	0.01	0.00
P(Stop at 3rd analysis)	Pocock	0.19	0.01	0.01	0.01
	MEMs(0.05)	0.21	0.01	0.01	0.00
	MEMs(0.1)	0.21	0.01	0.01	0.00
	MEMs*(0.2)	0.20	0.01	0.01	0.00
	MEMs*(0.5)	0.20	0.02	0.01	0.00
P(Stop at 4th analysis)	Pocock	0.44	0.97	0.97	0.97
	MEMs(0.05)	0.33	0.97	0.97	0.98
	MEMs(0.1)	0.27	0.95	0.97	0.98
	MEMs*(0.2)	0.37	0.97	0.97	0.99
	MEMs*(0.5)	0.37	0.97	0.97	0.99

True treatment effects for primary, secondary study are in parentheses for each scenario. Asterisk denotes truncated MEMs with ζ_i = 25 for both arms at each interim analysis.

Lastly, we again produce a figure comparing the MEMs methods, a matched calibrated traditional method and an uncalibrated traditional method in terms of power, expected sample size and MSE in Figure 4. In this Figure, we see that the MEMs methods again yield encouraging results with improvement in power even over the traditional methods calibrated to achieve the same type I error, along with generally higher expected sample size and considerably lower MSE of the point estimate close to the point of exchangeability.

FIGURE 4.

Simulation results: regression-based MEMs. MEMs(x) denotes MEMs method with x prior probability of exchangeability. Asterisk denotes truncated MEMs with ζ_i =25 for both arms at each interim analysis. Traditional (calib.) denotes a traditional no-borrowing analysis that is calibrated to achieve the same type I error rate as the corresponding MEM-based method. MSE=mean square error. In all plots the grey dashed line represents the traditional method that controls type I error at the nominal rate of 0.025. Solid black vertical line denotes the point where sources are exchangeable

5 |. DISCUSSION

Clinical investigation of pediatric or rare diseases is challenging using conventional methods for trial design, due to the fact such trials can be difficult to enroll^47;48;49. This means performing an adequately powered stand-alone trial may not be feasible⁵⁰. Databases facilitating real-world and clinical trial evidence sharing are becoming more accessible, which presents a unique opportunity for approaches that augment trial data using data from other sources. Innovations in adaptive trial design are needed to fully take advantage of these opportunities. Our method allows for adaptive use of both control and treatment arm borrowing at each interim analysis of a primary trial of interest. We illustrated sequential testing using MEM to facilitate dynamic borrowing for a normally distributed endpoint, binary endpoint, and linear regression. Our method attained lower expected sample size than a frequentist analysis that does not allow borrowing, with greater power under the alternative. Lower expected sample size could potentially lead to substantial cost-saving in the primary trial. We also note that running the primary and secondary studies concurrently would allow the trials to share infrastructure, study personnel and support, therefore saving costs. This advantage could result in further cost savings above and beyond the reduced sample size. The drawbacks of our method manifest themselves under the null hypothesis (specifically the “local” null) where hyperparameters can be tuned to control the type I error rate inflation. However, type I error rates were consistently reduced below the nominal level under a “global” null setting, which is in line with previous literature³⁹. The benefits and drawbacks are scenario-specific, which emphasizes the importance of completing extensive simulations to understand the operating characteristics under the desired scenario.

Borrowing can lead to inflated type I error rates under the “local” null setting, but our motivating example does not fit into the traditional drug approval setting. Instead, it seems prudent to combine data sources when answering the question of a policy impact. We showed that using our method could improve power, save sample size, as well as costs of the VLNC-CM trial as compared to the traditional analysis. Completing the VLNC-CM trial earlier would be desirable, considering the tobacco and nicotine replacement market is rapidly evolving and cutting edge research is needed to inform regulatory decisions.

We utilized results and theorems from previous literature to draw a parallel between frequentist group-sequential theory and Bayesian group-sequential methodology, which can be used for selection of the decision boundaries and for power and sample size calculations in Bayesian designs that utilize supplemental data borrowing. We also proposed a truncated modification of the MEM method which can control the operating characteristics of the design over time. Traditionally Bayesian models (especially model priors) have to be calibrated to achieve desired frequentist trial operating characteristics. The calibration is often achieved through modifications of the prior and simulations performed using a variety of priors. Such “calibrated priors” to some extent attenuate the purpose of a Bayesian analysis, which may be subjective. Our truncated MEM method may allow for the prior specifications to be more detached from the frequentist operating characteristics, as the tuning parameter ζ_i adds another layer of flexibility. In our simulations we demonstrated how using the truncated MEMs could help the researcher align trial operating characteristics with their preferences (for example when increased overall power is paramount, and earlier stopping of the trial is not as important, the truncated MEMs presented could be a better option).

We considered normally distributed and binary endpoints, as well as a linear regression analysis. Future work may extend the regression-based approach to other types of outcomes, such as time-to-event. Although we described the regression-based method on a linear model, it can be easily extended for example to generalized linear models. We only considered early stopping for superiority. A futility rule could be developed based on conditional power to detect a treatment effect at full enrollment given all data (both primary and supplemental data) available at the time of the interim analysis. Finally, one could consider adaptive randomization extension, which would change the randomization ratio in favor of the treatment group with smaller ESSS, as there is less information available for that group⁵¹.

In conclusion, our method showed several benefits over the standard method under the alternative hypothesis. One should consider borrowing of supplemental information in some cases, for example in the area of pediatric or rare disease trials. The primary trial will often have a small sample size, which naturally calls for methods that incorporate supplemental data. In these cases, our method can provide a favorable tradeoff between precision and type I error rate. Our method could be used to address both cost and ethical challenges in such trials.