Calibrated Dynamic Borrowing Using Capping Priors

Abstract

Multisource exchangeability models (MEMs), a Bayesian approach for dynamically integrating information from multiple clinical trials, are a promising approach for gaining efficiency in randomized controlled trials. When the supplementary trials are considerably larger than the primary trial, care must be taken when integrating supplementary data to avoid overwhelming the primary trial. In this paper, we propose “capping priors,” which controls the extent of dynamic borrowing by placing an a priori cap on the effective supplemental sample size. We demonstrate the behavior of this technique via simulation, and apply our method to four randomized trials of very low nicotine content cigarettes.

1. Introduction

Opportunities often arise to augment randomized controlled trials (RCTs) with data from other sources with the same control arm, treatment arm, or both. For example, interventions previously deemed effective may later be used as control arms of trials for new treatments(1,2). Alternatively, we may have data on the treatment arm from other trials in the comparative effectiveness setting, or on both arms in a different patient population. Integrating information into the analysis of a current trial allows researchers to decrease the study’s targeted enrollment (2), which in turn can reduce patient burden (1) and conserve time and financial resources. From a statistical standpoint, borrowing information can also increase precision and power while reducing type I error, assuming that the data sources are truly similar and exchangeable (2).

While synthesizing data from multiple sources has the potential to increase efficiency, it must be done in a statistically principled manner to avoid introducing bias or increasing the type I error rate (1-3). To address this issue, Pocock (4) proposed a method implementing shrinkage estimators to effectuate an appropriate amount of static borrowing, given an a priori estimate of the inter-trial heterogeneity. While Pocock’s method utilized an estimate of the inter-trial heterogeneity to calibrate borrowing from the supplemental data, the ultimate determination as to whether the primary and supplemental data could be combined was based on the trial settings (i.e. intervention, trial population, etc.). We don’t believe, however, that it is sufficient in practice to consider the trial setting, exclusively, as ambiguity may exist as to whether or not changes in the trial settings would impact the treatment effect or other trial results. Moreover, while participant eligibility may be similar by design, the ultimate enrollment distribution of important prognostic subpopulations may vary between trials. For example, a group of investigators may run multiple trials with the same intervention and control arms, but different trial populations that may or may not alter the treatment or expected outcomes. In these cases, we may alternatively consider dynamic borrowing, which evaluates the similarity between the primary and supplemental data, and borrows more information when the data are similar and less when there is evidence of heterogeneity (1,2).

Dynamic borrowing has been studied extensively in the statistical literature, primarily from a Bayesian perspective, and includes power priors (5), hierarchical models (6), and commensurate priors (7,8). These methods rely on a single parameter to control borrowing, which is estimated from the data (e.g. the power parameter in power priors, the variance parameter in the hierarchical model, and the commensurability parameter in commensurate priors). This results in a simple approach to dynamic borrowing but one that is limited if there are multiple supplemental sources with varying degrees of inter-trial heterogeneity. More recently, Kaizer et al. (3) proposed multisource exchangeability models (MEMs), a flexible Bayesian approach to dynamic borrowing that uses Bayesian model averaging (BMA) (9-11) to effectuate borrowing. MEMs consider all possible exchangeability relationships among a primary data source when combined with supplemental data sources. Information is smoothed over all possible exchangeability models using BMA. A major advantage of MEMs is that they allow the degree of dynamic borrowing to vary across supplemental sources, resulting in more flexible inference for identifying disjointed subpopulations comprised of meta-subtypes or singleton subtypes on the basis of accumulating evidence. This extent of flexibility is not allowed by standard ‘single-source’ hierarchical or commensurate prior modeling (wherein parameters for all sources are assumed necessarily exchangeable) (3).

As with many Bayesian procedures, a key challenge to the practical applications of MEMs is prior specification. An incorrectly specified prior can result in inappropriate borrowing and poor model performance. This phenomenon is particularly true in trials for which the sample size of the current trial is substantially smaller than the sample sizes of supplementary trials. Overborrowing can be disadvantageous for several reasons. From a clinical trial design standpoint, most trials aim to procure similar participant enrollment in the interventional versus control arms (12,13), which may not be achieved in this case. Furthermore, and perhaps more importantly, the larger supplementary cohorts’ information would likely overwhelm the signal from the current data, thus negating the benefits of an RCT in the current population. Ideally, we would like to use the supplemental data to improve precision, while still basing most of our conclusions on the primary data.

Consider the Center for the Evaluation of Nicotine in Cigarettes (CENIC) as a motivating example. CENIC is a multi-institution collaboration that aimed to explore the impact of reducing the nicotine content in cigarettes on tobacco use behavior (14-17). CENIC included four RCTs; the last of which randomized 58 smokers with serious mental illnesses— including schizophrenia, schizoaffective disorder, or bipolar disorder—to either normal nicotine content (NNC) cigarettes or very low nicotine content (VLNC) cigarettes (17). While we can directly estimate the treatment effect from these subjects’ data, data from over 900 additional participants on these same two treatment groups are available from the three other CENIC RCTs in different patient populations (14-16). Borrowing strength from the other three trials has the potential to improve precision, which motivates the use of dynamic borrowing approaches. A direct application of MEMs with default, non-informative priors results in substantial borrowing from the supplemental data to the extent that the supplemental data attenuates the signal from the primary data source. We would like an intuitive approach to prior specification that limits the amount of borrowing, while still improving efficiency.

We address these limitations through the introduction of “capping priors,” which allow us to control the degree of borrowing from supplemental data while still gaining efficiency. Throughout this manuscript, we characterize the extent of borrowing from supplemental data using the effective supplemental sample size (ESSS) (18), which extends the notion of effective sample size (19,20) to dynamic borrowing and summarizes the efficiency gained through borrowing by the increase in sample size that would be needed to achieve the observed improvement in precision. Capping priors allow the investigator to specify the maximum ESSS and derive the prior source inclusion probabilities that limit borrowing to the desired level. This results in an intuitive approach to prior specification, which allows investigators to control the extent of borrowing, while maintaining the efficiency gains that result from dynamic borrowing.

The remainder of this paper proceeds as follows. First, we provide some background and notation, including a brief introduction of MEMs and how to compute the ESSS, in Section 2. Then, we derive a method for specifying capping priors given a desired ESSS threshold in Section 3. Next, we conduct a simulation study to evaluate the ability of capping priors to restrict the ESSS at the desired threshold in Section 4. Finally, we demonstrate our method’s utility in a clinical trial setting by applying capping priors to our motivating data in Section 5 and conclude with a discussion in Section 6.

2. Background and Notation

Suppose we wish to determine the effect of an interventional treatment compared to a control on some outcome and that we have pertinent information from one primary and H supplementary sources. We assume that the response vector y_p for the n_p observations of the primary source are normally distributed with mean μ_p and variance ${\sigma }_p^2$; in this source, the true treatment effect is denoted by Δ_p. Similarly, for h = 1,2,…,H, we assume that the response vector y_h for the n_h observations of the hth supplementary source are normally distributed with mean μ_h and variance σ_h²; in source h, the true treatment effect is denoted by Δ_h. We assume that the data are independent, within and across all sources and, for the purposes of our theoretical development, assume that all variances are known. Thus, the vector of all response values, denoted by y = (y_p,y₁,y₂,…,y_H)^T, has a multivariate normal distribution with mean μ = (μ_p,μ₁,μ₂,…,μ_H)^T and variance-covariance matrix Σ = diag{Σ_p,Σ₁,Σ₂,…,Σ_H} where Σ_p = σ_p²I_np and Σ_h = σ_h²I_nh for h = 1,2,…,H. Furthermore, for each subject, we assume that we have information on the randomized treatment assignment x_Trt and c covariate measures x₁,x₂,…,x_c with no missing data. We denote the collection of all data as D.

2.1. Multisource Exchangeability Models

MEMs enable cross-trial data integration using BMA (9-11) to facilitate borrowing from multiple supplemental sources (3). Each of the H supplemental sources can be either exchangeable or nonexchangeable with the primary source. Thus, there are K = 2^H unique combinations of the data, denoted by Ω_k for k = 1,2,…,K, each describing a potential set of supplementary sources that are truly exchangeable with the primary source. MEMs effectuate borrowing by averaging over each of the K models, resulting in an estimate of the parameter of interest that utilizes different amounts of supplemental data from each source (3,11).

To illustrate our concept, we will first consider a simplified case for which we only have one supplementary source, so that H = 1, resulting in two possible ways to combine the data. The first MEM, Ω₁, assumes that the supplementary data are not exchangeable with the primary data. Under this model, the baseline value, treatment effect, and covariate effects are allowed to vary across sources. Thus, the conditional mean of the response value can be estimated using a linear model that regresses the response y on a design matrix X₁ containing a vector of ones (for the intercept) and the randomized treatment assignment x_Trt, covariates x₁,x₂,…,x_c, supplementary source indicator γ₁, interaction between the supplementary source indicator and the randomized treatment assignment, and interactions between the supplementary source indicator and each of the covariates. That is, the conditional mean response model is

$$E(y\mid X_1,{\Omega }_1)={\beta }_0^{(1)}+{\beta }_{Trt}^{(1)}{x}_{Trt}+{\sum }_{i=1}^c{\beta }_i^{(1)}{x}_i+{\gamma }_1\left({\beta }_{0,1}^{(1)}+{\beta }_{Trt,1}^{(1)}{x}_{Trt}+{\sum }_{j=1}^c{\beta }_{j,1}^{(1)}{x}_j\right)$$ (1)

The second MEM, Ω₂, assumes that the primary and supplementary sources are exchangeable. Our linear model of the conditional mean of the response value regresses the response y on a simplified design matrix X₂ containing a vector of ones (for the intercept) and the randomized treatment assignment x_Trt, covariates x₁,x₂,…,x_c, supplementary source indicator γ₁, and interactions between the supplementary source indicator and each of the covariates, but excludes the interaction between the supplementary source indicator and the treatment effect. This specification allows the covariate effects to vary by source, while keeping the treatment effect the same for the primary and supplemental data sources. In this case, the conditional mean response can be modeled as

$$E(y\mid X_2,{\Omega }_2)={\beta }_0^{(2)}+{\beta }_{Trt}^{(2)}{x}_{Trt}+{\sum }_{i=1}^c{\beta }_i^{(2)}{x}_i+{\gamma }_1\left({\beta }_{0,1}^{(2)}+{\sum }_{j=1}^c{\beta }_{j,1}^{(2)}{x}_j\right)$$ (2)

More generally, we can express both models above as E(y∣X_k,Ω_k) = X_kβ^(k), where the design matrices X_k for k = 1,2 are described above, and the vector of regression coefficients is ${\beta }^{(k)}=\left({\beta }_0^{(k)},{\beta }_{Trt}^{(k)},{\beta }_1^{(k)},\dots ,{\beta }_c^{(k)},{\beta }_{0,1}^{(k)},{\beta }_{Trt,1}^{(k)},{\beta }_{1,1}^{(k)},\dots ,{\beta }_{c,1}^{(k)}\right)$. Note that, for the model in Equation 2, the term ${\beta }_{Trt,1}^{(2)}$ will simply be 0.

This method can easily be extended to scenarios with multiple supplementary sources (i.e., when H > 1). In these scenarios, we can construct models and design matrices similarly by considering K = 2^H design matrices with corresponding model parameters β_0,h,β_Trt,h,β_1,h, … , β_c,h for each additional supplementary source h.

Regardless of the number of sources, our primary aim is to estimate the element, β_Trt, of the parameter vector, β, corresponding to the treatment effect in the primary source. The model specific posterior of β^(k) given X_k, Ω_k, and Σ, follows a multivariate normal distribution with mean $\phantom{(}E({\beta }_{Trt}^{(k)}\mid X_k,{\Omega }_k,\Sigma )=(X_k^T{\Sigma }^{-1}{X}_k{)}^{-1}{X}_k^T{\Sigma }^{-1}y$ and variance $\phantom{(}Var({\beta }_{Trt}^{(k)}\mid X_k,{\Omega }_k,\Sigma )=(X_k^T{\Sigma }^{-1}{X}_k{)}^{-1}$.

MEMs use BMA to smooth over the model-specific posteriors for β_Trt, resulting in a posterior distribution that is a weighted average of a posterior assuming no borrowing and a posterior assuming borrowing (3,9-11). BMA requires that we specify a prior distribution for each scenario, which is most easily done by assuming prior independence on the prior probability p_h that each supplemental data source h (for h = 1,2,…,H) is exchangeable with the primary source. Then, when H = 1, the prior weights are π(Ω₁) = 1 – p₁ and π(Ω₂) = p₁. When H > 1, the prior weight of Ω_k for k = 1,2,…,K is

$$\pi ({\Omega }_k)={\prod }_{h=1}^H{p}_h^{s_{h,k}}(1-p_h{)}^{1-s_{h,k}},$$ (3)

where s_h,k is an indicator function that equals 1 if supplementary source h is exchangeable with the primary source under model Ω_k, and 0 otherwise. We note that the indicator function, s_h,k, is neither observed data or a parameter to be estimated, but rather notation we use to specify the sub-model, Ω_k, That is, each sub-model is defined by a different combination of “s_h,k”s representing all of the different ways that the data can be combined.

Given the prior weights, the posterior weight for the kth model is a function of the prior weights and the marginalized likelihood, p(D∣Ω_k), for each MEM, i.e.,

$$p({\Omega }_k\mid D)=\frac{\pi ({\Omega }_k)p(D\mid {\Omega }_k)}{{\sum }_{j=1}^K\pi ({\Omega }_j)p(D\mid {\Omega }_j)},$$ (4)

as detailed in Raftery (11). Moreover, as described by Kotalik et al. (21) and Burnham and Anderson (22), when the model-specific marginals have no closed form solution, it can be useful to instead approximate the posterior weights by

$${\omega }_k=\frac{\pi ({\Omega }_k)\text{exp}(-0.5{\eta }_k)}{({\sum }_{j=1}^K\pi ({\Omega }_j)\text{exp}(-0.5{\eta }_j)},$$ (5)

with η_k = BIC_k – min{BIC₁,BIC₂,…,BIC_K} where BIC_k is the Bayesian Information Criterion for the kth MEM. We opt to utilize this approximation in our analysis.

Given the posterior weights ω_k for k = 1,2,…,K, the overall posterior for the primary source treatment effect, β_Trt, is a weighted average of the MEM-specific posteriors, i.e.,

$$P({\beta }_{Trt}\mid D)={\sum }_{k=1}^K{\omega }_{k}P({\mathrm{B}}_{Trt}^{(k)}\mid X_k,{\Omega }_k,\Sigma ),$$ (6)

and is consequently a mixture of normal distributions. That is,

$${\mathrm{B}}_{Trt}\mid D\sim N({\sum }_{k=1}^K{\omega }_{k}E\left({\mathrm{B}}_{Trt}^{(k)}\mid X_k,{\Omega }_k,\Sigma \right),{\sum }_{k=1}^K{\omega }_{k}E\left(({\beta }_{Trt}^{(k)}{)}^2\mid X_k,{\Omega }_k,\Sigma \right)-[{\sum }_{k=1}^K{\omega }_{k}E\left({\beta }_{Trt}^{(k)}\mid X_k,{\Omega }_k,\Sigma \right){]}^{\wedge }2)$$ (7)

where $\phantom{(}E({\left({\beta }_{Trt}^{(k)}\right)}^2\mid X_k,{\Omega }_k,\Sigma )=\phantom{(}Var({\beta }_{Trt}^{(k)}\mid X_k,{\Omega }_k,\Sigma )+E{\left({\beta }_{Trt}^{(k)}\mid X_k,{\Omega }_k,\Sigma \right)}^2$.

This mixture distribution is sensitive to the MEM-specific prior weights π(Ω_k) for k = 1,2,…,K, which dictate the degree of borrowing. Prior weights that favor little to no borrowing (e.g., π(Ω₁) = 1,π(Ω_k) = 0 ∀ k ≠ 1) will result in a mixture distribution that is close to the distribution of response variable in the primary source, alone. As the prior weights change to encourage more borrowing (e.g., π(Ω_K) = 1,π(Ω_k) = 0 ∀ k ≠ 1), the mixture distribution will become more similar to the response distributions in the supplementary sources and, potentially, more dissimilar from that of the primary source. Thus, in many situations, the investigator may wish to cap the total amount of borrowing from the H supplemental sources to be less than a pre-specified desired threshold. In order to better quantify this threshold and the true level of borrowing, we use a quantity called the effective supplemental sample size (ESSS) and refer to the desired threshold as ESSS_t.

2.2. Computation of ESSS Given Priors

As described above, the extent of borrowing can be characterized by the effective supplemental sample size (ESSS) (18,23), which extends the concept of effective sample size (19,20) to account for dynamic borrowing, and refers to the additional number of observations that would be required to achieve the same precision as was observed through dynamic borrowing. When the posterior precision is approximately linear in the sample size (as in the case of linear models), ESSS is defined as

$$ESSS=n_p\left(\frac{P_{Joint}}{P_{Reference}}-1\right),$$ (8)

where P_Reference is the posterior precision of the treatment effect under the reference model (which does not utilize any borrowing) and P_Joint is the posterior precision of the treatment effect under the joint model (which does utilize borrowing).

In our case, the reference model corresponds to Ω₁, for which the treatment effect is estimated independently for each source. Thus, the posterior precision for the reference model is

$$P_{Reference}={\phantom{[}\left[Var({\beta }_{Trt}^{(1)}\mid X_1,{\Omega }_1,\Sigma \right)]}^{-1}=X_1^T{\Sigma }^{-1}{X}_1.$$ (9)

The joint model assumes that the overall posterior, P(β_Trt∣D), for the treatment effect β_Trt of the primary source has a distribution equal to a weighted mixture of the K MEM specific posteriors for β_Trt. As described in Equation 7, this posterior precision is

$$P_{Joint}={\left\{{\sum }_{k=1}^K{\omega }_{k}E\phantom{(}({\left({\beta }_{Trt}^{(k)}\right)}^2\mid X_k,{\Omega }_k,\Sigma )-[{\sum }_{k=1}^K{\omega }_{k}E\phantom{(}({\beta }_{Trt}^{(k)}\mid X_k,{\Omega }_k,\Sigma ){]}^{\wedge }2\right\}}^{-1}.$$ (10)

Our goal is to find capping priors π(Ω_k) for k = 1,2,…,K that restrict the induced ESSS in Equation 8 to be at most some a priori specified desired threshold ESSS_t.

3. Capping Priors

In this section, we first consider the simplified scenario for which there is only one supplementary source and thus only one prior inclusion probability, p₁, of interest. Note that π(Ω₁) = 1 – p₁ and π(Ω₂) = p₁, which implies that the prior weights for each model can be written in terms of p₁. Given the direct connection between the prior model weights and the ESSS, we can derive the value of p₁ such that ESSS is less than some pre-specified threshold, ESSS_t. However, ESSS is also a function of the data, particularly the elements in y_p, which are unobserved a priori. Hence, we must derive p₁ such that the ESSS is less than ESSS_t for all y_p. We start by discussing our approach to deriving p₁ assuming that all data are known and then discuss how our approach can be relaxed in the next section.

Assuming all data are known, we can conduct a grid search by iterating across a fine grid of possible values over the support of p₁, considering values from smallest to largest. At each iteration, we compute the induced ESSS. Then, we flag the largest value for p₁ such that the corresponding induced ESSS for that value and for all considered p₁ values before it are not greater than ESSS_t, and we choose it to be $p_1^{\ast }$. Finally, we assign the capping priors as $\pi ({\Omega }_1{)}^{\ast }=1-p_1^{\ast }$ and $\pi ({\Omega }_2)=p_1^{\ast }$.

3.1. Estimating Responses from Primary Source

In a Bayesian data analysis, the prior cannot be a function of the outcome data. Thus, instead of using the primary source responses, y_p, we must use a proxy estimate, $y_p^{\ast }$, and use $y^{\ast }={\left(y_p^{\ast },y_1\right)}^T$ as a proxy for y in our computations of the capping prior above. We wish to choose y* such that the true induced ESSS (based on y) will be at most that which we would obtain with y*. Because the ESSS is maximized when the primary data have the same conditional mean as the supplementary data, we wish to generate $y_p^{\ast }$ from the conditional mean of the supplemental data.

To find this conditional mean, we first find an estimate, $\widehat{\beta }(1)$, of the vector of regression coefficients, β(1) = (β_0,1, β_Trt,1,β_1,1, … , β_c,1)^T. That is, we simply regress the responses y₁ from the one supplementary source on a design matrix, X(1) containing information on the intercept, treatment assignment, and covariate values of the supplementary source. We then let $y_p^{\ast }=X(P)\widehat{\beta }(1)$, where X(P) is a design matrix containing information on the intercept, treatment assignment, and covariate values in the primary source.

3.2. Multiple Supplementary Sources

If H > 1, we must find the prior inclusion probability for each supplementary source one by one, in sequence. Consider source h, where h = 1,2,…,H. To find $p_h^{\ast }$, we first regress the vector of supplementary source responses y_h on a design matrix X(h) containing information from the hth supplementary source on the intercept, treatment assignment, and covariate values to obtain a vector of estimated regression coefficients: $\widehat{\beta }(h)={\left({\widehat{\beta }}_{0,h},{\widehat{\beta }}_{Trt,h},{\widehat{\beta }}_{1,h},\dots ,{\widehat{\beta }}_{c,h}\right)}^T$ for h = 1,2,…,H. Next, we let $y_p^{\ast ,h}=X(P)\widehat{\beta }(1)$, and $y_a^{\ast ,h}=X(a)\widehat{\beta }(1)$ for a ∈ {1,2,…,H}\{h}, where X(P) is a design matrix containing information on the intercept, treatment assignment, and covariate values of the primary source and X(a) is a design matrix containing information on the intercept, treatment assignment, and covariate values of the ath supplementary source. We then use $y^{\ast ,h}={\left(y_p^{\ast ,h},y_1^{\ast ,h},\dots ,y_h,\dots ,y_H^{\ast ,h}\right)}^T$ as a proxy for y while searching for $p_h^{\ast }$. When there are multiple supplementary sources, we typically do not have a priori reason to believe that any source is more likely to be exchangeable with the primary source than the rest. Thus, we assume for simplicity that all prior inclusion probabilities are equal, i.e., that $p_1^{(h)}=p_2^{(h)}=\dots =p_H^{(h)}$. Then, we can complete a grid search, as described above, setting all prior inclusion probabilities equal to ${\tilde{p}}_h^{(h)}$. Then we let $p_h≔{\tilde{p}}_h^{(h)}$. After repeating the above process for all supplementary sources, we will have found all of the prior inclusion probabilities p_h for h = 1,2,…,H. Note that p₁ = p₂ = … = p_H will be true by nature of our algorithm, if the assumptions of normality and homogeneity within each source are met. Finally, we will then we compute the prior model weights using Equation 3.

3.3. Estimating Variance-Covariance Matrix

To compute the capping priors above, we assumed that the variance-covariance matrix Σ is fixed and known. In practice, however, this quantity is typically unknown and must be accounted for before we can solve for the capping prior. Ideally, we would specify a prior on Σ and then derive the capping priors after integrating out Σ. However, this approach is not analytically tractable. Thus, we propose the use of a plug-in estimator, $\widehat{\Sigma }=diag({\widehat{\sigma }}_p^2,{\widehat{\sigma }}_1^2,\dots ,{\widehat{\sigma }}_H^2)$, for Σ, which we obtain by fitting separate models for each source and setting ${\widehat{\sigma }}_a^2$ equal to the residual variance for model a. While this approach is not fully Bayesian, our simulation study illustrates that this simpler estimation technique is adequate for meeting our objectives.

4. Simulation Study

We completed a thorough simulation study to evaluate whether capping priors effectively control borrowing via MEMs. For our simulation study, we consider scenarios for which there are 1, 2, or 3 supplementary sources. For all scenarios, we consider one covariate (so that c = 1), with values originating from a standard normal distribution. We also fix the primary cohort’s sample size to be n_p = 50, the primary cohort’s variance to be ${\sigma }_p^2=50$, and the total number of supplementary observations to be ${\sum }_{h=1}^H{n}_h=900$. In all scenarios, we vary the true treatment effect in the primary source between 1 and 3 in increments of 0.5, and we consider six scenarios where we set ESSS_t at equally spaced values between 0 to 900.

For the different scenarios, we consider when the supplementary sources have equal treatment effects (with Δ_h = 2 for h = 1,…,H) or unequal treatment effects (with Δ_h = 2^h for h = 1,…,H.) We also consider when the supplementary sources have equal sample sizes (with n_h = 900/H for h = 1,…,H) or unequal sample sizes (with $n_h=\frac{900h}{{\sum }_{j=1}^{H}j}$ for h = 1,…,H). Finally, we also consider scenarios where the primary and supplementary sources all have equal variance (i.e. ${\sigma }_p^2={\sigma }_1^2=\cdots ={\sigma }_H^2=50)$), as well as scenarios where the variance in the supplemental sources vary. The sets of parameters specific to each scenario are enumerated in Table 1.

Table 1:

Scenarios considered in simulation study. We vary the number H of supplementary sources, as well as whether the treatment effects Δ₁,Δ₂,…,Δ_H in the supplementary sources are equal, whether the sample sizes n₁,n₂,…,n_H in the supplementary sources are equal, and whether the variances ${\sigma }_1^2$, ${\sigma }_2^2,\dots ,{\sigma }_H^2$ in the supplementary sources are equal.

Scenario	H	Equal Δh	Equal nh	Equal ${\sigma }_h^2$	Δ1, Δ2, … , Δh	n1, n2, … , nH	${\sigma }_1^2$, ${\sigma }_2^2,\dots ,{\sigma }_H^2$
1	1	T	T	T	2	900	50
2	1	T	T	F	2	900	40
3	2	T	T	T	2, 2	450, 450	50, 50
4	2	T	T	F	2, 2	450, 450	40, 60
5	2	T	F	T	2, 2	300, 600	50, 50
6	2	T	F	F	2, 2	300, 600	40, 60
7	2	F	T	T	2, 4	450, 450	50, 50
8	2	F	T	F	2, 4	450, 450	40, 60
9	2	F	F	T	2, 4	300, 600	50, 50
10	2	F	F	F	2, 4	300, 600	40, 60
11	3	T	T	T	2, 2, 2	300, 300, 300	50, 50, 50
12	3	T	T	F	2, 2, 2	300, 300, 300	40, 50, 60
13	3	T	F	T	2, 2, 2	150, 300, 450	50, 50, 50
14	3	T	F	F	2, 2, 2	150, 300, 450	40, 50, 60
15	3	F	T	T	2, 4, 8	300, 300, 300	50, 50, 50
16	3	F	T	F	2, 4, 8	300, 300, 300	40, 50, 60
17	3	F	F	T	2, 4, 8	150, 300, 450	50, 50, 50
18	3	F	F	F	2, 4, 8	150, 300, 450	40, 50, 60

To compose the vector of true regression coefficients, β, for each scenario, we let the element of β corresponding to the primary source’s treatment effect be β_Trt = Δ_p, the element corresponding to the hth source’s indicator be β_0,h = Δ_h, and the element corresponding to the interaction between the hth source indicator and primary source treatment effect be B_Trt,h = Δ_h – Δ_P. All other elements of β are set at the arbitrarily chosen value of 1.

Data are generated by first simulating covariate values from a standard normal distribution. The outcome vector, y, is then simulated from a normal distribution with parameters μ = Xβ and $\Sigma =ag({\sigma }_p^2,{\sigma }_1^2,\dots ,{\sigma }_H^2)$. A grid size 0.001 was used for determining the capping priors, and 1000 simulated data sets were considered for each scenario.

We assess the performance of capping priors by evaluating the extent of borrowing for each simulated data set, as measured by the ESSS. First, we check that our algorithm produces ESSS values of at most each of the desired ESSS_t thresholds for all simulations. Next, we verify that a lower ESSS_t corresponds to a lower mean and maximum ESSS, across all simulations. Finally, we verify that our algorithm borrows more information (i.e., has a higher induced ESSS) in situations for which the data are truly exchangeable—that is, when the true treatment effect, Δ_P, of the primary source is relatively close to the true treatment effects, Δ_h for h = 1,2,3, of the supplementary sources.

4.1. Results

For each scenario in Table 1, we visualize our results with three plots, which are shown in Figures 1 - 4. The first of each set of plots shows ESSS_t versus the maximum induced ESSS among all simulations for various possible values of Δ_p. From these plots, we note that the maximum induced ESSS across all simulations is always less than or equal to ESSS_t, which indicates that the capping priors have successfully limited the amount of borrowing to be at most the desired threshold. We also note that a lower ESSS_t value corresponds to a lower induced ESSS, as desired.

Figure 1:

Simulation results for Scenarios 1 – 5. Plots of (1) target ESSS threshold ESSS_t versus maximum induced ESSS among all simulations for various primary source treatment effects Δ_p, (2) target ESSS threshold ESSS_t versus mean induced ESSS among all simulations for various primary source treatment effects Δ_p, (3) treatment effect Δ_p in primary source versus mean induced ESSS among all simulations for various target ESSS thresholds ESSS_t, and (4) target ESSS threshold ESSS_t versus mean prior inclusion probabilities π(Ω_h) for h = 1,2,…,H.

Figure 4:

Simulation results for Scenarios 16 – 18. Plots of (1) target ESSS threshold ESSS_t versus maximum induced ESSS among all simulations for various primary source treatment effects Δ_p, (2) target ESSS threshold ESSS_t versus mean induced ESSS among all simulations for various primary source treatment effects Δ_p, (3) treatment effect Δ_p in primary source versus mean induced ESSS among all simulations for various target ESSS thresholds ESSS_t, and (4) target ESSS threshold ESSS_t versus mean prior inclusion probabilities π(Ω_h) for h = 1,2,…,H.

In the second of each set of plots, we have ESSS_t versus the mean induced ESSS among all simulations for various possible values of P. We again note that lower ESSS_t values correspond to lower induced ESSS.

The last of each set shows a plot of the treatment effect Δ_p in the primary source against the mean induced ESSS among all simulations for various possible threshold values, ESSS_t. In plots for which the supplementary cohorts’ treatment effects are equal (at Δ_h = 2 for h = 1,…,H), the average induced ESSS is higher when Δ_p is closer to the supplementary sources’ shared treatment effect. This illustrates the desired performance of MEMs, which should borrow more information when the treatment effects of the primary and supplementary sources are similar and ignore the supplementary data when there is heterogeneity between the primary and supplemental data set.

These findings demonstrate that the use of capping priors for MEMs is effective at capping the amount of overall supplementary information borrowed in the analysis at the desired ESSS_t threshold and that a lower ESSS_t value corresponds to a lower induced ESSS. Moreover, our results suggest that our method encourages more borrowing when the treatment effects in the supplementary sources are truly similar to the primary source.

5. Application to CENIC Project

We illustrate the use of capping priors through an application to four RCTs from the Center for the Evaluation of Nicotine in Cigarettes (CENIC), which examined the impact of nicotine reduction on smoking behavior. The last trial, titled Very Low Nicotine Cigarettes in Smokers With Schizophrenia (PS), was a double-blind randomized trial that collected data from 2014-2017 on smokers with serious mental illnesses, such as schizophrenia, schizoaffective disorder, and bipolar disorder (17). Subjects were randomized to receive either normal nicotine content (NNC) cigarettes with 15.8 mg nicotine/g tobacco or very low nicotine content (VLNC) cigarettes with 0.4 mg nicotine/g tobacco. The primary outcome of interest was the total number of cigarettes smoked per day (CPD) at the end of a 6-week period. The primary analysis used linear regression to compare the primary outcome between treatment groups, adjusting for baseline CPD as a covariate. Fifty-eight participants were randomized, and data are available for 51 subjects at study completion.

In addition to PS, CENIC completed three previous trials of VLNC cigarettes that could be used as supplemental data. The first trial, called Project 1, Study 1 (P1S1) explored the effect of VLNC cigarettes compared to NNC cigarettes with respect to tobacco dependence, measured by the total CPD and other variables (14). The second trial, called Project 1, Study 2 (P1S2), examined the impact of VLNC cigarettes with and without transdermal nicotine replacement therapy (i.e nicotine patch) on cigarette smoking, measured through the same variables as in P1S1 (16). The third trial, called Project 2 (P2), evaluated differences in biomarkers of tobacco exposure and smoking patterns among three participant groups: those given NNC cigarettes, those given VLNC cigarettes, and those given NNC cigarettes at the beginning of the study and were, over time, given cigarettes corresponding to a gradual reduction in nicotine content until receiving VLNC cigarettes for the last month of the intervention (15). In total, these trials contained information on 2,329 subjects; 1,111 of which were in treatment arms that administered NNC or VLNC cigarettes and, thus, were similar to the arms in PS (14,15,24). After removing two subjects’ data for having unfeasible measurements of over 180 CPD, there were 978 complete case observations, with information on both baseline and total CPD at the end of 6 weeks (for P1S1 and P1S2) or 8 weeks (for P2, since data on CPD at 6 weeks is unavailable for P2). Table 2 shows a detailed breakdown of these sample sizes, and Figure 5 has boxplots for the distributions of baseline CPD and ending total CPD in each source.

Table 2:

Summary of CENIC data by trial. Information on each trial’s abbreviation, defining characteristics of treatment arms, number of subjects randomized to each arm, and number of subjects in each arm used in our analysis. Subjects in all arms were given cigarettes with constant nicotine content throughout the study duration, except for those in arms labeled “gradual” (for which the cigarettes’ nicotine content was gradually decreased during the study). “NRT” denotes nicotine-replacement therapy.

Abbreviation	Arms	Randomized	Used in Analysis
PS	15.8 mg/g (NNC)	28	25
	0.4 mg/g (VLNC)	30	26
P1S1	Usual cigarette brand	118	0
	5.2 mg/g tobacco filler	122	0
	2.4 mg/g tobacco filler	119	0
	1.3 mg/g tobacco filler	119	0
	0.4 mg/g tobacco filler with 13 mg tar	123	0
	15.8 mg/g tobacco filler (NNC)	119	110
	0.4 mg/g tobacco filler (VLNC)	119	109
P1S2	15.8 mg/g (NNC) and NRT	59	0
	0.4 mg/g tobacco filler (VLNC) and NRT	60	0
	15.8 mg/g tobacco filler (NNC)	61	54
	0.4 mg/g tobacco filler (VLNC)	60	49
P2	15.8 to 0.4 mg/g, gradual	498	0
	15.8 mg/g tobacco filler (NNC)	249	234
	0.4 mg/g tobacco filler (VLNC)	503	422

Figure 5:

Boxplots of data from CENIC project used in this analysis. There are separate boxplots for the primary source PS (labeled “p”) and each of the three supplementary sources P1S1, P1S2, and P2 (labeled “1”, “2”, and “3,” respectively).

As PS was much smaller than the other trials, we hope to gain precision by borrowing information from the other trials using MEMs. We note that borrowing all the information from the three substantially larger data sets could potentially obscure the most important information from the much smaller primary cohort. Thus, we set a desired ESSS threshold of ESSS_t = 102, which is twice the number of subjects in the primary cohort. We choose a grid sizes of 0.001 for finding $p_1^{\ast }$.

For comparison, we conduct two additional analyses. The first analysis uses flat prior inclusion probabilities of p_h = 0.5 for h = 1,2,3, which lead to equal priors π(Ω_k) = 0.125 for all MEMs k = 1,2,…,8. In this analysis, we are cautious of a large ESSS, potentially much larger than the primary source’s sample size. The second analysis is a standard analysis of the PS data that ignores the supplemental information. Using only data from the primary cohort, we fit a linear model regressing the total ending CPD on treatment assignment and baseline CPD. We expect to see that this method’s point estimate for β_Trt (compared to that which we will find using capping priors and MEMs) will be similar but that the uncertainty surrounding this estimate will be larger.

5.1. Results

Based on models with only each of the secondary sources (P1S1, P1S2, and P2) separately, estimates of the supplementary-source-specific treatment effects are −6.120, −5.367, and −4.791, respectively. Using our algorithm, we compute the prior inclusion probabilities corresponding to the capping priors to be $p_1^{\ast }=p_2^{\ast }=p_3^{\ast }=0.20$. Ultimately, we achieve an ESSS of 58, which is less than the pre-specified threshold of 102. We compute the overall average treatment effect estimate to be E(β_Trt∣D) = −4.776 and its variance to be V ar(β_Trt∣D) = 1.984. Assuming a normal distribution, this finding corresponds to an entirely negative 95% confidence interval of (−7.537, −2.016), suggesting that the use of VLNC (as opposed to NNC) cigarettes is significantly associated with a reduction in the total number of CPD.

In comparison, an analysis that uses uninformative priors leads to MEM-specific priors and posteriors that yield an ESSS of 797.0, which is over 15 times the sample size in the primary cohort of 51. The average treatment effect is estimated to be −5.063, with a variance 0.2560 and a confidence interval of (−6.054, −4.071). While this estimation is very precise, the results are essentially an analysis of the supplemental data, thus undermining the purpose of conducting a RCT in the primary cohort.

Furthermore, an analysis with using only data from the primary cohort yields an average treatment effect estimate of −4.014, variance of 4.434, and 95% confidence interval of (−12.70, 4.677). As expected, compared to the result using capping priors and MEMs, this point estimate is similar, but the variance is higher and the confidence interval is wider.

6. Discussion

In this paper, we described capping priors as a means for calibrating the amount of supplementary information that is dynamically borrowed using MEMs. Our simulation results suggest that our method allows MEMs to cap the amount of borrowed information at an a priori specified threshold. Furthermore, the use of capping priors preserves an important property of MEMs—that more supplementary information is borrowed when the source-specific treatment effects are more similar between the primary and supplementary sources and that less supplementary information is borrowed, otherwise.

An application to data from CENIC demonstrates the utility of our method. In Section 5, we used MEMs with capping priors to analyze the last in a series of four trials, while borrowing a proportion of the information from the first three trials. Ultimately, we conclude that, among smokers with serious mental illnesses, the use of very low nicotine content (instead of normal nicotine content) cigarettes is associated with a significant reduction in smoking, with an expected smoking reduction of 2.016 to 7.537 cigarettes (with mean 4.776) per day. These results align with the previous literature, which suggest that decreased nicotine content in cigarettes may encourage smokers (including those with mental health conditions, such as schizophrenia) to smoke less (25,26). Moreover, this analysis using capping priors yielded an ESSS of 58, which is smaller and much more reasonable than an analysis using uninformative priors, which leads to an ESSS of 797.0. Furthermore, compared to an analysis of the primary trial alone (i.e., with no data borrowing), our results using capping priors with MEMs have a similar point estimate and a 23.52% increase in precision.

In our applied example, we specified ESSS_t to be twice the size of the primary trial and in our simulation study, we considered scenarios where ESSS_t was 18 times the size of the primary trial. In all cases, capping priors were able to control the amount of borrowing, resulting in an ESSS less that the pre-specified threshold. ESSS_t is an important parameter that must be pre-specified and that will likely have a substantial impact on trial operating characteristics (i.e. type-I error rate, power, etc.). In practice, we recommend an ESSS_t of no more than 2 – 3 times the sample size of the primary trial, which could potentially result in a situation where 66% - 75% of statistical information could come from the supplemental data. Alternately, ESSS_t could be viewed as a tuning parameter, with simulation used to evaluate the impact of different values on trial operating characteristics, as has been proposed in other applications of MEMs(27). Additional research is needed to understand the impact of varying ESSS_t on trial operating characteristics. Throughout this manuscript, we made the simplifying assumption of equal prior inclusion probabilities for all supplemental sources. This is justified because, in most cases, there will be little a priori information that would result in a preference for borrowing from one supplemental data source over the others. We previously considered the possibility of allowing the prior inclusion probabilities to vary by source but this resulted in poor performance because there is no unique solution for controlling the total ESSS. This indicates that, while unequal prior inclusion probabilities could be accommodated using the approach outlined in this paper, the investigators would be required the pre-specify the relationship between the prior inclusion probabilities between sources (i.e. specifying the prior inclusion probability for one source to be twice the other sources).

Nevertheless, this study has several limitations. Currently, our method relies on generalized least squares regression and thus assumes that the data from each source are normally distributed. This assumption may be unreasonable in many applications, for which the response variable may be bimodal or have some other non-Gaussian shape. Nevertheless, assuming that the data are not normally distributed leads to a more complicated problem, where the posterior may no longer be available in closed form, meaning that the another method (such as the Expectation-Maximization algorithm and other forms of numerical integration) might be required. We will consider extending capping priors to accommodate other data types in future research. Finally, a limitation of MEMs is that the number of sub-models grows quickly with the number of supplemental trials. In practice, we anticipate that the number of supplemental trials will be small (5 or less) and that capping priors would work well in most settings. Recently, iterated MEMs were proposed as a solution for applying MEMs with a large number of supplemental data sources (28). In principle, capping priors could be extended to iterated MEMs but further research is needed to understand how this would be implemented, in practice.

Figure 2:

Simulation results for Scenarios 6 – 10. Plots of (1) target ESSS threshold ESSS_t versus maximum induced ESSS among all simulations for various primary source treatment effects Δ_p, (2) target ESSS threshold ESSS_t versus mean induced ESSS among all simulations for various primary source treatment effects Δ_p, (3) treatment effect Δ_p in primary source versus mean induced ESSS among all simulations for various target ESSS thresholds ESSS_t, and (4) target ESSS threshold ESSS_t versus mean prior inclusion probabilities π(Ω_h) for h = 1,2,…,H.

Figure 3:

Simulation results for Scenarios 11 – 15. Plots of (1) target ESSS threshold ESSS_t versus maximum induced ESSS among all simulations for various primary source treatment effects Δ_p, (2) target ESSS threshold ESSS_t versus mean induced ESSS among all simulations for various primary source treatment effects Δ_p, (3) treatment effect Δ_p in primary source versus mean induced ESSS among all simulations for various target ESSS thresholds ESSS_t, and (4) target ESSS threshold ESSS_t versus mean prior inclusion probabilities π(Ω_h) for h = 1,2,…,H.