Fusion Designs and Estimators for Treatment Effects

Abstract

While randomized trials remain the best evidence for treatment effectiveness, lack of generalizability often remains an important concern. Additionally, when new treatments are compared against existing standards of care, the potentially small benefit of the new treatment may be difficult to detect in a trial without extremely large sample sizes and long follow-up times. Recent advances in ‘data fusion’ provide a framework to combine results across studies that are applicable to a given population of interest and allow treatment comparisons that may not be feasible with traditional study designs. We propose a data fusion-based estimator that can be used to combine information from two studies: 1) a study comparing a new treatment to the standard of care in the local population of interest, and 2) a study comparing the standard of care to placebo in a separate, distal population. We provide conditions under which the parameter of interest can be identified from the two studies described and explore properties of the estimator through simulation. Finally, we apply the estimator to estimate the effect of triple- versus monotherapy for the treatment of HIV using data from two randomized trials. The proposed estimator can account for underlying population structures that induce differences in case mix, adherence, and outcome prevalence between the local and distal populations, and the estimator can also account for potentially informative loss to follow-up. Approaches like those detailed here are increasingly important to speed the approval and adoption of effective new therapies by leveraging multiple sources of information.

Experiment remains the best judge of scientific truths. Two leading shortcomings of experimental science are that 1) results are local to the setting of the experiment (i.e., possible lack of generalizability), and 2) certainty of results is limited by the sample size, as indicated through the resultant random error. Below we concentrate on an increasingly common experimental setting where both of these shortcomings play major roles.

The approval of new drugs by regulatory agencies (such as the FDA or EMA) often requires, due to appropriate ethical concerns, comparing new treatments to existing standard treatments, rather than placebo. While a new treatment may be more effective than a standard treatment, and offer an important alternative, standard treatments are typically partially effective. It is therefore often the case that new treatments provide only incremental benefit over existing, partially effective standard treatments, and therefore the demonstration of superiority (or non-inferiority) for a new treatment compared to the standard can require unfeasibly large or long studies. Unfortunately, this may lead to delays in development and approval of new treatments that provide incremental benefit over an existing partially effective standard treatment. Consider the following example.

We have a new treatment for HIV infection (e.g., three-drug antiretroviral therapy, or triple-therapy). We would like to demonstrate the effectiveness of this new treatment in a local setting. There is an existing standard treatment to prevent progression of HIV infection (e.g., two-drug antiretroviral therapy, or dual-therapy). The existing standard was tested, by experiment, against a placebo in a setting that is distal in geographical space. In our example, the distal setting differs from the local setting in at least three respects. First, the case mix of patients indicated for HIV treatment differs between the local and distal settings. Second, the adherence of HIV treatments is expected to differ between local and distal settings. Third, the risk of progressing to AIDS or death is lower in the local setting compared to the distal setting.

Ethical constraints preclude comparing the new treatment to placebo given the established effectiveness of a standard treatment. Additionally, the combination of a lower risk of HIV progression in the local setting with a (typically expected) small incremental improvement of the new treatment makes a simple direct randomized comparison of the new versus standard treatment infeasible in the local setting, as the study would require a prohibitively large sample size¹. Considering these difficulties, a possible solution is to combine, or “fuse,” information from a randomized comparison of the new versus standard treatment in the local setting with existing information from a randomized comparison of the standard versus placebo treatment from the distal setting². Here, fusion means combining information from multiple settings while accounting for important structural differences between the settings. Such a fusion of information must account for underlying causal structures that can induce differences in case mix, adherence, and HIV progression between the local and distal settings³.

In prior work, Rudolph et al. described an estimation approach that adapts a treatment comparison from a distal to the local setting while accounting for differences between settings⁴. This procedure, in which the study population is not a subset of the target population, is often referred to as “transporting” the effect estimate⁵. That prior approach required having information on both arms of the comparison of interest available from the distal setting, and only used information on case mix and adherence from the local setting. Often, however, the treatment comparison available from the distal setting is not of direct interest, but it might be combined with additional sources of information from the local setting to estimate a different contrast involving a newly available treatment. Such an approach can allow for estimation of the effect of the new treatment compared to placebo among patients in the local setting. Here we describe an approach to obtaining evidence regarding a new treatment by fusing a chain of information from local and distal randomized studies. Through simulation, we explore several properties and operating characteristics of the proposed approach. Finally, we apply the approach to estimate the effect of triple- versus monotherapy for HIV using data from two randomized experiments conducted by the AIDS Clinical Trials Group (ACTG).

METHODS

Notation and the Parameter of Interest

Assume the study consists of n subjects. Let T be the time from randomization to HIV progression, D be the time from randomization to drop out, A be the time from randomization to first protocol deviation, τ be the length of the study period, and T*=min⁡(T,D,A,τ). If no protocol deviation occurs prior to the outcome or τ it is assumed that treatment ends and, using notation similar to Tsiatis⁶, A = ∞. Let W denote setting, with W = d denoting distal and W = l denoting local. Let R = 0 denote randomization to placebo, R = 1 standard treatment, and R = 2 new treatment. Collect measured covariates, other than W, into a vector denoted by X(t). Drop out, protocol deviations, HIV progression, and data collection occur at discrete points in time, t ∈ {0,1,…,τ}. Let δ_T(t) = 1 if an individual has the outcome at time t (e.g. T = t), δ_A(t) = 1 if an individual has a first protocol deviation at time t (e.g. A = t), and δ_D(t) = 1 if an individual drops out at time t (e.g. D = t). Finally, let T^r,a denote the time at which HIV progression would occur had, possibly counter to fact, the patient received treatment R = r and deviated from the protocol at time A = a. We denote the history of a random variable through t with an overbar, e.g. $\overline{X}\left(t\right)=\left\{X\left(0\right),X\left(1\right),\dots X\left(t\right)\right\}$, and we denote $\overline{X}\left(\tau \right)=\overline{X}$. The observed data consist of independent and identically distributed samples of $\left\{R,W,T^{*},\overline{X},{\overline{\delta }}_T,{\overline{\delta }}_A,{\overline{\delta }}_D\right\}$. Note that if W = l then P(R = 0) = 0, and if W = d then P(R = 2) = 0.

The parameter of interest is a difference, at the end of the study period τ, between the probability of HIV progression in the local setting under the protocol for new treatment P(T^2,∞≤τ|W = l), compared to the protocol for placebo P(T^0,∞≤τ|W = l).

Given the information from pending and existing randomized studies described above, respectively, along with identification conditions detailed below, first, the identified intent-to-treat parameters are

$$P\left(T^{2,A}\le \tau |W=l\right)-P\left(T^{1,A}\le \tau |W=l\right)$$ eq. 1

and

$$P\left(T^{1,A}\le \tau |W=d\right)-P\left(T^{0,A}\le \tau |W=d\right)$$ eq. 2

Note that here, T^r,A is the potential outcome under treatment level r and the observed time of protocol deviation. The first of these compares new treatment to standard treatment under observed adherence in the local setting, and the second compares standard treatment to placebo in the distal setting.

Second, we identify the per-protocol parameters

$$P\left(T^{2,\mathrm{\infty }}\le \tau |W=l\right)-P\left(T^{1,\mathrm{\infty }}\le \tau |W=l\right)$$ eq. 3

and

$$P\left(T^{1,\mathrm{\infty }}\le \tau |W=d\right)-P\left(T^{0,\mathrm{\infty }}\le \tau |W=d\right)$$ eq. 4

using measured information on protocol deviation times A to censor subjects when they first deviate from the protocol, and information on covariate history $\overline{X}$ to account for possible induced informative censoring due to protocol deviation⁷. Here, the per-protocol effect means the effect had all individuals followed their assigned treatment protocol⁸.

Third, because of possible differences in the populations in the local and distal environments that can impact risk, it is necessary to compare all treatments in the same population. We therefore map eq. 4, which is defined among the population in the distal environment (W = d), to the local environment (W = l) using methods for generalizability (or transportability). Essentially, this amounts to a change in the population over which the parameter is defined. This mapping is achieved through the use of inverse odds of sampling weighting⁵. We thus obtain the parameter

$$P\left(T^{1,\mathrm{\infty }}\le \tau |W=l\right)-P\left(T^{0,\mathrm{\infty }}\le \tau |W=l\right)$$ eq. 5

Fourth, we note that eq. 3 and eq. 5, both contain the parameter P(T^1,∞≤τ|W = l), through which, by transitivity, we have identified a data “fusion” parameter

$$\mathrm{\psi }=P\left(T^{2,\mathrm{\infty }}<\tau |W=l\right)-P\left(T^{0,\mathrm{\infty }}<\tau |W=l\right)$$ eq. 6

which is exactly the parameter of interest originally stated – a per-protocol comparison between the new treatment and the placebo in the local setting.

Identification Conditions

Identification of eq. 1 and eq. 2, the intent-to-treat parameters, is given by randomization. Identification of eq. 3 and eq. 4, the per-protocol parameters, can be achieved in two ways. First, one may conduct two separate trials, each involving subjects randomly sampled from the same target population, in which individuals are randomized to the new treatment vs. standard of care (in one trial) or standard of care vs. placebo (in the other trial). In each trial, individuals would be closely followed and all drop-out and protocol deviation would be prevented. Second, one may identify the per-protocol parameters by assuming a pair of exchangeability and positivity assumptions^7,9 and using data from two trials conducted in different populations with possible drop-out and protocol deviation. Specifically, for eq. 3 and eq. 4, we require sequential ignorability of protocol deviations and dropout conditional on measured covariates, i.e.,

$${\delta }_Z\left(t\right)⫫T^{r,\mathrm{\infty }}|\overline{X}\left(t\right),R,W,{\overline{\delta }}_A\left(t-1\right)={\overline{\delta }}_D\left(t-1\right)={\overline{\delta }}_T\left(t-1\right)=0,\forall t\in \left\{\mathrm{0,1},\dots ,\tau \right\},Z\in \{A,D\}$$ eq. 7

and associated positivity conditions. In words, at each time, dropout and protocol deviations must be independent of the potential time to event conditional on covariate history, treatment, and setting, among those who have not yet experienced the outcome, dropped out, or had a protocol deviation by that time. Identification of eq. 5 is given by a pair of exchangeability and positivity assumptions for generalizability^10–12. Specifically, we require ignorability of “selection” to the local setting conditional on measured baseline covariates, i.e.,

$$W⫫T^{r,\mathrm{\infty }}|X\left(0\right)$$ eq. 8

and an associated positivity condition¹⁰. In words, setting is independent of potential time to event conditional on baseline covariates. Finally, we assume independence between protocol deviations and drop out.

Fusion Estimator

Given the identification conditions above, the risk of the outcome in the local setting under treatment r and full adherence can be rewritten as

$$P\left(T^{r,\mathrm{\infty }}<\tau |W=l\right)=E{\left[\frac{I(W=d){\pi }_W\left(l\right)}{{\pi }_W\left(d\right)}\right]}^{-1}E\left[\frac{I\left(W=d\right)I\left(R=r\right)I\left(T^{\mathrm{*}}\le \tau \right){\delta }_T\left(T^{\mathrm{*}}\right)}{{\pi }_A\left(T^{\mathrm{*}}\right){\pi }_D\left(T^{\mathrm{*}}\right){\pi }_R}\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(d\right)}\right],$$

where the nuisance functions, indexed by parameters α, β, and γ, are π_W(w) = P(W= w|X(0), R; α), protocol deviations ${\pi }_A\left(t\right)=P\left[A>t|W,\overline{X},R;\beta \right]$, and drop out ${\pi }_D\left(t\right)=P\left[D>t|W,\overline{X},R;\gamma \right]$. The parameters of the nuisance functions are summarized as θ= (α, β, γ). We assume known randomization probability π_R = P(R|W). This is equivalence is demonstrated in discrete time as follows:

Beginning with the second expectation, we note that the indicator function I(T*<τ) can be rewritten as a sum of indicator functions, with at most one of those indicator functions taking value 1,

$$\begin{gathered} E\left[\frac{I\left(W=d\right)I\left(R=r\right)I\left(T^{\mathrm{*}}\le \tau \right){\delta }_T\left(T^{\mathrm{*}}\right)}{{\pi }_A\left(T^{\mathrm{*}}\right){\pi }_D\left(T^{\mathrm{*}}\right){\pi }_R}\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(d\right)}\right]= \\ E\left[\frac{I\left(W=d\right)I\left(R=r\right)\left[{\sum }_{t=1}^{\tau }I\left(T^{\mathrm{*}}=t\right)\right]{\delta }_T\left(T^{\mathrm{*}}\right)}{{\pi }_A\left(T^{\mathrm{*}}\right){\pi }_D\left(T^{\mathrm{*}}\right){\pi }_R}\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(d\right)}\right]= \\ {\sum }_{t=1}^{\tau }E\left[\frac{I\left(W=d\right)I\left(R=r\right)I(T^{\mathrm{*}}=t){\delta }_T\left(T^{\mathrm{*}}\right)}{{\pi }_A\left(t\right){\pi }_D\left(t\right){\pi }_R}\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(d\right)}\right] \end{gathered}$$

Focusing on each element of the sum, we note that I(T*=t)δ_T(T*)=I(T=t)I(D>t)I(A>t), so

$$E\left[\frac{I\left(W=d\right)I\left(R=r\right)I\left(T^{\mathrm{*}}=t\right){\delta }_T\left(T^{\mathrm{*}}\right)}{{\pi }_A\left(t\right){\pi }_D\left(t\right){\pi }_R}\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(W\right)}\right]=$$

$$E\left[\frac{I\left(W=d\right)I\left(R=r\right)I(T=t)I\left(D>t\right)I(A>t)}{{\pi }_A\left(t\right){\pi }_D\left(t\right){\pi }_R}\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(W\right)}\right].$$

Then by causal consistency we can replace T with the potential outcome T^r,∞, so the previous quantity equals

$$E\left[\frac{I\left(W=d\right)I\left(R=r\right)I\left(T^{r,\mathrm{\infty }}=t\right)I\left(D>t\right)I\left(A>t\right)}{{\pi }_A\left(t\right){\pi }_D\left(t\right){\pi }_R}\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(W\right)}\right],$$

which by randomization equals

$$E\left[\frac{I\left(W=d\right)I\left(T^{r,\mathrm{\infty }}=t\right)I\left(D>t\right)I\left(A>t\right)}{{\pi }_A\left(t\right){\pi }_D\left(t\right)}\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(W\right)}\right].$$

Using the law of total expectation

$$\begin{gathered} E\left[\frac{I\left(W=d\right)I\left(T^{r,\mathrm{\infty }}=t\right)I\left(D>t\right)I\left(A>t\right)}{{\pi }_A\left(t\right){\pi }_D\left(t\right)}\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(W\right)}\right]= \\ E\left[E\left[\frac{I\left(W=d\right)I\left(T^{r,\mathrm{\infty }}=t\right)I\left(D>t\right)I\left(A>t\right)}{{\pi }_A\left(t\right){\pi }_D\left(t\right)}\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(W\right)}|R,\overline{X}\left(t-1\right),{\overline{\delta }}_D\left(t-1\right),{\overline{\delta }}_A\left(t-1\right),{\overline{\delta }}_T\left(t-1\right),W\right]\right]. \end{gathered}$$

Next, we note that both I(D>t) and I(A>t) can be replaced by products of indicator terms as follows,

$$E\left[E\left[\frac{I\left(W=d\right)I\left(T^{r,\mathrm{\infty }}=t\right){\prod }_{k=0}^{t}I({\delta }_D\left(k\right)=0){\prod }_{k=0}^{t}I\left({\delta }_A\left(k\right)=0\right)}{{\pi }_A\left(t\right){\pi }_D\left(t\right)}\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(W\right)}|R,\overline{X}\left(t-1\right),{\overline{\delta }}_D\left(t-1\right),{\overline{\delta }}_A\left(t-1\right),{\overline{\delta }}_T\left(t-1\right),W\right]\right].$$

With sequential ignorability, this latter quantity equals

$$E\left[E\left[\frac{I\left(W=d\right)I\left(T^{r,\mathrm{\infty }}=t\right){\prod }_{k=0}^{t-1}I({\delta }_D\left(k\right)=0){\prod }_{k=0}^{t-1}I\left({\delta }_A\left(k\right)=0\right)}{{\pi }_A\left(t-1\right){\pi }_D\left(t-1\right)}\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(W\right)}|R,\overline{X}\left(t-1\right),{\overline{\delta }}_D\left(t-1\right),{\overline{\delta }}_A\left(t-1\right),{\overline{\delta }}_T\left(t-1\right),W\right]\right]=$$

$$E\left[\frac{I\left(W=d\right)I\left(T^{r,\mathrm{\infty }}=t\right)I\left(D>t-1\right)I\left(A>t-1\right)}{{\pi }_A(t-1){\pi }_D(t-1)}\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(W\right)}\right].$$

Iterating these steps t−1 more times gives

$$E\left[\frac{I\left(W=d\right)I\left(R=r\right)I\left(T^{\mathrm{*}}=t\right){\delta }_T\left(T^{\mathrm{*}}\right)}{{\pi }_A\left(t\right){\pi }_D\left(t\right){\pi }_R}\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(W\right)}\right]=$$

$$E\left[I\left(W=d\right)I\left(T^{r,\mathrm{\infty }}=t\right)\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(W\right)}\right].$$

Next, again using the law of total expectation, we note that

$$\begin{gathered} E\left[I\left(W=d\right)I\left(T^{r,\mathrm{\infty }}=t\right)\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(d\right)}\right]= \\ E\left[E\left[I\left(W=d\right)I\left(T^{r,\mathrm{\infty }}=t\right)\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(d\right)}|X\left(O\right)\right]\right]. \end{gathered}$$

From ignorability of region, the above equals

$$E\left[({\pi }_W\left(l\right)I\left(T^{r,\mathrm{\infty }}=t\right)\right],$$

Which by an application of Bayes’ theorem equals

$$E\left[\frac{P\left(\left(X\left(0\right)|W=l\right)\right)P(W=l)}{P\left(X\left(0\right)\right)}I\left(T^{r,\mathrm{\infty }}=t\right)\right],$$

which by the law of total expectation and region ignorability equals

$$E\left[\frac{P\left(\left(X\left(0\right)|W=l\right)\right)P(W=l)}{P\left(X\left(0\right)\right)}E\left[I\left(T^{r,\mathrm{\infty }}=t\right)|X\left(0\right),W=l\right]\right].$$

By the definition of the expectation of an indicator, the three terms in the numerator of this latter quantity can be combined as

$$E\left[\frac{P\left(X\left(0\right),T^{r,\mathrm{\infty }}=t,W=1\right)}{P\left(X\left(0\right)\right)}\right].$$

By definition, this latter quantity is

$$E\left[P\left(T^{r,\mathrm{\infty }}=t,W=l|X\left(0\right)\right)\right],$$

Which is equal to

$$P\left(W=l\right)P(T^{r,\mathrm{\infty }}=t|W=l).$$

Therefore,

$$\begin{gathered} {\sum }_{t=1}^{\tau }E\left[\frac{I\left(W=d\right)I\left(R=r\right)I(T^{\mathrm{*}}=t)\delta }{{\pi }_A\left(T^{\mathrm{*}}\right){\pi }_D\left(T^{\mathrm{*}}\right){\pi }_R}\times \frac{{\pi }_W\left(l\right)}{{\pi }_W\left(d\right)}\right]= \\ {\sum }_{t=1}^{\tau }P\left(W=l\right)P(T^{R=r,A=\mathrm{\infty }}=t|W=l)= \\ P\left(W=l\right)P(T^{R=r,A=\mathrm{\infty }}\le \tau |W=l). \end{gathered}$$

Finally, note that

$$E{\left[\frac{I(W=d){\pi }_W\left(l\right)}{{\pi }_W\left(d\right)}\right]}^{-1}=E{\left[{\pi }_W\left(l\right)\right]}^{-1}=P{\left(W=l\right)}^{-1},$$

So

$$E{\left[\frac{I\left(W=d\right){\pi }_W\left(l\right)}{{\pi }_W\left(d\right)}\right]}^{-1}P\left(W=l\right)P\left(T^{R=r,A=\mathrm{\infty }}\le \tau |W=l\right)=P\left(T^{R=r,A=\mathrm{\infty }}\le \tau |W=l\right).$$

A fusion estimator combines, or fuses, data sources. Specifically, a per-protocol fusion estimator for the risk of the outcome under treatment r and full adherence in the local setting is obtained by replacing expectations with sample averages as

$${\widehat{P}}_{fusion}\left(T^{r,\mathrm{\infty }}<\tau |W=l;\widehat{\theta }\right)={\left({\sum }_{i=1}^n\frac{I(W_i=d){\widehat{\pi }}_{W,i}\left(l\right)}{{\widehat{\pi }}_{W,i}\left(d\right)}\right)}^{-1}{\sum }_{i=1}^n\frac{I\left(W_i=d\right)I\left(R_i=r\right)I\left(T_i^{\mathrm{*}}\le \tau \right){\delta }_T\left(T^{\mathrm{*}}\right)}{{\widehat{\pi }}_{A,i}\left(T_i^{\mathrm{*}}\right){\widehat{\pi }}_{D,i}\left(T_i^{\mathrm{*}}\right){\pi }_{R,i}}\times \frac{{\widehat{\pi }}_{W,i}\left(l\right)}{{\widehat{\pi }}_{W,i}\left(d\right)},$$

with $\widehat{\theta }=\left(\widehat{\alpha },\widehat{\beta },\widehat{\gamma }\right)$ fit by maximum likelihood. For instance, α can be estimated with logistic regression and β and γ can be estimated using proportional hazards models with the Breslow estimator of the baseline hazard function¹³. Note that the fusion occurs in the estimation of α, as this parameter is used to standardize information from the distal setting to the local setting. Similarly, the local estimator for the risk of the outcome under treatment r and full adherence in the local setting is

$${\widehat{P}}_{local}\left(T^{r,\mathrm{\infty }}<\tau |W=l;\widehat{\theta }\right)={\left({\sum }_{i=1}^{n}I(W_i=l)\right)}^{-1}{\sum }_{i=1}^n\frac{I\left(W_i=l\right)I\left(R_i=r\right)I\left(T_i^{\mathrm{*}}\le \tau \right){\delta }_T\left(T^{\mathrm{*}}\right)}{{\widehat{\pi }}_{A,i}\left(T_i^{\mathrm{*}}\right){\widehat{\pi }}_{D,i}\left(T_i^{\mathrm{*}}\right){\pi }_{R,i}}.$$

Then the estimator for the parameter of interest is estimated with

$$\widehat{\psi }=\left({\widehat{P}}_{local}\left(T^{2,\mathrm{\infty }}<\tau |W=l;\widehat{\theta }\right)-{\widehat{P}}_{local}\left(T^{1,\mathrm{\infty }}<\tau |W=l;\widehat{\theta }\right)\right)+\left({\widehat{P}}_{fusion}\left(T^{1,\mathrm{\infty }}<\tau |W=l;\widehat{\theta }\right)-{\widehat{P}}_{fusion}\left(T^{0,\mathrm{\infty }}<\tau |W=l;\widehat{\theta }\right)\right).$$

Standard errors can be estimated using an influence function-based estimator of the form

$$SE\left(\widehat{\psi }\right)={\left[{\left({\sum }_{i=1}^{n}I\left(W_i=l\right)\right)}^{-2}{{\sum }_{i=1}^n\left(\left[\left(\frac{I\left(W_i=l\right)I\left(R_i=2\right)I\left(T_i^{\mathrm{*}}\le \tau \right){\delta }_T\left(T^{\mathrm{*}}\right)}{{\widehat{\pi }}_{A,i}\left(T_i^{\mathrm{*}}\right){\widehat{\pi }}_{D,i}\left(T_i^{\mathrm{*}}\right){\pi }_{R,i}}-\frac{I\left(W_i=l\right)I\left(R_i=1\right)I\left(T_i^{\mathrm{*}}\le \tau \right){\delta }_T\left(T^{\mathrm{*}}\right)}{{\widehat{\pi }}_{A,i}\left(T_i^{\mathrm{*}}\right){\widehat{\pi }}_{D,i}\left(T_i^{\mathrm{*}}\right){\pi }_{R,i}}\right)+\left(\frac{I\left(W_i=d\right)I\left(R_i=1\right)I\left(T_i^{\mathrm{*}}\le \tau \right){\delta }_T\left(T^{\mathrm{*}}\right)}{{\widehat{\pi }}_{A,i}\left(T_i^{\mathrm{*}}\right){\widehat{\pi }}_{D,i}\left(T_i^{\mathrm{*}}\right){\pi }_{R,i}}\times \frac{{\widehat{\pi }}_{W,i}\left(l\right)}{{\widehat{\pi }}_{W,i}\left(d\right)}-\frac{I\left(W_i=d\right)I\left(R_i=0\right)I\left(T_i^{\mathrm{*}}\le \tau \right){\delta }_T\left(T^{\mathrm{*}}\right)}{{\widehat{\pi }}_{A,i}\left(T_i^{\mathrm{*}}\right){\widehat{\pi }}_{D,i}\left(T_i^{\mathrm{*}}\right){\pi }_{R,i}}\times \frac{{\widehat{\pi }}_{W,i}\left(l\right)}{{\widehat{\pi }}_{W,i}\left(d\right)}\right)\right]-\left[\left({\widehat{P}}_{local}\left(T^{R=2,A=\mathrm{\infty }}<\tau |W=l\right)-{\widehat{P}}_{local}\left(T^{R=1,A=\mathrm{\infty }}<\tau |W=l\right)\right)+\left({\widehat{P}}_{fusion}\left(T^{R=1,W=\mathrm{\infty }}<\tau |W=l\right)-{\widehat{P}}_{fusion}\left(T^{R=0,W=\mathrm{\infty }}<\tau |W=l\right)\right)\right]\right)}^2\right]}^{\frac{1}{2}}.$$

Simulation Design and Analysis

To explore the operating characteristics of the proposed fusion estimator, we designed a simulation to estimate the effect of a new drug compared with placebo. The available information comes from a randomized trial in a distal setting (W = d) comparing standard of care (R = 1) to placebo (R = 0) and a separate randomized trial in a local setting (W = l) comparing a new treatment (R = 2) to standard of care (R = 1).

Treatment was randomized in each setting with equal numbers of subjects allocated to each treatment group. Participants in each setting differ by a single baseline binary covariate, X₁, with those in the distal setting having a higher probability of X₁=1. Treatment is received and outcomes are measured at two time points, t ∈ {1,2}. At each time, a patient may drop out (δ_D(t)=1) or deviate from the medication protocol (δ_A(t)=1). Dropout is completely at random, with P(δ_D(t)=1|δ_D(t−1)=0)=0.1 at each time.

We simulated two different scenarios. In Scenario A, there is no protocol deviation and there is no difference in baseline risk of the outcome between the two settings. In scenario B, the distribution of protocol deviation is a function of treatment, setting, and a time-varying binary covariate X₂. Those in the local setting are more likely to follow the protocol than those in the distal setting, those receiving placebo are more likely to follow protocol than those receiving active treatment, and those receiving the new treatment are more likely to follow protocol than those receiving standard of care. Those with the time-varying covariate X₂=1 at a given time point are more likely to follow protocol than those with X₂=0.

In scenario B, at each time point, the distribution of the outcome is a function of treatment, X₁, and X₂. Those receiving placebo are at the highest outcome risk, and those receiving the new treatment are at the lowest risk.X₁=0 and X₂=1 are associated with a lower risk of the outcome, so individuals in the local setting and individuals who follow the protocol have lower risk of the outcome. The specific distributions of the variables are detailed in the appendix. Using these distributions, in scenario A, the marginal incidences of the outcome in the local and distal settings under placebo treatment were both 33%. In scenario B, the local and distal setting incidences under placebo treatment were 7.7% and 20.4%, respectively.

We simulated sample sizes of 100 and 1000 in the local and distal setting for each scenario. In each scenario, the true effect was determined by averaging the actual values of the simulated potential outcomes in a large simulation of n=100,000. In each simulation, we compared two estimators: the fusion estimator and a “naïve” estimator that does not include any covariates in the nuisance models and ignores protocol deviations (in this case, the estimator is equivalent to Kaplan-Meier estimators that right censor individuals at the time of dropout¹⁴). Each simulation involved 4000 simulated datasets, and bias, average standard error, Monte Carlo standard error, and root mean squared error were compared.

SIMULATION RESULTS

As seen in Table 1 and the Figure, the proposed fusion estimator appears unbiased across both scenarios A and B, and across all sample sizes. With sample sizes of 1000 in both the local and distal settings, the bias of the fusion estimator was 0.009 in scenario A and 0.001 in scenario B, both substantially smaller than the standard error of the estimates. The naïve estimator is unbiased only in scenario A, where sampling and drop out are completely at random and there are no protocol deviations. With the same sample sizes as above, the bias of the naïve estimator was 0.008 in scenario A and −0.072 in scenario B. The bias of the naïve estimator in scenario B was more than twice the magnitude of the standard error of the estimate. Root MSE is also always better for the proposed estimator in scenario B where sampling is differential and protocol deviations occur, but root MSE is lower for the naïve estimator in scenario A, where the simpler naïve estimator is consistent. With 1000 subjects in each setting in scenario A, the root MSEs were 0.05 and 0.04 for the fusion and naïve estimators, respectively. However, in scenario B, the root MSEs were 0.02 and 0.08 for the fusion and naïve estimators, respectively.

TABLE 1:

Bias and root MSE for the naïve and fusion estimators, 4000 simulation trials

	N		Bias		Root Mean Squared Error
Scenario	Local	Distal	Fusion	Naïve	Fusion	Naïve
A	100	100	0.003	0.004	0.156	0.113
A	100	500	0.005	0.006	0.091	0.078
A	100	1000	0.007	0.008	0.079	0.073
A	500	100	0.008	0.008	0.142	0.097
A	500	500	0.007	0.007	0.069	0.051
A	500	1000	0.005	0.005	0.053	0.041
A	1000	100	0.009	0.008	0.136	0.090
A	1000	500	0.007	0.007	0.065	0.045
A	1000	1000	0.009	0.008	0.049	0.036
B	100	100	0.002	−0.069	0.056	0.108
B	100	500	0.001	−0.071	0.044	0.085
B	100	1000	0.003	−0.070	0.041	0.081
B	500	100	0.003	−0.069	0.042	0.103
B	500	500	0.001	−0.072	0.024	0.080
B	500	1000	0.001	−0.072	0.021	0.077
B	1000	100	0.001	−0.073	0.040	0.106
B	1000	500	0.001	−0.072	0.021	0.079
B	1000	1000	0.001	−0.072	0.018	0.077

Figure 1: Results of 4000 simulations of scenarios A and B.

The solid lines represent the true treatment effect.

As shown in Table 2, the estimated SE for the proposed estimator appears to well approximates the Monte Carlo simulation standard error in scenario B, but is slightly greater than the Monte Carlo simulation standard error in scenario A. The maximum ratio of the estimated SE to the Monte Carlo SE for the fusion estimator was 1.40, which occurred with 100 individuals in the local setting and 1000 in the distal setting for scenario A. In all other scenarios this ratio was less than 1.2. The SE shrinks as a function of both the local and distal sample sizes, but it is more strongly related to the distal sample size in both scenarios. For the fusion estimator in scenario B, sample sizes of 100 in the low-incidence local and 1000 in the high-incidence distal setting yielded a SE of 0.043, while sample sizes of 1000 in the low-incidence local and 100 in the high-incidence distal yielded a SE of 0.039. In scenario A, in which both estimators are unbiased, the SE for the naïve estimator is always smaller than the fusion estimator.

TABLE 2:

Average and Monte Carlo standard errors for the naive and fusion estimators, 4000 simulation trials

	N		Average Standard Errora		Monte Carlo Standard Errorb
Scenario	Local	Distal	Fusion	Naïve	Fusion	Naïve
A	100	100	0.163	0.123	0.156	0.113
A	100	500	0.108	0.096	0.091	0.078
A	100	1000	0.110	0.105	0.079	0.073
A	500	100	0.149	0.105	0.141	0.096
A	500	500	0.073	0.055	0.068	0.050
A	500	1000	0.057	0.046	0.053	0.041
A	1000	100	0.147	0.103	0.136	0.090
A	1000	500	0.069	0.050	0.065	0.044
A	1000	1000	0.051	0.039	0.048	0.035
B	100	100	0.054	0.087	0.056	0.082
B	100	500	0.043	0.056	0.044	0.047
B	100	1000	0.043	0.057	0.041	0.040
B	500	100	0.041	0.081	0.042	0.076
B	500	500	0.025	0.039	0.024	0.037
B	500	1000	0.022	0.030	0.021	0.028
B	1000	100	0.039	0.080	0.040	0.077
B	1000	500	0.021	0.037	0.021	0.034
B	1000	1000	0.018	0.028	0.018	0.026

^aAverage standard error is defined as the average of the standard errors across Monte Carlo simulations, estimated with the standard error estimator described in the text.

^bMonte Carlo standard error is defined as the standard deviation of the estimates across Monte Carlo simulations.

APPLICATION

To demonstrate the use of the proposed approach, we applied it to estimate the effect of triple- vs. monotherapy for HIV treatment. The data came from two randomized experiments conducted by the AIDS Clinical Trials Group (ACTG): ACTG 175¹⁵, which compared mono- with dual-therapy for HIV, and ACTG 320¹⁶, which compared dual- with triple-therapy for HIV. Specifically, we included subjects who were randomized to either Zidovudine (monotherapy), Zidovudine + Didanosine (dual-therapy), or Zidovudine + Zalcitabine (dual-therapy) from ACTG 175, and subjects who were randomized to Zidovudine + Lamivudine (dual-therapy) or Indinavir + Zidovudine + Lamivudine (triple therapy) from ACTG 320. We note that these trials had non-overlapping CD4+ cell count inclusion criteria, and therefore the results presented here should be not be considered meaningful clinical effects, but rather a demonstration of the proposed analytic approach. The outcome of interest was the 1-year risk of AIDS, death, or a 50% drop in CD4+ cell count.

A total of 821 subjects (33% monotherapy, 67% dual-therapy) from ACTG 175 and 1156 subjects (50% dual-therapy, 50% triple-therapy) from ACTG 320 were included. A sample of the baseline characteristics of the subjects, stratified by study, are presented in Table 3. Of note, subjects in ACTG 320 were slightly older, more likely to be of black race, more likely to inject drugs at baseline, and had a lower Karnofsky score at baseline.

TABLE 3:

Select characteristics of subjects from AIDS Clinical Trials Group studies 175 and 320¹

	ACTG 175		ACTG 320
	Monotherapy N = 271	Dual-therapy N = 550	Dual-therapy N = 579	Triple-therapy N = 577
Male Sex	221 (81.5)	448 (81.5)	485 (83.8)	471 (81.6)
Age (Median, IQR)	35 (30, 41)	35 (30, 41)	38 (33, 44)	38 (33, 44)
Black race	70 (25.8)	138 (25.1)	165 (28.5)	163 (28.2)
Injection drug use	33 (12.2)	79 (14.4)	93 (16.1)	91 (15.8)
Karnofsky Score (Mean, SD)	95.0 (6.3)	95.4 (6.0)	91.4 (7.7)	91.2 (7.7)

¹All numbers are N (%) unless otherwise noted

Because data on adherence were not available, we did not consider non-adherence in our estimates, and instead used the proposed analytic approach to estimate the intention-to-treat effect. There remained two nuisance models to be fit – a model for the probability of being in ACTG 320 v. ACTG 175 and a model for the probability of remaining under follow-up. The former model was specified as a logistic regression adjusted for male sex, black race, injection drug use, age (modeled with B-splines with 3 knots), and Karnofsky score (modeled with B-splines with 3 knots). The latter model was specified as a Cox proportional hazards model with a Breslow estimator¹³ of the baseline hazard, adjusted for the same variables as the previous model, but additionally stratified by treatment group. The model for dropout was fit separately for each study. For comparison, we also fit each model without adjusting for covariates in order to estimate the ‘crude’ effect of triple- versus monotherapy.

Using the proposed estimator, the crude effect was triple- versus monotherapy on the 1-year risk of AIDS, death, or a 50% drop in CD4+ cell count was a risk difference of −0.17 (95% CI −0.23, −0.11). Adjusting for male sex, black race, injection drug use, age, and Karnofsky score in each model yielded a risk difference of −0.20 (−0.29, −0.10). Of note, the standard error of the crude risk difference, 0.033, was smaller than standard error of the adjusted risk difference, 0.048, thus demonstrating the “cost” associated with using the proposed approach.

DISCUSSION

It is possible to fuse evidence from multiple sources to estimate unbiased treatment contrasts that would not be possible using information from either source in isolation. The estimator we propose here uses information from randomized trials in local and distal settings to compare a new treatment to a placebo, even though such a comparison would be unethical in any single trial. Our proposed estimator is quite general, and can accommodate time-varying treatments and covariates, time to event outcomes, informative dropout and nonadherence, and differences in patient mix and disease risk between the local and distal setting. Moreover, the general approach of fusion estimators and designs (combining sources of information) is widely applicable beyond the motivating example described here. Through simulation, we have found that, given the identification conditions we described, the fusion estimator appears unbiased in the settings we explored, and the proposed estimator for the standard error appears mildly conservative. Notably, for the set of scenarios explored here, the standard error of the fusion estimator was more sensitive to the sample size and number of events in the distal setting compared with the local setting, suggesting that, if data collection costs were equal, committing resources to the higher incidence distal setting may be preferred.

There is a ‘cost’, however, to using the fusion estimator. In particular, in scenarios where the fusion estimator is not necessary for unbiased effect estimation, such as our simulation scenario A, the standard error of the fusion estimator is larger than the standard error of the naïve estimator. The confidence afforded by the smaller standard error may be misplaced, however, as it is generally not possible to determine from the data alone when the naïve estimator will be unbiased. The loss of precision of the fusion estimator is thus the cost of insurance to avoid bias when fusing data sources.

As demonstrated by our applied example, implementation of the fusion estimator is straightforward and can be achieved by using many standard procedures from readily available software. In our example, the estimator required fitting logistic regressions and Cox proportional hazard models, making predictions from the models, and combining the predictions to form the estimator. In this example, the estimate using the proposed approach was slightly larger in magnitude than the crude estimate, albeit with a wider confidence interval. Additionally, the risk difference of −0.20 is notably larger than the risk differences reported originally by each trial, where the effects of incremental improvements to treatment were estimated. By fusing results across trials, we were able to estimate a parameter that would not have been feasible using data from any available individual trial. Notably, a trial comparing triple- to monotherapy would not have been ethical to conduct due to knowledge that monotherapy is inferior to dual-therapy. The fusion approach therefore allowed for the estimation of a parameter that could not have been estimated easily otherwise.

We note that the conditions under which we identified the parameter of interest are sufficient but not necessary. Specifically, our derivation relied on conditional exchangeability with respect to the outcome across study locations. Essentially, this means that the risk of the outcome is the same in each location within levels of covariates. This is a stronger condition than is needed, as it may be possible to identify the parameter of interest by only requiring the effect measure of interest, rather than the risk itself, be the same in each location within levels of covariates. Identification under this weaker condition may allow the proposed approach to be used in a wider set of scenarios, but it should be noted that it also requires parametric model form knowledge that may not be available in all settings.

This work builds extensively on existing literature related to generalizing, transporting, and fusing effect estimates. We often refer to generalizing causal effects when the study population is a subset of the target population. Prior work by Lesko et al. described sufficient conditions estimators, expressed in terms of potential outcomes, for generalizing study results to a target population¹⁰. When the study population is not a subset of the target population, we often refer to “transporting” effects from one setting to another. In prior work by Rudolph and van der Laan¹⁷, conditions, identification results, and estimators (including semiparametric efficient, doubly robust estimators) for transporting effects are described, however these results are limited to single time point treatment and outcomes and do not account for dropout or non-adherence. Rudolph et al. present additional results that also allow for differences in non-adherence between the study and target populations⁴. When both generalizing and transporting results, no experimental evidence from the local setting is used. An alternative to the potential outcomes framework used here is the graphical framework for transporting and fusing information established by Pearl and Bareinboim¹⁸², however, the literature on these graphical approaches has not yet included estimators for data fusion parameters or demonstrated their operating characteristics through simulation.

Our proposed approach is subject to limitations. First and foremost, the identification results rely on a set of untestable conditions described in equations 7 and 8. Though graphical approaches have been developed to aid in choosing sufficient adjustment sets and establishing identifiability¹⁸, the validity of the underlying causal structure is not testable given the data typically observed. Second, the proposed estimator relies on correctly specified models for the nuisance parameters π_A, π_D, π_W. If any of these models are misspecified, then the proposed estimator may be biased. A possible extension would be the development of multiply robust estimators that use additional nuisance parameters and are consistent if models for certain subsets of those parameters are correctly specified. Additionally, these multiply robust estimators achieve fast convergence rates and thus allow for extremely flexible, data-adaptive estimation of the nuisance parameters, enhancing the likelihood of correct model specification. Doubly robust methods have been developed for estimating effects of time-varying treatments on time-to-event outcomes in a study population^19,20 and for transporting the effects of single time point treatments on single time point outcomes¹⁷, but to our knowledge no doubly robust methods have been developed for the setting described in the present work. Third, our estimator for the standard error treats the nuisance parameters as known and therefore may deviate from the true standard error of the estimator. Correct standard errors that account for the estimation of the nuisance parameters could be estimated using stacked estimating equations²¹ or using the bootstrap²². Fourth, our estimator can only be used to estimate per-protocol effects, not intention to treat effects. The approach described by Rudolph et al.⁴ that uses an outcome-model based approach could be adapted to estimate such effects. Finally, further work is needed to determine what improvements in statistical power can be achieved by using the fusion approach to compare a new versus placebo treatment over a traditional comparison of a new versus standard treatment in the local setting. The relative power of each approach is likely strongly related to the relative effect sizes of each treatment and the characteristics of the local and distal populations, and thus extensive simulations will be needed to characterize the power of each approach. We anticipate that in many scenarios the fusion approach can be used to overcome issues related to incremental treatment effects that may lead to the need for untenably large studies by allowing new treatments to be compared with a placebo rather than existing standard treatments.

In addition to the scenario we introduced in the introduction, fusion estimators and designs may be widely applicable in settings of diseases that are changing rapidly with respect to case mix and geography, such as COVID-19. Trials for treatments or vaccines may take place in one location, but because of the movement of the disease, newly arising outbreaks, and multiple waves of infections, subsequent trials for other treatments will likely be conducted in new locations where incidence is highest. As new treatments are discovered and become standard, each subsequent trial will need to use the most recent standard identified from the previous location in the control arm. Fusion designs and estimators are well-equipped for such a scenario to allow information to be consistently fused across locations to facilitate the rapid development of new treatments for emerging, fast-moving diseases.

The proposed approach allows for the fusion of information across settings to estimate contrasts of interest that would not be feasible with standard study designs and analyses. In the era of ‘real-world evidence’ and ‘big data,’ such approaches will become increasingly important to speed the approval and adoption of effective new therapies by leveraging multiple sources of information. Additionally, given the high cost of conducting clinical trials, the ability to reuse the information generated and augment it with results from smaller local studies is an attractive approach to assessing treatment effects in populations that may not be included in large studies. Fusion approaches, such as the one presented here, provide an important path towards maximizing the value, accuracy, and impact of clinical research.

Appendix

Let $\mathrm{e}\mathrm{x}\mathrm{p}\mathrm{i}\mathrm{t}\left(x\right)=\frac{{\mathrm{e}}^{\mathrm{x}}}{1+{\mathrm{e}}^{\mathrm{x}}}$ and I(x) be the indicator function. The variable distributions for the simulation are:

Scenario A:

$$P\left(R=0|W=l\right)=0$$

$$P\left(R=1|W=l\right)=0.5$$

$$P\left(R=2|W=l\right)=0.5$$

$$P\left(R=0|W=d\right)=0.5$$

$$P\left(R=1|W=d\right)=0.5$$

$$P\left(R=2|W=d\right)=0$$

$$P\left(X_1=1|W\right)=expit\left(-1+2\mathrm{*}I\left(W=d\right)\right)$$

$$P\left(X_{2,t=1}=1\right)=0.5$$

$$P\left(X_{2,t=2}=1|X_{2,t=1}\right)=expit(0.3+0.3X_{2,t=1})$$

$$P\left(C_{t=1}=1\right)=0.1$$

$$P\left(C_{t=2}=1|A_{t=1}=C_{t=1}=Y_{t=1}=0\right)=0.1$$

$$P\left(A_t=1|W,R,X_{2,t}\right)=0$$

$$P\left(Y_{t=1}^{r,\mathrm{\infty }}|W,X_1,X_{2,t=1}\right)=expit\left(-3+0.5\mathrm{*}I\left(R=1\right)+1.5\mathrm{*}I\left(R=0\right)\right)$$

$$P\left(Y_{t=2}^{r,\mathrm{\infty }}|W,X_1,X_{2,t=2},Y_{t=1}^{r,\mathrm{\infty }}=0\right)=expit\left(-3+0.5\mathrm{*}I\left(R=1\right)+1.5\mathrm{*}I\left(R=0\right)\right)$$

Scenario B:

$$P\left(R=0|W=l\right)=0$$

$$P\left(R=1|W=l\right)=0.5$$

$$P\left(R=2|W=l\right)=0.5$$

$$P\left(R=0|W=d\right)=0.5$$

$$P\left(R=1|W=d\right)=0.5$$

$$P\left(R=2|W=d\right)=0$$

$$P\left(X_1=1|W\right)=expit\left(-1+2\mathrm{*}I\left(W=d\right)\right)$$

$$P\left(X_{2,t=1}=1\right)=0.5$$

$$P\left(X_{2,t=2}=1|X_{2,t=1}\right)=expit(0.3+0.3X_{2,t=1})$$

$$P\left(C_{t=1}=1\right)=0.1$$

$$P\left(C_{t=2}=1|A_{t=1}=C_{t=1}=Y_{t=1}=0\right)=0.1$$

$$P\left(A_t=1|W,R,X_{2,t}\right)=1-expit(I\left(W=l\right)+0.5\mathrm{*}I\left(R=2\right)+2\mathrm{*}I\left(R=0\right)+X_{2,t})$$

$$P\left(Y_{t=1}^{r,\mathrm{\infty }}|W,X_1,X_{2,t=1}\right)=expit\left(-8+0.5\mathrm{*}I\left(R=1\right)+1.5\mathrm{*}I\left(R=0\right)+5\mathrm{*}{X}_1-0.5\mathrm{*}{X}_{2,t=1}\right)$$

$$P\left(Y_{t=2}^{r,\mathrm{\infty }}|W,X_1,X_{2,t=2},Y_{t=1}^{r,\mathrm{\infty }}=0\right)=expit\left(-8+0.5\mathrm{*}I\left(R=1\right)+1.5\mathrm{*}I\left(R=0\right)+5\mathrm{*}{X}_1-0.5\mathrm{*}{X}_{2,t=2}\right)$$

Data availability:

The data that support the findings of this study are available from the corresponding author upon reasonable request.