Efficient and robust methods for causally interpretable meta-analysis: transporting inferences from multiple randomized trials to a target population

Summary:

We present methods for causally interpretable meta-analyses that combine information from multiple randomized trials to estimate potential (counterfactual) outcome means and average treatment effects in a target population. We consider identifiability conditions, derive implications of the conditions for the law of the observed data, and obtain identification results for transporting causal inferences from a collection of independent randomized trials to a new target population in which experimental data may not be available. We propose an estimator for the potential (counterfactual) outcome mean in the target population under each treatment studied in the trials. The estimator uses covariate, treatment, and outcome data from the collection of trials, but only covariate data from the target population sample. We show that it is doubly robust, in the sense that it is consistent and asymptotically normal when at least one of the models it relies on is correctly specified. We study the finite sample properties of the estimator in simulation studies and demonstrate its implementation using data from a multi-center randomized trial.

1. Introduction

When examining a body of evidence that consists of multiple trials, decision makers are typically interested in learning about the effects of interventions in some well-defined target population. In other words, they are interested in synthesizing the evidence across trials and transporting causal inferences from the collection of trials to a target population in which further experimentation may not be possible. Typically, each trial samples participants from a different underlying population, by recruiting participants from centers with different referral patterns or in different geographic locations. The goal of evidence synthesis in this context is to use the information from these diverse trials to draw causal inferences about the target population, accounting for any differences between the target population and the populations underlying the trials.

“Meta-analysis,” an umbrella term for statistical methods for synthesizing evidence across multiple trials (Cooper et al., 2009), traditionally focuses on modeling the distribution of treatment effects (effect sizes) across studies or on obtaining unbiased and minimum variance summaries of data from multiple trials (Higgins et al., 2009; Rice et al., 2018). Standard meta-analysis methods produce estimates that do not have a clear causal interpretation outside of the sample of participants enrolled in the trials because the estimates do not pertain to any well-defined target population (Dahabreh et al., 2020). Recent work on “generalizability” and “transportability” has considered methods for extending causal inferences to a target population, when data are available from a single randomized trial (Westreich et al., 2017; Rudolph and van der Laan, 2017; Dahabreh et al., 2020). When multiple trials are available, methods have been proposed for assessing case-mix heterogeneity in individual patient data meta-analyses without specifying or using data from a target population of substantive interest (Vo et al., 2019, 2021).

Here, we propose methods for causally interpretable meta-analysis that can be used to extend inferences about potential (counterfactual) outcome means and average treatment effects from a collection of randomized trials to a target population in which experimental data may not be available (Dahabreh et al., 2020). We consider identifiability conditions, derive implications of the conditions for the law of the observed data, and obtain identification results. We then propose novel estimators for potential outcome means and average treatment effects in the target population. The estimators use data on baseline covariates, treatments, and outcomes from the collection of trials, but only data on baseline covariates from the sample of the target population. We show that the estimators are doubly robust, in the sense that they remain consistent and asymptotically normal provided at least one of the two working models on which they rely is correctly specified. Last, we study the finite sample properties of the estimators in simulation studies and demonstrate the implementation of the methods using data from a multi-center trial of treatments for hepatitis C infection.

2. Data, sampling scheme, and causal estimands

Suppose we have data from a collection of randomized trials $𝒮$, indexed by $s=1,\dots ,m$. For each trial participant we have information on the trial $S$ in which they participated, treatment assignment $A$, baseline covariates $X$, and outcomes $Y$. We assume that the same finite set of treatments, $𝒜$, has been compared in each trial of the collection (extensions to cases where only a subset of treatments are evaluated in each trial are straightforward but require more cumbersome notation). From each trial $s\in 𝒮$, the data are independent and identically distributed random tuples $\left(X_i,S_i=s,A_i,Y_i\right),i=1,\dots ,n_s$, where $n_s$ is the total number of randomized individuals in trial $s$.

We also obtain a simple random sample from the target population (of individuals not participating in the trials), in which treatment assignment may not be under the control of investigators (e.g., individuals may self-select into treatment). We use the convention that $S=0$ for individuals from the target population. The data from the sample of the target population consists of independent and identically distributed tuples $\left(X_i,S_i=0,A_i,Y_i\right)$, $i=1,\dots ,n_0$, where $n_0$ is the total number of individuals sampled from the target population. The total number of observations from the trials and the sample of the target population is $n={\sum }_{s=0}^m{n}_s$. We define a new random variable $R$, such that $R=1$, if $S\in 𝒮$; and $R=0$, if $S=0$.

Informally, we assume that the observations from the trials in the collection $𝒮$ and from the target population $S=0$ are Bernoulli-type (independent random) samples (Breslow and Wellner, 2007; Saegusa and Wellner, 2013) from (near-infinite) underlying populations (Robins, 1988). Specifically, we assume that the sample from the target population is representative of a well-specified population of substantive interest. In contrast, we do not require that the trial samples are obtained through a formal sampling process; rather, that the investigators are willing to model the data from each trial as if sampled from a hypothetical underlying population. This modeling choice is consistent with the fairly standard super-population approach to the analysis of individual randomized trials (Robins, 1988). Though alternative frameworks (e.g., randomization-based inference) can be appealing in some cases, we find the super-population approach attractive when the goal is to extend inferences from a collection of trials to a new target population (Dahabreh and Hernán, 2019). Investigators conducting meta-analyses usually do not have control over the selection of participants of each trial or the relative sample size of different trials. Thus, we view the observations available for analysis as obtained by sampling the corresponding underlying populations with sampling fractions that are not under the control of the investigators and are unknown to them. In Section 4 we further formalize this sampling scheme; for now it suffices to say that all expectations and probabilities below are defined under the sampling scheme. Throughout, we use $f(\cdot )$ to generically denote densities. In Section 4 we discuss the implications of the sampling scheme for identifiability of these densities.

To define causal estimands, let $Y^a$ denote the potential (counterfactual) outcome under intervention to set treatment to $a\in 𝒜$ (Rubin, 1974; Robins and Greenland, 2000). We are interested in the potential outcome mean in the target population, $\text{E}\left[Y^a\mid R=0\right]$ for each $a\in 𝒜$, as well as the average causal effect, $\text{E}\left[Y^a-Y^{a^{\prime }}\mid R=0\right]$ for each pair of treatments $a\in 𝒜$ and $a^{\prime }\in 𝒜$. The treatments used in the target population need not be the same as the treatments used in the trials (e.g., some treatments may not be available outside experimental settings). Furthermore, as we show below, data on treatments and outcomes from the target population are not necessary for identification and estimation of the potential outcome means and average treatment effects of interest.

3. Identification

3.1. Identifiability conditions

The following are sufficient conditions for identifying the potential outcome mean in the target population $\text{E}\left[Y^a\mid R=0\right]$.

A1. Consistency of potential outcomes: if ${\mathit{A}}_{\mathit{i}}=\mathit{a}$, then ${\mathit{Y}}_{\mathit{i}}^{\mathit{a}}={\mathit{Y}}_{\mathit{i}}$, for every individual $\mathit{i}$ and every treatment $\mathit{a}\in \mathcal{A}$.

Implicit in condition A1 are assumptions that (i) there is no direct effect of participation in any trial $(R=1)$ or participation in some specific trial $(S=s)$ on the outcome (Dahabreh et al., 2019; Dahabreh and Hernán, 2019); (ii) there are no multiple versions of treatment (or that treatment variation is irrelevant with respect to the outcome (VanderWeele, 2009)); and (iii) there is no treatment-outcome interference (Rubin, 1986, 2010). These assumptions may be most plausible in the case of large pragmatic trials where fidelity to the assigned treatment can be high across settings. Note that consistency of potential outcomes is distinct from the notion of consistency used in so-called network meta-analyses; consistency of potential outcomes refers to the relationship between observed (factual) and potential (counterfactual) outcomes under each treatment, not the relationship between contrasts (“direct” and “indirect”, in meta-analytic parlance) of different treatments.

A2. Exchangeability over treatment A in each trial: for each trial $\mathit{s}\in \mathcal{S}$ and each $\mathit{a}\in \mathcal{A}$, ${\mathit{Y}}^{\mathit{a}}⫫\mathit{A}\mid (\mathit{X},\mathit{S}=\mathit{s})$.

Condition A2 is typically plausible because of randomization (marginal or conditional on $X$) in each of the trials . The condition is not equivalent to the condition $Y^a⫫A\mid (X,R=1)$ because the trials may have different randomization ratios and trial participation may have direct effects on the outcome or share unmeasured common causes with the outcome (the last two possibilities, however, are precluded by conditions A1 and A4, respectively). It is also worth noting that condition A2 would also hold if $𝒮$ was a collection of observational studies in which the covariates $X$ were sufficient to adjust for baseline confounding. Thus, our results can also apply to pooled analyses of observational studies, provided that background knowledge suggests “adjustment” for $X$ is sufficient to control confounding.

A3. Positivity of treatment assignment in each trial: for each treatment $\mathit{a}\in \mathcal{A}$ and for each trial $\mathit{s}\in \mathcal{S}$, if $\mathit{f}(\mathit{x},\mathit{S}=\mathit{s})\ne 0$, then $\mathbf{P}\mathbf{r}[\mathit{A}=\mathit{a}\mid \mathit{X}=\mathit{x},\mathit{S}=\mathit{s}]>0$.

Condition A3 is also plausible in marginally and conditionally randomized trials.

A4. Exchangeability over $\mathit{S}$: for each $\mathit{a}\in \mathcal{A},{\mathit{Y}}^{\mathit{a}}⫫\mathit{S}\mid \mathit{X}$.

In applied work, condition A4 will typically be a critical assumption connecting the randomized trials in the collection $𝒮$, between them and with the target population $(S=0)$. On its own, condition A4 is not testable; we will show, however, that it has testable implications when combined with conditions A1 and A2). In practice, condition A4 will need to be examined in light of substantive knowledge and examining the impact of violations of the condition may require undertaking sensitivity analyses (Robins et al., 2000).

A5. Positivity of trial participation: for each $\mathit{s}\in \mathcal{S}$, if $\mathit{f}(\mathit{x},\mathit{R}=0)\ne 0$, then $\mathbf{P}\mathbf{r}[\mathit{S}=\mathit{s}\mid \mathit{X}=\mathit{x}]>0$.

Informally, condition A5 states that covariate patterns in the target population can also occur in each trial. This condition only involves the observable data and thus is in principle testable. Formal examination of the condition, however, is challenging when $X$ is high dimensional (Petersen et al., 2012).

3.2. Implications of the identifiability conditions

We now explore some implications of the identifiability conditions. By Lemma 4.2 of Dawid (1979), condition A4 implies exchangeability over $S$ among the collection of trials, that is,

$$Y^a⫫S\left|X⟹Y^a⫫S\right|\left(X,R=1\right)\text{, for every }a\in 𝒜,$$ (1)

and also implies exchangeability of participants in the collection of trials and the target population,

$$Y^a⫫S\left|X⟹Y^a⫫I(S=0)\right|X⟺Y^a⫫R\mid X.$$ (2)

The first of these results will be useful to derive restrictions on the law of the observed data imposed by the identifiability conditions; the second result will be useful in obtaining identification results for causal quantities in the target population, using information from the collection of trials.

By Lemma 4.3 of Dawid (1979), the result in (1) and condition A2 imply conditional exchangeability of potential outcomes over $(S,A)$ in the collection of trials, that is,

$$\left.\left\{\begin{array}{c}{Y}^a⫫A\mid (X,R=1,S)\\ Y^a⫫S\mid (X,R=1)\end{array}\right\}⟹Y^a⫫(S,A)\right|(X,R=1).$$ (3)

Noting that $Y^a⫫(S,A)\mid (X,R=1)$ implies $Y^a⫫S\mid (X,R=1,A=a)$, and using condition A1, we obtain

$$Y⫫S\mid (X,R=1,A=a).$$ (4)

Thus, among individuals participating in any trial $(R=1)$, the observed outcome $Y$ is independent of the trial random variable $S$, within treatment groups $(A=a)$ and conditional on covariates $X$. Because this condition does not involve potential outcomes, it is testable using the observed data (e.g., using methods for comparing conditional densities). Furthermore, because $\{R=0\}⟺\{R=0,S=0\}$, we also obtain that $Y⫫S\mid (X,R,A=a)$. The independence condition in (4) implies that for every $a\in 𝒜$ and every $x$ such that $f(x,S=0)\ne 0$,

$$\text{E}\left[Y∣X=x,S=1,A=a\right]=\dots =\text{E}\left[Y∣X=x,S=m,A=a\right].$$ (5)

This restriction is testable using widely available parametric or non-parametric approaches for modeling the conditional expectations (e.g., (Racine et al., 2006; Luedtke et al., 2019)). Such statistical testing may be a useful adjunct to assessments of the causal assumptions based on substantive knowledge.

In practical terms, violations of the restrictions in displays (4) or (5) can occur, for example, when there are unmeasured common causes of trial participation $S$ and the outcome $Y$ (failure of condition A4), when trial participation directly effects the outcomes (failure of condition A1), or when there exists outcome-relevant variation in the treatments evaluated across trials (again, failure of condition A1). Though formal statistical assessments of the restrictions are possible, they will often be challenging when $X$ is high-dimensional. Furthermore, formal assessments of the restrictions in (4) and (5) will in general not be able to pinpoint the specific identifiability condition whose failure explains the violation of the observed data restrictions. Thus, considerable background knowledge and scientific judgment will be needed to decide on the appropriateness of combining information across trials to learn about the target population.

3.3. Identification of potential outcome means using the collection of trials

When conditions A1 through A5 hold, the potential outcome mean in the target population can be identified using covariate, treatment, and outcome data from the collection of trials, and baseline covariate data from the target population.

Theorem 1 (Identification of potential outcome means): Under conditions A1 through A5, the potential outcome mean in the target population under treatment $a\in 𝒜,\text{E}\left[Y^a\mid R=0\right]$, is identifiable by the observed data functional

$$\psi \left(a\right)\equiv \text{E}\left[\text{E}\left[Y∣X,R=1,A=a\right]∣R=0\right],$$ (6)

which can be equivalently expressed as

$$\psi \left(a\right)=\frac{1}{\text{Pr}\left[R=0\right]}\text{E}\left[\frac{I\left(R=1,A=a\right)Y\text{Pr}\left[R=0∣X\right]}{\text{Pr}\left[R=1∣X\right]\text{Pr}\left[A=a∣X,R=1\right]}\right].$$ (7)

The proof is given in Web Appendix A.

It also follows that, under the conditions of Theorem 1, the average treatment effect in the target population comparing treatments $a$ and $a^{\prime }$ in $𝒜$ is also identifiable: $\text{E}\left[Y^a-Y^{a^{\prime }}\mid R=\right.0]=\text{E}\left[Y^a\mid R=0\right]-\text{E}\left[Y^{a^{\prime }}\mid R=0\right]=\psi (a)-\psi \left(a^{\prime }\right)$.

3.4. Identification under weaker conditions

Practitioners of meta-analysis believe that by combining evidence from multiple randomized trials it should be possible to draw inferences about target populations that are – in some vague sense – broader than the population underlying each randomized trial. To give a concrete example: suppose that we have data from two trials of the same medications, one recruiting individuals with mild disease and the other recruiting individuals with severe disease. Suppose also that our target population includes some individuals with mild and some with severe disease and that effectiveness varies by disease severity. Intuition suggests that if inferences are “transportable” from each of the two trials to each subset of the target population defined by disease severity, then it should be possible to draw some conclusions about treatment effectiveness in the target population. Yet, in this setting, condition A5 is grossly violated (e.g., individuals with mild disease have zero probability of participating in one of the two trials). In the following two sub-sections, we give identification results that rely on weaker identifiability conditions and capture the intuition that when combining evidence from multiple trials we can draw inferences about target populations that are broader than the population underlying each randomized trial.

3.4.1. Weakening the positivity conditions of Theorem 1.

Suppose that the independence conditions $Y^a⫫R\mid X$ and $Y^a⫫A\mid (X,R=1)$ hold, taken either as primitive conditions or as implications of identifiability conditions A2 and A4 (as was done for Theorem 1).

Furthermore, consider the following positivity conditions:

A3*. Positivity of the probability of treatment in the collection of trials: for each treatment $a\in 𝒜$, if $f(x,R=1)\ne 0$, then $\text{Pr}[A=a\mid X=x,R=1]>0$.

A5*. Positivity of the probability of participation in the collection of trials: if $f(x,R=0)\ne 0$, then $\text{Pr}[R=1\mid X=x]>0$.

It is worth noting that these positivity conditions are weaker than the corresponding conditions A3 and A5, in the sense that A3* is implied by, but does not imply, A3; and A5* is implied by, but does not imply, A5. The difference between the two sets of assumptions is practically important: informally, condition A3 requires every treatment to be available in every trial and for every covariate pattern that can occur in that trial, and condition A5 requires every covariate pattern that can occur in the target population to be represented in every trial. In contrast, condition A3* requires every treatment to be available in the aggregated collection of trials and for any covariate pattern that can occur in the collection (but not necessarily in every trial); condition A5* requires every covariate pattern that can occur in the target population to occur in the aggregated collection of trials (but not necessarily in every trial).

Using these modified conditions, we obtain the following result (we give the formal arguments in Web Appendix A and Web Appendix B):

Theorem 2 (Identification of potential outcome means under weaker positivity conditions): If $Y^a⫫R\mid X$ and $Y^a⫫A\mid (X,R=1)$ and conditions A1, A3*, and A5* hold, then the potential outcome mean in the target population under treatment $a\in 𝒜,\text{E}\left[Y^a\mid R=0\right]$, is identifiable by $\psi \left(a\right)$.

3.4.2. Identification under weaker overlap and exchangeability conditions.

We will now show that the exchangeability conditions for potential outcome means can also be relaxed, alongside the positivity conditions; to do so, we need to introduce some additional notation in order to keep track of the subsets of the collection of trials where different covariate patterns can occur.

Let ${𝒳}_j$ for $j\in \{\mathrm{0,1},\dots ,m\}$ denote the support of the random vector $X$ in the subset of the population with $S=j$. That is, ${𝒳}_j\equiv \{x:f(x\mid S=j)>0\}$. For each covariate pattern $X=x$ that can occur in the collection of trials, that is, for each $x\in \underset{s\in 𝒮}{\cup }{𝒳}_s$, define ${𝒮}_x$ as the subset of trials in $𝒮$ such that $x$ belongs in their support. That is, ${𝒮}_x\equiv \left\{s:s\in 𝒮,x\in {𝒳}_s\right\}$. Intuitively, ${𝒮}_x$ denotes the subset of trials in the collection $𝒮$ where the covariate pattern $X=x$ can occur. Using this additional notation, consider the following identifiability conditions:

A4^†. Exchangeability in mean over: For every $x$ such that $f(x,S=0)\ne 0$ and every $s\in {𝒮}_x,\text{E}\left[Y^a\mid X=x,S=0\right]=\text{E}\left[Y^a\mid X=x,S=s\right]$.

A5^†. Overlap of the collection $𝒮$ with the target population: $\bigcup _{s\in 𝒮}\left({𝒳}_s\cap {𝒳}_0\right)={𝒳}_0$. Using the above two conditions in the place of conditions A4 and A5 still allows for identification of the potential outcome means, using the following result.

Theorem 3 (Identification of potential outcome means under weaker conditions): Under identifiability conditions A1 through A3, A4^†, and A5^†, for every $a\in 𝒜,\text{E}\left[Y^a\mid R=0\right]$ is identifiable by

$$\varphi \left(a\right)\equiv \int \text{E}\left[Y\mid X=x,I\left(S\in {𝒮}_x\right)=1,A=a\right]f\left(x∣S=0\right)dx.$$

The proof is given in Web Appendix B.

In simple cases, as in our hypothetical example of effect modification by disease severity where the positivity violation was due to a single binary covariate, the above result simply suggests that the conditional mean of the outcome $Y$ can be modeled separately in each group of trials defined by that binary covariate. For more complicated violations of positivity condition A5, especially those involving multiple continuous covariates, the above result is mostly useful, not for suggesting a particular modeling strategy, but for showing that some degree of interpolation when modeling the conditional expectation may be relatively benign, when there is adequate overlap between the collection of trials and the target population, in the sense of condition A5^†.

3.5. Identification under exchangeability in effect measure

Up to this point, we have examined conditions that are sufficient for identifying both potential outcome means and average treatment effects. If interest is restricted to average treatment effects, identification is possible under a weaker condition of exchangeability in effect measure over $S$ given baseline covariates, essentially, a requirement that conditional average treatment effects, but not necessarily potential outcome means, can be transported from each trial in the collection $S$ to the target population; see Dahabreh et al. (2018, 2020) for a similar argument in the context of transporting inferences from a single trial. Under this weaker condition, however, the individual potential outcome means are not identifiable. Specifically, consider the following condition:

A4^‡. Exchangeability in effect measure over $S$: for every pair of treatments $a$ and $a^{\prime }$, with $a\in 𝒜$ and $a^{\prime }\in 𝒜$, for every $s\in 𝒮$, and for every $x$ such that $f(x,S=s)\ne 0$, we have $\text{E}\left[Y^a-Y^{a^{\prime }}\mid X=x,S=s\right]=\text{E}\left[Y^a-Y^{a^{\prime }}\mid X=x,R=0\right]$.

It is easy to see that this condition, combined with conditions A1 through A3, and A5, implies that the observed data trial-specific conditional mean difference comparing treatments $a\in 𝒜$ and $a^{\prime }\in 𝒜,\text{E}[Y\mid X,S=s,A=a]-\text{E}\left[Y\mid X,S=s,A=a^{\prime }\right]$ does not vary over $s\in 𝒮$ for all $X$ values with positive density in the target population. Using this implication, we now give an identification result for the average treatment effect under the weaker condition of exchangeability in effect measure.

Theorem 4 (Identification under exchangeability in effect measure): Under conditions A1 through A3, A4^‡, and A5, the average treatment effect in the target population comparing treatments $a\in 𝒜$ and $a^{\prime }\in 𝒜,\text{E}\left[Y^a-Y^{a^{\prime }}\mid R=0\right]$, is identifiable by the observed data functional $\rho \left(a,a^{\prime }\right)\equiv \text{E}\left[\tau \left(a,a^{\prime };X\right)\mid R=0\right]$, where $\tau \left(a,a^{\prime };X\right)\equiv \text{E}[Y\mid X,S=s,A=a]-\text{E}\left[Y\mid X,S=s,A=a^{\prime }\right]$ does not vary over $s\in 𝒮$. Furthermore, $\rho \left(a,a^{\prime }\right)$ can be re-expressed as $\rho \left(a,a^{\prime }\right)=\frac{1}{\text{Pr}[R=0]}\text{E}\left[w\left(a,a^{\prime };R,X,S,A\right)Y\right]$, where

$$w\left(a,a^{\prime };R,X,S,A\right)\equiv \left(\frac{I\left(R=1,A=a\right)}{\text{Pr}\left[A=a∣X,S,R=1\right]}-\frac{I\left(R=1,A=a^{\prime }\right)}{\text{Pr}\left[A=a^{\prime }\mid X,S,R=1\right]}\right)\frac{\text{Pr}\left[R=0∣X\right]}{\text{Pr}\left[R=1∣X\right]}.$$

The proof is given in Web Appenidx C. Note in passing that positivity condition A5 in Theorem 4 can be relaxed in a way analogous to what was done in Theorem 3.

Because the potential outcome means $\text{E}\left[Y^a\mid R=0\right]$ in the target population often are of inherent interest, in the rest of this paper we focus on estimating the identifying functionals $\psi \left(a\right)$, for $a\in 𝒜$, following Theorems 1 and 2.

4. Remarks on the sampling model

We now revisit the assumed sampling scheme that we described in Section 2 and introduce some notation to help us distinguish between the population sampling model, where data are obtained by simple random sampling from a common super-population, and a biased sampling model, where data are obtained by stratified random sampling from the target population and the populations underlying the trials, with unknown and possibly variable sampling fractions.

Under a nonparametric model for the observed data, the density of the law of the observable data $O=(X,S,R,A,Y)$, using $p$ to generically denote densities, can be written as $p(r,x,s,a,y)=p\left(r\right)p(x\mid r)p(s\mid x,r)p(a\mid r,x,s)p(y\mid r,x,s,a)$. By definition, $S$ contains all the information contained in $R$, which implies that $p(a\mid r,s,x)=p(a\mid s,x)$. Furthermore, by (4), $p(y\mid r=1,x,a)=p(y\mid r=1,s,x,a);$ and the equivalence $\{R=0,S=0\}⟺\{S=0\}$ gives $p(y\mid r=0,x,a)=p(y\mid r=0,s,x,a)$; thus, $p(y\mid r,x,a)=p(y\mid r,s,x,a)$. We conclude that, under simple random sampling, the density of the observable data would be

$$p\left(r,x,s,a,y\right)=p\left(r\right)p\left(x∣r\right)p\left(s∣x,r\right)p\left(a∣x,s\right)p\left(y∣r,x,a\right).$$

This density can be viewed as reflecting a population sampling model.

The data collection approach described in Section 2, however, induces a biased sampling model (Bickel et al., 1993). What we mean here is that the sampling fraction from the sub-population underlying each randomized trial $(S=1,\dots ,m)$ and the target population sample $(R=S=0)$ is not under the control of the investigators (and, typically, unknown to them) and reflects the particular circumstances (e.g., recruiting practices) of how sampling was conducted (e.g., convenience sampling is typical in randomized trials). Thus, in the data, the ratios $\frac{n_j}{n}$, for $j\in \{\mathrm{0,1},\dots ,m\}$, do not reflect the population probabilities of belonging to the subset of the population with $S=j$, because the sampling fraction from each subset is unknown (and possibly variable between subsets). As a technical condition, we require that as $n⟶\infty ,\frac{n_j}{n}⟶{\pi }_j>0$, for $j\in \{\mathrm{0,1},\dots ,m\}$. Nevertheless, under the biased sampling model, the limiting values, ${\pi }_j$, are not necessarily equal to the super-population probabilities under the population sampling model.

Under this more plausible sampling model, the density of the observable data can be written as

$$\begin{gathered} q(r,x,s,a,y)=q\left(r\right)q(x\mid r)q(s\mid x,r)p(a\mid x,s)p(y\mid r,x,a) \\ =q\left(r\right)\left\{q\right(x\mid r=1\left){\}}^r\right\{q(x\mid r=0){\}}^{1-r}q(s\mid x,r)p(a\mid x,s)p(y\mid r,x,a) \\ =q\left(r\right)\left\{q\right(x\mid r=1\left){\}}^r\right\{p\left(x∣r=0\right){\}}^{1-r}q\left(s∣x,r\right)p\left(a∣x,s\right)p\left(y∣r,x,a\right). \end{gathered}$$

Here, $q\left(r\right)\ne p\left(r\right)$ and $q(s\mid x,r)\ne p(s\mid x,r)$ to reflect the fact that the sampling fractions for participants in the trials and members of the target population are not equal to the super-population probabilities under the population sampling model, but instead depend on the complex processes of randomized trial design and conduct (i.e., the biased sampling model). For the same reason, $q(x\mid r)\ne p(x\mid r)$, because in general we do not expect $q(x\mid r=1)=p(x\mid r=1)$; the distribution of $X$ in the collection of trials depends on the sampling from the different population underlying each trial. Nevertheless, we expect $p(x\mid r=0)=q(x\mid r=0)$ because we take a simple random sample from the target population under the biased sampling model. In contrast, the terms $p(a\mid x,s)$ and $p(y\mid a,x,r)$ are the same in both densities, reflecting the property of stratified (by $S$) random sampling of the population underlying each trial or the target population, and the independence condition in (4).

The functional $\psi \left(a\right)$ in Theorems 1 and 2 only depends on components of $p(r,x,s,a,y)$ that are also identifiable under the distribution $q(r,s,x,a,y)$ induced by the biased sampling. Specifically, using the representation in (6), $\psi \left(a\right)$ depends on $p(y\mid r=1,x,a)$, and $p(x\mid r=0)=p(x\mid s=0)$, both of which are identifiable under the biased sampling model.

5. Estimation and inference

We now turn our attention to the estimation of the functional $\psi \left(a\right)$, for $a\in 𝒜$. The results presented in this section apply to data obtained under the population sampling model $p(r,x,s,a,y)$ and the biased sampling model $q(r,x,s,a,y)$, because $\psi \left(a\right)$ is identifiable under both. In fact, the results in Breslow et al. (2000) imply that influence functions for $\psi \left(a\right)$ under sampling from $q(r,x,s,a,y)$ are equivalent to those under sampling from $p(r,x,s,a,y)$, but with densities from $q(r,x,s,a,y)$ replacing those under $p(r,x,s,a,y)$; see Kennedy et al. (2015) for a similar argument in the context of matched cohort studies and Dahabreh et al. (2020) in the context of transporting inferences from a single trial. In the remainder of the paper, we assume that analysts will be working under $q(r,x,s,a,y)$, because the biased sampling model is more realistic for applied meta-analyses.

5.1. Proposed estimator

In Web Appendix D we show that the first-order influence function (Bickel et al., 1993) of $\psi \left(a\right)$, for $a\in 𝒜$, under the nonparametric model for the observable data, is

$${\Psi }_{q_0}^1\left(a\right)=\frac{1}{{\pi }_{q_0}}\left\{\frac{I(R=1,A=a){\text{Pr}}_{q_0}[R=0\mid X]}{{\text{Pr}}_{q_0}[R=1\mid X]{\text{Pr}}_{q_0}[A=a\mid X,R=1]}\left\{Y-{\text{E}}_{q_0}[Y\mid X,R=1,A=a]\right\}\left.+I(R=0)\left\{{\text{E}}_{q_0}[Y\mid X,R=1,A=a]-{\psi }_{q_0}\left(a\right)\right\}\right\},\right.$$

where ${\pi }_{q_0}={\text{Pr}}_{q_0}[R=0]$ and the subscript $q_0$ denotes that all quantities are evaluated at the “true” data law. Specifically, we show that ${\Psi }_{q_0}^1\left(a\right)$ satisfies ${\left.\frac{\partial {\psi }_{q_t}\left(a\right)}{\partial t}\right|}_{t=0}=\text{E}\left[{\Psi }_{q_0}^1\left(a\right)u\left(O\right)\right]$, where $u\left(O\right)$ denotes the score of the observable data $O=(R,X,S,A,Y)$ and the left hand side of the above equation is the pathwise derivative of the target parameter $\psi \left(a\right)$. Theorem 4.4 in (Tsiatis, 2007) shows that ${\Psi }_{q_0}^1\left(a\right)$ lies in the tangent set; it follows (see, e.g., (Van der Vaart, 2000), page 363) that ${\Psi }_{q_0}^1\left(a\right)$ is the efficient influence function under the nonparametric model (it is in fact the unique influence function under that model). For a more thorough discussion of semiparametric efficiency theory and precise definitions of pathwise derivatives and the tangent space we refer to Chapter 25 in (Van der Vaart, 2000). Furthermore, in Web Appendix D we show that ${\Psi }_{q_0}^1\left(a\right)$ is the efficient influence function under semiparametric models incorporating the restriction $Y⫫S\mid (X,A=a,R)$ or where the probability of treatment conditional on covariates and trial participation status is known (e.g., if all trials have the same randomization probabilities).

The influence function above suggests the estimator

$${\widehat{\psi }}_{\text{aug}}\left(a\right)=\frac{1}{n\widehat{\pi }}\sum _{i=1}^n\left\{I\left(R_i=1,A_i=a\right)\frac{1-\widehat{p}\left(X_i\right)}{\widehat{p}\left(X_i\right){\widehat{e}}_a\left(X_i\right)}\left\{Y_i-{\widehat{g}}_a\left(X_i\right)\right\}+I\left(R_i=0\right){\widehat{g}}_a\left(X_i\right)\right\},$$ (8)

where $\widehat{\pi }=n^{-1}{\sum }_{i=1}^{n}I\left(R_i=0\right),{\widehat{g}}_a\left(X\right)$ is an estimator for $\text{E}[Y\mid X,R=1,A=a],{\widehat{e}}_a\left(X\right)$ is an estimator for $\text{Pr}[A=a\mid X,R=1]$, and $\widehat{p}\left(X\right)$ is an estimator for $\text{Pr}[R=1\mid X]$. Note that ${\widehat{\psi }}_{\text{aug}}\left(a\right)$ involves data on $(R=1,X,A,Y)$ from trial participants and data on $(R=0,X)$ from the sample of the target population; thus, treatment and outcome data from the target population are not necessary. Furthermore, it is natural to estimate the average treatment effect in the target population for comparing treatments $a$ and $a^{\prime }$ in $𝒜$, that is, $\text{E}\left[Y^a-Y^{a^{\prime }}\mid R=0\right]$, using the contrast estimator $\widehat{\delta }\left(a,a^{\prime }\right)={\widehat{\psi }}_{\text{aug}}\left(a\right)-{\widehat{\psi }}_{\text{aug}}\left(a^{\prime }\right)$.

In practical applications, analysts may be able to improve the performance of the estimator by normalizing the weights so that their sum is equal to the total number of observations in the target population sample (Dahabreh et al., 2018, 2019, 2020). This normalization, which was originally proposed for survey analyses (Hájek, 1971), may be particularly useful when $\frac{1-\widehat{p}\left(X\right)}{\widehat{p}\left(X\right){\widehat{e}}_a\left(X\right)}$ is highly variable over the observations in the trials (Robins et al., 2007).

Asymptotic properties of estimators for potential outcome means: Let $g_a^{*}\left(X\right),e_a^{*}\left(X\right)$, and $p^{*}\left(X\right)$ denote the asymptotic limits (assumed to exist) of ${\widehat{g}}_a\left(X\right),{\widehat{e}}_a\left(X\right)$, and $\widehat{p}\left(X\right)$, respectively. Finally, define $\widehat{\gamma }={\left\{\frac{1}{n}{\sum }_{i=1}^{n}I\left(R_i=0\right)\right\}}^{-1}={\widehat{\pi }}^{-1}$ as an estimator for ${\gamma }^{*}=\text{Pr}[R=0{]}^{-1}$.

For general functions ${\gamma }^{\prime },g_a^{\prime }\left(X\right),e_a^{\prime }\left(X\right)$, and $p^{\prime }\left(X\right)$ define $H\left({\gamma }^{\prime },g_a^{\prime }\left(X\right),e_a^{\prime }\left(X\right),p^{\prime }\left(X\right)\right)={\gamma }^{\prime }\left\{I(R=0)g_a^{\prime }\left(X\right)+I(R=1,A=a)\frac{1-p^{\prime }\left(X\right)}{p^{\prime }\left(X\right)e_a^{\prime }\left(X\right)}\left\{Y-g_a^{\prime }\left(X\right)\right\}\right\}$. Using notation from van der Vaart and Wellner (1996), define ${\mathbb{P}}_n\left(v\right(W\left)\right)=n^{-1}{\sum }_{i=1}^{n}v\left(W_i\right)$ and ${\mathbb{G}}_n\left(v\right(W\left)\right)=\sqrt{n}\left({\mathbb{P}}_n\left(v\right(W\left)\right)-\text{E}\left[v\right(W\left)\right]\right)$, for some function $v$ and a random variable $W$. Using this notation, ${\widehat{\psi }}_{\text{aug}}\left(a\right)={\mathbb{P}}_n\left(H\left(\widehat{\gamma },{\widehat{g}}_a\left(X\right),{\widehat{e}}_a\left(X\right),\widehat{p}\left(X\right)\right)\right)$.

To establish asymptotic properties of $\widehat{\psi }\left(a\right)$, we make the following assumptions:

1. (i) The sequence $H\left(\widehat{\gamma },{\widehat{g}}_a\left(X\right),{\widehat{e}}_a\left(X\right),\widehat{p}\left(X\right)\right)$ and its limit $H\left({\gamma }^{*},g_a^{*}\left(X\right),e_a^{*}\left(X\right),p^{*}\left(X\right)\right)$ fall in a Donsker class (van der Vaart and Wellner, 1996).

2. (ii) ${∥H\left(\widehat{\gamma },{\widehat{g}}_a\left(X\right),{\widehat{e}}_a\left(X\right),\widehat{p}\left(X\right)\right)-H\left({\gamma }^{*},g_a^{*}\left(X\right),e_a^{*}\left(X\right),p^{*}\left(X\right)\right)∥}_2\stackrel{\text{a.s.}}{\to }0$.

3. (iii) $\text{E}\left[H{\left({\gamma }^{*},g_a^{*}\left(X\right),e_a^{*}\left(X\right),p^{*}\left(X\right)\right)}^2\right]<\infty$.

4. (iv) At least one of the following two assumptions holds: (a) $\widehat{p}\left(X\right)\stackrel{\text{a.s.}}{\to }{p}^{*}\left(X\right)=\text{Pr}[R=1\mid X]$ and ${\widehat{e}}_a\left(X\right)\stackrel{\text{a.s.}}{\to }{e}_a^{*}\left(X\right)=\text{Pr}[A=a\mid X,R=1]$; or $\left(\text{b}\right){\widehat{g}}_a\left(X\right)\stackrel{\text{a.s.}}{\to }{g}_a^{*}\left(X\right)=\text{E}[Y\mid X,R=1,A=a],p^{*}\left(X\right)⩾\epsilon$, and $e_a^{*}\left(X\right)⩾\epsilon$ a.s. for some $\epsilon >0$.

The following Theorem gives the asymptotic properties of ${\widehat{\psi }}_{\text{aug}}\left(a\right)$; a detailed proof is given in Web Appendix E.

Theorem 5: If assumptions (i) through (iv) hold, then

(1) ${\widehat{\psi }}_{\text{aug}}\left(a\right)\stackrel{\text{a.s.}}{\to }\psi \left(a\right)$; and

(2) ${\widehat{\psi }}_{\text{aug}}\left(a\right)$ has the asymptotic representation

$$\sqrt{n}\left({\widehat{\psi }}_{\text{aug}}\left(a\right)-\psi \left(a\right)\right)={\mathbb{G}}_n\left(H\left({\gamma }^{*},g_a^{*}\left(X\right),e_a^{*}\left(X\right),p^{*}\left(X\right)\right)\right)+\mathit{\text{Rem}}+o_P\left(1\right),$$ (9)

where ${\mathbb{G}}_n\left(H\left({\gamma }^{*},g_a^{*}\left(X\right),e_a^{*}\left(X\right),p^{*}\left(X\right)\right)\right)$ is asymptotically normal and

$$Rem⩽\sqrt{n}{O}_P\left(\left(\parallel \text{Pr}[R=1\mid X]-\widehat{p}\left(X\right){\parallel }_2+{∥\text{Pr}[A=a\mid X,R=1]-{\widehat{e}}_a\left(X\right)∥}_2\right)\left.\times {∥{\widehat{g}}_a\left(X\right)-\text{E}[Y\mid X,R=1,A=a]∥}_2\right).\right.$$ (10)

If ${\widehat{g}}_a\left(X\right),\widehat{p}\left(X\right),{\widehat{e}}_a\left(X\right)$ and $g_a^{*}\left(X\right),p^{*}\left(X\right),e_a^{*}\left(X\right)$ are Donsker; $p^{*}\left(X\right)$ and $e_a^{*}\left(X\right)$ are uniformly bounded away from zero – assumption (iv); and $g_a^{*}\left(X\right)$ and $Y$ are uniformly bounded, then $H\left(\widehat{\gamma },{\widehat{g}}_a\left(X\right),{\widehat{e}}_a\left(X\right),\widehat{p}\left(X\right)\right)$ and its limit are Donsker (Kosorok, 2008).

Assumptions (i), (ii), and (iii) are standard assumptions used to show asymptotic normality of M-estimators (Van der Vaart, 2000). Assumption (i) restricts the flexibility of the models used to estimate the nuisance parameters (and their corresponding limits). But, it still covers a wide range of commonly used estimators such as parametric Lipschitz classes and VC classes (van der Vaart and Wellner, 1996). For data-adaptive estimators, Donsker assumptions can be relaxed using sample splitting (Robins et al., 2008).

Assumption (iv) indicates that the estimator ${\widehat{\psi }}_{\text{aug}}\left(a\right)$ is doubly robust, in the sense that it converges almost surely to $\psi \left(a\right)$ when either (1) the model for the conditional outcome mean $\text{E}[Y\mid X,R=1,A=a]$ is correctly specified, so that ${\widehat{g}}_a\left(X\right)$ converges to the true conditional expectations almost surely; or (2) the model for the probability of participation in any trial $\text{Pr}[R=1\mid X]$ and the model for the probability of treatment assignment $\text{Pr}[A=a\mid X,R=1]$ are correctly specified, so that $\widehat{p}\left(X\right)$ and ${\widehat{e}}_a\left(X\right)$ converge to the true conditional probabilities almost surely. When the treatment assignment is the same for all the trials $\text{Pr}[A=a\mid X,R=1]$ is known and when the treatment assignment only depends on a few categorical covariates $\text{Pr}[A=a\mid X,R=1]$ is easy to estimate consistently. But, when the treatment assignment mechanism depends on continuous covariates, as would be the case in pooled analyses of observational studies, estimating $\text{Pr}[A=a\mid X,R=1]$ may be more challenging.

The asymptotic representation given by (9) gives several useful insights into how estimation of the nuisance parameters affect the asymptotic distribution of ${\widehat{\psi }}_{\text{aug}}\left(a\right)$. By the central limit theorem, the first term on the right hand side of (9) is asymptotically normal. Hence, the asymptotic distribution of the estimator relies on the behavior of the term given in equation (10). If both the pair of nuisance parameters $\widehat{p}\left(X\right)$ and ${\widehat{g}}_a\left(X\right)$ and the pair of nuisance parameters ${\widehat{e}}_a\left(X\right)$ and ${\widehat{g}}_a\left(X\right)$ converge combined at a rate fast enough such that term (10) is $o_P\left(1\right)$, then the estimator is $\sqrt{n}$-consistent, asymptotically normal, and has asymptotic variance equal to the variance of the efficient influence function.

The asymptotic representation gives the rate of convergence result

$$∥{\widehat{\psi }}_{\text{aug}}\left(a\right)-\psi \left(a\right)∥=O_P\left(\frac{1}{\sqrt{n}}+\left(\parallel \text{Pr}[R=1\mid X]-\widehat{p}\left(X\right){\parallel }_2+{∥\text{Pr}[A=a\mid X,R=1]-{\widehat{e}}_a\left(X\right)∥}_2\right)\left.\times {∥{\widehat{g}}_a\left(X\right)-\text{E}[Y\mid X,R=1,A=a]∥}_2\right)\right.$$

Thus, if both the combined rate of convergence of $\widehat{p}\left(X\right)$ and ${\widehat{g}}_a\left(X\right)$ and the combined rate of convergence of ${\widehat{e}}_a\left(X\right)$ and ${\widehat{g}}_a\left(X\right)$ are at least $\sqrt{n}$, then ${\psi }_{\text{aug}}\left(a\right)$ is $\sqrt{n}$ convergent.

5.2. Inference

To construct Wald-style confidence intervals for $\psi \left(a\right)$, when using parametric models, we can easily obtain the sandwich estimator (Stefanski and Boos, 2002) of the sampling variance for $\widehat{\psi }\left(a\right)$. Alternatively, we can use the non-parametric bootstrap. If the remainder term in equation (9) of Theorem 5 is $o_P\left(1\right)$, then the estimator ${\widehat{\psi }}_{\text{aug}}\left(a\right)$ is asymptotically normally distributed. The asymptotic normality combined with Assumption (iii) ensure that the confidence intervals calculated using the non-parametric bootstrap asymptotically have the correct coverage rate.

6. Simulation study

We performed simulation studies to evaluate the finite sample performance of the augmented estimators described in the previous section and compare them with alternative approaches.

6.1. Data generation

The data generation process involved six steps: generation of covariates, selection for trial participation, sampling of individuals from the target population, allocation of trial participants to specific trials, random treatment assignment, and potential/observed outcomes. Simulations contained either $n=\mathrm{10,000}$ or $n=\mathrm{100,000}$ individuals, including both trial participants and the sample of the target population.

1. Covariates: Three covariates $X_1,X_2,X_3$ for each individual in the target population were drawn from a mean-zero multivariate normal distribution with all marginal variances equal to 1 and all pairwise correlations equal to 0.5.

2. Selection for trial participation: We considered three trials, $𝒮=\{\mathrm{1,2},3\}$, a reasonable number given the requirement that all trials have examined the same treatments. We examined scenarios where the total number of trial participants, ${\sum }_{s=1}^3{n}_s$, was 1,000, 2,000, or 5,000. We “selected” observations for participation in any trial using a logistic-linear model, $R~\text{Bernoulli}\left(\text{Pr}\right[R=1\mid X\left]\right)$ with $\text{Pr}[R=1\mid X]=\frac{\text{exp}\left(\beta X^T\right)}{1+\text{exp}\left(\beta X^T\right)},X=\left(1,X_1,\dots ,X_3\right)$, $\beta =\left({\beta }_0,\text{ln}\left(2\right),\text{ln}\left(2\right),\text{ln}\left(2\right)\right)$, where we solved for ${\beta }_0$ to result (on average) in the desired total number of trial participants given the total cohort sample size (exact numerical values for ${\beta }_0$ are available for all scenarios in the code to reproduce the simulations in Web Appendix H).

3. Sampling of individuals from the target population: We used baseline covariate data from all remaining non-randomized individuals in the sample, $n_0=n-{\sum }_{s=1}^3{n}_s$; this corresponds to taking a census of the non-randomized individuals in the simulated cohort and treating them as the sample from the target population.

4. Allocation of trial participants to specific trials: We allocated trial participants $(R=1)$ to one of the three randomized trials in $𝒮$ using a multinomial logistic model, $S\mid (X,R=1)~\text{Multinomial}\left(p_1,p_2,p_3;{\sum }_{s=1}^3{n}_s\right)$, with $p_1=\text{Pr}[S=1\mid X,R=1]=1-p_2-p_3,p_2=\text{Pr}[S=2\mid X,R=1]=\frac{e^{\xi X^T}}{1+e^{\xi X^T}+e^{\zeta X^T}}$, and $p_3=\text{Pr}[S=3\mid X,R=1]=\frac{e^{\zeta X^T}}{1+e^{\xi X^T}+e^{\zeta X^T}}$, where $\xi =\left({\xi }_0,\text{ln}\left(1.5\right),\text{ln}\left(1.5\right),\text{ln}\left(1.5\right)\right)$ and $\zeta =\left({\zeta }_0,\text{ln}\left(0.75\right),\text{ln}\left(0.75\right),\text{ln}\left(0.75\right)\right)$. We evaluated a scenario in which the trials had the same sample size and one where the sample size varied across trials. We used Monte Carlo methods to obtain intercepts ${\xi }_0$ and ${\zeta }_0$ that resulted in approximately equal-sized trials or in unequal-sized trials with a 4:2:1 ratio of samples sizes (exact numerical values for ${\xi }_0$ and ${\zeta }_0$ are available for all scenarios in the code to reproduce the simulations) (Robertson et al., 2021).

5. Random treatment assignment: We generated an indicator of unconditionally randomized treatment assignment, $A$, among randomized individuals. In one scenario the treatment assignment mechanism was marginally randomized and constant across trials, $A~\text{Bernoulli}\left(\text{Pr}\right[A=1\mid S=s\left]\right)$ with $\text{Pr}[A=1\mid S=s]=1/2$. In the second scenario, the treatment assignment mechanism varied across trials and was marginally randomized, with probabilities $\text{Pr}[A=1\mid S=1]=1/2,\text{Pr}[A=1\mid S=2]=1/3$, and $\text{Pr}[A=1\mid S=3]=2/3$.

6. Outcomes: We generated potential outcomes as $Y^a={\theta }^a{X}^T+{ϵ}^a$, for $a\in \left\{\mathrm{0,1}\right\},{\theta }^0=(1.5,1,1,1)$, and ${\theta }^1=(0.5,-1,-1,-1)$. In all simulations, ${ϵ}^a$ had an independent standard normal distribution for $a=\mathrm{0,1}$. We generated observed outcomes under consistency, such that $Y=A{Y}^1+(1-A)Y^0$.

6.2. Methods implemented and comparisons

Estimators:

In each simulated dataset, we applied the estimators from Section 5. Heuristically, the identification results in Theorems 1 and 2 suggest two non-augmented (and non-doubly robust) estimators: the g-formula estimator,

$${\widehat{\psi }}_{\text{g}}\left(a\right)={\left\{\sum _{i=1}^{n}I\left(R_i=0\right)\right\}}^{-1}\sum _{i=1}^{n}I\left(R_i=0\right){\widehat{g}}_a\left(X_i\right),$$

and the weighting estimator

$${\widehat{\psi }}_{\text{w}}\left(a\right)={\left\{\sum _{i=1}^{n}I\left(R_i=0\right)\right\}}^{-1}\sum _{i=1}^{n}I\left(R_i=1,A_i=a\right)\frac{1-\widehat{p}\left(X_i\right)}{\widehat{p}\left(X_i\right){\widehat{e}}_a\left(X_i\right)}{Y}_i,$$

where ${\widehat{g}}_a\left(X\right),\widehat{p}\left(X\right)$, and ${\widehat{e}}_a\left(X\right)$ were previously defined. The estimators ${\widehat{\psi }}_{\text{g}}\left(a\right)$ and ${\widehat{\psi }}_{\text{w}}\left(a\right)$ can be viewed as special cases of the augmented estimator ${\widehat{\psi }}_{\text{aug}}\left(a\right)$: we obtain ${\widehat{\psi }}_{\text{g}}\left(a\right)$ by identically setting $\widehat{p}\left(X\right)$ to one, and we obtain ${\widehat{\psi }}_{\text{w}}\left(a\right)$ by identically setting ${\widehat{g}}_a\left(X\right)$ to zero, in equation (8). It follows that ${\widehat{\psi }}_{\text{g}}\left(a\right)$ and ${\widehat{\psi }}_{\text{w}}\left(a\right)$ are not robust to misspecification of the models used to estimate ${\widehat{g}}_a\left(X\right)$, or $\widehat{p}\left(X\right)\times {\widehat{e}}_a\left(X\right)$, respectively (Dahabreh et al., 2020).

Model specification:

All working models included main effects for the observable covariates $X_j,j=\mathrm{1,2},3$. Models for the probability of participation in the trial and models for the probability of treatment included main effects for all covariates. We used logistic regression to model the probability of participating in any trial $(R=1$ vs 0). In our simulation scenarios, when the probability of treatment (the randomization ratio) varies across trials, a logistic regression model of treatment with main effects of the covariates (and no adjustment for trial $S$) is not correctly specified. To avoid misspecification of the model for the probability of treatment, we used the fact that $\text{Pr}[A=a\mid X,R=1]={\sum }_{s=1}^m\text{Pr}[A=a\mid X,S=s]\text{Pr}[S=s\mid X,R=1]$, and modeled the probability of treatment in each trial $\text{Pr}[A=a\mid X,S=s]$ using logistic regression (separately in each trial) and the probability of participation in a particular trial among the collection of trials $\text{Pr}[S=s\mid X,R=1]$ using a multinomial logistic regression model (in the collection of trials). We refer to this approach as “averaging the trial-specific treatment probabilities.” For comparison, in the Appendix, we report results using a (misspecified) logistic regression model with main effects only. In our simulation setup, when the randomization ratio varies across trials, use of this model is expected to have some bias when using ${\widehat{\psi }}_{\text{w}}\left(a\right)$, should not affect ${\widehat{\psi }}_{\text{g}}\left(a\right)$ at all, and should only affect the precision, but not induce bias for ${\widehat{\psi }}_{\text{aug}}\left(a\right)$, when the outcome model is correctly specified. Outcome models were fit separately in each treatment group using linear regression to allow effect modification by all covariates.

Comparisons:

We compared the bias and variance of estimators over 10,000 runs for each scenario and each treatment.

6.3. Simulation results

Tables 1 and 2 present selected results from our simulation study for scenarios with $n=\mathrm{10,000}$ and total trial sample size, ${\sum }_{s=1}^3{n}_s$, of 2000 individuals, when averaging the trial-specific treatment probabilities; complete simulation results are presented in Web Appendix F, including results for modeling the probability of treatment using logistic regression across all trials. Bias estimates for ${\widehat{\psi }}_{\text{aug}}\left(a\right)$ and ${\widehat{\psi }}_{\text{g}}\left(a\right)$ were near-zero for all estimators, regardless of how the probability of treatment was modeled; ${\widehat{\psi }}_{\text{w}}\left(a\right)$ also had near-zero bias except when the randomization ratio varied across trials and the probability of treatment was modeled using logistic regression fit across all trials (see Appendix Table S3). The low bias of the g-formula estimator ${\widehat{\psi }}_{\text{g}}\left(a\right)$ and the weighting estimator ${\widehat{\psi }}_{\text{w}}\left(a\right)$ (when the model for the probability of treatment was correctly specified) indirectly verify the double robustness property of the augmented estimator ${\widehat{\psi }}_{\text{aug}}\left(a\right)$; this is also supported by the near-zero bias of ${\widehat{\psi }}_{\text{aug}}\left(a\right)$ even when the randomization ratio varied across trials and the model for the probability of treatment was misspecified (Appendix Table S3). Across all scenarios, the sampling variance of the augmented estimator ${\widehat{\psi }}_{\text{aug}}\left(a\right)$ was larger than that of the outcome model-based estimator ${\widehat{\psi }}_{\text{g}}\left(a\right)$, but substantially smaller than that of the weighting estimator ${\widehat{\psi }}_{\text{w}}\left(a\right)$.

Table 1:

Bias estimates based on 10,000 simulation runs; selected sample size scenarios; the probability of treatment in the collection of trials was estimated by averaging the trial-specific treatment probabilities.

$\mathit{a}$	$\mathit{n}$	${\sum }_{\mathit{s}=1}^3{\mathit{n}}_{\mathit{s}}$	Balanced	TxAM varies	${\widehat{\mathit{\psi }}}_{\text{aug}}\left(\mathit{a}\right)$	${\widehat{\mathit{\psi }}}_{\mathbf{g}}\left(\mathit{a}\right)$	${\widehat{\mathit{\psi }}}_{\mathbf{w}}\left(\mathit{a}\right)$
1	10000	2000	Yes	No	0.0022	0.0003	−0.0011
1	10000	2000	Yes	Yes	0.0004	−0.0003	−0.0026
1	10000	2000	No	No	−0.0006	−0.0001	−0.0082
1	10000	2000	No	Yes	0.0011	0.0002	−0.0067
1	100000	2000	Yes	No	−0.0003	0.0002	−0.0104
1	100000	2000	Yes	Yes	−0.0002	0.0008	0.0102
1	100000	2000	No	No	−0.0012	−0.0002	−0.0074
1	100000	2000	No	Yes	0.0009	0.0013	0.0039
0	10000	2000	Yes	No	0.0001	0.0008	0.0083
0	10000	2000	Yes	Yes	0.0006	0.0004	0.0060
0	10000	2000	No	No	0.0010	−0.0003	0.0002
0	10000	2000	No	Yes	0.0002	0.0005	0.0106
0	100000	2000	Yes	No	0.0013	0.0005	0.0016
0	100000	2000	Yes	Yes	−0.0009	−0.0000	0.0034
0	100000	2000	No	No	−0.0009	−0.0003	0.0064
0	100000	2000	No	Yes	0.0012	0.0006	0.0109

In the column titled Balanced, Yes denotes scenarios in which the trials had on average equal sample sizes; No denotes scenarios with unequal trial sample sizes. In the column titled TxAM varies, Yes denotes scenarios in which the treatment assignment mechanism varied across trials; No denotes scenarios in which the mechanism did not vary.

Table 2:

Variance estimates based on 10,000 simulation runs; the probability of treatment in the collection of trials was estimated by averaging the trial-specific treatment probabilities.

$\mathit{a}$	$\mathit{n}$	${\sum }_{\mathit{s}=1}^3{\mathit{n}}_{\mathit{s}}$	Balanced	TxAM varies	${\widehat{\mathit{\psi }}}_{\text{aug}}\left(\mathit{a}\right)$	${\widehat{\mathit{\psi }}}_{\mathbf{g}}\left(\mathit{a}\right)$	${\widehat{\mathit{\psi }}}_{\mathbf{w}}\left(\mathit{a}\right)$
1	10000	2000	Yes	No	0.0105	0.0038	0.2893
1	10000	2000	Yes	Yes	0.0084	0.0035	0.1945
1	10000	2000	No	No	0.0101	0.0038	0.2167
1	10000	2000	No	Yes	0.0088	0.0038	0.1925
1	100000	2000	Yes	No	0.0144	0.0039	0.3316
1	100000	2000	Yes	Yes	0.0120	0.0035	0.3610
1	100000	2000	No	No	0.0143	0.0039	0.4028
1	100000	2000	No	Yes	0.0128	0.0037	0.2925
0	10000	2000	Yes	No	0.0103	0.0039	0.1165
0	10000	2000	Yes	Yes	0.0131	0.0042	0.1509
0	10000	2000	No	No	0.0101	0.0039	0.1136
0	10000	2000	No	Yes	0.0118	0.0040	0.1551
0	100000	2000	Yes	No	0.0151	0.0039	0.1873
0	100000	2000	Yes	Yes	0.0198	0.0043	0.2699
0	100000	2000	No	No	0.0146	0.0038	0.1663
0	100000	2000	No	Yes	0.0218	0.0040	0.3243

In the column titled Balanced, Yes denotes scenarios in which the trials had on average equal sample sizes; No denotes scenarios with unequal trial sample sizes. In the column titled TxAM varies, Yes denotes scenarios in which the treatment assignment mechanism varied across trials; No denotes scenarios in which the mechanism did not vary.

7. Application of the methods to the HALT-C trial

7.1. Using the HALT-C trial to emulate meta-analyses

The HALT-C trial data:

The HALT-C trial (Di Bisceglie et al., 2008) enrolled 1050 patients with chronic hepatitis C and advanced fibrosis who had not responded to previous therapy and randomized them to treatment with peginterferon alfa-2a $(a=1)$ versus no treatment $(a=0)$. Patients were enrolled at 10 research centers and followed up every 3 months after randomization. Here, we used the secondary outcome of platelet count at 9 months of follow-up as the outcome of interest; we report all outcome measurements as platelets×10³/ml. We used the following baseline covariates: baseline platelet count, age, sex, previous use of pegylated interferon, race, white blood cell count, history of injected recreational drugs, ever received a transfusion, body mass index, creatine levels, smoking status, previous use of combination therapy (interferon and ribavirin), diabetes, serum ferritin, hemoglobin, aspartate aminotransferase levels, ultrasound evidence of splenomegaly, and ever drinking alcohol. For simplicity, we restricted our analyses to patients with complete baseline covariate and outcome data $(n=948)$.

Using the HALT-C trial to emulate a meta-analysis and evaluate the proposed methods:

We treated observations from the largest center with complete data in the HALT-C trial, with sample size $n_0=199$, as a sample from the target population, $S=R=0$. We then used the data from the remaining 9 centers as a collection of “trials,” $𝒮=\{1,\dots ,9\}$, with a total sample size of ${\sum }_{j=1}^9{n}_j=749$. Appendix Table S5 summarizes covariate information, stratified by $R$; Appendix Tables S6 and S7 summarize covariate information stratified by S. Because HALT-C used 1:1 randomized allocation, all trials had the same treatment assignment mechanism. Our goal was to transport causal inferences from the nine trials to the population represented by the target center. We were able to use the randomization in $R=0$ to empirically benchmark the methods we propose using as a reference standard the unadjusted estimate of the average treatment effect in the target population. In view of how we configured the data for this illustrative analysis, our results should not be clinically interpreted.

Methods implemented and comparisons:

We implemented the augmented estimator ${\widehat{\psi }}_{\text{aug}}\left(a\right)$, along with the g-formula estimator ${\widehat{\psi }}_{\text{g}}\left(a\right)$ and the weighting estimator ${\widehat{\psi }}_{\text{w}}\left(a\right)$, defined in Section 6.2. These estimators use covariate, treatment, and outcome data from the collection of trials $𝒮$ but only covariate data from the sample of the target population $R=0$. In these analyses we specified linear regression models for the conditional expectation of the outcome in each treatment group, and logistic regression models for the probability of trial participation (in any trial) and the probability of treatment among randomized individuals. All models included main effects for the baseline covariates listed above. For comparison, we also used an unadjusted linear regression analysis to estimate the potential outcome means and average treatment effects in $R=0$ only; this analysis is justified by randomization and uses treatment and outcome data from the target population.

To provide an assessment of the observed data independence condition in equation (5), we used analysis of covariance (ANCOVA) to compare a linear regression model that included the main effects of baseline covariates, treatment, and all possible product terms between baseline covariates and treatment, against a linear regression model that additionally included the main effect of the trial indicators and all product terms between baseline covariates, treatment, and the trial indicators.

7.2. Results in the emulated meta-analysis

Table 3 summarizes results from the emulation of meta-analyses using HALT-C trial data. All transportability estimators, which only use baseline covariate data from $R=0$ and covariate, treatment, and outcome data from $R=1$, produced estimates that were very similar to the benchmark estimates from the unadjusted estimator that uses treatment and outcome data from the target center $R=0$. The three transportability estimators produced similar point estimates, which suggests that the transported inferences are not driven by model specification choices. The three transportability estimators also produced estimates similar to those of the benchmark unadjusted estimator in $R=0$, which suggests (but does not prove) that the identifiability conditions needed for the different transportability estimators hold, at least approximately, and that model specification is approximately correct. The ANCOVA p-value of 0.425 did not indicate gross violations of the observed data implication in display (5).

Table 3:

Results from analyses using the HALT-C trial data.

Estimator	$\mathit{a}=1$	$\mathit{a}=0$	Mean difference
$R=0$ (benchmark)	121.6 (111.0, 133.3)	164.4 (151.2, 178.1)	−42.8 (−60.2, −25.7)
${\widehat{\psi }}_{\text{aug}}\left(a\right)$	124.5 (116.3, 133.5)	167.1 (157.3, 177.7)	−42.6 (−51.2, −34.4)
${\widehat{\psi }}_{\text{g}}\left(a\right)$	124.6 (116.6, 133.0)	167.1 (157.3, 177.3)	−42.5 (−50.5, −34.9)
${\widehat{\psi }}_{\text{w}}\left(a\right)$	123.2 (113.6, 136.0)	165.9 (152.6, 182.9)	−42.7 (−60.1, −26.2)

$R=0$ denotes analyses using data only from the sample of the target population. Numbers in parentheses are 95% quantile-based confidence intervals from 10,000 bootstrap samples; 95% normal confidence intervals from the sandwich estimator were similar.

8. Discussion

We have described methods to transport causal inferences from a collection of randomized trials to a target population in which baseline covariate data are available but no experimentation has been conducted. Our results provide a solution to one of the leading problems in the field of “evidence synthesis”: how to use a body of evidence consisting of multiple randomized trials to draw causal inferences about the effects of the interventions investigated in the trials for a new target population of substantive interest.

Traditional approaches to evidence synthesis, including meta-analysis, focus on modeling aspects of the distribution of study effects or producing statistically optimal summaries of available data (Higgins et al., 2009; Rice et al., 2018). In general, such analyses do not produce causally interpretable estimates of the effect of well-defined interventions on any target population because the contributions of individual studies to the meta-analysis summary are weighted by the precision of the study-specific estimates, without considering the relevance of the studies to any target population or the strong assumptions needed to combine information across studies to estimate treatment effects.

In contrast, our approach explicitly targets a well-defined population chosen on scientific or policy grounds, which may be different from the populations underlying the trials. In our experience, policy-makers who use evidence syntheses to inform their decisions are not interested in the populations represented by the completed trials and almost always have a different target population in mind. Thus, the statistical methods we propose can be viewed as a form of causally interpretable meta-analysis, consistent with the conceptual framework outlined in (Sobel et al., 2017). An important aspect of our approach is the specification of the target population and the use of sample data from that population to estimate treatment effects. In practical applications, obtaining data from a target population can be challenging (Barker et al., 2021), but we expect that challenge will become easier to address with increasing availability of administrative and registry data from populations of clinical or policy relevance. Another important aspect of our approach is the explicit statement of the assumptions needed for the causal interpretation of estimates produced by the estimators we propose. Though the assumptions are fairly strong even for the simple case we study here (point treatments with complete adherence and outcomes assessed at the end of the study), our approach naturally connect with the broad literature on causal inference and thus extensions to address time varying treatments, incomplete adherence, and longitudinal or failure-time outcomes are possible. We believe that making assumptions explicit allows reasoning about them and can be the basis of future work on sensitivity analyses. Future work may also address differential covariate measurement error in the randomized trials and the sample of the target population, systematically missing data (e.g., when some covariates are not collected in some trials), and variable selection methods (e.g., methods incorporating the independence in display (4)).

Our methods can are a generalization of methods for extending inferences from a single trial to a target population (e.g., Cole and Stuart (2010); Dahabreh et al. (2018, 2020); Rudolph and van der Laan (2017)). They also relate to the growing literature on matching-adjusted indirect comparison (e.g., Signorovitch et al. (2010, 2012); Cheng et al. (2020); Jackson et al. (2021)). This literature has focused on the common situation where data on the target population are only available in summary form (e.g., estimates of the first moments of the covariates in $X$), typically does not use explicit causal (counterfactual) notation (with some recent exceptions, e.g., Cheng et al. (2020)), relies heavily on parametric assumptions (or other investigator-imposed constraints), and only applies to a single source trial (not multiple trials in meta-analysis) (Jackson et al., 2021). Our identification results suggest obvious extensions of matching-adjusted indirect comparison methods to the multiple trial case, provided investigators are willing to make the necessary causal assumptions. Conversely, methods from the literature on matching-adjusted indirect comparison and related matching methods (Jackson et al., 2021) can be used to relax the requirement of individual participant data from the target population inherent in our approach (at the cost of reliance on additional parametric assumptions or reduced efficiency). It may also be interesting to combine our approach with recent extensions of meta-regression methods to incorporate matching-adjusted indirect comparisons (Phillippo et al., 2020).

We caution that our methods will be most useful when the available trials are quite similar (e.g., in terms of treatments, outcomes, and follow-up protocols), as may be the case in prospective research programs investigating a particular intervention or close replication efforts. It should come as no surprise that a strong degree of similarity is needed in order for a “pooled” analysis to produce estimates with a clear causal interpretation with respect to a particular target population; after all, our approach focuses on differences in participant characteristics across the populations underlying the trials and the target population, not variations in treatment. Extensions of our approach to address variations in the randomized treatments, as well as non-randomized co-interventions, would be valuable. We suggest, however, that summarizing bodies of evidence much larger and diverse than a few closely related studies, particularly when individual participant data are only available from a minority of sources, is best undertaken with a descriptive or predictive attitude, focusing on modeling the response surface of study effects (Rubin, 1992), rather than on obtaining a single causally interpretable estimate. That said, we sincerely hope that future work will find ways to improve on this rather unambitious view.