Estimating target population treatment effects in meta-analysis with individual participant-level data

Abstract

Meta-analysis of randomized controlled trials (RCTs) is commonly used to evaluate treatments and inform policy decisions because it provides comprehensive summaries of all available evidence. However, meta-analyses are limited to draw population inference of treatment effects because they usually do not define target populations of interest specifically, and results of the individual RCTs in those meta-analyses may not generalize to the target populations. To leverage evidence from multiple RCTs in the generalizability context, we bridge the ideas from meta-analysis and causal inference. We integrate meta-analysis with causal inference approaches estimating target population average treatment effect. We evaluate the performance of the methods via simulation studies and apply the methods to generalize meta-analysis results from RCTs of treatments on schizophrenia to adults with schizophrenia who present to usual care settings in the United States. Our simulation results show that all methods perform comparably and well across different settings. The data analysis results show that the treatment effect in the target population is meaningful, although the effect size is smaller than the sample average treatment effect. We recommend applying multiple methods and comparing the results to ensure robustness, rather than relying on a single method.

Introduction

Evidence-based practice is commonly used to evaluate treatments or interventions by synthesizing results from multiple clinical trials and such data synthesis provides the best summaries of the available evidence¹. A meta-analysis is frequently used to combine multiple randomized controlled trials (RCTs), and it helps gain more power and provide reliable and precise effect estimates². However, meta-analysis is limited to draw population inference of treatment effects because it does not consider how well the RCTs represent a target population of interest. To leverage evidence from multiple RCTs in the generalizability context, developing methods that calibrate meta-analysis in a proper way in order to draw population-level inference is timely and needed³.

A few methods that estimate the Target Population Average Treatment Effect (TATE) with a single RCT have been developed by using a causal inference framework. Similar to the concept of propensity score weighting approaches in non-experimental studies, weighting approaches were implemented to make individuals in an RCT resemble a target population^4,5. In addition, Kern et al.⁶ proposed using Bayesian Additive Regression Trees (BART), a flexible machine learning approach predicting potential outcomes, to estimate the TATE. Furthermore, targeted maximum likelihood estimation (TMLE) approaches were adapted to the generalizability context^7,8.

However, these causal inference methods need to be extended and integrated with meta-analysis to leverage multiple RCTs to estimate the TATE comprehensively and accurately. In many settings a single RCT may not reflect the target population well, but there are often multiple RCTs, each representing somewhat different samples, which together can be combined to better estimate the TATE.

In the generalizability with meta-analysis context, our primary focus is on using statistical methods to assess and account for heterogeneity that can be modeled given available data sources. Heterogeneity can arise from multiple factors, including: (1) known study design attributes, such as inconsistent outcome definitions, variations in covariate measurement, or differences in study sample eligibility criteria and (2) unknown sources related to the underlying sampling mechanism. Generally, known sources of heterogeneity can be managed by ensuring that outcomes and covariates are measured consistently across all data sources and that the RCTs included in the meta-analysis have comparable eligibility criteria, making their samples reasonably homogeneous, as assumed in our work. Once this level of consistency is achieved, statistical methods can then be applied to account for residual heterogeneity stemming from the underlying sampling mechanism. Specifically, in the generalizability framework, such heterogeneity refers to differences in treatment effects due to variations in the characteristics of individuals between the RCTs and the target population. In the meta-analysis framework, it refers to differences in treatment effects across multiple RCTs due to variations in sample characteristics.

In this paper, we propose methods to help generalize the results of meta-analyses and estimate the TATE by integrating these causal inference approaches in meta-analyses. Our methods will help translate results into appropriate decisions regarding broad implementation and clinical practice using combining information from multiple studies.

The remainder of this paper is structured as follows. First, Section introduces the required data structure and key model assumptions and Section provides details of meta-analysis model integrated with causal inference methods. Section reports the settings and results of our extensive simulation study. Then we apply all methods to a real data example and Section presents the results. Finally, Section discusses our work and unmet methodological challenges.

Background and assumptions

Observed data structure and notation

Figure 1 (a) exhibits the underlying data generating mechanism that we assume in this paper. The observed target population, $\Omega$, is assumed to be a simple random sample from a true target population, ${\Omega }^{*}$, where the true target population is not directly observed. We assume that the observed target population represents the true target population well. For brevity, the target population stands for the observed target population, unless otherwise specified. Each RCT is assumed to be a sample from the true target population. In practice, the RCT sampling mechanism may not be random because of its restricted eligibility criteria, limited recruitment procedure, and ethical considerations. In this case, an RCT sample may not perfectly represent the true target population.

Figure 1.

Illustrations of (a) underlying data generating mechanism and (b) observed data structure. In (a), we denote a true hypothetical target population by ${\Omega }^{*}$, a dataset representing that target population by $\Omega$, and multiple RCTs by $R_1,\dots ,R_M$, where $M$ is the total number of RCTs. In (b), $S_{ij}$ is an indicator of population membership, $\mathbf{\text{X}}$ is a set of covariates, $A$ is the treatment assignment indicator, and $Y$ is the outcome.

Figure 1 (b) shows the structure of the observed data that we consider in this paper. The target population of interest $\Omega$ has a sample size of $n_{\Omega }$ and RCTs are denoted by $R_1,\dots ,R_M$ with corresponding sample sizes $n_1,\dots ,n_M$, where $M$ is the total number of RCTs. We use $i$ and $j$ to index studies and individuals, respectively. We define $S_{ij}$ as the indicator of population membership: $S_{ij}=1$ for individuals in the target population and $S_{ij}=0$ for individuals in one of the RCTs. We consider the case where the RCT samples and the target population are disjoint (i.e., an individual is in only one of the data sources – either one RCT or the target population data). Both RCTs and the target population data include a set of covariates X that are observed and measured consistently across all RCTs and population data. On the other hand, we observe the treatment assignment $A_{ij}$, where 1 for the treated and 0 for the untreated (i.e., control), and the outcome $Y_{ij}$ only in the RCTs. This data structure is common in medicine and public health, especially when studying new interventions not yet widely available and thus not yet observed in population datasets such as electronic health records, national surveys, or other administrative data sources.

Estimand

The estimand of interest is the Target Population Average Treatment Effect (TATE). To define the TATE formally, we use the potential outcomes, $Y_{ij}\left(A_{ij}=a\right)$ for the $j^{\mathit{\text{th}}}$ individual in the $i^{\mathit{\text{th}}}$ dataset under the treatment (A=1) or control (A=0). The TATE is defined as comparison of potential outcomes between treatment groups in the target population,

$$\frac{1}{n_{\Omega }}\sum _{j=1}^{n_{\Omega }}\left(Y_{\Omega j}(1)-Y_{\Omega j}(0)\right).$$ (1)

However, we cannot directly estimate the TATE because we do not observe any outcomes in $\Omega$. Therefore, the goal of the proposed methods is to estimate the TATE using the observed target population and RCTs.

As a side note, the TATE should be defined using the entire true target population ${\Omega }^{*}$ by replacing $\Omega$ with ${\Omega }^{*}$ in (1). However, we assume that inferences drawn from $\Omega$ accurately reflect inferences drawn from ${\Omega }^{*}$ as we assumed above that $\Omega$ is a simple random sample from ${\Omega }^{*}$. That is, we do not need to consider adjustment for the sampling process between $\Omega$ and ${\Omega }^{*}$. If $\Omega$ is not a simple random sample of ${\Omega }^{*}$ and the sampling process is known or estimable, additional adjustments (e.g., weighting) could be implemented to create a pseudo target population (e.g., weighted $\Omega$) that represents ${\Omega }^{*}$, but for simplicity we do not consider this case in this paper. In addition, we consider a continuous outcome and define the treatment effect on the mean difference scale; the risk difference, odds ratio, or relative risk can be used for binary or count outcomes.

The RCT samples observe outcomes and we can estimate the Sample Average Treatment Effect (SATE) in the $i^{\mathit{\text{th}}}$ RCT, defined as

$$SAT{E}_i=\frac{1}{n_i}\sum _{j=1}^{n_i}\left(Y_{ij}(1)-Y_{ij}(0)\right),$$ (2)

Each RCT provides an unbiased estimate of the SATE for the sample in that trial due to randomization (known as “internal validity”)⁹. However, each RCT does not necessarily provide an unbiased estimate of the TATE, if the RCT sample and the target population differ on covariates related to treatment effects (i.e., effect moderators). Such imbalance of the covariates between the RCT and the target population may lead to a biased estimate of the TATE (analogous to confounders which are associated with both the treatment assignment and the outcome and lead to bias when estimating causal effects in non-experimental studies). SATE is an unbiased estimate of the TATE if (1) the RCT sample is a random sample of the target population, and/or (2) there are no effect moderators, and/or (3) the RCT sample differs from the population, but only on variables unrelated to effects (non-moderators)⁵. However, it is unrealistic to expect this to be the case in practice, and thus understanding the amount of effect moderation and comparability of participants’ characteristics between RCTs and population is important when estimating the TATE.

In practice, meta-analysis usually pools $SAT{E}_i$ and concludes that the resulting estimate is the ultimate finding about the treatment effect. However, strictly speaking, the meta-analysis results are only applicable within the RCT samples and the inferences may or may not be stretched to a target population. In addition, if each of the RCTs represents a somewhat different population, the meta-analysis results may not reflect a well-defined common target population. In this case, calibrating each of the RCT samples to make them resemble a pre-specified target population helps ensure that the meta-analysis is estimating a common estimand, the TATE. Therefore, integrating meta-analysis with the methods for generalizing results of an RCT will complement both methods and provide accurate and reliable inferences about population treatment effects.

Assumptions

To identify and estimate the TATE by leveraging the information we obtained from the RCTs, several assumptions are required between the target population and each of the RCTs. Our methods described in Section estimate the TATE using data from each of the RCTs included in a meta-analysis. Therefore, the following assumptions need to hold for each pair of an RCT and population data.

Positivity.

The distribution of baseline characteristics in the $i^{\mathit{\text{th}}}$ RCT and the target population need to overlap with respect to all level of effect moderators in the target population, $0<\mathit{\text{Pr}}\left(S_{ij}=0\mid {\mathbf{\text{X}}}_{\Omega }\right)\le 1,\forall x\in {\mathbf{\text{X}}}_{\Omega }$. This implies that every category or value of each effect moderator must be represented in both the RCTs and the target population.

Sample ignorability for treatment effects.

The distribution of treatment effects is the same in the $i^{\mathit{\text{th}}}$ RCT and the target population given the observed covariates. That is, the sample selection is independent of the treatment effect, $S_{ij}⫫Y_{ij}(1)-Y_{ij}(0)\mid {\mathbf{\text{X}}}_{ij}$ for all $i$⁶. This implies that effect moderators that predict the sample selection are the variables that need to be adjusted for or calibrated when estimating TATE.

Random treatment assignment.

In each RCT, treatment assignment is random and independent of sample selection given covariates, $A_{ij}⫫S_{ij},Y_{ij}\mid {\mathbf{\text{X}}}_{ij}$ for all $i$¹⁰. This is generally satisfied in an RCT.

Consistency.

Each of the potential outcomes of the $j^{\mathit{\text{th}}}$ subject in the $i^{\mathit{\text{th}}}$ study under a given treatment is the same as the observed outcome if individual $j$ receives that treatment, $Y_{ij}=\left(1-A_{ij}\right)Y_{ij}(0)+A_{ij}{Y}_{ij}(1)$.

Trial outcome consistency.

There is only one version of each treatment across all RCTs and each participant has only one version of potential outcomes for $a=0\text{and}1$ no matter which RCT the participant is in.

No interference.

The treatment assignment in an RCT does not affect the potential outcomes of individuals in the other studies¹¹. In addition, within an RCT one person’s treatment assignment does not influence another person’s potential outcomes.

The last two assumptions are related to the stable unit treatment value assumption (SUTVA)¹².

Two-stage meta-analysis with individual participant data

We consider a two-stage meta-analysis model to analyze IPD from multiple RCTs¹³. In the first stage, we estimate TATE in the $i^{\mathit{\text{th}}}$ RCT, denoted by $TAT{E}_i$. Then, the second stage pools the study-specific TATE estimates accounting for variability of TATEs across RCTs.

First stage: Estimating TATE from each RCT

The first stage is to analyze each RCT separately using the same outcome model. In the context of generalizability, the first stage estimates a TATE using a dataset pairing the $i^{\mathit{\text{th}}}$ RCT and the target population, created by essentially stacking the two datasets on top of one another. We consider five different methods: three weighting approaches and two outcome prediction approaches as described in the following sections. Note that we ignore the $i$ subscript indicating RCTs because the same methods are applied to all RCTs.

Weighting approaches

An unbiased estimate of TATE can be obtained when an RCT is representative of the target population. However, it is common that patients’ characteristics in the RCT are not comparable with those in the target population. In this case, weighting approaches can help equate the characteristics between the weighted RCT and the target population^4,5. Then, TATE is obtained as a weighted treatment effect estimated using the RCT with the weights, analogous to Horvitz-Thompson weighting method often used in sample surveys¹⁴. We denote these weights using the term selection weights.

Using the stacked dataset of an RCT and the target population, we assign a population indicator $S_j=1$ for the participants in the target population and 0 for the participants in the RCT, following the notation used in Kern et al.⁶. The selection weight is calculated by

$$w_j=\left\{\begin{array}{ll}\frac{{\widehat{e}}_j}{1-{\widehat{e}}_j}& \text{if}{S}_j=0\\ 0& \text{if}{S}_j=1,\end{array}\right.$$ (3)

where $e_j=P\left(S_j=1\mid {\mathbf{\text{X}}}_j\right)$ is the selection score defined as the probability of being in the target population given observed covariates.

Note that outcomes among participants in the target population are not available so the target population data are only used for estimating the selection score and weight, using the observed covariates. These weights are then used in subsequent analyses of the weighted regressions using the RCT data. We fit the same model to each RCT included in the meta-analysis and collect study-specific TATEs and their standard errors.

The selection score, $e_j=P\left(S_j=1\mid {\mathbf{\text{X}}}_j\right)$, is estimated by fitting a selection model. We consider three models: logistic regression, generalized boosted model (GBM), and super learner models. First, the logistic regression selection model is written as $\mathit{\text{logit}}\left(e_j\left({\mathbf{\text{x}}}_j\right)\right)={\mathbf{\text{X}}}_j\mathit{\beta }$. Here, we consider only first-order terms for the covariates, but higher-order terms (e.g., age-squared) can be included. Second, GBM estimates the selection probability by fitting a piecewise constant model, where the constant function is learned by fitting regression trees iteratively^15,16. As such, GBM allows for flexible non-linear associations between the population membership and covariates, and often performs better than a simple logistic regression model¹⁷. Finally, a Super Learner model is used to estimate the selection probability¹⁸. The Super Learner model uses an ensemble technique that merges results from multiple candidate learners so that the ensemble results minimize a prediction error. A library of candidate learners can be specified by researchers. We choose the library proposed by Luedtke and van der Laan¹⁹ based on the findings in Schmid et al.²⁰. The library consists of five learners including logistic regression, logistic regression with two-way interactions, generalized additive model, neural network, and recursive partitioning. In our simulation study and data analysis, we had to modify the library by dropping the last two learners because these two methods either failed to converge numerically, producing no results, or generated numerically unreliable estimates that were significantly different from the true selection score in the simulation study and data application. The unstable performance of these methods is due to the neural network typically requiring a large set of covariates and observations, which our application did not have, and the recursive partitioning method being more suited for clustering rather than estimating selection scores.

After calculating weights in (3), a weighted regression incorporating survey weights is conducted to estimate $TAT{E}_i$ and its standard error⁶.

Assessment of covariate balance between RCTs and target population

To assess the similarity of the distributions of covariates between RCTs and the target population, we employ absolute standardized mean differences (SMDs) defined formally in Appendix A. We compare absolute SMDs calculated before and after weighting and assess how much the weighing approach improve the covariate balance. An absolute SMD less than 0.1 is indicative of good balance in covariates between an RCT and the target population^21,22. We first calculate absolute SMDs for each covariate when comparing each RCT to the population. Then, we summarized the absolute SMDs in two ways: (1) absolute SMD for each covariate averaging over RCTs, and (2) absolute SMD for each RCT averaging over covariates. The first absolute SMD shows which covariates achieve the desired level of similarity to the population after weighting; and the second absolute SMD shows which RCTs struggle with generalizability.

Outcome prediction approaches

We also consider outcome prediction approaches that directly model $E\left(Y_j\mid A_j,X_j\right)$ and then predict the potential outcomes of participants in the target population. The TATE can be estimated by the predicted potential outcomes such as $E\left(Y_j(1)\mid S_j=1\right)-E\left(Y_j(0)\mid S_j=1\right)$. Furthermore, outcome prediction can be also integrated with the selection weights. We apply two outcome prediction approaches: one ignoring selection weights, Bayesian Additive Regression Trees (BART), and the other one integrating selection weights, targeted maximum likelihood estimation (TMLE). BART and TMLE is applied to the previously described stacked dataset of an RCT and the target population.

BART is a flexible and robust Bayesian nonparametric method which is a “sum-of-trees” model allowing multivariate features including all possible interactions²³. BART generates a binary tree by fitting a model such as $Y_j=E\left[Y_j\mid A_j,X_j\right]+{ϵ}_j$, where ${ϵ}_j~N\left(0,{\sigma }_j^2\right)$ and $E\left[Y_j\mid A_j,X_j\right]=\sum _{h=1}^{V}g\left(a_j,x_j;T_h,M_h\right)$. Here, each pair of $\left(T_h,M_h\right)$ is the fit from a single bottom node of the binary tree, where $T_h$ is the $h^{th}$ tree, $M_j$ is the mean of the terminal node from the $h^{th}$ tree, and $V$ is the total number of trees. BART has been adapted in the causal inference context to detect treatment effect heterogeneity²⁴ and estimate the TATE⁶. In the context of generalizability, BART predicts potential outcomes for participants in the population data based on the relationship between covariates and the outcome observed in the RCT. Then, we estimate the treatment effect for each individual in the target population by calculating $d\left(x_j\right)=Y_j^{*}\left(1,x_j\right)-Y_j^{*}\left(0,x_j\right)$, where the asterisk indicates a predicted outcome, and the TATE is estimated by averaging $d\left(x_j\right)$ across individuals in the target population. Using a Markov chain Monte Carlo method, the BART model provides a posterior distribution of the TATE and its mean and standard deviation are used to estimate $TAT{E}_i$ and its standard error.

TMLE is a semi-parametric model that has been used to predict potential outcomes for causal inference²⁵ and has been applied to the generalizability context^7,8. TMLE often delivers robustness to model misspecification and efficiency⁸. Distinct from BART, TMLE uses both selection and outcome models, where the selection model is used to account for residual selection bias in the predicted outcomes. Rudolph et al.⁸ introduced the TMLE estimator for TATE, and it is obtained as follows. First, the potential outcomes in the target population, ${\widehat{Y}}_j(a)$, is predicted using the conditional expectation of the outcome estimated in the RCT. Second, the selection probability $e_j$ is estimated as described in Section. Third, the so-called “clever covariates” is calculated as

$$H_{aj}=\frac{I\left(A_j=a\right)P\left(S_j=1\mid {\mathbf{\text{X}}}_j\right)}{P\left(A_j=a\right)*P\left(S_j=0\mid {\mathbf{\text{X}}}_j\right)P\left(S_j=1\right)},$$

where $P\left(A_j=a\right)=0.5$ because random treatment assignment in the RCT and $P\left(S_j=1\mid {\mathbf{\text{X}}}_j\right)=e_j$. The initial predicted potential outcome in the first step, ${\widehat{Y}}_j(a)$, is updated by $\left.g\left(Y_j^{*}(a)\right)=g\left({\widehat{Y}}_j(a)\right)\right)+{\widehat{ϵ}}_j{H}_{aj}$, where $g(\cdot )$ is the appropriate link function (e.g., the identity link function with a continuous outcome). Here, ${\widehat{ϵ}}_j$ is estimated by regressing $Y_j$ in the RCT on $H_{aj}$ with offset of $g\left({\widehat{Y}}_j(a)\right)$ using the RCT. The TATE is estimated as the mean difference between $Y_j^{*}(1)$ and $Y_j^{*}(0)$ for individuals in the target population and the variance of the estimator is estimated by a closed form using the efficient influence curve^8,26.

Second stage: Pooling study specific TATEs

The second stage is to meta-analyze the study-specific TATE estimates and the associated standard errors obtained in the first stage, ${\widehat{\mathit{\text{TATE}}}}_i$ and $\widehat{\mathit{\text{se}}}\left({\widehat{\mathit{\text{TATE}}}}_i\right)$, respectively. We consider a widely-used meta-analysis method known as the inverse variance method under the assumption of variability of the treatment effects across RCTs, commonly referred to as a random effects meta-analysis². First, we need to calculate a study-specific weight for meta-analysis. This weight is a function of within-study variability (i.e., $\widehat{\mathit{\text{se}}}\left({\widehat{\mathit{\text{TATE}}}}_i\right)$) and between-study variability (i.e., between-RCT variability of ${\widehat{\mathit{\text{TATE}}}}_i$, denoted by ${\tau }^2$). The meta-analysis weight is defined as

$$w_i^{*}=\frac{1}{\frac{1}{{\widehat{\mathit{\text{se}}}\left({\widehat{\mathit{\text{TATE}}}}_i\right)}^2}+{\tau }^2},$$

where ${\tau }^2$ is estimated as proposed by DerSimonian and Laird². Then, the inverse variance meta-analysis estimator of TATE and its variance are defined as

$$\widehat{\mathit{\text{TATE}}}=\frac{\sum _i{w}_i^{*}{\widehat{\mathit{\text{TATE}}}}_i}{\sum _i{w}_i^{*}}\text{and}\widehat{\mathit{\text{var}}}(\widehat{\mathit{\text{TATE}}})=\frac{1}{\sum _i{w}_i^{*}}.$$

In the second stage, any meta-analysis model that fits the given data well can be used to pool study-specific TATEs. Given that the focus of this paper is on comparing generalizability methods rather than meta-analysis methods, we only consider one meta-analysis approach. However, the DerSimonian and Laird method may be biased when the number of studies is small or the outcome is rare²⁷. In such cases, alternative models, such as Bayesian meta-analysis models, can be applied^28,29. Nonetheless, we expect our findings and conclusions about generalizability in meta-analysis to remain consistent regardless of the meta-analysis model used.

Computation

Simulations and data analyses were conducted in R³⁰. To implement the weighted regressions for weighting approaches, we used the survey package³¹. This package yields a design-based standard error of the TATE. The GBM selection model was fitted with the twang package³² with the average absolute standardized effect size as the statistic optimizing the GBM iteration with stop.method=“es.mean”. The Super Learner selection model was implemented with the SuperLearner package³³. For BART, the BayesTree package³⁴ was used with its default settings, including $V=200\text{and}1000$ draws of Markov chain Monte Carlo samples after discarding 100 burn-in samples. Convergence of the BART model was assessed using trace plots of the posterior samples for the standard deviation of the outcome, as suggested by Sparapani et al.³⁵. We implemented TMLE by modifying the R code provided by Rudolph et al.⁸. When estimating TATE using TMLE, the initial outcome prediction was conducted using a Super Learner model and the selection probability was estimated using a logistic regression model. The meta-analysis was fitted using a user-defined R function, but it can be fitted the metafor package³⁶. The R code used for our simulation study is available at https://github.com/HwanheeHong/MetaAnalysisGeneralizability.

Simulation study

In this simulation, we compare the performance of six meta-analysis methods: unweighted meta-analysis, meta-analyses integrating the three weighting approaches (logistic regression, GBM, and superleaner), and the two outcome prediction approaches (BART and TMLE). Table 1 summarizes key parameter settings for the data generating scenarios. Bias, mean squared error (MSE), and coverage probabilities are calculated as measures of performance for all methods. In addition, for the weighting approaches, we calculate absolute SMD to assess how different selection models balance covariates after weighting.

Table 1.

Outcome and selection models depending on two covariates, $X_1$ and $X_2$, used to generate data under different scenarios in the simulation study. ${\varphi }_1$ and ${\varphi }_2$ are coefficients for effect moderators in the outcome models and ${\gamma }_{1i}$ and ${\gamma }_{2i}$ are coefficients in the selection model corresponding to $X_1$ and $X_2$, respectively.

Scenario	Parameter setup
Outcome Model (OM)
OM1: No effect moderator (EM)	${\varphi }_1=0$ and ${\varphi }_2=0$
OM2: $X_1$ is an EM	${\varphi }_1=1$ and ${\varphi }_2=0$
OM3: $X_2$ is an EM	${\varphi }_1=0$ and ${\varphi }_2=1$
OM4: Both $X_1$ and $X_2$ are EMs	${\varphi }_1=1$ and ${\varphi }_2=1$
OM5: $X_1$ is a stronger EM than $X_2$	${\varphi }_1=2$ and ${\varphi }_2=1$
OM6: $X_2$ is a stronger EM than $X_1$	${\varphi }_1=1$ and ${\varphi }_2=2$
Selection Model (SM)
SM1: RCT $i$ is a random sample	${\gamma }_{1i}=0$ and ${\gamma }_{2i}=0$
SM2: Selection model depends on $X_1$	${\gamma }_{1i}=1$ and ${\gamma }_{2i}=0$
SM3: Selection model depends on $X_2$	${\gamma }_{1i}=0$ and ${\gamma }_{2i}=1$
SM4: Selection model depends on both $X_1$ and $X_2$ equally	${\gamma }_{1i}=1$ and ${\gamma }_{2i}=1$
SM5: Selection model depends more on $X_1$ than $X_2$	${\gamma }_{1i}=2$ and ${\gamma }_{2i}=1$
SM6: Selection model depends more on $X_2$ than $X_1$	${\gamma }_{1i}=1$ and ${\gamma }_{2i}=2$

Setup

In our simulation, we follow the data generating mechanism and observed data structure depicted in Figure 1 with a binary treatment indicator $A$, a continuous outcome $Y$, and two covariates $X_1~N(0,1)$ and $X_2~\mathit{\text{Bernoulli}}(0.5)$.

First, we generate a true target population ${\Omega }^{*}$ with sample size 11500. The true target population data contain $X_1,X_2,Y(0)$, and $Y(1)$. To generate potential outcomes, we use the following outcome models:

$$\begin{array}{lll}{Y}_j(0)& ~& N\left(X_{1j}+X_{2j},1\right)\\ Y_j(1)& ~& N\left(X_{1j}+X_{2j}+\eta +{\varphi }_1{X}_{1j}+{\varphi }_2{X}_{2j},1\right),\end{array}$$

where $j=1,\dots ,11500$ indicates individuals and $\eta +{\varphi }_1{X}_{1j}+{\varphi }_2{X}_{2j}$ is the true conditional TATE for the $j^{\mathit{\text{th}}}$ individual. The true marginal TATE is $\eta +0.5{\varphi }_2$ as $E\left(X_1\right)=0$ and $E\left(X_2\right)=0.5$. We set $\eta =2$. Here, ${\varphi }_1$ and ${\varphi }_2$ are coefficients for effect moderation corresponding to $X_1$ and $X_2$, respectively. We use six different values for the pair of (${\varphi }_1,{\varphi }_2$) to consider different true outcome models, denoted them by “OM” (Table 1). Here, OM1 has no effect moderator, OM2 and OM3 have a single effect moderator, OM4 - OM6 have two effect moderators, where OM4 has an equal-sized moderation effect in the two effect moderators, whereas OM5 and OM6 have one effect moderator is stronger than the other. Note that although $\eta =2$ is set arbitrarily in our simulation, what matters is the relative magnitude between $\eta$ and (${\varphi }_1,{\varphi }_2$). We consider an effect moderator to be strong if its effect size equals the conditional treatment effect $\eta$. Table 1 shows various values of ${\varphi }_1$ and ${\varphi }_2$ used to represent different strengths of the effect moderators.

Second, we randomly sample an observed target population dataset $\Omega$ with $n_{\Omega }=1500$ from ${\Omega }^{*}$. This ensures that the observed target population is a good representation of the true target population. After sampling $\Omega$, we exclude the potential outcomes from $\Omega$ to make $\Omega$ contain covariates only.

Third, we now create a meta-analysis dataset including 5 RCTs ($M=5$), where each RCT has around 300 participants. The 5 RCTs are sampled from ${\Omega }^{*}-\Omega$ with replacement, so that the observed target population and the RCT samples do not overlap. Note that this sampling approach causes a few individuals (approximately 7%) are sampled for more than one RCT. To reduce such overlap, we tried $n_{{\Omega }^{*}}=50000$ and the simulation results were very similar.

The $i^{th}$ RCT is sampled by using the following selection model:

$$\mathit{\text{logit}}\left(\mathit{\text{Pr}}\left(S_{ij}=0\right)\right)={\gamma }_{0i}+{\gamma }_{1i}{X}_{1j}+{\gamma }_{2i}{X}_{2j}.$$

In this selection model, ${\gamma }_{0i}$ is selected to control the sample size of each RCT, and ${\gamma }_{1i}$ and ${\gamma }_{2i}$ control the similarity of the covariate distribution between the RCT and the target population with respect to $X_1$ and $X_2$, respectively. We use six different values for the pair of $\left({\gamma }_{1i},{\gamma }_{2i}\right)$ for different selection models, denoted by “SM” (Table 1). SM1 is a simple random sample selection model while SM2 - SM6 are conditionally random (i.e., probabilistic sampling) selection models, where the contribution of $X_1$ and $X_2$ varies. After obtaining the RCT samples, the treatment group in each RCT is assigned randomly with a 1:1 ratio, and the observed outcome is defined as $Y=AY(1)+(1-A)Y(0)$.

To study scenarios where selected RCTs included in the meta-analysis do not represent the target population well, we apply the simple random sampling selection model (SM1) to different numbers of RCTs, 5, 4, 3, or 0, out of a total of 5 RCTs, denoted by random samples (RS), and then let the remaining RCTs follow other selection models (SM2 - SM6), which depend on $X_1$ and $X_2$. For example, suppose we want to study SM2 with 3 RCTs under simple random sampling. Then, we apply SM1 to 3 RCTs and SM2 to the remaining 2 RCTs, and the 5 RCTs are meta-analyzed. eFigures 1 and 2 visualize distributions of $X_1$ in a simulated dataset under selected scenarios. eFigure 1 presents the combination of OM2 and SM2 with 5, 3, 2, and 0 RCTs under SM1, and shows that the overall level of overlap of $X_1$ between the RCTs and target population reduces from Panel (a) to (d). In addition, eFigure 2 compares the distribution of $X_1$ under different selection models fixing OM2, Panels (b) to (d), with 5 RCTs sampled under SM1 Panel (a). Panel (c) shows the poorest overlap of $X_1$ between the RCTs and target population.

To sum up, we consider 120 simulation scenarios (6 outcome models × 20 meta-analysis data generating scenarios with various selection models). For each scenario, we simulate 1000 datasets.

In the simulation, we expect that the unweighted method provides biased estimates of TATE when there are effect moderators and the effect moderators are associated with the RCT sample selection probabilities, including the scenario combinations of (OM2, OM4, OM5, OM6) × (SM2, SM4, SM5, SM6) and (OM3, OM4, OM5, OM6) × (SM3, SM4, SM5, SM6). In addition, we expect that when the number of RCTs under simple random sampling is smaller, the bias will be greater. Our simulation investigates how much the bias would decrease when the weighting and outcome prediction methods are integrated with meta-analysis.

Results

Figure 2, eFigure 3, and eFigure 4 exhibit bias, mean squared error (MSE), and 95% confidence interval coverage of the true TATE for all combination pairs of OMs and SMs with different numbers of simple random sample RCTs. Note that a positive (or negative) bias means that the method overestimates (or underestimate) the TATE. As bias and MSE showed similar patterns, we focus on interpreting bias and coverage probability in this section.

Figure 2.

Bias of TATE in the simulation study. A positive (or negative) bias means that the method overestimates (or underestimate) the TATE. The different colored characters correspond to various data generating scenarios, where 5, 4, 2, or 0 RCTs in meta-analysis data are randomly sampled from the true target population out of a total of 5 RCTs. OM=outcome model; SM=selection model; RS=random sample.

As expected, the unweighted meta-analysis provided unbiased TATE in three settings: (1) no effect moderator exists (regardless of the selection model) (top row in Figure 2), (2) $X_1$ (or $X_2$) is an effect moderator but it is not associated with the RCT selection probability (OM3 × SM2 and OM2 × SM3), or (3) all 5 RCTs included in a meta-analysis are simple random samples of the true target population (regardless of the outcome or selection models; all red dots show zero bias). In all other cases the unweighted meta-analysis was biased, with larger bias when fewer RCTs were simple random samples of the population. The cases with a single effect moderator (OM2 and OM3) tended to yield less bias than the cases with two effect moderators (OM4 - OM6). Comparing OM2 with OM3 and OM5 with OM6, to study the impact of the type of an effect moderator (continuous vs. binary), a continuous effect moderator tended to provide more bias under the unweighted meta-analysis when the coefficients of moderators were equal.

eFigure 5 shows the average of absolute SMDs for $X_1$ and $X_2$ under different weighting models. In general, weighted RCTs resembled the target population as they provided absolute SMDs less than 0.1 under most scenarios, except the selection model 5 scenario.

Overall, the five meta-analysis models integrated with weighting or outcome prediction approaches helped reduce bias in TATE. All three weighting approaches and BART showed similar patterns of bias with some noticeable bias when 0 or 2 simple random sample RCTs existed under SM5, the most challenging selection model scenario. TMLE outperformed all other methods in most cases even with zero simple random sample RCTs. Among the three weighting methods, GBM produced slightly larger bias than logistic regression and super learner, but the difference in performance was small. Although GBM is known for its flexibility and better performance over the simple logistic regression selection model¹⁷, our simulation used a simple logistic regression selection model including only two covariates as the true model and thus may not provide a setting that would show the benefits of GBM.

When it comes to confidence interval coverage (eFigure 4), TMLE outperformed the other methods across all scenarios. The other methods showed poor coverage probabilities with zero simple random sample RCTs. Among the three weighting methods, GBM produced slightly lower coverage probability than logistic regression and super learner. All methods except TMLE did not satisfy the nominal level of 95% when less than or equal to 2 simple random sample RCTs were included in a meta-analysis. Furthermore, meta-analysis data simulated under SM4 - SM6 provided poorer coverage probabilities than SM2 or SM3, a similar trend observed in bias. This implies that including some RCTs well-representative of the target population in a meta-analysis is crucial to estimate TATE accurately and draw inferences reliably.

Data analysis

Schizophrenia trials

We illustrated our methods using a meta-analysis of four RCTs, $i=1,\dots ,4$, (total n=1,738 participants with 1,241 on paliperidone palmitate and 497 on placebo)³⁷ investigating the efficacy of paliperidone palmitate compared with placebo for patients having schizophrenia. Additionally, the Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) study (n=1,458)³⁸, a pragmatic effectiveness trial supported by the National Institute of Mental Health, serves as target population data ($\Omega$) representing schizophrenia patients in usual care settings in the Unites States. The four RCTs were identified and available in the Yale University Open Data Access (YODA) Project³⁹. The RCTs share comparable eligibility criteria including common schizophrenia diagnostic code by Diagnostic and Statistical Manual and disease severity at baseline. Detailed description of the trials can be found in Hong et al.³⁷. The goal of this analysis is to estimate the TATE of paliperidone palmitate compared with placebo among schizophrenia patients in usual care settings, as represented by the CATIE population.

The outcome of interest is the Positive and Negative Syndrome Scale (PANSS) total score measuring schizophrenia symptoms⁴⁰. Higher PANSS total scores indicate more severe symptoms and we expect taking paliperidone palmitate to reduce the PANSS total score at the end of trial. As such, we analyzed the change in PANSS total score from baseline to the end of the study, where a large positive value means a greater reduction of PANSS total score. The treatment effect is defined as the mean difference in the PANSS score change between paliperidone palmitate and placebo.

Six baseline covariates that could be potential effect moderators were included in the analyses: sex, race, age, age at the first diagnosis of schizophrenia (onset age), weight, and the baseline PANSS total score. The baseline PANSS score is continuous and the other covariates are binary or categorical.

Results

eFigure 6 presents the distributions of baseline characteristics across the four RCTs and CATIE. The distributions of age, and onset age were comparable between CATIE and RCTs. On the other hand, CATIE tended to have a slightly lower proportion of females and a higher proportion of patients with weight over 90 kg than the RCTs. The racial distribution varies across all studies. For the baseline PANSS score, CATIE had patients with a wide range of the PANSS score, with each of the RCTs having a range that fell within the distribution of values seen in CATIE. All RCTs had mean baseline PANSS slightly higher (i.e., more severe schizophrenia symptoms) than that in CATIE. Although we found that variability in distributions across studies and imbalance in distributions of weight and PANSS between CATIE and RCTs, no strong evidence of violation of the positivity assumption presented because all categories or continuous values of covariates in the target population are observed in each RCT. That is, patients in each RCT represented the patients in CATIE though each RCT revealed different degrees of representativeness for different covariates.

Panels (a) and (b) of Figure 3 plot absolute SMDs for each covariate and study, respectively. Panel (a) shows large absolute SMDs for race, weight and PANSS before weighting, indicating that their distributions were somewhat different from those in CATIE. However, the weighted RCT samples under all weighting methods improved such imbalance to the desirable level for sex, race, age, and onset age with absolute SMDs < 0.1, and improved it substantially but not fully to the desirable level for weight and PANSS. These results were expected because a few RCTs showed large differences in their distributions from CATIE before weighting. Panel (b) shows that all RCTs are improved their representativeness with reduced SMDs averaging over all covariates after weighting, though only one study (NCT00590577) reached the desired level of balance. In both Panels (a) and (b), to compare the three weighting methods, the super learner method tended to provide small SMDs overall, but the differences among the methods was minimal.

Figure 3.

(a) Absolute SMD of covariates averaged across trials and (b) average absolute SMDs of covariates in each trial.

Figure 4 presents the results of the meta-analysis using unweighted, weighted, and outcome prediction methods (with the BART model converged, as shown in eFigure 7). eFigure 8 presents all estimated $TAT{E}_i$ and the variability in the TATE estimates across studies, ${\tau }^2$. The ${\tau }^2$ estimate under the unweighted meta-analysis was 17.7. The calibrated meta-analysis provided estimated ${\tau }^2$ from 6.88 under the BART method to 21.21 under the TMLE method.

Figure 4.

TATE estimates and the associated 95% confidence intervals for the mean difference of the change in PANSS between paliperidone palmitate and placebo under different methods.

In Figure 4, the unweighted meta-analysis estimated SATE as 9.07 with an associated 95% confidence interval (CI) of (4.43, 13.71), indicating that paliperidone palmitate is significantly better than placebo to reduce PANSS score. All weighting and outcome prediction methods provided significant results, but with smaller TATE point estimates. The three weighting methods provided reasonably comparable TATE estimates, as expected given the similar covariate balancing abilities. Comparing the two outcome prediction methods, the TMLE approach provided a TATE estimate of 4.92 (95% CI: −0.00, 9.84), and the BART approach provided 7.81 (95% CI: 4.35, 11.27). While one may argue that the point estimates of 4.92 and 7.81 are quite different, their 95% credible intervals show considerable overlap. The BART method provides the narrowest credible interval, but as our simulation results indicate, this can lead to undercoverage due to overly narrow interval estimates in certain settings.

Given the similar results obtained across all methods, we can conclude that the treatment effect in the target population is meaningful and significant, although the effect size becomes smaller than that of the SATE (i.e., the unweighted meta-analysis). In this example, it is difficult to determine if any single method is significantly better or preferred. Therefore, we recommend applying multiple methods and comparing the results to ensure robustness, rather than relying on a single method.

Discussion

In this paper, we proposed and compared methods to estimate target population average treatment effects given data from multiple RCTs and population data, bridging ideas from meta-anlaysis and generalizability. We integrated meta-analysis with widely-used causal inference methods including weighting and outcome prediction approaches. We evaluated their performance via simulation and a data example. The simulation results showed that the TMLE method performed the best across all simulation scenarios. All other methods also helped reduce bias of the TATE compared with an unweighted meta-analysis, but they struggled a bit when few RCTs included in the meta-analysis were simple random samples of the target population. Our data analysis results showed that the TATE estimates were smaller than the unweighted SATE estimate, though the treatment was significantly better than placebo under all methods. That is, the expected treatment effect in the target population may be smaller than the treatment effects estimated in the RCT samples in our schizophrenia meta-analysis data example.

To implement our methods in meta-analysis, a few criteria need to be met. First, individual participant data must be available for both RCTs and the target population data. In addition, covariates must be recorded in a consistent format. For example, in our schizophrenia meta-analysis, some RCTs provided continuous values for age and weight available, while others only shared categorized versions. To include all available RCTs, we categorized the continuous variables and used these categorized versions, which may result in a minor loss of information. Recent advocacy of transparency of clinical research and development of many data sharing platforms are enabling researchers to access IPD sources much more easily than before. Although harmonizing and merging multiple IPD require time and effort, utilizing IPD in the context of generalizability allows us to fully leverage and utilize abundant individual-specific information to assess representativeness of each RCT to the target population and to estimate the TATE properly. A big challenge and a growing area of research in data harmonization is how to handle systematically missing covariates (e.g., some covariates are available in a subset of studies). In the related context of multilevel data and individual patient-level data meta-analysis, several different multiple imputation approaches have been proposed by a few researchers^41,42. Recently, Steingrimsson et al.⁴³ proposed a causal inference method to handle systematically missing covariates in estimating TATE using meta-analysis. An interesting extension of our methods could be to develop methods that allow different sets of covariates in each study; this will be important areas for additional investigation and research.

Second, the sample ignorability for treatment effects assumption should hold. However, this assumption is arguably difficult to test or validate in practice. This is even more challenging in meta-analyses because it is difficult to collect all covariates needed to satisfy the assumption across all trials, in part because trials do not always collect extensive covariates, and in addition the data owners share only deidentified data, which may exclude certain covariates to protect personal health information. In our data example, after data hamonization, we ended up with only six covariates. As such, our results need to be interpreted with caution as they may provide biased estimate of the TATE if there are unobserved covariates that are associated with both treatment effect heterogeneity and the selection process. Conducting sensitivity analysis is a great suggestion to assess the robustness of the estimates to potential violation of this assumption. Although such methods exist for individual studies⁴⁴, sensitivity analyses in the meta-analysis setup do not yet exist and may not be a straightforward extension, thus requiring further research.

Third, the positivity assumption should be met for each RCT. If a subset of RCT samples does not represent the target population at all, we recommend to exclude the subset from the meta-analysis to avoid extreme selection weights in the weighting approaches and too much extrapolation in the outcome model prediction. For example, a target population may include adult participants only, but an RCT includes a small number of teenagers. In this case, the teenagers data would be excluded.

Finally, the target population data should represent the true target population well. Note that our TATE estimate is applied to the population characterized by the target population data used to fit the selection and outcome models. Therefore, the quality of the target population data is crucial. Researchers are recommended to define the target population data based on the data source and expert opinions.

Our simulation study has a few limitations. First, we generated data only assuming linear associations in both selection and outcome models. Schmid et al.²⁰ conducted extensive simulations examining methods for estimating the TATE using a single RCT and considered linear and non-linear outcome models. They concluded that BART was the best method overall as the TMLE and Super Learner approaches did not perform well under non-linear outcome models. Second, we did not study selection model misspecification, a potential concern for the weighting approaches. Our logistic regression selection model was correctly specified, so it performed as well as the other flexible GBM and Super Learner methods. However, we would expect a larger bias under the logistic regression selection model if it is misspecified. Third, we did not explore a wide range of Super Learner libraries. The model performance of the Super Learner method may depend on the choice of library and this requires further research.

Our data analysis revealed that the logistic regression, GBM and Super Learner weighting methods may not be able to provide perfect balance between RCTs and the target population for the baseline PANSS score. As shown in eFigure 6, the overlap of baseline PANSS score between RCTs and the target population was not as good as the overlap for the other variables, and thus the three widely-used weighting methods were not able to improve the balance of the baseline PANSS score to the desired level. In this case, the outcome prediction models extrapolate to some extent to predict outcomes for the individuals in the target population whose characteristics are poorly represented in RCTs. This highlights the importance of how RCT participants are sampled to improve representativeness to the target population. If an RCT is designed to study a treatment effect for a specific subgroup of patients, the results would inform less about the population treatment effect and statistical methods would not be helpful to provide accurate TATE estimates. Alternatively, recently proposed weighting approaches pursuing exact balance such as the overlap weighting^45,46 and entropy weighting⁴⁷ approaches can be considered, and this remains as future work.

Machine leaning methods such as GBM and BART have been gaining popularity in causal inference and selection score estimation due to their flexibility. Compared with parametric models (e.g., logistic regression), machine learning models have two key advantages: (1) they do not require parametric model assumptions and (2) in settings with many covariates the ones to be included in the selection score model may not need to be pre-specified because machine learning methods automatically down weight unnecessary covariates⁴⁸. In addition, machine learning models can outperform logistic regression when estimating propensity scores by providing better covariate balance and treatment effect estimators with smaller mean squared errors⁴⁹. However, machine learning methods can suffer from overfitting, criteria of model tuning, and the number of covariates. Machine learning models usually handle a large set of covariates, but it requires a moderately large sample size⁵⁰. In practice, the choice of methods between machine learning and parametric models will depend on the specific dataset. As both approaches have strong and weak points and are easy to implement using standard statistical software, we recommend applying both approaches and comparing results. This is also an important direction for additional methodological work, to provide guidance for applied practice.

In conclusion, our methods aim to draw better inferences about treatment effects for a target population by integrating findings from RCTs via meta-analysis. The results help researchers and policymakers translate the results of meta-analyses to broader community settings.