Accounting for Post-randomization Variables in Meta-analysis: A Joint Meta-Regression Approach

Summary:

Meta-regression is widely used in systematic reviews to investigate sources of heterogeneity and the association of study-level covariates with treatment effectiveness. Existing meta-regression approaches are successful in adjusting for baseline covariates, which include real study-level covariates (e.g., publication year) that are invariant within a study and aggregated baseline covariates (e.g. mean age) that differ for each patient but are measured before randomization within a study. However, these methods have several limitations in adjusting for post-randomization variables. Although post-randomization variables share a handful of similarities with baseline covariates, they differ in several aspects. First, baseline covariates can be aggregated at the study level presumably because they are assumed to be balanced by the randomization, while post-randomization variables are not balanced across arms within a study and are commonly aggregated at the arm level. Second, post-randomization variables may interact dynamically with the primary outcome. Third, unlike baseline covariates, post-randomization variables are themselves often important outcomes under investigation. In light of these differences, we propose a Bayesian joint meta-regression approach adjusting for post-randomization variables. The proposed method simultaneously estimates the treatment effect on the primary outcome and on the post-randomization variables. It takes into consideration both between- and within-study variability in post-randomization variables. Studies with missing data in either the primary outcome or the post-randomization variables are included in the joint model to improve estimation. Our method is evaluated by simulations and a real meta-analysis of major depression disorder treatments.

1. Introduction

Meta-regression has become a widely used meta-analytic technique to explore the sources of heterogeneity in treatment effect estimates and to investigate the relationship between the effect size and one or more study-level covariates. A covariate included in a meta-regression can be a real study-level characteristic that is the same for all subjects in a study (e.g., publication year) or an individual-level characteristic that varies between subjects but is aggregated as a study-level summary statistic (e.g., mean age). Real study-level covariates are typically related to the study design, such as length of follow-up, study design (e.g., randomized vs. observational), the geographical region of study, etc. Aggregated covariates can be either baseline variables such as mean age or proportion of male or post-randomization variables such as treatment discontinuation rate, compliance measured through average pill counts and other biochemical markers, and cross-over to an alternative therapy.

While including a real study-level covariate is straightforward, adjusting for a covariate that also varies at the patient level raises several concerns. First, the conclusion from the meta-regression is subject to aggregation bias: the estimated coefficient is attenuated by measurement error in the covariate. When the variation in a covariate between studies is small compared to variation within studies, it is difficult to identify the underlying between-study trends in a covariate. Researchers have discussed the impacts of between- and within-study heterogeneity on statistical inference and power (Thompson and Higgins, 2002; Lambert et al., 2002; Schmid et al., 2004). Second, as in other meta-analytic techniques, some degree of heterogeneity between studies in the effect size is expected, because the studies are conducted at different locations, by different researchers, or at different times. A fixed effects model (Greenland, 1987) can be used when the researchers believe that between-study variability in the effect sizes is due to differences in the study-level covariates only. Random effects models have been proposed to account for the residual heterogeneity, i.e., the part of variation in the effect sizes that can not be explained by study-level covariates (Berkey et al., 1995; Thompson and Sharp, 1999).

Finally, one issue in meta-regression that has received little attention is missing covariate data. Here, we need to carefully distinguish two types of missing data in the context of meta-analysis: (1) missing data at the individual patient level and (2) missing data at the study level. The aforementioned missingness refers to missing covariate data at the study level. Meta-analytic methods generally assume that individual level missingness has already been handled with appropriate methods within each study and the reported aggregated data are unbiased or consistent estimates of the true study-specific treatment effect. Yuan and Little (2009) showed that the widely used DerSimonian and Laird estimate (DerSimonian and Laird, 1986) can yield biased estimates when individual-level missingness depends on the size of the underlying study-specific treatment effect. Ignorable missingness at the individual level may have a non-ignorable impact in a meta-analysis. In this paper, the missingness generally refers to missing aggregated values at the study level. Meta-regression is primarily based on published reports. A study may fail to report full information on relevant covariates, especially when the covariates are not significant in that study. Although a meta-analyst may contact the researchers to obtain unreported data, this is not always successful. Also, in an early investigation of a treatment, important confounding factors may not yet be well recognized, so data on these covariates may not even be collected in early studies. Conclusions based on a subset of studies with complete covariate information can be misleading.

Although the scientific value and limitations of meta-regression adjusting for real study-level covariates or aggregated baseline covariates have been thoughtfully investigated (Thompson, 1993; Schmid et al., 2004), little attention has been devoted to post-randomization variables, i.e., variables measured after the randomization within each study. Post-randomization variables share a handful of similarities with baseline covariates, so adjusting for a post-randomization variable in a meta-regression also faces challenges and limitations analogous to those in traditional meta-regression adjusting for aggregated baseline covariates. In addition, post-randomization variables often serve as predictors of the primary outcome and as responses to the treatment in their own right. Because of this, they differ from baseline covariates in important ways. This article briefly reviews some existing meta-regression models and discusses their limitations in handling post-randomization covariates, then proposes a joint meta-regression approach to adjusting for post-randomization variables. The proposed method not only hurdles the aforementioned obstacles in traditional meta-regression methods, but also takes account for the distinctive features of post-randomization variables. Throughout, we focus on meta-analyses of two-arm, parallel group trials.

2. Meta-regression methods accounting for post-randomization variables

This section is organized as follows. Section 2.1 reviews commonly used meta-regression approaches and discusses the limitations of existing methods. Section 2.2 proposes a novel joint meta-regression model adjusting for post-randomization variables. A Bayesian estimation approach is given in Section 2.3.

2.1. Existing meta-regression approaches

Like multiple regression, meta-regression can be used to assess the relationship between the effect size and covariates at the study level. Fixed and random effects models have been proposed (Greenland, 1987; Berkey et al., 1995). Here, we review a commonly used random effects meta-regression approach. The model can be specified as:

$$\begin{gathered} y_i\mid {\Delta }_i,{\zeta }_i\sim f_y(y_i\mid {\Delta }_i,{\zeta }_i), \\ h({\Delta }_i)={\beta }_0+{\mathit{X}}_i\beta +{\omega }_i,{\omega }_i\sim N(0,{\sigma }^2),i=1,2,\cdots ,N \end{gathered}$$ (1)

where y_i is the reported relative effect size of study i, Δ_i is the true underlying effect size of study i, ζ_i is a known nuisance parameter available in each study (typically variance), X_i is study i’s vector of study-level confounding covariates, ω_i is the random effect, which follows a normal distribution with mean zero and standard deviation σ, and β₀ and β are the fixed effects of interest. For example, if the primary outcome is binary, y_i can be the log odds ratio of treatment to control reported from study i. The intercept β₀ is interpreted as the overall log odds ratio of treatment efficacy when X_i = 0, and β corresponds to the effects of study-level covariates. When σ² = 0, this is the fixed effects meta-regression model.

Although a broad consensus considers it acceptable to use such a meta-regression for real study-level covariates and aggregated baseline covariates, existing meta-regression approaches have several drawbacks in adjusting for post-randomization variables, such as compliance measured through average pill counts and other biochemical markers, the proportion of missing data during patient follow-up, cross-over to an alternative therapy, or premature withdrawal from study treatment. First, existing methods are restricted to studies with balanced covariates in which no systematic difference between arms is expected within a study. As in Eq (1), such a meta-analysis often investigates the impact of study-level covariates on a study-specific relative effect size. Post-randomization variables, which are measured following the randomization, generally differ between treatment arms within a study and are commonly aggregated at the arm level. Proper consideration of these post-randomization variables is needed. Second, unlike baseline covariates, post-randomization variables can themselves be outcomes of interest. For example, in a meta-analysis of cancer trials where the primary goal is to evaluate the treatment effect in a patient reported pain outcome, opioid usage in the trials might be a confounder of the treatment effect on the primary outcome and a measure of acceptability of the treatment at the same time. Therefore, post-randomization variables are generally considered random variables instead of fixed values. For studies with relatively small sample sizes, within-study variation in post-randomization variables may be non-ignorable compared with between-study variation. Third, because these variables are observed post-randomization, they may interact dynamically with the primary outcome measure over the course of patient follow-up at the study level. For example, an unsatisfactory treatment may induce poor compliance, which further attenuates the response to treatment. Even if each individual trial uses the same strategy to construct the estimand, the primary outcome can still be confounded by intercurrent events at the aggregated/study level. Finally, traditional meta-regression methods cannot include studies with missing covariate information. Implicitly, traditional meta-regression approaches assume the covariate data is missing completely at random (MCAR), i.e., failure to observe the value of a confounding covariate is independent of both the observed and unobserved data (Little and Rubin, 2002). However, the decision to report the value of a covariate is often related to the observed effect size. As we discussed in Section 1, any analysis using only studies for which complete covariate data is available is potentially biased.

2.2. Joint meta-regression approach using Bayesian hierarchical modeling

This subsection proposes a joint meta-regression approach. The goals of the proposed method are: (1) to estimate the overall effect of a treatment relative to a control for the main outcome and the post-randomization variables; (2) to estimate the overall absolute treatment effects in both treatment groups; and (3) to quantify the relationship between the main effect size and the post-randomization variables. The major difference between the proposed method and existing meta-regression methods is that it treats post-randomization covariates as random variables and jointly models them with the outcome of primary interest. The proposed method also allows the main outcome or the post-randomization variables to be missing at random (MAR) at the study level and naturally accounts for uncertainty due to missing data. Here, data is called MAR if, given the observed data, the missing covariate or outcome data does not depend on unobserved data (Little and Rubin, 2002).

We consider a meta-analytic dataset with a collection of I clinical trials to compare an experimental treatment to an active control or a placebo. For any type (i.e., binary, categorical, or continuous) of aggregate data (i.e., summarized over individuals), assume that the observations of a primary outcome variable from each treatment arm of each trial arise independently from a parametric model: Y_ik ~ f_y(y_ik∣θ_ik, ϒ_ik); i = 1, ……, I, k = 1, 0, where y_ik is the observed aggregate primary outcome for arm k in trial i, f_y(·) is a probability density having an unknown location parameter θ_ik and a known nuisance (typically variance) parameter ϒ_ik. For example, when the measurement Y_ik is continuous, f_y(·) is often assumed to be the normal density. A binary Y_ik is often assumed to follow a binomial distribution Bin(n_ik, θ_ik), where n_ik is the sample size for treatment group k in the i^th study and the unknown location parameter θ_ik is the unknown event probability p_ik, i.e., θ_ik = p_ik. We define ${\delta }_i^y$ to be missingness indicators for the primary outcome variable at the study level (not at the arm level), taking value 1 when y_i = (y_i1, y_i0) is available from study i and 0 otherwise.

As mentioned above, this method treats post-randomization variables as random. Let W_ik be a random aggregated post-randomization variable for the k^th treatment arm of the i^th trial. We assume that W_ik arises independently from a parametric model: W_ik ~ f_w(w_ik∣q_ik, ψ_ik), i = 1, …, I; k = 0, 1, where f_w(·) is a probability density having an unknown location parameter q_ik and a known nuisance parameter (typically variance) ψ_ik. We define ${\delta }_i^w$ as the missingness indicators for the post-randomization variable at the study level (not at the arm level), taking value 1 when w_i = (w_i1, w_i0) is available from study i and 0 otherwise.

Let ${\mathit{X}}_i^q$ and ${\mathit{X}}_i^{\theta }$ be row vectors containing the study-level baseline covariates (e.g., average age and location) associated with the post-randomization variables and the primary outcome for the i^th trial, respectively. Similar to traditional meta-regression approaches, ${\mathit{X}}_i^q$ and ${\mathit{X}}_i^{\theta }$ are assumed to be fixed because they are not affected by the random treatment allocation. We assume ${\mathit{X}}_i^q$ and ${\mathit{X}}_i^{\theta }$ are available for all i. To allow the study-level baseline covariates ${\mathit{X}}_i^{\theta }$ and the latent location parameter q_ik of the post-randomization variables to have differential effects on the primary outcome, i.e., to let the ${\mathit{X}}_i^{\theta }$ and W_ik affect absolute treatment effects on the primary outcome, we consider a very general joint model:

$$\begin{gathered} h_{\theta }({\theta }_{ik})={\mu }_{0k}+{\mathit{X}}_{\mathit{i}}^{\theta }{\phi }_k^{\theta }+g(q_{ik}){\mu }_{1k}+{\omega }_{ik}; \\ {h}_q(q_{ik})={\alpha }_k+{\mathit{X}}_{\mathit{i}}^q{\phi }_k^q+{\nu }_{ik},k=0,1 \end{gathered}$$ (2)

where h_θ(·), h_q(·) and g(·) are link functions associated with the primary outcome and the post-randomization variables, respectively. The random effects ω_i = (ω_i1, ω_i0) and ν_i = (ν_i1, ν_i0) are mutually independent and follow bivariate normal distributions with covariance matrices Σ_θ and Σ_q, respectively. The off-diagonal elements and diagonal elements of Σ_q capture the potential correlation between arms and heterogeneity across studies respectively in the post-randomization variable. The between-arm correlation and between-study variability in the primary outcome can be partially explained by the post-randomization variable, while the diagonal and off-diagonal elements of Σ_θ capture the residual heterogeneity across studies and correlation between arms, respectively. The adjusted overall absolute effect (arm-based effect) on the primary outcome in arm k is μ_0k when the covariate terms are zero, while μ_1k measures the strength of the relationship between primary outcome and the post-randomization variable in arm k. If the post-randomization variable is assumed to have a non-differential effect on the primary outcomes in the two arms, μ_1k (k = 0, 1) in Eq (2) can be replaced by a single parameter μ₁. If μ_1k = 0, treatment efficacy is exchangeable without conditioning on the post-randomization variable and standard meta-analysis models provide consistent estimates as well. The parameters ${\phi }_k^{\theta }$ and ${\phi }_k^q$, which account for confounding effects, can also be simplified to φ under analogous assumptions.

Implicitly, our modeling strategy assumes the primary effect size of interest underlying the data with g(q_ik) = 0 is similar across trials, so that systematic deviation from the overall effect size is due, e.g., to the fraction of participants who discontinue study treatment. By jointly modeling the post-randomization variable and the main outcome, the proposed method takes into consideration the randomness in the outcome of post-randomization variable.

2.3. Estimation

Let y_i = (y_i1, y_i0), w_i = (w_i1, w_i0), μ₀ = (μ₀₁, μ₀₀), μ₁ = (μ₁₁, μ₁₀), α₀ = (α₀₁, α₀₀), ${\phi }^{\theta }=({\phi }_1^{\theta },{\phi }_0^{\theta })$ and ${\phi }^q=({\phi }_1^q,{\phi }_0^q)$. Unlike traditional meta-regression, the underlying assumption for the joint modeling approach is that missing data are missing at random at study level, i.e., given the observed data, failure to observe the primary outcome or the post-randomization variable in a study is independent of the unobserved data. Given this assumption, the likelihood contribution conditional on the model parameters from study i with observed primary outcome and post-randomization variable is

$$\begin{gathered} l_{i1}=p({\mathit{y}}_i,{\mathit{w}}_i,{\delta }_i^w=1,{\delta }_i^y=1\mid {\mu }_0,{\alpha }_0,{\omega }_i,{\nu }_i,{\lambda }_m,{\mu }_1,{\mathit{X}}_{\mathit{i}}^q,{\mathit{X}}_{\mathit{i}}^{\theta },{\phi }^{\theta },{\phi }^q),\text{by MAR assumption} \\ =p({\delta }_i^w=1,{\delta }_i^y=1\mid {\mathit{y}}_i,{\mathit{w}}_i,{\lambda }_{\mathit{m}},{\mathit{X}}_{\mathit{i}}^q,{\mathit{X}}_{\mathit{i}}^{\theta })p({\mathit{y}}_i,{\mathit{w}}_i\mid {\mu }_0,{\alpha }_0,{\omega }_i,{\nu }_i,{\mu }_1,{\mathit{X}}_{\mathit{i}}^q,{\mathit{X}}_{\mathit{i}}^{\theta },{\phi }^{\theta },{\phi }^q) \\ \propto \prod _k{f}_y(y_{ik}\mid {\theta }_{ik},{{\rm Y}}_{ik})f_w(w_{ik}\mid q_{ik},{\psi }_{ik}), \end{gathered}$$

where θ_ik and q_ik are functions of μ₀, α₀, ω_i, ν_i, ${\mathit{X}}_i^q$, φ^θ, φ^q, ${\mathit{X}}_i^{\theta }$ and μ₁, as shown in Eq (2). The parameter λ_m is a vector of parameters related only to the missingness pattern. As shown above, the distribution of the observed data can be factored into a marginal density for (y_i, w_i) and a conditional density for (${\delta }_i^y$, ${\delta }_i^w$) given (y_i, w_i). As a result, the missingness mechanism $p({\delta }_i^y=1,{\delta }_i^w=1\mid {\mathit{y}}_i,{\mathit{w}}_i,{\lambda }_{\mathit{m}},{\mathit{X}}_i^q,{\mathit{X}}_i^{\theta })$ is ignorable and inferences about the model parameters are based solely on $p({\mathit{y}}_i,{\mathit{w}}_i\mid {\mu }_0,{\alpha }_0,{\omega }_i,{\nu }_i,{\mu }_1,{\mathit{X}}_i^q,{\mathit{X}}_i^{\theta },{\phi }^{\theta },{\phi }^q)$. This suggests there is no need to specify a model for the mechanism generating missing data, under the missing-at-random assumption. Similarly, we can show that the conditional likelihood contribution from study i with observed primary outcome but missing post-randomization variable is $l_{i2}=p({\mathit{y}}_i,{\delta }_i^w=0,{\delta }_i^y=1\mid {\mu }_0,{\alpha }_0,{\omega }_i,{\nu }_i,{\lambda }_m,{\mu }_1,{\mathit{X}}_i^q,{\mathit{X}}_i^{\theta },{\phi }^{\theta },{\phi }^q)\propto {\prod }_k{f}_y(y_{ik}\mid {\theta }_{ik},{{\rm Y}}_{ik})$. Detailed derivation is available in Web Appendix C. Finally, the conditional likelihood contribution for a study with missing outcome data but observed post-randomization variable is l_i3 ∝ ∏_k f_w(w_ik∣q_ik, ψ_ik). The total conditional likelihood given $({\mu }_0,{\alpha }_0,{\omega }_{\mathit{i}},{\nu }_{\mathit{i}},{\mu }_1,{\mathit{X}}_{\mathit{i}}^q,{\mathit{X}}_{\mathit{i}}^{\theta })$ is then $\mathcal{L}={\prod }_i(l_{i1}{)}^{{\delta }_i^w{\delta }_i^y}(l_{i2}{)}^{1-{\delta }_i^w}(l_{i3}{)}^{1-{\delta }_i^y}$. The marginal likelihood integrated over the multi-dimensional random effects is: $p({\mu }_0,{\alpha }_0,{\sum }_{\theta },{\sum }_q,{\mu }_1\mid \text{Data})={\prod }_i{\iint }_{}(l_{i1}{)}^{{\delta }_i^w\ast {\delta }_i^y}(l_{i2}{)}^{1-{\delta }_i^w}(l_{i3}{)}^{1-{\delta }_i^y}\text{MVN}({\omega }_i\mid {\sum }_{\theta })\text{MVN}({\nu }_i\mid {\sum }_q)d{\omega }_{i}d{\nu }_i$. To account for the complexity of the model, we adopt a full Bayesian approach using Hamiltonian Monte Carlo (HMC) algorithms implemented using the STAN software in the rstan package in R (Stan Development Team, 2016).

To complete the model specification, we assign vague normal priors to the fixed effects α_0k, μ_0k and the coefficient μ_1k. The covariance matrices Σ_θ and Σ_q can be decomposed as Σ_θ = PΩ_θP and Σ_q = QΩ_qQ respectively, where P and Q are diagonal matrices with diagonal element {σ_k, k = 1, 0} and {τ_k, k = 1, 0} respectively. Ω_θ and Ω_q are correlation matrices, which capture the correlation between the two arms. We place LKJ priors (Lewandowski et al., 2009) on the correlation matrices and place half-Cauchy priors on the standard deviation parameters τ_k and σ_k (Gelman et al., 2006). Although we use unstructured covariance matrices here, these can be simplified if relevant prior knowledge is available or through a model selection procedure. For example, Σ_θ and Σ_q may be reduced to diagonal matrices, which implies that the arms in a study have independent effect sizes.

If both the primary outcome and the post-randomization variable are binary, i.e., Y_ik ~ Bin(n_ik, θ_ik = p_ik), W_ik ~ Bin(n_ik, q_ik), and h_θ(·) and h_q(·) are the logit link function, the adjusted relative treatment effect on the primary outcome and post-randomization variable with ${\mathit{X}}_{\mathit{i}}^q={\mathit{X}}_{\mathit{i}}^{\theta }=0$ can be summarized by $\log {\text{OR}}_y^0={\mu }_{01}-{\mu }_{00}$ and $\log {\text{OR}}_w^0={\alpha }_{01}-{\alpha }_{00}$ respectively. Here, $\log {\text{OR}}_y^0$ can be interpreted as the adjusted log odds ratio of the treatment’s effect on the primary outcome when g(q_ik) = 0 and the covariates are equal to 0, which reflects the treatment effect under ideal circumstances. More generally, we can estimate the overall conditional adjusted log odds ratio when the post-randomization variable takes a pre-specified value l and study-level baseline variables take pre-specified values ${\mathit{X}}_{\mathit{i}}^{\theta }=\mathit{x}$, i.e., $\log {\text{OR}}_y^{l,\mathit{x}}={\mu }_{01}-{\mu }_{00}+g(l)({\mu }_{11}-{\mu }_{10})+\mathit{x}({\phi }_1^{\theta }-{\phi }_0^{\theta })$. For example, $\log {\text{OR}}_r^{E(q_{i0}),\mathit{x}}$ gives a conditional adjusted log odds ratio of the treatment’s effect on the primary outcome when the post-randomization variables in both groups are equal to the underlying population risk and the study-level covariate vector is equal to x. Other interesting quantities are the marginal means of the primary outcome in the two groups given a post-randomization variable value l, i.e., ${\pi }_k(l,\mathit{x})=E(p_{ik}\mid {\mu }_{0k},{\sigma }_k,l)=\int h_{\theta }^{-1}({\mu }_{0k}+{\mu }_{1k}\ast g(l)+\mathit{x}{\phi }_k^{\theta }+{\omega }_{ik})dF({\omega }_{ik})$, k = 1, 0, where l is the pre-specified value of the post-randomization variable. This can be interpreted as the population average treatment effect on the primary endpoint in arm k for a given post-randomization value. Summary statistics for the relative treatment effect can be derived from π_k(l, x). For example, the overall marginal relative risk can be derived as RR^y(l) = π₁(l, x)/π₀(l, x). Although computation of π_k(l, x) involves integration, there is a closed form solution for the probit random effects model, ${\pi }_k(l,\mathit{x})=\Phi \left(({\mu }_{0k}+{\mu }_{1k}\ast g(l)+\mathit{x}{\phi }_k)∕\sqrt{1+{\sigma }_1^2}\right)$, and a well-established approximation

$${\pi }_k(l,\mathit{x})=\frac{exp\left(({\mu }_{0k}+{\mu }_{1k}\ast g(l)+\mathit{x}{\phi }_k^{\theta })∕\sqrt{c^2{\sigma }_1^2+1}\right)}{\left[1+exp\left(({\mu }_{0k}+{\mu }_{1k}\ast g(l)+\mathit{x}{\phi }_k^{\theta })∕\sqrt{c^2{\sigma }_1^2+1}\right)\right]}$$

for the logit random effects model, where $c=16\sqrt{3}∕(15\pi )$. We calculate these statistics in each iteration of the HMC algorithm and inference is based on the means of their posterior samples. As a graphic presentation of the results, we can plot the estimated π_k(l, x) over a series of l in the observed range of w_ik for a given x to show how the primary effect size changes with the underlying post-randomization variable. Extrapolation beyond the available data is not recommended. The posterior samples also give a 95% pointwise credible band for the estimated curve to show the uncertainty in the estimates of π_k(l, x).

If Y_ik is continuous and f_y(·) is the normal density, the adjusted relative treatment effect on the primary outcome and post-randomization variable with ${\mathit{X}}_{\mathit{i}}^q={\mathit{X}}_{\mathit{i}}^{\theta }=0$ can be summarized by ${\Delta }_y^0={\mu }_{01}-{\mu }_{00}$ and ${\Delta }_w^0={\alpha }_{01}-{\alpha }_{00}$ respectively. The overall conditional adjusted treatment effect when the post-randomization variable takes a pre-specified value l and study-level baseline variables take pre-specified values ${\mathit{X}}_{\mathit{i}}^{\theta }=\mathit{x}$ is ${\Delta }_y^{l,\mathit{x}}={\mu }_{01}-{\mu }_{00}+g(l)({\mu }_{11}-{\mu }_{10})+\mathit{x}({\phi }_1^{\theta }-{\phi }_0^{\theta })$. The interpretations are analogous to those for a binary outcome.

The observed data may contain little information to precisely estimate μ_1k when the number of studies in the meta-analysis is small. In such situations, we suggest conducting a sensitivity analysis. Specifically, we can assign a series of informative normal priors with different means for μ_1k and investigate how the inference shifts with the prior. Alternatively, we can fix μ_1k at a series of values (instead of estimating it from the data) and examine how the estimates change.

The method’s performance was extensively examined in simulation studies, compared to regular meta-regression and the unadjusted meta-analysis. The detailed simulation setup, results and interpretations can be found in Web Appendix A. Based on Web Table 2-4, our proposed method generally provides estimates with almost zero bias and coverage probabilities close to the nominal level. It outperforms existing methods in complete data, MCAR and MAR scenarios and performs similarly in MNAR scenarios. Two important findings of the simulation results are (1) ignoring the association between the post-randomization variable and the primary outcome can lead to an estimate with large bias and low coverage probability and (2) the regular meta-regression method based only on the subset of studies with complete observations can be even more misleading than not adjusting for the post-randomization variables.

3. Case Study

This example was chosen because it shares characteristics with a wide variety of examples that involve other post-randomization variables. One issue in clinical trials is participants discontinuing their assigned treatments. Common reasons for treatment discontinuation are that the treatment is ineffective or has unacceptable side effects (Council et al., 2011). Clearly, treatment discontinuation is a post-randomization variable (intercurrent event) that occurs after treatment initiation and affects the interpretation and indeed measurement of the treatment effect. Meta-analysis of the primary outcome based on the intention-to-treat analysis may be confounded by a difference in treatment discontinuation at the study level, therefore the study-specific effect sizes may be exchangeable only after adjusting for treatment discontinuation. In cases where participants discontinue study treatment because of ineffectiveness, treatment discontinuation may also affect the effect size differentially in different arms. Also, clinical studies often aim to investigate both efficacy and acceptability of the experimental treatment compared to a placebo or active control. At the study level, treatment discontinuation itself is a measure of treatment acceptability or tolerability (Cipriani et al., 2009, 2018; Leucht et al., 2013). This section exemplifies the proposed method through a meta-analysis of randomized trials of the antidepressant paroxetine.

Cipriani et al. (2018) conducted a meta-analysis on 50 double-blind clinical trials comparing the anti-depressant paroxetine with placebo for acute treatment of adults (18 years of age or older and both sexes) with a primary diagnosis of major depressive disorder (MDD). Cipriani et al. (2018) noted that larger all-cause treatment discontinuation rate were associated with a lower response to treatment. We revisited this meta-analysis to exemplify the proposed method. The primary estimand was the treatment effect and acceptability of paroxetine compared to placebo in the population defined by the inclusion criteria specified in Cipriani et al. (2018). Here, efficacy was measured by the proportion of patients who responded to treatment and acceptability was measured by the proportion of patients who discontinued the allocated treatment for any reason. Response was defined as a reduction of at least 50% from baseline on a standardized observer-rating scale for depression. Although last observation carried forward (LOCF) was used to impute missing outcome data in the individual studies, it is not a prerequisite of the proposed method.

Table 1 summarizes the observed response and treatment discontinuation rates from the 50 studies. The response rate was missing in 8 studies and the treatment discontinuation rate was missing in 2 studies. To explore the potential bias due to dependence between the study-specific response rate and treatment discontinuation rate, Figure 1 plots the observed response rates against the observed treatment discontinuation rates for the 40 studies that provided both. This figure shows a clear negative association between the two outcomes, i.e., studies/arms reporting larger treatment discontinuation rates tended to have lower response rates. Therefore, the unadjusted response rates may not be exchangeable and an investigation of the effect of acceptability on efficacy is warranted. We performed meta-regression analyses to (1) compare the treatments for the two outcomes simultaneously, (2) estimate the overall treatment efficacy of paroxetine adjusted for acceptability, and (3) assess the association between the two outcomes.

Table 1.

Descriptive summaries of studies in the anti-depressant drug meta-analysis: Percentiles of response rate and treatment discontinuation rate, and number of studies that did not report these rates.

	Drug	Min.	2.5%	Median	97.5%	Max.	N missing
Response Rate	Paroxetine	0.14	0.23	0.49	0.73	0.82	8
	Placebo	0.07	0.13	0.37	0.53	0.62	8
Treatment discontinuation Rate	Paroxetine	0.11	0.11	0.30	0.53	0.60	2
	Placebo	0.09	0.11	0.30	0.69	0.82	2

Figure 1.

Graphical summary of the data. The y-axis is log(odds)of the response, computed as log(y_ik/(n_ik – y_ik)). The dashed line is the simple linear regression of log(odds) on percent treatment discontinuation for placebo while the solid line is the regression line for paroxetine).

We implemented the proposed joint meta-regression method as described in Section 2. Figure 1 shows that the association between the treatment discontinuation and log odds of response rates are similar for the two treatments. Therefore, it is reasonable to assume that treatment discontinuation has non-differential effect between the two arms, i.e, μ₁₁ = μ₁₀ = μ₁, giving a more parsimonious model. Logit and identity link functions were used for h_θ(·), h_q(·) and g(·), repectively. We placed a vague normal prior with mean 0 and variance 2.25 on the fixed effects α_0k and μ_0k, k = 1, 0, which corresponds to a prior distribution with 95% credible interval (CrI) approximately (0.05, 0.95) for the p_ik and q_ik. LKJ(2) is used as the prior for the correlation matrices Ω_θ and Ω_q. We assigned half Cauchy distributions with scale 0.25 for the priors of σ_k and τ_k, which corresponds to a prior distribution with 95% CrI approximately (0.01, 6.4), large enough to cover all plausible values. A vague normal prior with mean 0 and variance 100 was placed on μ₁.

The posterior mean of the adjusted log odds ratio when the two arms have the same treatment discontinuation rates (i.e., $\log {\text{OR}}_w^0$ under μ₁₁ = μ₁₀) was estimated to be 0.50 for treatment efficacy. The 95% CrI excludes zero, indicating that paroxetine is significantly better than the placebo in treating MDD if no patients discontinue treatment. The estimated log odds ratio for treatment acceptability is −0.04, with a 95% CrI including zero. The point estimate of μ₁, −2.96, means that one unit decrease in the treatment discontinuation rate is association with an increase of 2.96 in the log odds, a negative association. The 95% CrI excludes 0, which indicates a significant study-level association between treatment discontinuation rates and response rates. These results imply that a lower treatment discontinuation rate is associated with a better observed treatment effect at the study level. Figure 2 shows how the treatment discontinuation rate affects the marginal mean of the response rate in the two treatment groups. In particular, the estimated marginal mean of the response rate in the paroxetine and placebo group are 0.50 and 0.38, respectively at the observed median treatment discontinuation rate 0.30. The posterior mean of ${\text{logOR}}_y^0$ was estimated to be 0.54 using the naive unadjusted analysis, with 95% CrI (0.42, 0.66). Comparing the results from the two methods, the estimate of ${\text{logOR}}_y^0$ is slightly smaller in the joint meta-regression method. Although the two methods give similar estimates for the relative effects, the absolute effect estimates of the response rates are quite different. Table 2 gives more detailed results on the estimates. It is important to recognize that even after adjusting for treatment discontinuation rate, response rates may still be confounded by other study-level or patient-level characteristics that are unknown or unavailable.

Figure 2.

Case study results: estimated marginal means of the response rate (“E(Response Rate)”) in the two groups given the treatment discontinuation rate.

Table 2.

Case study results from the proposed method adjusting for treatment discontinuation rates and the naive method without adjustment: Posterior mean (95% credible interval).

	Adjusted analysis	Unadjusted analysis
${\text{logOR}}_y^0$	0.50 (0.39, 0.62)	0.54 (0.42, 0.66)
${\text{logOR}}_w^0$	−0.04 (−0.18, 0.10)	−0.05 (−0.19, 0.09)
μ 01	0.91 (0.60, 1.21)	−0.05 (−0.23, 0.12)
μ 00	0.40 (0.10, 0.71)	−0.59 (−0.76, −0.43)
α 01	−0.81 (−0.99, −0.63)	−0.81 (−0.98, −0.63)
α 00	−0.76 (−0.97, −0.56)	−0.76 (−0.96, −0.54)
μ 1	−2.96 (−3.85, −2.08)	−

We conducted sensitivity analyses to assess the impact on the estimates of the prior distributions and MAR assumption. Detailed descriptions and results can be found in Web Appendix B. To understand the effect of the post-randomization variable on the primary outcome in each arm, one can start with a model assuming differential effects of the post-randomization variable in each arm. Posterior samples of μ₁₀ and μ₁₁ can then be used to formally test whether μ₁₀ = μ₁₁.

4. Discussion and Future Work

Existing meta-regression methods are insufficient to model post-randomization variables that can differ substantially between treatment arms. This paper proposed a joint meta-regression approach to fill this gap. The proposed method jointly models the primary outcome and the post-randomization variables in each study arm so that both within- and between-study variation in both the primary outcome and the post-randomization variables are taken into account. It also allows either the primary outcome or the post-randomization variable to be missing at random. A Bayesian hierarchical approach naturally accounts for uncertainty arising from the missing data. The proposed model is readily applicable to a wide range of post-randomization variables, such as compliance measured through average pill counts and other biochemical markers, the proportion of missing data during patient follow-up, cross-over to an alternative therapy, or premature withdrawal from study treatment. The method can also serve as a mediation analysis in cases where the post-randomization variables may confound estimation of primary treatment effect but may not themselves be of direct clinical interest (e.g., intercurrent events).

Although we focused on meta-analysis of randomized trials, the proposed method is also relevant to meta-analysis of observational studies. Unlike clinical trials, observational studies do not have the benefit of randomization. Common reasons for being unable to do a meta-analysis on observational studies are that: (1) not all studies report the same set of confounding variables; (2) many confounding variables are not real study-level covariates and are subject to measurement error; and (3) confounding variables may modify the treatment effect both within and between studies. It is easy to see that confounding variables in a meta-analysis of observational studies share similarities with post-randomization variables in meta-analysis of randomized controlled trials. Therefore, this paper’s method can also be used to adjust for confounding variables in meta-analysis of observational studies. For the same reasons, the proposed method may also be applicable to meta-analysis in genome-wide association studies. We leave this to future work.

The proposed meta-regression method models absolute effects in the treatment and control arms. Developments in NMA have stimulated a debate about modeling relative treatment effects versus modeling the absolute effects in each treatment arm of each study (Hong et al., 2016; Dias and Ades, 2016). Recently, modeling the absolute effects has been found to be mathematically tidier with performance similar to methods modeling relative treatment effects, and both models are suitable for the analysis of NMA data with certain requirements on the data (Senn, 2010; White et al., 2019). Although that debate pertains mainly to NMA and this paper focuses on meta-regression evaluating only two treatments with outcomes missing at study level (not at arm level), readers may still wonder why this paper modeled absolute effects instead of modeling the relative effects. In a meta-regression adjusting for post-randomization variables, one obvious benefit of modeling the absolute effect is that it allows values of post-randomization variables that are imbalanced between arms, which is the key attribute that differentiates post-randomization variables from baseline variables. An analogous method that models relative effects may be possible if post-randomization variables have non-differential effects on the treatment and control arms; this warrants future investigation.

The proposed method has several limitations. Meta-regression has been widely used to determine whether the treatment effect depends on the control group risk. Control group risk is an effective measure of baseline differences between studies. It is possible that differences between studies in a post-randomization variable are also associated with underlying risk. Although the proposed method can be extended to include some study-level baseline covariates, we did not investigate the relationships among the primary outcome, control group risk and post-randomization variables. When considering the underlying risk in the control group, we need to account for regression dilution bias (Thompson et al., 1997), i.e., the artificial negative association arising from having correlated measurement error in the observed control group risk and the observed treatment effect. Several functional and structural approaches have been proposed to extend traditional meta-analysis methods to account for underlying risk (Sharp et al., 1996; Thompson et al., 1997; Sharp and Thompson, 2000). Adjusting for both post-randomization variables and underlying risk can be complicated. The proposed methods may be extended to adjust for underlying risk in line with the methods presented in Thompson et al. (1997) and Van Houwelingen et al. (2002). Applying them to the joint meta-regression method requires further investigation.

The results from the proposed method should be interpreted with caution. Conclusions derived from a meta-regression analysis are observational, even when all included studies are randomized trials. The proposed method — and all other existing meta-regression methods — cannot establish causality although they are helpful in generating new hypotheses and motivating new clinical trials. Essentially, meta-regression is an epidemiologic investigation of the relationship between the treatment effect size and the covariate, so its results are subject to the same types of bias as observational epidemiologic analyses (Thompson and Higgins, 2002; Schmid et al., 2004). Also, aggregate (ecologic) data are known to be insufficient to characterize individual-level association. Meta-regression provides ecological inference, and relationships observed for groups do not necessarily hold for individuals. Therefore, results from using meta-regression methods (including the proposed method) without individual patient data should be interpreted with caution to avoid the ecological fallacy (Greenland and Morgenstern, 1989; Berlin et al., 2002).

Data Availability Statement

The data that support the findings in this paper are available from the corresponding author upon reasonable request.