Testing moderation in network meta-analysis with individual participant data

Summary

Meta-analytic methods for combining data from multiple intervention trials are commonly used to estimate the effectiveness of an intervention. They can also be extended to study comparative effectiveness, testing which of several alternative interventions is expected to have the strongest effect. This often requires network meta-analysis (NMA), which combines trials involving direct comparison of two interventions within the same trial and indirect comparisons across trials. In this paper, we extend existing network methods for main effects to examining moderator effects, allowing for tests of whether intervention effects vary for different populations or when employed in different contexts. In addition, we study how the use of individual participant data (IPD) may increase the sensitivity of NMA for detecting moderator effects, as compared to aggregate data NMA that employs study-level effect sizes in a meta-regression framework. A new network meta-analysis diagram is proposed. We also develop a generalized multilevel model for NMA that takes into account within- and between-trial heterogeneity, and can include participant-level covariates. Within this framework we present definitions of homogeneity and consistency across trials. A simulation study based on this model is used to assess effects on power to detect both main and moderator effects. Results show that power to detect moderation is substantially greater when applied to IPD as compared to study-level effects. We illustrate the use of this method by applying it to data from a classroom-based randomized study that involved two sub-trials, each comparing interventions that were contrasted with separate control groups.

1 Introduction

As intervention research advances, there is growing interest in moving from simple tests that compare intervention efficacy to more complex questions concerning which interventions work for whom and under what conditions. When analysts combine data from multiple trials using meta-analysis, they address these questions through meta-regression, testing whether study-level covariates predict systematic variation in intervention effects. In this paper we take up the question of how this work can be advanced through combining two general methods. These include network meta-analysis and the use of participant-level data.

Individual intervention trials randomly assign participants to controls and one or more interventions, testing whether outcomes differ between intervention and control, or among other forms of intervention. When an intervention has been tested in a number of trials, conventional meta-analysis techniques combine results across trials that have the same intervention arms, but test intervention effects across different contexts or populations. Meta-analysis techniques are available to combine summary measures such as effect sizes from a set of trials. Network meta-analysis, a more general method, allows for combining results across multiple trials that have varying intervention arms [1,2]. Two interventions of interest may be present in head-to-head trials, giving rise to a direct comparison, while other trials may only include one of these interventions, tested against a control. The overall effect that compares two active interventions can be estimated by taking into account both direct comparisons of the two interventions within the same trials, as well as indirect comparisons where one of these interventions is absent [3–6]. It is always helpful but not absolutely essential to have direct head-to-head comparisons. For example, consider two randomized trials where the first tests an intervention (B) against control (A) while the second tests a different intervention (C) against the same control. Even though there is no head-to-head comparison of B vs C, combining the comparisons of A vs B and A vs C should provide useful information on comparing B vs C. Provided nothing differs systematically between the two trials, the difference in these two estimates yields an unbiased estimate of the effect of B vs C. Extensions of this idea form the basis of network meta-analysis. Like pairwise meta-analysis, most network meta-analyses are based on aggregate data, generally from publications, to make comparisons between interventions.

Network meta-analysis has been used almost exclusively to study intervention main effects. However, as intervention research progresses, important questions emerge concerning which interventions work best for which populations, and under what general conditions. Such questions can only be answered through tests of moderation.

In pairwise meta-analysis, moderation is tested through meta-regression. Study effect sizes are regressed on study-level variables, such as type of intervention or percent of female participants. Although meta-regression techniques could in principle be used within network meta-analysis, we suggest that it will often be more pro table to employ new methods that combine participant-level data from multiple trials [7, 8]. As rich as network meta-analysis is, the use of summary statistics from reported findings limits its ability to examine more complex hypotheses involving moderators of intervention effects. As an example, consider the question of whether intervention effects may differ for participants who begin the trial with higher levels of targeted symptoms. This question is important as a growing number of studies suggest the importance of baseline moderators in determining who will benefit the most from an intervention [9]. The average estimates obtained from network meta-analysis say nothing about variation in the effects of an intervention across participants who differ on baseline characteristics.

Moderator effects in comparative effectiveness studies could be addressed by combining reports of moderator analyses of intervention effects in a pairwise meta-analytic framework. However, moderator analyses for single trials are not always reported in the published literature, especially when each individual trial has very low statistical power for detecting moderation. Such analyses in separate trials may be reported more often for those moderator variables that reach standard levels of statistical significance, and even when published in multiple trials the analyses may be based on different cutoff values (e.g., high or low risk). Thus selective reporting and lack of harmonization make it difficult to obtain useful syntheses of published moderation analyses. In many cases meta-analyses have used meta-regression based on trial-level covariates such as the percentage of high-risk individuals in a trial [10,11], to reflect how an individual covariate could moderate impact [12, 13]. Use of summarized participant-level covariates is problematic; it is well known that this approach can easily lead to erroneous conclusions when applied to participants (the ecological fallacy [14]).

The alternative we explore here involves testing for participant-level moderation of intervention effects [15] or more generally examining variation in impact by baseline characteristics [16, 17] by analyzing participant-level data across multiple trials. For this paper we assume that participant-level data from all trials are available and that the participant-level covariate and outcome are identical across trials. Analysis of individual-participant data (IPD) across multiple trials [12,18,19] is considered a type of integrative data analysis (IDA), which has seen extensive methodologic development in recent years both for combining data from passive longitudinal studies [20] as well as for randomized trials [21–23].

Analysis of IPD offers a number of advantages [24]. It allows for more precise harmonization of outcome measures, of particular importance when trials use somewhat different methods for measuring the same outcome [25, 26]. This may be less of an advantage when all trials use the same outcome measures, although even in this case IPD can be used to test the assumption that measurement is invariant over trials. IPD also makes maximal use of information on within-study variation in potential moderators, avoiding the ecological fallacy. The availability of IPD also facilitates a “one-stage” approach in the network meta-analysis. The use of IPD is limited to situations where participant-level data are available across trials. Federal policies now require that funded investigators make data available for other investigators, but obtaining and combining such data can require substantial resources (see Perrino et al. [9] for an in-depth discussion of methods for data sharing).

Recent work has established the benefits of studying whether participant characteristics moderate the effects of interventions with participant-level data as compared to aggregate data [15, 27–29]. This has recently been applied to network meta-analysis, with particular attention to combining trials when moderators are binary [1, 7]. We build on this work in 3 ways. Focusing on situations where IPD are available in all trials, we develop models for binary, categorical and continuous moderators. We also conduct simulations to assess how power is affected by this use of IPD.

In this article, we present multilevel models that allow for the analysis of moderator effects based on participant-level data aggregated across a pool of trials that involve both direct and indirect comparisons of interventions. This NMA framework is sufficiently general to allow the incorporation of non-randomized data such as additional samples of controls or intervened populations. It is noted that the NMA framework needs to adjust for confounders and other sources of heterogeneity when non-randomized data are included. We then use simulations to evaluate the statistical power for network meta-analytic comparisons of main effects and moderator effects for two interventions tested across the same and different trials. For main effect analyses that compare two active interventions, network meta-analysis with IPD is expected to improve statistical power slightly over network meta-regression, when variance estimates can be pooled [30]. For moderation analyses across multiple trials we anticipate greater statistical power when participant-level data are used compared with composite trial-level characteristics.

The remainder of this paper is structured as follows. Section 2 presents a two-level model specification of network meta-analysis based on IPD for both fixed and random effect modeling and provides a general definition of homogeneity and consistency, key concepts in network meta-analyses. Simulation results for power analyses for both main effect and moderation analyses are presented in Section 3. These models are then applied in Section 4 to two linked multilevel randomized trials in order to estimate an indirect comparison with two active interventions. Finally, Section 5 concludes with recommendations for this procedure's use and future work.

2 Modeling of network meta-analysis using individual-participant data

We consider how to make inferences about main effects and moderator effects across a collection of I studies (trials) each of which tests one or more intervention arms. In totality the trials involve K interventions, with K ≥ 3. We assume that all studies with more than one intervention are randomized at the participant level, and for convenience we use the term trial to refer to any single-arm or multi-arm study. Trials that are included need to be comparable, a term we use to refer to trials that involve similar populations (e.g., similar age group) and trial designs (e.g., similar follow-up time [31,32]). It is a basic premise of network meta-analysis that statistical comparisons of interventions across trials should be inferred only if the trials are clinically and methodologically comparable [33]. Inclusion/exclusion criteria for trials are often defined to identify eligible trials. It is not always easy to obtain participant-level data from multiple trials, but procedures for obtaining such data through a data sharing collaboration are discussed in Perrino et al. [34].

2.1. Modeling main effects and effect modifications in network meta-analysis

We begin with a generalized linear mixed model, accounting for different intervention arms in each trial. We assume that all trials have a common outcome measure Y and a common set of baseline covariates X. Measures need not be common, but issues of harmonizing different measures across trials raise tangential concerns to this paper's development and are discussed elsewhere [35]. Intervention moderator effects are given by the product of a subset Z of X covariates and these intervention indicators. We recommend following the standard practice of centering all moderator terms across trials at zero before calculating products, to reduce collinearity. In the generalized linear mixed model [36], defined in Equation (1) below with a link function g(·), we decompose each individual's response into a sum of three components corresponding to three sets of predictors X: intervention denoted by T, covariates Z, and effect modifiers².

$$g\left(E[Y_{ij}\mid T_{ij}=k]\right)={\mathit{X}}_{ij}\alpha +{\beta }_k+{\mathit{Z}}_{ij}{\gamma }_k+b_i+{\mathit{Z}}_{ij}{\mathit{c}}_i,i=1,\dots ,I,j=1,\dots ,n_i$$ (1)

where i and j refer to participant j in trial i. The expectation of the outcome for participant j, in trial i when assigned to the kth intervention, T_ij = k, is given by E[Y_ij|T_ij = k]. α is a q × 1 vector of fixed effects associated with baseline covariates X_ij which could also include trial-level covariates, although we focus here on individual-level covariates which extend the model when individual participant data is available, and a 1 × p row vector Z_ij are effect modifiers of intervention; n_i is the number of participants in trial i. Here T_ij = k indicates that participant j in trial i is assigned to intervention k, which can take at least one value in 1, ..., K. A scalar β_k and a p × 1 vector γ_k are fixed effects to denote the main effect of intervention and the effect modifiers. $b_i\sim N_1(0,{\sigma }_b^2)$ and c_i ~ N_p(0, Σ_cc) are the corresponding random effects for trial i. Of note, we consider random effects of trials instead of fixed effects to capture variability among trials. Please also see Section 3 on an example of such data. Note that β_k and b_i are scalar while γ_k and c_i are p-dimensional vectors of the same dimension as Z_ij. This generalized linear model can incorporate linear, logistic, and log-linear regression, as well as a wide range of other models.

Define the (p + 1) × (p + 1) covariance matrix of b_i and c_i as $\Sigma =\left(\begin{array}{cc}\hfill {\sigma }_b^2\hfill & \hfill {\Sigma }_{bc}\hfill \\ \hfill {\Sigma }_{cb}\hfill & \hfill {\Sigma }_{cc}\hfill \end{array}\right)$ which does not vary with trial i. The vectors (b_i, c_i) are assumed to be independent across trials. In generalized fixed-effects modeling, we also assume that the variance in Y is some function of its mean, Var(Y) = ϕh(E(Y)).

We define the trial level mean and effect modifier

$$m_{i,k}={\beta }_k+b_i$$ (2)

$${\mathit{w}}_{i,k}={\gamma }_k+{\mathit{c}}_i$$ (3)

Note that in this first model the trial affects all interventions identically by shifting the overall intercept and moderator effect. It is also noted that X_ijα in equation (1) produces an overall adjustment for the baseline values of the covariates in X.

As a final remark in this subsection on the level of moderator variables, we contrast our analyses with pairwise meta-analysis, which does not have access to participant-level data. Meta-analysis almost never includes participant level moderators as these analyses are generally not reported. In the absence of reported participant-level moderator analyses, meta- analysis sometimes resorts to the use of trial-level covariates, such as the average baseline level of risk, to provide some clues about how the intervention effect varies by, say participant level of risk. This can easily produce ecological fallacies, as we show later in the paper. But other subtleties arise when considering moderation both at participant level and trial level when IPD are available. For example, there are circumstances where it is appropriate to include trial-level covariates as having an effect on participants over and above participant- level covariates. As an illustration, it is possible that a trial conducted in hospitals that treat very sick children will use a more intensive behavioral intervention than one in a hospital with less sick children. We leave such investigations to a future paper.

2.2. Heterogeneity in interventions across trials

An extension to model (1) is to allow different random effects for each intervention arm in each trial, i.e., the trial by intervention interaction:

$$g\left(E[Y_{ij}\mid T_{ij}=k]\right)={\mathit{X}}_{ij}\alpha +{\beta }_k+b_i+b_{i,k}+{\mathit{Z}}_{ij}({\gamma }_k+{\mathit{c}}_i+{\mathit{c}}_{i,k}),i=1,\dots ,I,j=1,\dots ,n_i,$$ (4)

where α is a q × 1 vector of coefficients of baseline covariates X_ij; as with all moderator models, the parameters for any covariate in X_ij that also appears in Z_ij cannot be interpreted in isolation from its moderator effect. The random effects for trial by intervention interaction are distributed as $b_{i,k}\sim N_1(0,{\sigma }_{b,k}^2)$ and the random effects for trial × intervention × effect modifier are distributed as c_i,k ~ N_p(0, Σ_cc,k). Define the covariance matrix of b_i,k and c_i,k as ${\Sigma }_k=\left(\begin{array}{cc}\hfill {\sigma }_{b,k}^2\hfill & \hfill {\Sigma }_{bc,k}\hfill \\ \hfill {\Sigma }_{cd,k}\hfill & \hfill {\Sigma }_{cc,k}\hfill \end{array}\right)$. As before, these random effect vectors are assumed to be independent across trials. Random effects for the two interventions within the same trial can be correlated with one another, independent, or degenerate (e.g., b_i,k = b_i,k′ with probability one); however, we require that the covariance of b_i,k and b_i,k′ within a trial, Cov(k, k′), be the same across trials. Thus all the random effects for all trials are assumed to have the same underlying variance-covariance structure. The simplest case is to assume all the off-diagonal sub matrices are 0, but correlated random effects (e.g., random intercepts for control and active intervention) are often scientifically warranted [38] and also necessary for valid meta-analytic inferences for non-linear models [39]. It may be appropriate to equate some of the Σ elements, as we do in the example in Section 4.

The trial level main and modification effects for this more general model (4) are

$$m_{i,k}={\beta }_k+b_i+b_{i,k}$$ (5)

$${\mathit{w}}_{i,k}={\gamma }_k+{\mathit{c}}_i+{\mathit{c}}_{i,k}$$ (6)

The presence of random effects b_i,k and c_i,k allow heterogeneity in intervention effects across trials. Differences in these coefficients represent differences between interventions within a trial (e.g., m_i,k – m_i,k′) or differences between trials for the same intervention (e.g., m_i,k – m_i′,k). Contrasts involving mixed effects m_i,k – m_i,k′ over i correspond to moderation across trials, which are used in forming indirect comparative effectiveness estimates.

Model (4) can be specified in a number of ways. For example, one can replace fixed effects β_k by β_i,k and γ_k by γ_i,k and drop the random effects entirely. Other extensions, for example, introducing participant-level random effects as in growth modeling or random slopes for covariates X [40] are not discussed here as they add notational complexity without changing the underlying model.

2.3. Directly and indirectly estimable main effects and moderator effects

The realized model for a set of trials is determined by which interventions are available for each trial. We define an estimable direct effect as a simple contrast between two interventions that are realized in at least one trial. Define H to be the I × K indicator matrix of interventions for each trial. Then the contrast k, k′ is directly estimable if there exists at least one trial i such that H_i,k = H_i,k′ = 1.

2.3.1. Derived Contrasts along a Path

In addition to basic contrast parameters within a trial, the patterns of interventions tested across the collection of trials can also provide additional information about contrasts. We define a contrast between interventions k and k′ to be indirectly estimable if there is at least one pathway involving two or more trials connecting these endpoints. In Figure 1 a step from one intervention to another is horizontal, and a step from one trial to another is vertical. Specifically, a path of length u > 1 starting in trial i at intervention k and ending in trial i′ at intervention k′ can be represented by the following steps: let k₀ = k, i₀ = i, k_u = k′, and i_u = i′ and then S₀ = (i₀, k₀) and S_u = (i_u, k_u), where each step is either horizontal, i.e., i_l = i_l–1 (and k_l = k_l–1 – 1 or k_l–1 + 1), or vertical, i.e., k_l = k_l–1 (and i_l = i_l–1 – 1 or i_l–1 + 1), and all H_il,kl are 1, l = 1, . . ., u. The indirect value of the mean difference between k and k′ through a path S is defined in terms of the expected values m_kj,

$$d(k,k^{\prime };S)=\sum _{l=1}^u(m_{i_l,k_l}-m_{i_{l-1},k_{l-1}})$$ (7)

$$\delta (k,k^{\prime };S)\equiv E\left(d(k,k^{\prime };S)\right)=\sum _{l=1}^u({\beta }_{k_l}-{\beta }_{k_{l-1}})$$ (8)

where the expectation is over the trial level random effects. Similarly the indirect response difference in k and k′ involving moderation at covariate level Z is

$$r(k,k^{\prime };\mathit{Z},S)=\sum _{l=1}^u[(m_{i_l,k_l}+{Zw}_{i_l,k_l})-(m_{i_{l-1},k_{l-1}}+{Zw}_{i_{l-1},k_{l-1}})]$$ (9)

$$\rho (k,k^{\prime };\mathit{Z},S)\equiv E\left(r(k,k^{\prime };\mathit{Z},S)\right)=\sum _{l=1}^u\left[({\beta }_{k_l}+\mathit{Z}{\gamma }_{k_l})-({\beta }_{k_{l-1}}+\mathit{Z}{\gamma }_{k_{l-1}})\right]$$ (10)

Note that the covariates in Z must be available and have the same definition in all trials. In addition, their joint distributions must have sufficient overlap across trials.

Figure 1.

A diagram of network with five interventions labeled A – E

Figure 1 provides a convenient way to represent the set of trials and interventions, using a connected, non-directed graph. Unlike other figures for network meta-analysis [41, 42], this figure retains information on which interventions are tested within each separate trial. Interventions are labeled A-E and subscripted by the trial number. Horizontal ties represent directly estimable contrasts in the same trial, for any pair of interventions on the same row. Vertical ties represent trials having the same intervention. Any pair of interventions not directly estimable but connected through some path has an indirectly estimable contrast. In this example the directly estimable contrasts are AB, BC, and AC (from Trial 1), BD (from Trial 2), and DE (from Trial 3). One indirectly estimable contrast is AD, where B serves as a common comparator [33] in the three-step path from A₁ to B₁, B₁ to B₂, and B₂ to D₂. Other indirectly estimable contrasts are AE, BE, CD, and CE. The last trial only includes intervention D.

2.4. Definitions of Homogeneity and Consistency

The terms homogeneity and consistency have been used in network meta-analysis [41] to refer to the assumptions required for interpreting contrasts across trials as causal effects. Violations of these conditions would imply that biases exist that would confound causal inferences of effects across trials. The term consistency is used by Lu and Ades [43] and Cooper et al. [44] to describe the condition where direct and indirect comparisons of two interventions do not differ. We have reviewed previous references to these concepts [41, 44] and applied these ideas to both main effects and moderation, including fixed-effects and mixed-effects models. Our own definitions follow.

Definition

Weak homogeneity for contrast k, k′ occurs if for all trials i that assign participants to interventions k or k′, E(m_i,k – m_i,k′) ≡ β_k – β_k′ does not vary with trial. We say there is strong homogeneity if in addition Var(m_i,k – m_i,k′) = 0 for all i. Similarly weak homogeneity of moderation comparing k and k′ is said to occur if E(w_i,k – w_i,k′) ≡ γ_k – γ_k′ does not depend on the trial. Strong homogeneity of moderation comparing k and k′ also requires that Var(w_i,k – w_i,k′) = 0. Likewise we say there is weak homogeneity for intervention k if E(m_i,k) does not vary with trial and strong homogeneity for intervention k if Var(m_i,k) = 0 for all i. Weak and strong homogeneity of moderation for intervention k are defined similarly.

When we use the terms “weak or strong homogeneity” without any modifiers (e.g., leaving off contrasts and conditioning), this shorthand will denote that all available contrasts k, k′ satisfy the conditions on m_i,k – m_i,k′ above. For clarity, if there are no trials or only one trial that assigns participants to interventions k or k′ we say homogeneity still holds.

Definition

There is weak consistency for contrast k, k′ in a collection of trials if there is the same expectation across any direct or indirect path. Weak consistency thus requires β_k – β_k′ to be the same for every trial having both k and k′ interventions, and that these are equal to δ(k, k′; S) for any realized path S. We use a similar definition of consistency for moderation involving k, k′, namely ρ(k, k′, Z, S) is constant over all paths S. Strong consistency requires no between-trial level variance in these quantities.

2.4.1. Testing for and estimation under homogeneity and consistency

Formal testing for homogeneity under model (4), e.g., verifying that E(m_i,k – m_i,k) ≡ β_k – β_k′ is constant over trials, can be done under certain circumstances where assumptions about trial level variance can be made. Define the subset of trials J_k,k′ = {i : H_i,k = H_i,k′ = 1} where two interventions are tested. By equation (5) the quantities m_i,k – m_i,k′ for i ∈ J_k,k′ have means β_k – β_k′ and between trial variance given by $v^2(k,k^{\prime })={\sigma }_{b,k}^2+{\sigma }_{b,k^{\prime }}^2-2Cov(k,k^{\prime })$ where Cov(k, k′) is the covariance of b_i,k and b_i,k′, which is the same across trials. Let m̂_i,k – m̂_i,k′ be an asymptotically unbiased estimate of m_i,k – m_i,k′ and se_i its within-trial standard error. If we are willing to make the assumption that v²(k, k′) = 0, then the sum of squares (SSQ) ratio over trials J_k,k′, $SSQ({\widehat{m}}_{i,k}-{\widehat{m}}_{i,k^{\prime }})∕{\sum }_{i}s{e}_i^2$ has asymptotically noncentral χ² distribution with noncentrality proportional to $SSQ({\beta }_k-{\beta }_{k^{\prime }})∕\sum \left(E\left(s{e}_i^2\right)\right)$. Thus a test of homogeneity for k, k′, i.e., SSQ(β_k – β_k′) = 0 can be made under the assumption of no trial-level random effects. If we make no assumptions about v² we cannot distinguish between the variation in the fixed effects versus trial level random effects and thereby cannot form a test of weak homogeneity. But if we assume that v²(k, k′) = constant over all k, k′ we can compute its marginal maximum likelihood estimate and form such a test. Thus a rejection of the hypothesis that all v²(k, k′) are constant may suggest disconfirmation of weak homogeneity. In section 4 we investigate both of these tests in the context of a real example.

Similarly we can test for consistency by comparing the fit of a consistency restricted model with an unrestricted one. Again restrictions on the random effects distribution are required.

Estimation under strong or weak homogeneity and consistency in a frequentist approach is based on a generalized mixed model using marginal maximum likelihood (MML) [36] for the fixed participant-level effects as well as regression coefficients at the trial level, and restricted maximum likelihood (REML) for the variance-covariance parameters of the random effects, subject to restrictions.

3 Power analysis simulations

We pose two basic questions in this section. First, are there benefits in statistical power for examining a main effect contrast between two interventions when using participant-level data network analysis compared with aggregate data network meta-regression analysis that is based on combining summary statistics across multiple trials? Secondly, how is statistical power for contrasting moderation effects for two interventions affected when we combine participant-level data across multiple trials? Is the gain in power for moderation analysis using indirect estimates as large as that for direct estimates [17]? For both questions we examine a comparison of two interventions, B versus C, within a network of two-arm trials involving interventions A, B, and C. We consider two types of networks of trials. We call the first network an open loop because no trials compare B and C directly. In these open loop trial networks only indirect estimation of the B-C contrast is possible; see Figure 2. The second type involves trials that form a closed loop trial network with three interventions as in Figure 3; here trials directly compare interventions with each other so that both direct and indirect comparisons can be combined.

Figure 2.

A scheme of an open loop trial network with three interventions A, B, and C. The broken line indicates B and C are not tested head-to-head, though an indirect comparison is estimable by μ_BC = μ_BA – μ_CA.

Figure 3.

A scheme of a closed loop trial network with three Interventions A, B, and C. The solid lines indicate that any pair of interventions are tested head-to-head. For Interventions B and C, for example, there are direct μ_BC = μ_B – μ_C and indirect μ_BC = μ_BA – μ_CA estimates.

Monte Carlo simulations were conducted to investigate the added value of using IPD in the context of a network meta-analysis which involves three interventions, A, B, and C. Our task was to estimate both main effects of the comparison of B vs. C and the moderating effects of a binary covariate on the same B vs. C comparison. We simulated situations involving a continuous outcome (e.g., depression score) in analytic models with and without a two-level moderator variable at the participant level (e.g., gender) and varying numbers of trials. Some simulation conditions included trials with arms B and C (closed loop), and others did not (open loop). These simulations were designed to assess the gain in statistical power attained by using participant-level data compared with summarized trial-level data when either or both main effects and moderator effects were allowed to vary across trials. Global homogeneity and path homogeneity were assumed in this mixed-model framework. The following parameters were manipulated in the simulation: number of trials, effect sizes, distribution of moderation effects across trials, levels of trial heterogeneity and sample sizes. The number of trials of each type (AB), (AC), and (BC) was set to R = 2, 3, 4, 8; the closed loop designs included (BC) trials while the open loop designs excluded such trials. The effect sizes for both main and moderating effects were set to 0.25, 0.5, or 0.75, representing small, medium and large effects, respectively. Sample sizes were selected for each of these three effect sizes based on having 80% power to detect the main effect in a single trial.

Below we specify a model for generating the response Y_rij from participant j in trial i (e.g., i = 1, . . ., 3 = I of trial types (AB), (AC), and (BC)) for the r-th replicate r = 1, . . ., R. For the main effects, we generate data according to the following model:

$$E[Y_{rij}\mid (T_{rij}=k)]={\alpha }_r+{\beta }_{r,k}+a_{ri}+b_{ri,k},i=1,\dots ,I,j=1,\dots ,n_i,r=1,\dots ,R$$ (11)

where $a_{ri}\sim N(0,{\sigma }_a^2)$, r = 1, . . ., R is the additional random effects for the ith trial for the rth replicate, and $b_{ri,k}\sim N(0,{\sigma }_b^2)$ has one more subscript r to denote that it is repeated r times. The conditional mean m_ri,k for intervention k is α_r + a_ri + β_r,k + b_ri,k. We model the observed arm-level means for each intervention in the ith trial for the rth replicate, Ȳ_ri,k as follows:

$$E\left[{\stackrel{‒}{Y}}_{ri,k}\right]={\alpha }_r+a_{ri}+{\beta }_{r,k}+b_{ri,k},k\in \{A,B,C\};$$ (12)

which is the same as model (11). The comparison between intervention arms, e.g., A vs. B, can be obtained by testing the contrast E[Ȳ_ri,k] – E[Ȳ_ri,k′] for k = A and k′ = B.

To investigate the differences between trial level and IDA under moderation, a binary covariate (gender) was used. Trial level analyses involved only the proportion of females in each trial; this proportion was set to π = 0.3, 0.5, 0.7 for trials that compared AB, AC and BC, respectively. For the moderation effect, the data were generated based on the following model:

$$E[Y_{rij}\mid (T_{rij}=k)]={\alpha }_r+{\beta }_{r,k}+a_{ri}+b_{ri,k}+Z_{rij}({\delta }_r+b_{ri}+{\gamma }_{r,k}+c_{ri,k}),$$ (13)

where Z_rij is an indicator for gender of participant j in the ith trial for the rth replicate. The values of Z_rij are generated from a Bernoulli distribution with probability π. The fixed effect for gender main effect and effect modification are denoted by δ_r and γ_r,k, respectively; the random effects for the gender main effect and modification effect between intervention and gender are $d_{ri}\sim N(0,{\sigma }_d^2)$ and $c_{ri,k}\sim N(0,{\sigma }_c^2)$, respectively. The remaining parameters were similarly defined in Model (4).

The trial-level moderator is the percent of female participants for each specific intervention arm within each trial. Thus the moderator is a continuous variable reflecting the percentage of females in each arm. However, in the participant-level analysis, the moderator is a dichotomous variable reflecting the gender of participant. The size of the aggregate dataset is twice the number of trials, each observation representing an arm-level mean of intervention within a trial. The arm-level aggregation allows us to fit a model which includes a trial effect, intervention effects, and the modification effects between interventions and gender proportion, and a mixed-effects model is specified as

$$E\left[{\stackrel{‒}{Y}}_{ri,k}\right]={\alpha }_r+{\beta }_{r,k}+a_{ri}+b_{ri,k}+{\stackrel{‒}{Z}}_{ri}({\delta }_r+d_{ri}+{\gamma }_{r,k}+c_{ri,k}),$$ (14)

where Z̄_ri is a proportion of females in trial i at the rth replicate.

We also varied the level of variation in impact across trials from small to large [45], setting the values of σ_a = 0.10 and σ_b = 0.25, or σ_a = 0.25 and σ_b = 0.10. We also set the values of σ_c = σ_d = 0 to simplify the computation. Participant-level standard deviation is set at 1.0. Trial sample sizes were fixed based on the overall effect size. The required sample sizes to achieve at least 0.8 power for direct comparison of B versus C in a single trial were set at 250 for effect size of 0.25, 65 for effect size of 0.5, and 30 for effect size of 0.75. We used 1000 datasets for computing the statistical power for comparing B versus C under direct and indirect comparisons. Figures 2 and 3 depict the two types of comparisons of interest. The open loop network in Figure 2 allowed for only an indirect comparison of B versus C since the two interventions had not been compared head-to-head (broken line). The closed loop network in Figure 3 allowed for both indirect and direct estimates of comparison of B versus C.

3.1. Results

Based on the parameter specifications given above, we fit the proposed models using a marginal maximum likelihood approach. The simulation results for power calculations of main effects of Intervention B versus Intervention C are presented in Table 1. Power estimates for testing the main effects at the trial-level are only modestly lower than those at the participant-level for all conditions with more than one trial per type. This finding is not surprising because an analysis based on the intervention means at the trial level is close to the two-stage IPD model when the sample sizes are the same [46]. In particular, the small decline in power for trial level analyses is due to treating the within study sample variances as fixed and known, whereas the participant-level analyses pool these estimates. Also in Table 1 we found virtually identical power for both the trial-level and participant-level analyses for a main effect on the closed loop trial data. This is expected to occur because the arm-level means used in the trial-level analyses are sufficient statistics. There exist, however, measurable differences in power for the open-loop trials because of the fact that there is 50% more data for the closed loop trials, resulting in greater power in estimating individual variances.

Table 1.

Mean and standard error (SE) of estimate of power for main effects in the comparison of Intervention B against Intervention C across multiple trials when the ratio of SD_trial×treat(σ_b) to SD_trial(σ_a) is 0.4 in Model (11)

		Open Loop				Closed Loop
		Trial Level		Participant Level		Trial Level		Participant Level
Effect	Trials per Type	Mean	SE	Mean	SE	Mean	SE	Mean	SE
.25 (n = 250)	2	.11	.01	.18	.012	.30	.015	.29	.015
	3	.41	.015	.59	.015	.75	.014	.75	.014
	4	.61	.015	.74	.014	.91	.009	.91	.009
	8	.92	.009	.97	.006	1.0	.001	1.0	.001
.5 (n = 65)	2	.09	.009	.12	.01	.49	.016	.48	.016
	3	.60	.015	.67	.015	.96	.006	.96	.006
	4	.83	.012	.88	.01	1.0	.001	1.0	.002
	8	.99	.002	1.0	.002	1.0	0	1.0	0
.75 (n = 30)	2	.09	.009	.11	.01	.56	.016	.55	.016
	3	.67	.015	.73	.014	.98	.004	.98	.004
	4	.90	.009	.92	.009	1.0	0	1.0	0
	8	1.0	.001	1.0	.001	1.0	0	1.0	0

SD = standard deviation.

For Table 2, where the total variance is the same as in Table 1 but the trial by intervention variance is much higher than the trial variance, we find an overall decline in statistical power but more benefit from participant-level analysis compared with study-level analysis in the open loop case than we found for Table 1.

Table 2.

Mean and standard error (SE) of estimate of power for main effects in the comparison of Intervention B against Intervention C across multiple trials when the ratio of SD_trial×treat(σ_b) to SD_trial(σ_a) is 2.5

		Open Loop				Closed Loop
		Trial Level		Participant Level		Trial Level		Participant Level
Effect	Trials per Type	Mean	SE	Mean	SE	Mean	SE	Mean	SE
.25 (n = 250)	2	.11	.01	.27	.014	.14	.011	.14	.011
	3	.23	.013	.50	.016	.27	.014	.27	.014
	4	.32	.015	.58	.015	.37	.015	.37	.015
	8	.47	.016	.73	.014	.69	.015	.69	.015
.5 (n = 65)	2	.13	.01	.22	.013	.27	.014	.26	.014
	3	.39	.015	.58	.015	.63	.015	.64	.015
	4	.55	.016	.72	.014	.82	.012	.82	.012
	8	.85	.011	.94	.008	.99	.003	.99	.003
.75 (n = 30)	2	.11	.01	.17	.012	.37	.015	.37	.015
	3	.52	.016	.65	.015	.86	.011	.86	.011
	4	.72	.014	.82	.012	.96	.006	.97	.005
	8	.95	.007	.98	.004	1.0	0	1.0	0

SD = standard deviation.

We also compared study-level and participant-level analyses without random effects of trial or trial by intervention. These simulations (not shown) produced identical results for study-level and participant-level analyses for both open loop and closed loop analyses, as one would expect.

We now summarize the findings on main effects, comparing study-level and participant-level main effects estimation. In terms of the utility of using participant-level data in estimating comparative effectiveness, we conclude that the only gain in power occurs when the number of trials of each type is small, there are no trials where the effect of interest (B vs. C) is directly estimable, and it is appropriate to pool variances.

Tables 3 and 4 show the calculated power of the test of moderation effects in the direct and indirect comparisons of B vs C. The results in these two tables show that the tests based on summary statistics at the trial level are very underpowered for both direct and indirect comparisons, and there is much higher power when participant-level data are used. When the number of trials of each type is increased, power increases much more rapidly for individual-level analysis, compared with that for study level. This implies that when conducting a network meta-analysis, one is almost certain to have substantially higher statistical power for detecting a moderation effect based on participant-level data compared with trial-level data. In addition, open-loop networks have somewhat lower power than closed-loop networks for the respective indirect and direct comparison of B and C.

Table 3.

Mean and standard error (SE) of estimate of power for moderation effects in the comparison of Intervention B against Intervention C across multiple trials (proportions of females: 0.3 (A vs B), 0.5 (A vs C), 0.7 (B vs C)) and the ratio of SD_trial×treat(σ_b) to SD_trial(σ_a) is 2.5

		Open Loop				Closed Loop
		Trial Level		Participant Level		Trial Level		Participant Level
Effect	Trials per Type	Mean	SE	Mean	SE	Mean	SE	Mean	SE
.25 (n = 250)	2*	.04	.006	.49	.016	.06	.007	.76	.014
	3	.10	.009	.29	.014	.58	.015	.64	.015
	4	.08	.009	.58	.015	.59	.015	.84	.012
	8	.07	.008	.94	.007	.60	.015	.99	.012
.5 (n = 65)	2*	.04	.007	.48	.016	.08	.009	.78	.013
	3	.07	.009	.35	.015	.33	.015	.71	.014
	4	.08	.009	.62	.015	.35	.015	.90	.009
	8	.08	.009	.96	.006	.36	.015	1.0	0
.75 (n = 30)	2*	.04	.006	.47	.016	.08	.009	.80	.013
	3	.05	.007	.33	.015	.17	.012	.69	.015
	4	.07	.008	.60	.015	.24	.013	.92	.009
	8	.09	.009	.96	.007	.32	.015	1.0	0

^*Calculated separately with fixed-effects assumptions.

SD = standard deviation.

Table 4.

Mean and standard error (SE) of estimate of power for moderation effects in the comparison of Intervention B against Intervention C across multiple trials (proportions of females: 0.3 (A vs B), 0.5 (A vs C), 0.7 (B vs C)) and the ratio of SD_trial×treat(σ_b) to SD_trial(σ_a) is 0.4

		Open Loop				Closed Loop
		Trial Level		Participant Level		Trial Level		Participant Level
Effect	Trials per Type	Mean	SE	Mean	SE	Mean	SE	Mean	SE
.25 (n = 250)	2*	.04	.006	.49	.016	.06	.007	.76	.014
	3	.03	.006	.30	.015	.21	.013	.70	.015
	4	.06	.008	.59	.015	.25	.014	.91	.009
	8	.07	.008	.95	.007	.27	.014	1.0	0
.5 (n = 65)	2*	.04	.007	.48	.016	.08	.009	.78	.013
	3	.03	.005	.36	.015	.10	.009	.73	.014
	4	.05	.007	.63	.015	.13	.01	.93	.008
	8	.07	.008	.97	.006	.20	.013	1.0	0
.75 (n = 30)	2*	.04	.006	.47	.016	.08	.009	.80	.013
	3	.04	.006	.34	.015	.08	.009	.72	.014
	4	.03	.005	.60	.015	.13	.011	.93	.008
	8	.06	.007	.96	.006	.24	.014	1.0	0

^*Calculated separately with fixed-effects assumptions.

SD = standard deviation.

4 Application to two simultaneous, parallel trials

In this section we present an example of network meta-analysis testing moderation of intervention effects using participant-level data. The data come from a large preventive field trial having two active interventions and one control. Interventions were assigned to first grade classrooms within schools in an imbalanced, incomplete block design where the two active interventions were never tested against one another. Each school can be considered a separate sub-trial as schools were randomly assigned to test classroom intervention B – the Good Behavior Game – against a control (A) or intervention C – Mastery Learning – against a control (A). First grade classes within schools were balanced on child's kindergarten grades and behavior prior to classes being randomly assigned to intervention within each school so that direct comparison of each intervention could be made against the control. Separate intervention teams provided training and supervision to teachers for both the Good Behavior Game and Mastery Learning. We chose not to allow two active interventions to be administered in classes within the same school as this would double the number of times intervention trainers would need to travel to that school [17]. In addition, having only one active intervention in a school reduces the possibility of leakage of intervention effects from one class to another.

Because there was no direct comparison of the two active interventions in the same school, this can be viewed as an open loop network where each school can be considered as if it were a separate cluster-randomized trial with classroom as the unit of randomization within school (see Figure 4). Random assignment of schools to the two trial designs, either A-B or A-C, allows us to treat the pool of schools as a collection of highly similar trials, thus permitting us to apply the methods for participant-level network meta-analysis. Interest here is on the indirect comparison of the two active interventions’ effects for the entire population of schools as well as testing whether these indirect effects were moderated by gender. Also of interest is the comparison of controls across schools as a test of partial homogeneity of this control; differences in these controls by type of design would suggest intervention contamination or leakage, which would complicate any findings. Network pathways allowing indirect comparisons of the Good Behavior Game and Mastery Learning all have vertical paths through the control classrooms. We have no direct comparisons of the two active interventions, so we cannot test consistency. Because of classrooms in the same school having the same intervention, it is possible to pool results for these similar classrooms to estimate the standard error at this level. We can use this to test for partial homogeneity of the contrast between the two active interventions, as well as assess their overall difference. If the test of weak heterogeneity indicated that control groups differed significantly, this would call into question inferences based on indirect comparisons.

Figure 4.

A diagram of network with three interventions labeled GBG (Good Behavior Game), ML (Mastery Learning), and Control.

More specifically, the Baltimore Prevention Trial (BPT) [16, 47] was a randomized controlled trial with two preventive interventions tested in parallel. The Good Behavior Game (GBG) is a group-contingent behavioral intervention aimed at reducing aggressive or disruptive behavior [48]. The second intervention, Mastery Learning (ML) is curriculum based and was aimed at improving reading achievement [49]. The study involved five large urban areas within Baltimore City, each having three or four similar elementary schools. The total number of students in the study was 1196. Assignment of school, and therefore of intervention trial type (GBG vs control, ML vs control) was equally balanced across these urban areas. The schools were randomly divided into 6 schools for testing GBG and 7 schools for ML. In the 13 schools with an active intervention, one first-grade classroom was randomly assigned to receive the control and the remaining one or two classes were given that particular intervention. A total of 8 GBG classes within the 6 GBG-testing schools received GBG, and 6 classrooms served as controls. Similarly, in the seven schools where ML was being tested, 9 classrooms were randomly selected to serve as ML intervention classrooms and 7 classrooms served as controls. Because GBG and ML were not compared head-to-head, our interest is to compare GBG and ML indirectly via an open-loop network meta-analysis method discussed in Section 2. For this paper we use as an outcome measure the proportion of 10-second time intervals that the child was observed to be off-task towards the end of first grade, obtained through direct observation of children during times where they were reading alone at their desks. Because of the relatively large number of observations available at the individual level (e.g., 40 minutes of 10-second observations), this estimated proportion is transformed with an arcsine square-root transformation [50] to stabilize variances and achieve approximate normality. Reduction of off-task behavior was a direct target of GBG and an indirect target of ML. Since some students were absent on some days during observation, we assumed missing at random for valid analysis using only observed data.

Below we present a multilevel model that is then used to make a partial test of weak and strong homogeneity. The results of this test are then used to conduct an appropriate analysis to compare both the main effect as well as a moderator effect of gender on the comparison of GBG and ML.

4.1. Multilevel model

We first fit a four-level mixed effects model with fixed intervention and random effects at the urban area, school, classroom and participant levels. Let Y_risj be an observation on off-task behavior from the rth urban area, ith school (or trial), in the sth class, and from the jth participant, where r = 1, . . ., R, i = 1, . . ., I_r, s = 1, . . ., F_ri, and j = 1, . . ., n_ris. Let X_risj represent a vector of baseline off-task behavior and gender (1 for male, 0 for female), and Z_risj is an indicator for gender. Following the same notational rule of Model (1), a multilevel model is given as

$$g\left(E[Y_{risj}\mid T_{risj}=k]\right)={\mathit{X}}_{risj}\alpha +{\beta }_{ris,k}+Z_{risj}{\gamma }_{ris,k}+Z_{risj}{c}_{ri,k}+b_r^I+b_{ri}^{II}+b_{ris}^{III}$$ (15)

where the parameters have the same definitions as (1) with the exception that c_ri,k is now a fixed effect; the random effects are distributed as

$$\begin{array}{cc}\hfill b_r^I& \sim N(0,{\sigma }_I^2)\hfill \\ \hfill b_{ri}^{II}& \sim N(0,{\sigma }_{II}^2)\hfill \\ \hfill b_{ris}^{III}& \sim N(0,{\sigma }_{III}^2)\hfill \end{array}$$

and are assumed to be independent. Note that this model assumes that the variance for each intervention across schools, ${\sigma }_{II}^2$ does not depend on the intervention.

In this example, neither of the assumptions of weak homogeneity for GBG versus its control and ML versus its control, can be tested directly because variation in these values across trials is confounded with a random effect that is used to capture variation in intervention differences across schools. This pooled variation at the school level, however, is quite small, ${{\widehat{\sigma }}^2}_{II}=0.003$, SE = 0.0123, suggesting that any departure from strong homogeneity is relatively small. We can conduct tests of weak homogeneity as well. The first of these tests whether the school level variation in GBG versus its control is the same as that for ML versus its control. If these variances differ, it may indicate one intervention in this parallel trial shows more variation in impact than the other. One possible cause for this could be some variation in leakage of an active intervention from one class into a control class in the same school, a potential that we have investigated elsewhere [51]. For this example the ratio of between school variances for ML and GBG effects relative to their own controls is F_4,4 = .0240/.0027 = 8.89, p = .0289, where the number of degrees of freedom, 4, is based on 5 school error contrasts per intervention effect nested within 5 urban areas. This shows that there is more variation between ML schools as compared with those schools under the GBG intervention, suggesting greater leakage in ML schools. Another potential explanation is that greater variation in ML classrooms occurred compared to that for GBG, but we found no evidence for this in the data.

We have also tested for homogeneity of controls in schools assigned to test the Good Behavior Game versus controls assigned to test Mastery Learning. It is well known that tests of strong homogeneity where the variance estimate is based on a small number of trials (i.e., schools here) are often underpowered. Thus we consider an alternative test that is based on means rather than variances. Using our multilevel model above, this estimate of the mean difference between GBG control and ML control is 0.138, se = 0.090, df = 16.4, p = 0.142. Thus there is no evidence to reject homogeneity in the controls. This finding of equivalent controls allows us to consider the indirect comparison of GBG vs. ML.

4.2. Model results

Tables 5 and 6 present results of a multilevel model based on an IPD network meta-analysis approach for comparing the moderated effect of gender on the comparison of GBG against ML. Since the similarity of the GBG and ML controls was tested above, and it was found that there was not any significant difference between the controls, we have reasonable con dence that an indirect test of differences of differences between these two trials will provide useful inferences about the impact of the Good Behavior Game versus Mastery Learning.

Table 5.

Specific contrasts of levels of interventions (Good Behavior Game (GBG), Mastery Learning (ML) and Control) in network meta-analysis by off-task behavior occurrences after adjusting for baseline off-task behavior

Effect	Estimate	Std Error	DF	Pvalue
Baseline off-task	.123	.046	388	.008
Gender (male)	.088	.033	379	.004
Intervention (contrasts)
GBG vs. ML	−.239	.084	8.96	.020
GBG vs. Control	−.287	.076	11.6	.003
ML vs. Control	−.047	.075	11.1	.545

Random effects	Variance (ICC)
Urban	.009 (.080)	.011
School	.004 (.037)	.013
Classroom	.011 (.100)	.012

ICC = intra-class correlation coefficient.

Table 6.

Gender as effect modifier of interventions (Good Behavior Game (GBG), Mastery Learning (ML) and Control) in a network meta-analysis

Effect	Estimate	Std Error	DF	Pvalue
Baseline off-task	.124	.046	386	.007
Gender (male)	.103	.048	377	.031
GBG vs. Control	−.247	.086	18.7	.003
ML vs. Control	−.055	.083	16.2	.519
GBG*Gender (male)	−.074	.074	376	.32
ML*Gender (male)	.017	.072	376	.813

Random effects	Variance (ICC)
Urban	.009 (.080)	.011
School	.003 (.031)	.013
Classroom	.012 (.105)	.012

ICC = intra-class correlation coefficient.

After establishing the similarity between the two controls, we combined the effects for controls of GBG and ML, effectively allowing us to use them as a single control in estimating the indirect effect of GBG versus ML. As is depicted in Table 5, the estimated indirect effect of GBG versus ML is –.239 and a standard error of .084, providing strong evidence that GBG is more effective in reducing classroom off-task behavior compared with the ML intervention. We did not find any significant effect modifications involving gender (see Table 6).

5 Discussion

In this paper we examined how much power is needed to detect main and moderator effects in pairwise meta-analysis and network meta-analysis. We found that power is shaped by three factors: inclusion of both direct and indirect effects using network meta-analytic models, inclusion of moderator variables, and use of participant-level data. The findings indicate that the power of detecting effects depends on all three factors. Synthesis analyses have greater power when direct and indirect effects are combined, as compared with analyses that only include indirect effects. This holds for both main effect and moderator analyses, and occurs when analyses involve either summary (i.e., aggregate data meta-analytic) or participant-level data. Trials with participant-level data have a far greater statistical power for detecting moderator effects regardless of whether we are testing direct or indirect effects. Use of meta-regression with arm-level or trial-level summary data results in much lower power: under some conditions the ratio is on the order of 12:1 in favor of participant-level data. These power differences will also be in uenced by other factors including actual effect sizes, the number of trials, the number of participants per trial, and the amount of variability in the aggregated moderator across trials. In our example the empirically based power estimates are low for both methods, with a ratio favoring moderated IDA of only 1.5, largely because of the weak and nonsignificant moderator effects found here.

For comparative effectiveness using network meta-analysis, assessing homogeneity and consistency assumptions is very important. We used these assumptions in both main effects and moderation effects under mixed-effects models, separately for IPD and study-level analysis. In the case of main effects, trial-level and participant-level analyses of networks of trials give similar results. This is expected since the summary values are sufficient statistics [46].

The situation is quite different for testing participant-level moderator effects using participant-level data versus meta-regression that uses trial-averaged scores to test for moderation. Based on our simulations we urge extreme caution in interpreting trial-level moderation analysis at the participant level. We note that Donegan et al. [7] point out that IPD models can be specified such that within-trial moderation and between-trial moderation are forced to be equal. The model we present here does allow for tests of this equivalence across participant and trial level. They also note that this equivalence can be extended to other synthesis projects that integrate aggregate data and participant-level data. While this combining of IPD and aggregate data where some trials have IPD only and the remaining trials have only AD [1, 7] provides a valuable strategy for synthesis, we believe testing homogeneity in such hybrid pairwise meta-analysis and network meta-analysis warrants further study.

Our example included gender as a binary moderator, but the methods described here can also include continuous moderator variables. We have yet to investigate how these methods perform on two different moderator analyses, involving latent or unobserved measures at baseline. In particular, with multiple outcome measures we can analyze the data using growth models, which would summarize participant-level growth into a random or latent intercept term and one or more random or latent slope parameters. The moderating effect of interventions as a function of baseline then involves a effect modification term between intervention status and the latent intercept, as they predict changes in a latent slope over time. General methods for this type of analysis have been developed [52, 53], and can be readily extended to the case of multiple trials using multilevel growth models. Further work is necessary to investigate moderation of comparative effectiveness that may occur across interventions. In our example, the Good Behavior Game was hypothesized to affect aggressive/disruptive and off-task behavior in the classroom, and through the reduction of these disruptions to increase learning and comprehension. Mastery learning was designed to do the opposite, focusing first on reading improvement and its consequent outcomes. It is not surprising then that we concluded that the Good Behavior Game was superior to Mastery Learning in reducing off-task behavior. We also found no evidence that this differential effect of these two interventions differed by gender.

There are further approaches that deserve continued investigation. The presentation focuses on participant-level covariates, but we recognize that useful synthesis projects often examine arm-level and trial-level measured covariates. We have only examined unmeasured trial-level covariates in this paper. We employed a frequentist approach to modeling, but these estimation methods may be taxed as models grow more complex. In those instances it will likely be pro table to explore Bayesian methods [54]. In the example, we introduced a type of potential outcomes framework that could be used to represent outcomes for a participant on interventions not even tested in that particular trial. We believe this is a fruitful approach for future work in obtaining inferences generalized over the collection of trials; we chose not to introduce this framework at the beginning of this paper as it would have made the notation unwieldy. Finally, we focus on determining the power of detecting moderator effects as functions of network types, IPD or trial-level summaries. It is a topic of further research to investigate how to improve the precision of estimates.

In summary, this paper presented a formalized conceptualization of and methodology for investigating homogeneity and consistency that is quite general, allowing these terms to be examined in the context of an arbitrary set of trials whose participant-level data are used for data synthesis. Definitions of weak and strong homogeneity and consistency were introduced in the context of a generalized mixed model, and we applied these terms to both contrasts between interventions as well as single interventions themselves. Defining these terms for single interventions allows us to include non-randomized observational studies with randomized trials. In particular, a similar study of participants not exposed to an intervention can often serve as an external control group [55]; likewise when a program is scaled up and delivered to everyone as in an implementation study [56, 57], such data can potentially be combined with those in a trial who were randomly assigned to that intervention.