Review of Recent Methodological Developments in Group-Randomized Trials: Part 2—Analysis

Abstract

In 2004, Murray et al. reviewed methodological developments in the design and analysis of group-randomized trials (GRTs). We have updated that review with developments in analysis of the past 13 years, with a companion article to focus on developments in design.

We discuss developments in the topics of the earlier review (e.g., methods for parallel-arm GRTs, individually randomized group-treatment trials, and missing data) and in new topics, including methods to account for multiple-level clustering and alternative estimation methods (e.g., augmented generalized estimating equations, targeted maximum likelihood, and quadratic inference functions).

In addition, we describe developments in analysis of alternative group designs (including stepped-wedge GRTs, network-randomized trials, and pseudocluster randomized trials), which require clustering to be accounted for in their design and analysis.

In a group-randomized trial (GRT), the unit of randomization is a group, and outcome measurements are obtained for members of those groups.¹ Also called a cluster-randomized trial or community trial,^2–5 a GRT is the best comparative design available if the intervention operates at a group level, manipulates the physical or social environment, or cannot be delivered to individual members of the group without substantial risk of contamination; it is also the best available design in other circumstances such as a desire for herd immunity in studies of infectious disease.^1–5

In GRTs, outcomes for members of the same group are likely to be more similar to each other than to outcomes for members from other groups.¹ Such clustering must be accounted for in the design to avoid an underpowered study and in the analysis to avoid underestimated standard errors and inflated type I error for the intervention effect.^1–5 In analyses, regression modeling approaches are generally preferred and most commonly used because of their ease of implementation.⁶ Several textbooks now address these and other issues.^1–5

In 2004, Murray et al.⁷ published a review of methodological developments in both the design and analysis of GRTs. In the 13 years since, there have been many developments in each area. Here we focus on developments in analytic methods, including those relevant to our companion article that focuses on developments in GRT design.⁸ (The glossary of terms is available as a supplement to the online version of this article at http://www.ajph.org.) As a pair, these articles update the 2004 review. In both, our goal is to provide a broad and comprehensive review to guide readers in seeking out appropriate materials for their own circumstances.

ANALYSIS OF PARALLEL-ARM GRTS

In GRTs, superiority trials are more common than equivalence or noninferiority trials: a PubMed search by one of the authors (D. M. M.) of studies published in 2015 identified 562 superiority GRTs but only 1 equivalence GRT and 2 noninferiority GRTs. Similarly, developments in the methods literature have focused on superiority GRTs, with developments for equivalence and noninferiority GRTs limited to small sections in 2 of the more recent textbooks^2,5 and a review article on sample size methods.⁹ As a consequence, we focus here on superiority GRTs.

Methods for Intervention Effects

In GRTs, protocol violations can lead to noncompliance at either the group or member level.⁵ As a means of minimizing bias, intention-to-treat principles are recommended at both levels rather than are “on-treatment” and “per-protocol” analyses.^2,4,5 Although group-level protocol violations are usually easy to identify, member-level compliance may be more difficult to ascertain in practice.² Jo et al. demonstrated that analyses ignoring compliance information may be underpowered to detect an intention-to-treat effect, and they proposed a multilevel model combined with a mixture model.¹⁰ The implications of group-level noncompliance can be considerable in GRTs given the small number of groups randomized in many such trials.

Methods Based on Randomization Scheme

Matching or stratification in designs has been recommended for some time as a way to ensure baseline balance in terms of important potential confounders,¹ with constrained randomization more recently developed.¹¹ Recent reports suggest that this advice is followed in most GRTs.^6,12–15 Matching and stratification in designs can be ignored in analyses of intervention effects without harm to the type I error rate, and often the saved degrees of freedom will improve power.^16,17

Recently, Donner et al. reported that ignoring matching can adversely affect other analyses, such as analyses examining the relationship between a risk factor and an outcome¹⁸; for this reason, investigators considering pair matching should consider small strata instead (e.g., strata of 4). Li et al.¹⁹ compared model-based and permutation methods in the context of constrained randomization adjusting for group-level covariates. They found that both the adjusted F test and the permutation test maintained the nominal size and exhibited improved power under constrained randomization relative to simple randomization.

Model-Based Methods

Model-based methods can be broadly classified according to the interpretation of the model parameters. Conditional model parameters, typically estimated with mixed-effects regression via maximum likelihood estimation (MLE), are referred to as cluster-specific effects (or as subject-specific effects in the longitudinal analysis literature). Effects are conditional on the random effects used to account for clustering and on other covariates included in the analysis. Conditional models are often recommended for studies focused on within-member changes or on mediation analyses.⁷

Parameters of marginal models are usually estimated via generalized estimating equations (GEE).^20,21 They define the marginal expectation of the dependent variable as a function of the independent variables and assume that the variance is a function of the mean; they separately specify a working correlation structure for observations made on members of the same group. Marginal models are often preferred for analyses of population-level effects because the intervention effect coefficient is interpreted as a population-averaged effect. In practice, marginal models are less frequently used than conditional models.⁶

Marginal and conditional intervention effects are equal for identity and log links,²² and the distinction between them is important only for link functions such as the logit for binary outcomes. Although some authors have advocated for the log instead of logit link for binary outcomes,²³ this approach is not widely used, possibly because of model convergence problems for certain types of data.^24,25 Alternatively, a modified Poisson approach with log link and robust standard errors could be used in the GEE framework²⁶ because it does not suffer from the same convergence problems as the binomial model with log link²⁷; however, its use may be less common because of the familiarity of logistic regression among epidemiologists and biostatisticians.

In practice, the question about which types of effects, conditional or marginal, are desired depends on the research question. It is essential to understand the underlying assumptions of each method: conditional models rely on correct specification of untestable aspects of the data distribution, whereas marginal models rely on a correct definition of the population of interest, which can make it difficult to generalize results to other populations.²⁸ We address each of the 2 approaches in more detail in the sections to follow.

Conditional approaches.

If the mixed-effects model used to estimate conditional effects is misspecified, the estimates are difficult to interpret and, even if regression diagnostics can help,²⁹ standard errors are not robust. Fortunately, Murray et al.³⁰ and Fu³¹ have shown that mixed models are robust to substantial violations of the normality assumptions for member- and group-level errors so long as equal numbers of groups are randomized to each arm. Parameter estimation via restricted maximum likelihood estimation is preferred to MLE when few groups are available.^32–34 In the case of binary outcomes, alternative methods for specifying test degrees of freedom have been examined in small-sample GRTs, and the between–within method is recommended.^32,35

Multiple levels of clustering in conditional models.

GRTs may involve multiple levels of clustering as a result of repeated measures on individuals or groups or additional hierarchical levels in the design. Murray¹ distinguished between mixed-effects models based on the number of measurements included in the analysis and recommended mixed-effects analysis of variance (ANOVA) or covariance (ANCOVA), or mixed-effects repeated measures ANOVA or ANCOVA, for analyses involving 1 or 2 measurements per person or per group; these models can account for all sources of random variation in such data if they are properly specified.³⁶

However, this is not the case in analyses involving 3 or more measurements per person or per group, wherein the sources of random variation may be different; instead, such analyses require a random coefficients model in which random trends and intercepts are calculated for each member (in cohort GRT designs) or group (in cohort and cross-sectional GRT designs), average trends and intercepts are calculated for each study arm, and the intervention effect is the net difference in the average study-arm trends.³⁶ Trends are often estimated as linear slopes but can take another form.

Variable group size in conditional models.

Johnson et al. focused on analysis of Gaussian outcomes from GRTs with variable group sizes.³⁷ They compared 10 model-based approaches and found that a 1-stage mixed model with Kenward–Roger³² degrees of freedom and unconstrained variance components performed well in GRTs with 14 or more groups per study arm. A 2-stage model weighted by the inverse of the estimated theoretical variance of the group means and with unconstrained variance components performed well in GRTs with 6 or more groups per study arm. A number of other models resulted in an inflated type I error rate when there was substantial variability in group size.

Marginal approaches.

When the GEE approach is used to estimate marginal effects, unbiased intervention effects can be estimated even if the working correlation structure is incorrect (e.g., via robust standard errors with the sandwich estimator), although precision is increased if the working matrix is correct. When degrees of freedom are limited for the test of interest, as often occurs in GRTs, standard error estimation is frequently biased downward and no method corrects for this issue in all cases, although several have been proposed.^38–44

Multiple levels of clustering in marginal models.

Multilevel clustering is easy to account for in mixed-effects regression, but there is less literature for the GEE approach. The alternating logistic regression approach⁴⁵ for binary and ordinal outcomes can be used to account for correlation attributable to repeated measures on individuals within groups, and this approach can be implemented within a GEE framework in both R (the alr package) and SAS (PROC GEE).⁴⁶ The second-order GEE approach, which (by contrast with regular GEE) models the working correlation structure as a function of covariates, can be implemented in R (geepack in R⁴⁷).⁴⁸ For more general working correlation matrices, users typically need to perform additional programming to provide the appropriate covariance matrix, and convergence may not be achieved.

In addition, although the intervention effect is unbiased when the marginal model is not correctly specified, standard errors estimated via GEE may be too small. A robust sandwich estimator of the variance can be used to correct this problem, but such an approach leads to loss of power.⁴⁹ Because of this accuracy–power trade-off, mixed-effects models may be a better option in GRTs involving more than 2 levels, although the effects estimated in such models are conditional rather than marginal effects.

Variable group size in marginal models.

Although GEE analysis can accommodate variable group sizes, informative group size can negatively affect efficiency. In this case, Williamson et al.⁵⁰ showed that GEE weighted by group size can correct bias in estimated intervention effects. This approach is equivalent to and less computationally demanding than within-cluster resampling.⁵¹

Advanced GEE approaches to improve efficiency.

For binary outcomes, GEE is more conservative (i.e., the intervention effect will be estimated closer to the null) than mixed-effects models.^28,52 Moreover, the standard error of the estimated intervention effect is typically larger when GEE is used, so much recent effort has focused on efficient estimation. GEE is most efficient when the true correlation structure of the data is selected as the working correlation structure. Hin et al. compared multiple selection criteria for the working correlation matrix.⁵³

An alternative approach is augmented GEE (AU-GEE), a method developed for independent data in a causal inference framework⁵⁴ that has been extended to clustered data.⁵⁵ AU-GEE uses covariate information to improve efficiency in a 2-stage approach that specifies a model for the potential outcomes under the treatment not received. AU-GEE is unbiased and robust to misspecification of the potential outcome model, although correct specification improves efficiency. As for analysis of all trials, only baseline covariates should be included in AU-GEE for analysis of GRT data because adjustment for postbaseline covariates may lead to bias.⁵⁶ Alternative methods are available to account for postbaseline, time-varying confounding.^57–59

Alternatives to GEE.

The quadratic inference function (QIF) method is an alternative to GEE for estimation of marginal effects. Song et al.⁶⁰ demonstrated that QIF has advantages over GEE: it is more efficient and more robust to outliers, it includes a goodness-of-fit test of the marginal mean model, and it permits straightforward extensions to model selection. In large samples, QIF is more efficient than GEE when the working correlation structure for the data is misspecified.⁶¹ However, the standard errors may be underestimated for small and medium sample sizes or for variable group sizes.⁶² More recent work by Westgate^63,64 provides improvements; Westgate used a bias-corrected sandwich covariance estimate and simultaneously selected the QIF or GEE while selecting the best working correlation structure.⁶⁵ Despite the many attractive properties of QIF, at this time there are few applications in public health.^66–68

A second alternative estimation method is targeted maximum likelihood estimation (tMLE),⁶⁹ a maximum likelihood–based G-computation estimator that targets the fit of the data-generating distribution to reduce bias in the parameter of interest. It is based on a machine learning approach that fluctuates an initial estimate of the conditional mean outcome and minimizes a loss function to provide an estimate of the parameter of interest.⁷⁰ The approach has been used in public health^71,72 and shows much promise for GRTs^73,74 because it can improve efficiency by simultaneously accounting for missing data and chance baseline covariate imbalance without committing to a specific functional form.⁷⁵

Permutation Methods

Gail et al. introduced permutation analysis for GRTs.⁷⁶ They found that the permutation test had nominal type I error rates across a variety of settings common to GRTs when the member-level errors were Gaussian or binomial—and even when very few heterogeneous groups were randomized to each study arm and the intraclass correlation coefficient was large—so long as equal numbers of groups are randomized to each arm. Murray et al.³⁰ extended this work, and their results showed that unadjusted permutation tests offer no more protection against confounding than unadjusted model-based tests, whereas the adjusted versions of both tests perform similarly. The permutation test was more powerful than the model-based test when the data were binomial and the intraclass correlation coefficient was 0.01 or above. Fu³¹ extended the work to heavy-tailed and very skewed distributions and reported similar results.

Li et al. compared model-based and permutation methods in the context of constrained randomization adjusting for group-level covariates. They found that the adjusted F test and the permutation test maintained the nominal size and had similar power but cautioned that the randomization distribution must be calculated within the constrained randomization space to prevent inflation of the type I error rate.¹⁹

DEVELOPMENTS IN THE ANALYSIS OF ALTERNATIVES

Alternative group designs can be used in place of a traditional parallel-arm GRT.⁸ Four of these alternatives involve randomization and some form of clustering that must be appropriately accounted for in both their design and analysis. Thus, they share key features of the standard parallel-arm GRT, yet all have distinct and different features that are important to understand.

Stepped-Wedge GRTs

Both between- and within-group information is available to estimate the intervention effect from a stepped wedge GRT (SW-GRT).^77,78 However, because the control condition is usually observed earlier than the intervention condition, time is a potential confounder and should be accommodated in analyses of SW-GRTs, typically by accounting for time as a predictor.⁷⁹ As with parallel GRTs, clustering by group must be taken into account, and longitudinal measures for individuals can be accommodated within either the mixed-effects or the GEE framework, although more easily with mixed-effects models (see the sections on multiple levels of clustering). Conditional approaches are more commonly used in practice and reported on in the methods literature.^79,80 Several authors have highlighted other characteristics specific to SW-GRTs, including lagged intervention effects⁸¹ and fidelity loss over time.⁷⁹

Network-Randomized GRTs

Because the network properties of a network-randomized GRT are primarily used at the design stage,⁸² and because they differ from regular GRTs only in the novel way in which groups are defined, theories regarding analysis of parallel-arm GRTs can be applied to parallel-arm network-randomized GRTs.⁸³ For example, in a ring trial of an Ebola vaccine⁸³ in which a network was defined as all individuals who had regular physical contact with the incident (index) case of Ebola and in which all contacts received the vaccine (placebo or active), standard GRT methods were used.

For network-randomized GRTs in which the intervention is not directly administered to all individuals and it is expected that the intervention will spread over the network (e.g., snowball trials of an HIV prevention intervention for drug users⁸⁴ or a microfinance intervention⁸⁵), methods^86,87 are available to estimate both the direct and indirect effects of the intervention. When network information is available and the outcome of interest is known to be a disseminated process, adjusting for network features such as information on the location of each individual within the network (i.e., group) can improve the efficiency as well as the power of the analysis.⁸⁸

Pseudocluster Randomized Trials

Teerenstra et al.⁸⁹ compared analytic methods for continuous outcomes in pseudocluster randomized trials, and Campbell and Walters discussed principles in their recent textbook.⁵ Clustering by the unit of randomization at the first stage (e.g., provider) must be taken into account in both the design and analysis of pseudocluster randomized trials. No explicit sample size or analytic methods are known to be available for noncontinuous outcomes.

Individually Randomized Group-Treatment Trials

Baldwin et al. compared 4 analytic models for individually randomized group treatment trials and 3 methods for calculating degrees of freedom.⁹⁰ A multilevel model adapted to reflect clustering in only 1 study arm, combined with either Satterthwaite⁹¹ or Kenward–Roger³² degrees of freedom, resulted in better type I error control, better efficiency, and less bias, even with heteroscedasticity at the member level. This finding is consistent with earlier reports by Pals et al.⁹² and Roberts and Roberts.⁹³ More recently, Roberts and Walwyn⁹⁴ and Andridge et al.⁹⁵ considered circumstances in which members are associated with more than 1 small group or change agent. Both found that ignoring membership in multiple groups further inflates the type I error rate. Roberts and Walwyn reported that multiple-member multilevel models maintained the nominal type I error rate; they also provided sample size and power formulas.⁹⁴

DEVELOPMENTS IN ADDRESSING DATA CHALLENGES

Data challenges include those related to missing outcome data, baseline imbalance of covariates, and practical implementation in software.

Missing Outcome Data

Two recent reviews^6,96 indicate that missing outcome data are common in GRTs, although investigators frequently analyze only available data without accounting for the missing data pattern. In cases in which the covariate-dependent missingness (CDM) assumption is plausible, both mixed-effects and GEE models provide unbiased estimates of the intervention effect when the CDM covariates are included in analyses of all available data.^97,98 AU-GEE also can provide unbiased effects through inclusion of all CDM covariates in the augmentation component,⁵⁵ and it has the advantage that all estimates can still be interpreted as marginal effects. Other 2-stage approaches such as multiple imputation (MI) and inverse probability weighting (IPW) can provide unbiased intervention effects under certain conditions for more general missing-at-random patterns and may provide increased precision relative to covariate-adjusted conditional or marginal models for CDM.^97,99

Although there is less literature on how to address missing-not-at-random data,¹⁰⁰ sensitivity analyses are recommended.¹⁰¹ A recent review showed that very few GRTs incorporated sensitivity analyses for missing data assumptions.⁶

To avoid possible type I error, MI should account for the clustered data structure.^102,103 Fixed group effects should not be used owing to reduced power.¹⁰⁴ For binary outcomes, Ma et al.¹⁰⁵ and Caille et al.¹⁰⁶ showed that the preferred MI method depends on the number of groups and the design effect, and they noted that bias may arise for some approaches (including CDM). Using group-specific mean imputation may be adequate for continuous outcomes.^98,102 Hossain et al.⁹⁸ showed that if the missing data mechanism includes an interaction between a covariate predictive of the outcome and the study arm, the imputation strategy must account for this interaction if it is to be unbiased.

Whereas MI requires specifying the distribution of the missing data conditional on covariates, IPW requires specifying the probability of missingness depending on covariates. Theoretically, both approaches can be used for any type of outcome and for CDM as well as more general missing-at-random mechanisms.⁹⁹ Although IPW requires an additional assumption of positivity (all participants have a nonzero probability of being observed), it may be viewed as easier to define, particularly in the presence of nonintermittent missingness.¹⁰⁷ Importantly, and as with MI, if the missing data mechanism includes an interaction between a covariate predictive of the outcome and the study arm, the weights must be generated by accounting for this interaction if the strategy is to be unbiased.¹⁰⁸

Prague et al.^109,110 developed a doubly robust estimator in the context of IPW that provides an unbiased estimate of the intervention effect if either the marginal mean model or the missing data model is correctly specified. They demonstrated that a doubly robust augmented GEE approach can simultaneously account for both CDM and baseline covariate imbalance in GRTs when the parameter of interest is a marginal effect. Combining MI and IPW is a promising new approach that may be superior in performance to IPW or MI alone when there are missing covariates in addition to missing outcomes.¹¹¹

Baseline Imbalance of Covariates

Although design strategies such as restricted randomization⁸ can help to achieve baseline covariate balance, they may not be easy to implement (e.g., if group characteristics are unknown in advance), and chance imbalance may arise regardless. In this case, some form of model-based covariate adjustment could be used, such as standard multivariable regression for conditional models or AU-GEE for marginal models.⁵⁵ The advantage of AU-GEE in this case is that it is doubly robust: the consistency of the intervention effect estimate requires correct specification of either the marginal mean structure or the treatment model, and covariate adjustment is separated from intervention effect estimation, thereby reducing the risk of selecting the models that produce the most significant results. The standard multivariable regression adjustment approach does not offer either of these benefits.

Alternatively, Hansen and Bowers¹¹² proposed a balancing criterion and studied its randomization distribution to simultaneously test for balance of multiple covariates in both randomized controlled trials and GRTs. Leyrat et al.¹¹³ suggested using the c-statistic of the propensity score model to measure covariate balance at the individual level. Leon et al.¹¹⁴ recommended propensity score matching to correct for baseline imbalance; in a simulation study, they reported a median 90% reduction in bias. Nevertheless, the CONSORT (Consolidated Standards for Reporting of Trials) recommendation is that the adjustment covariates be specified a priori for primary analyses so that the sensitivity of the primary findings to adjustment for covariates identified post hoc can be tested in secondary analyses.¹¹⁵

Software

Table 1 presents information on 3 software programs that can be used to analyze data from GRTs. The table is organized around topics considered here. None of the 3 programs can readily implement both QIF and tMLE for GRTs; however, the R program offers the most ready-to-use functionality given its broad applicability to the methods we have described.

TABLE 1—

Summary of Known Functions and Procedures for Analyzing Group-Randomized Trials (GRTs) via the Methods Described for Three Commonly Used Software Programs

	Software
Method	SAS	Stata	R
Outcome analysis of all available data
Mixed-effects models	PROC MIXED	mixed	lme4
	PROC NLMIXED	melogit	nlme
	PROC GLIMMIX	mepoisson
GEE	PROC GENMODa	xtgee	geepack/geeM
tMLE	NA	NA	NAb
QIF	%qif	NA	qifc
Permutation tests	%ptest	NA	NA
Accounting for missing outcomes
Multiple imputation for clustered data	%mmi_imputed	REALCOM-IMPUTE	pan
	%mmi_analyze	mi imputed	jomoe
Inverse probability weighting	PROC GENMODf	NAg	CRTgeeDR
Causal inference–based methodsh
AU-GEE	NA	NA	CRTgeeDR
Doubly robust AU-GEE	NA	NA	CRTgeeDR

Note. AU-GEE = augmented GEE; GEE = generalized estimating equations; NA = not applicable; QIF = quadratic inference function; tMLE = targeted maximum likelihood.

^aPROC GEE is another option, but it is in the experimental phase and has limited usefulness for GRTs over and above PROC GENMOD.

^bIn R, tmle is available for tMLE; at the time of writing, however, it did not allow for clustering.

^cAt the time of writing, we were unable to load the package, and it allows only equal cluster sizes; however, Westgate modified the code for GRTs with variable cluster sizes in the appendix of his article.⁶³

^dOnly useful for continuous outcomes.

^eIn R, mice is available for multiple imputation; at the time of writing, however, it did not account for clustering.

^fCannot account for imprecision in weights.

^gxtgee cannot accommodate individual-level weights but, rather, only group-specific weights.

^hThe 2 listed methods are related: AU-GEE accounts for baseline covariate imbalance, and doubly robust AU-GEE, an extension of AU-GEE, accounts for both baseline covariate imbalance and missing data.

REPORTING OF RESULTS

The CONSORT guidelines for individually randomized trials were extended to GRTs in 2004,¹¹⁵ and most journals now require authors to conform to these guidelines. Ivers et al. reviewed 300 GRTs published between 2000 and 2008 and reported that 60% and 70%, respectively, accounted for clustering in sample size calculation and in the analysis; 56% involved restricted randomization, and most (86%) allocated more than 4 groups per arm.¹⁴ A more recent review of 86 trials published in 2013 and 2014 showed that 77% and 78% accounted for clustering in sample size calculation and in the analysis, respectively, and 51% involved some form of restricted randomization.⁶

Recent work on conduct and reporting has focused on the ethics of GRTs given concerns regarding this issue.^116,117 For example, Sim and Dawson discussed the challenges associated with obtaining informed consent in GRTs.¹¹⁸ The Ottawa statement on ethical design and conduct of GRTs was published in 2012,¹¹⁹ with a reevaluation in 2015.¹²⁰

CONCLUSIONS

In this review, we have summarized many of the most important advances in the analysis of GRTs during the 13 years since the publication of the earlier review by Murray et al.⁷ Much of our discussion has focused on marginal model parameter estimation (e.g., AU-GEE, QIF, tMLE) and missing data methods. Some topics that could not be included owing to space limitations are survival outcomes,^2,121–125 measurement bias,^126,127 validity,^128,129 Bayesian methods,^4,130–132 cost-effectiveness analyses,^4,133–136 mediation analyses seeking to uncover mechanisms of action,^137–140 and methods to analyze alternative GRT designs such as crossover GRTs.^141–144

Our aim here has been to remind readers of the value of well-thought-out analyses of GRTs and of keeping up to date with the many recent developments in this area. We hope that this review, paired with our companion review of developments in GRT design,⁸ will lead to continued improvements in the design and analysis of GRTs.