Finite-Sample Corrected GEE of Population Average Treatment Effects in Stepped Wedge Cluster Randomized Trials

Abstract

Stepped wedge designs are increasingly commonplace and advantageous for cluster randomized trials (CRTs) when it is both unethical to assign placebo and it is logistically difficult to allocate an intervention simultaneously to many clusters. We study marginal mean models fit with generalized estimating equations (GEE) for assessing treatment effectiveness in stepped wedge CRTs. This approach has advantages over the more commonly used mixed models that (1) the population-average parameters have an important interpretation for public health applications and (2) they avoid untestable assumptions on latent variable distributions and avoid parametric assumptions about error distributions, therefore providing more robust evidence on treatment effects. However, CRTs typically have a small number of clusters, rendering the standard GEE sandwich variance estimator biased and highly variable and hence yielding incorrect inferences. We study the usual asymptotic GEE inferences (i.e., using sandwich variance estimators and asymptotic normality) and four small-sample corrections to GEE for stepped wedge CRTs and for parallel CRTs as a comparison. We show by simulation that the small-sample corrections provide improvement, with one correction appearing to provide at least nominal coverage even with only 10 clusters per group. These results demonstrate the viability of the marginal mean approach for both stepped wedge and parallel CRTs. We also study the comparative performance of the corrected methods for stepped wedge and parallel designs, and describe how the methods can accommodate interval censoring of individual failure times and incorporate semiparametric efficient estimators.

1 Introduction

Cluster randomized trials (CRTs) randomize groups of individuals to interventions and assess population-level treatment effects. Examples of CRTs include smoking prevention trials where the unit of randomization is school or city^1,2 and HIV prevention trials where the unit of randomization is community or workplace.^3,4

We consider stepped wedge (SW) CRTs and standard parallel CRTs for comparison. The SW design is a one-way crossover CRT design that follows enrolled clusters for at least two “steps” of time intervals. Typically the clusters are crossed over from control to active vaccine/treatment. In our formulation at least one cluster receives vaccine in the first step while the remainder receive control and then are crossed over at randomly assigned steps to receive vaccine. Figure 1 (adapted from Hussey and Hughes⁵) illustrates the differences between the parallel, crossover, and SW designs.

Figure 1.

Treatment schedules for basic parallel, crossover, and stepped wedge cluster randomized trial designs. A “0” represents control and a “1” represents active treatment/vaccine.

While most CRTs have used parallel designs, recently SW designs have received attention for two main reasons. First, parallel designs assign some communities the control condition for the entire study, which poses an ethical and enrollment challenge if there is prior evidence for intervention/vaccine efficacy. Second, parallel designs implement the intervention simultaneously in all the randomized communities, but this is sometimes logistically impossible. For example, in an HIV prevention trial that evaluates circumcision versus control in 20 versus 20 villages, it may be difficult for medical/surgical teams to deploy immediately in all 20 villages, whilst a traveling team could service four villages at a time in five steps. In addition, there is a large need to develop a scientific framework to guide Phase 4 post-licensure health-care scale-up in developing countries, due to the “formidable gap between innovations in health and their delivery to communities in the developing world”.⁶ The SW design is a recent tool under development that contributes to this scientific framework as well as to the scientific framework for Phase 3 licensure trials.

SW trials are increasingly being conducted. Since it was introduced in the Gambia Hepatitis B study,⁷ several other trials have used this design for many different types of outcomes including those involving HIV infection,^8–10 waterborn diseases,¹¹ and childhood malnutrition.¹² Statistical methods for SW designs, including power and sample size calculations, have been developed by Hussey and Hughes⁵, Moulton et al.,¹³ and others.¹⁴

Statistical analysis of SW designs must accommodate both within-cluster correlation and time effects (as treatment is stepped up in a one-way crossover manner). Traditional analyses of CRTs handle within-cluster correlation by either a cluster-level or individual-level analysis.¹⁵ Cluster-level analysis compares cluster-level summary statistics (e.g., averages) between groups using two-sample methods for independent data. Individual-level analysis uses the individual as the analysis unit and accounts for within-cluster correlation using either marginal mean models (typically fit by GEE with a sandwich variance estimator) or mixed models. The mixed models account for the within-cluster correlation either by including random cluster effects or by conditioning out the random cluster effects.

For handling time effects, cluster-level analyses in SW trials are complicated by the fact that the treatment effect may change over time. Hussey and Hughes⁵ conducted cluster-level analysis using a linear mixed model that summarized cluster responses with estimated means for each cluster at each time. Hussey and Hughes⁵ also described individual-level analyses via linear models using both random effects models and marginal mean models. Moulton et al.^13,16 regard event times of individuals as right censored and used a Cox regression-type of analysis, where at each observed failure time they compare treatment groups by conditioning on the number of events and the number at risk. To account for within-cluster correlation, either a sandwich or bootstrap estimator of variance is used, with cluster as the unit.¹³ While methods using sandwich estimators of variance have been shown useful for large cluster trials not requiring bias-correction,^17,18 standard sandwich estimators of variance are generally anti-conservative when there are a small number of clusters. Although various methods have been investigated to reduce the bias and under-coverage with small cluster sizes in CRTs,^19–24 none have been evaluated in SW designs.

Whereas the predominant approach to analyzing CRTs has been to use individual-level outcomes and random effects models, e.g.,^5,13 we focus on the marginal mean modeling approach that has target parameter “total vaccine efficacy” (V E_T).²⁵ This parameter combines direct and indirect vaccine efficacies in comparing outcomes for individuals in vaccinated populations versus those for individuals in unvaccinated populations, and is defined mathematically in Section 2.3.²⁵ We focus on the V E_T parameter (and its time-varying version) because in Phase 3 and 4 trials it is of particular interest to use population-average estimands of group-level summary statistics that are most relevant for guiding public health policy decisions. The population-average approach also has advantages in the weaker and more testable assumptions it requires compared to mixed model approaches. In particular, mixed models require untestable assumptions about latent variable distributions, require correctly specified error distributions for consistent estimation, and standard error estimates are not robust to model mis-specifications.²⁶ In contrast, marginal mean models do not make assumptions about latent variables and do not require correctly specified error distributions for consistent estimation, and, consequently, results from the latter models may be interpreted as carrying a greater weight of evidence.²⁶ A potential pitfall of the marginal mean models is insufficient numbers of clusters to allow unbiased and stable standard error estimation, however, which has limited its use in practice.²⁷ The significance of this work is that we show that small cluster size methods for marginal mean models can be used for correct inference about total vaccine efficacy overall and over time, thereby justifying the restored use of the highly interpretable population-average target parameter that is appropriate for stepped wedge and parallel CRTs.

The article is organized as follows. Section 2 describes a marginal mean model for the cluster-step outcomes for the SW and parallel designs, and defines the intervention/vaccine efficacy estimands of interest in terms of parameters in the model. This model allows interval censored time to events. Section 3 summarizes five approaches to estimation and testing of the estimands, using standard sandwich SEs and four small-sample methods that better protect the type I error and improve the accuracy of confidence intervals^19–22 with additional details in the Web Supplementary Materials. Section 4 provides a simulation study to evaluate the five methods and compare their operating characteristics in SW and parallel designs. Section 5 provides discussion. R code implementing the developed methods for CRTs is provided at the last author's website.

2 Marginal mean model

2.1 Notation

We consider a prospective cohort CRT. Subjects testing HIV negative are enrolled during a fixed accrual period. Following the accrual period subjects are followed through J additional fixed calendar time intervals, each of duration M months, and are tested for HIV infection once within each calendar interval. Upon enrollment within the initial/first calendar interval, a subject's HIV testing date within the second calendar interval is scheduled, and evenly-spaced HIV testing dates within the subsequent calendar intervals 3, …, J are scheduled.

As an example used throughout, we consider a 72-month study with 12-month accrual period and ten subsequent 6-month calendar intervals, such that J = 10 and M = 6 months. Corresponding to the J calendar intervals, each subject is followed through J “steps,” with Step 1 defined as the interval between enrollment and his/her scheduled HIV test in calendar interval 2, Step 2 defined as the interval between the scheduled HIV tests in calendar intervals 2 and 3, and so on. Thus the steps are defined based on study time and the HIV testing schedule, conforming to the usual approach for HIV prevention trial design. For the SW design, the i^th cluster starts the intervention (crosses over from the control condition) at a randomly assigned calendar interval $C_i^0\in \{1,\dots ,J-1\}$. Subjects in cluster i start vaccination a day or two after their HIV test (if it is negative) within calendar interval $C_i^0$. The two-group parallel design is the same except the i^th cluster is assigned either to $C_i^0=1$ or to the control condition throughout the follow-up period of the study. Figure 2 describes the follow-up and HIV testing schedule for the SW design.

Figure 2.

A sample of three participants from each of five clusters are shown, each randomized to start vaccine in different steps. The dashed lines represent the non-vaccinated follow-up while the solid lines represent the vaccinated follow-up. The hash marks during the accrual period represent participants' dates of entry into the study. The open circles denote the times at which participants are tested for HIV infection and the filled circles denote the times at which the participants are tested for HIV infection and vaccinated if HIV negative.

Let Y_ijk be the indicator of whether individual k in cluster i is HIV infected during step j for i = 1, …, I and j = 1, …, J. Let X_1ij be the treatment indicator (1=vaccine; 0=control), and X_2ij be a vector of other cluster-step level covariates that may be useful for bias-correction and/or improving precision. We study marginal mean models of cluster-step level responses Y_ij, with Y_ij an estimate of E[Y_ij], the incidence of new infections in cluster i during step j. (Recall that our approach analyzes cluster-level summary statistics, not the individual outcomes Y_ijk.) In Section 3.1 we discuss different possible estimators of the cluster-step incidences E[Y_ij].

2.2 Generalized linear model

We model the marginal mean μ_ij = E[Y_ij] with a generalized linear model (glm)

$$g({\mu }_{ij})={\beta }_0+{\beta }_1{X}_{1ij}+{\beta }_2\ast j+{\beta }_3{X}_{1ij}{s}_{ij}+{\beta }_{4j}^T{X}_{2ij},\text{for}j=1,\dots ,J,$$ (1)

where g(·) is a link function, β₂ adjusts for calendar/secular trends, and $s_{ij}\equiv {(j-t_i^0)}^{+}$, where $t_i^0$ is the step (j) at which cluster i is randomized to start vaccination, and a⁺ = a if a > 0 and 0 otherwise. While β₂ measures how the marginal mean varies across steps defined based on an individual's HIV testing schedule, it approximately measures how the marginal mean varies over calendar time, because the accrual period is a small fraction of the total study follow-up period. The term s_ij is the number of steps since cluster i started vaccination, and ranges from 0 to $J-t_i^0$.

In model (1) with β₃ = 0, β₁ represents the treatment effect averaged over all clusters (i.e., the expected change in the response in the population when all clusters change from the control condition to the intervention). If the treatment effect changes with time, then the interaction coefficient β₃ is nonzero. We use the coding s_ij for time effects because this allows assessment of how the treatment effect varies with time since vaccination. We focus on the identity link g(y) = y, although the method can also be implemented straightforwardly by modeling Y_ij as a count variable (the number of new infections in cluster i during step j) and using a log link, as was done in the first author's PhD dissertation.²⁸ These approaches lead to different types of treatment efficacy estimands, as described in Section 2.3.

While standard glm modeling fit by GEE can be used for inference on β ≡ (β₀, β₁, β₂, β₃, β₄)′, the fact that the number of clusters is small implies that the typical sandwich variance estimators for β̂ tend to underestimate the variance, which leads to anti-conservative inference.^29,30 Accordingly, we will implement and evaluate four small-sample corrected SE methods for fitting model 1 for the CRT application with a small number of clusters.

2.3 Treatment efficacy/vaccine efficacy estimands

Vaccine efficacy, V E, is a measure of the form 1 − RR (one minus some measure of relative risk, see Halloran et al.²⁵ Chapter 2). We use the term treatment efficacy, T E, as a more general term to denote any parameter that measures a treatment effect (e.g., a difference, a ratio, or 1 − RR). We focus on the glm with identity link and Y_ij the estimated incidence of cluster i in step j. From model (1), in the case that β₃ = 0, the population-average parameter of interest is the total treatment efficacy T E_T ≡ β₁, which measures a combination of the direct and indirect effects of the intervention, as an additive difference (T E_T is interpreted as the mean difference in cluster-step incidences for treated versus control clusters). In the general case where β₃ may be nonzero, for a fixed s ∈ {1, …, J − 1}, define T E_Ts = β₁ + β₃ * s, which is the treatment/vaccine effect s steps (i.e., sM months) after vaccination. The parameters T E_Ts measure how the treatment efficacy changes with time since vaccination.

While we do not evaluate it here, the glm with log link and Y_ij the estimated mean number of infections is also of interest. In this model the number of person-years at risk PY_ij is included as a fixed covariate (entered as an offset term). If β₃ = 0, the population-average total treatment efficacy parameter of interest is T E_T = 1 − exp(β₁), the multiplicative reduction in the mean cluster-step incidences for treated versus control clusters. If the intervention is vaccine, then T E_T equals V E_T as defined by Halloran et al.³¹ The parameter T E_Ts = 1 − exp(β₁ + β₃ * s) is the treatment/vaccine effect s steps after treatment/vaccination.

3 Estimation and testing

In this section we first describe approaches to estimating the HIV infection incidence for a given cluster-step. Second we summarize the standard sandwich SE and small-sample corrections.

3.1 Estimation of cluster-step incidences

Estimates of the cluster-step incidences, E[Y_ij], are used as the summary statistic responses in the glm. As such the estimation of the E[Y_ij] and of the parameters in the glm are completely separate steps, and the goal in the first step is to choose an estimator that performs well in bias and variance. In the following, the beginning and end of step j for a subject is the time of the scheduled HIV test in calendar interval j − 1 and in calendar interval j, respectively. Let N_ij be the number of subjects at-risk (HIV uninfected) and under follow-up at the beginning of step j, for j = 1, …, J − 1. If all subjects in cluster-step (i,j) have an HIV test at the beginning and end of the step, then N_ij is known (based on HIV negative results at the beginning of the step) and E[Y_ij] may be estimated by p̂_ij, the fraction of the N_ij subjects with a positive test result at the end of the step. If some subjects are missing HIV test results at either edge of the step and if the missingness is completely at random, then the above estimator computed in subjects with complete testing data is consistent. However, if a missing at random (MAR) mechanism is more tenable and hence the complete-case estimator is not consistent, a method designed for MAR data is superior. (Here MAR means that whether HIV testing results are missing depends only on collected information from the subject, which could include individual- and cluster-level data.)

One MAR method would model the incidence of infections parametrically, and estimate E[Y_ij] by maximum likelihood. Because consistent estimation for this approach would rely on a correct parametric model, an alternative approach that would provide consistent estimation under a mis-specified parametric model would be augmented inverse probability weighting (AIPW)³². This method, while advantageous as a doubly robust method, could perform poorly if there is no reasonable model predicting whether a subject's HIV test results are observed, or if some of these estimated probabilities are outliers near zero³³. The collaborative targeted maximum likelihood (collaborative tMLE) method is another option for a doubly robust method. It has been shown in simulations to often perform well in settings with outlying weights near zero, which is due to its targeting of the mean-variance tradeoff on the parameter of interest and to the fact that it is a substitution estimator that is guaranteed to fall in the parameter space³⁴. Augmented GEE³⁵ is another approach, which, like the above approaches, can increase precision for estimating E[Y_ij] by incorporating individual-level covariates that predict whether individuals are infected during the step.

A limitation of the methods described above is that HIV tests have a time-lag between the date of HIV acquisition and the time at which the test would yield a positive result (for antibody-based tests about 3 weeks and for antigen-based tests such as HIV-specific PCR about 1 week). Therefore, the most accurate way to assess HIV infection is to consider the infection time as interval censored, where there is a known window period during which each diagnosed infection is known to occur (for a typical HIV prevention trial the window would be 1 week before the last negative PCR test to 1 week before the first PCR positive test). To explicitly handle the interval censoring, a nonparametric maximum likelihood estimator of the distribution of all subjects may be used to obtain a consistent estimate of E[Y_ij] for each i, j.³⁶

3.2 Standard sandwich variance and corrections for handling a small number of clusters

We consider GEE for estimation and inference in model (1). This approach, introduced by Liang and Zeger,³⁷ estimates β as the solution to an estimating equation (see Web Appendix A). GEE analysis typically uses the sandwich estimator of the variance matrix of β, which is consistent and asymptotically normal even when the working correlation is mis-specified; however, the estimator is biased (see Theorem 5.4 of Ziegler).²⁹ This bias tends to underestimate the variance which leads to under-coverage of confidence intervals and inflated type I errors, particularly for the Wald test in small sample settings.

Numerous studies have been conducted and several solutions proposed to correct this bias and undercoverage (e.g., see Lu et al.³⁸, the papers cited below, and the reviews of Ziegler²⁹ and Dahmen and Ziegler³⁰). One solution is to abandon the use of the sandwich estimator and resort to either a Jackknife or bootstrap estimator of the variance. Other approaches use the sandwich estimator with adjustments to the Wald test, and less commonly to the score test³⁹. We focus on the more common Wald tests in this paper.

There are three main ways of adjusting the Wald tests for small samples. One solution implements some form of bias correction on the variance, with approaches developed by Fay and Graubard,¹⁹ Mancl and DeRouen,²⁰ Kauermann and Carroll,²¹ and Morel et al.²² Theorem 5.21 of Ziegler²⁹ shows that the Mancl and DeRouen,²⁰ and modified Fay and Graubard²⁹ (mFG) estimators are less biased than the usual sandwich variance estimator. This proof extends to the Kauermann and Carroll²¹ (KC) estimator since the KC variance-covariance estimator is identical to that of mFG (proved in Web Appendix B). While the mFG and KC procedures are identical analytically, their implementation requires numerical inversion of different matrices. Typically this matrix has lower dimension for the mFG approach, such that the mFG method may be preferred to provide more numerically stable inference. Although we ran simulations with KC, the results are so close to those with mFG that the KC results are not presented. Based on the estimator formulas the Mancl and DeRouen²⁰ (MD) variance estimator will tend to be larger than those of mFG and KC,³⁸ which is confirmed in the simulations. The Fay and Graubard variance estimator (see Web Appendix A) is similar to the mFG approach. The adjusted estimator of Morel et al.²² (MBN) is additive and always positive such that the type I error rate of Wald tests is always smaller than Wald tests based on the uncorrected sandwich estimator. In summary, all three bias-corrected SE estimators (MD, mFG/KC, MBN) are guaranteed to confer better Type I error control of Wald tests in GEE than the standard sandwich estimator, and the simulation study provides insight into the comparative performance. Web Appendices A and B provide additional details on the four corrected SE methods that we study.

A second way of adjusting the Wald test is to assume that the working variance is correctly specified and there is a common correlation structure, and then to reduce the variability of the sandwich estimator by smoothing. For example, Pan (2001)⁴⁰ smoothed by replacing individual Pearson residuals with their mean, Gosho et al. (2014)⁴¹ added a simple bias correction to Pan's method, and Wang and Long (2011)⁴² used regularization for smoothing. Since these approaches are less robust, we do not pursue these in this paper.

A third adjustment takes into account the variability of the sandwich estimator and bases inferences on a t-distribution (or F-distribution) instead of a normal (or chi-square) distribution, an idea that parallels using a t-test instead of a Z-test.⁴³ Combinations of the first and third solutions have been proposed^{19,21,43–45}. Because (1) these combination approaches are often very similar, (2) the most recent work of Fan et al. (2012)⁴⁵ does not show substantial improvement over the combination approach of Fay et al. (2001)¹⁹, and (3) the software is readily available in R to implement Fay et al. (2001), we focus on the latter combination approach in this article. In particular, in addition to studying three (mFG, MD, MBN) of the corrected-SE methods mentioned above using a reference normal distribution, we also study the Fay and Graubard¹⁹ δ₅ (FG d5) approach (method=d5 in the saws R function). This approach uses the FG bias-corrected SE estimator and a reference t-distribution with degrees of freedom estimated by a weighted average of covariance estimates of the terms of the estimating equation (d̃_H in the paper). In models with more than one parameter, the FG d5 approach in general has different degrees of freedom for different parameters, which addresses the different amounts of variability in the variance estimator for each parameter.

Other small sample adjustments not explored in this paper are an adjustment to the mean parameters (as opposed to adjustments to the variance estimators noted above), see Paul and Zhang⁴⁶. In addition to the F-statistic adjustments mentioned above, McCaffrey and Bell (2006)⁴⁴ studied saddlepoint approximations for calculating p-values with bias corrected sandwich estimators.

4 Simulation study

4.1 Objectives of the simulation study

We address several scientific questions via a simulation study for a glm with identity link g(y) = y and hence a mean difference treatment/vaccine efficacy parameter T E. Negative values of T E indicate treatment/vaccine efficacy. For SW and parallel CRTs, we evaluate the following properties of inference based on GEE assuming asymptotic normality with the standard SE estimator and GEE with a small sample correction:

• Standard errors and coverage probabilities of Wald-based confidence intervals for the regression parameters in the glm.

• Size of tests for $H_0^0:T{E}_T=0$ and for $H_0^1:T{E}_{Ts}=T{E}_T$ for s = 0, …, J − 1.

• Power of tests for $H_0^0$ and $H_0^1$ at various alternatives.

We study standard sandwich SE and MD-, MBN-, and mFG-corrected SEs; results for KC-corrected SEs are not reported because the results were almost identical to those for mFG. We also study the FG d5 method.

We study three cluster sizes, 10, 20 and 50. The first choice was based on the need to have at least 10 clusters to allow adequately stable inference. The second and third numbers were chosen to represent a typical number and a number that should be representative of asymptotic results and is near the maximum of what is typically used in stepped wedge CRTs. A literature review of all published stepped wedge CRTs with at least 50 individuals per cluster on average showed that the mean and median number of clusters was 33 and 12, respectively, with interquartile range 7–29 (Supplementary Table 1 in Web Appendix C).

4.2 Simulation of vaccine trials

We study the following three scenarios for the true vaccine efficacy parameters:

Scenario 1:T E_Ts = T E_T = 0 for all s = 0, …, J − 1 (complete null)

Scenario 2:T E_Ts = T E_T for all s = 0, …, J − 1 and T E_T < 0 (beneficial efficacy that is time-constant)

Scenario 3:T E_Ts decreasing in time s = 0, …, J − 1 (efficacy increases over the steps) CRTs are simulated in R version 2.14.0 by generating the number of infections for each cluster step from the linear marginal mean model in (1) without any extra covariates (X_2ij):

$$E[Y_{ij}]={\beta }_0+{\beta }_1{X}_{1ij}+{\beta }_2\ast j+{\beta }_3{X}_{1ij}\ast s_{ij},$$ (2)

where X_1ij is defined as in Section 2.2 for cluster i and step j = 1, …, J. For parallel CRTs, $t_i^0=1$ for all i, such that s_ij = j − 1 for all i, j, whereas for SW CRTs, $s_{ij}\equiv {(j-t_i^0)}^{+}$ as described in Section 2.2.

The parameter β₂ specifies how much the background incidence changes over time, and for simplicity we set β₂ = 0. We choose β₀ such that the mean HIV incidence over a step in non-vaccinated cluster-steps is either 0.04 (Scenario 1) or 0.05 (Scenarios 2–3). Based on the fact that the additive difference vaccine efficacy parameters equal

$$T{E}_{Ts}={\beta }_1+s\times {\beta }_3\text{for}s=0,\dots ,J-1$$

in terms of model (2), we choose β₁ and β₃ to create the three scenarios: (1) T E_Ts = 0 for all s = 0, …, J − 1; (2) T E_Ts = −0.015 for all s = 0, …, J − 1; and (3) T E_T0 = −0.008, T E_T(J−1) = −0.03, and T E_Ts decreases linearly with s = 1, …, J − 2. Scenario 2 represents an antibody-based vaccine that protects only through a reduction in HIV acquisition, with protection established quickly without waning, whereas scenario 3 represents an antibody/T-cell based combination vaccine that protects through indirect effects (on infectiousness and disease progression) that take time to accrue. Setting T E_T0 near zero for scenario 3 is reasonable for a vaccine with no effect to reduce susceptibility, T E_S = 0, given the time needed for indirect efficacy to accrue.

Vaccine trials are simulated to satisfy model (2). For each cluster i, the vector of estimated infection incidences (Y_i1, …, Y_iJ)′ is generated from a multivariate normal distribution with E[Y_ij] = θ_ij, Var[Y_ij] = 0.0005, and an AR-1 correlation structure (with linear correlation ρ = 0.8) to account for the anticipated within-cluster correlation of the Y_ij's over the steps. In addition, the case of independent Y_ij, j=1, …, J, for each i is considered to study sensitivity of inference to the within-cluster correlation structure (results presented in Web Appendix D).

To simulate SW and parallel trials via model (2) that follow each of the three scenarios above, simple math determines maps between values of θ_ij and values of the true regression parameters β₀, β₁, and β₃. In particular:

Scenario 1:θ_ij = 0.04 for all i,j; β₀ = 0.04, β₁ = β₃ = 0.

Scenario 2:θ_ij = 0.05 for all i,j with X_1ij = 0; θ_ij = β₀ + β₁ = 0.035 for all i,j with X_1ij = 1; β₀ = 0.05, β₁ = −0.015, β₃ = 0

Scenario 3:θ_ij = 0.05 for all i,j with X_1ij = 0; θ_i1 = β₀ + β₁ = 0.042 with X_1i1 = 1; β₀ = 0.05, β₁ = −0.008. For the parallel design, θ_ij = 0.05 − 0.008 + β₃(j − 1) for all i with X_1ij = 1; β₃ = (0.02 − β₀ − β₁)/(J − 1) = −0.022/(J − 1). For the SW design, θ_ij = 0.05 − 0.008 + β₃(j − 1) for all i,j with s_ij = j − 1 and X_1ij = 1; β₃ = (0.02 − β₀ − β₁)/(J − 1) = −0.022/(J − 1).

In practice, lacking evidence of the interaction effect in scenarios 1–2 (i.e., analysis consistent with β₃ = 0) would result in fitting a reduced form of model (2) excluding the interaction term. Consequently, we report operating characteristics for β₁ in scenarios 1–2 based on this reduced model.

This simulation approach supposes that complete HIV testing results are available; we focus on this relatively simple setting to focus attention on the performance of estimation and inference in the marginal mean model. Given the separation of the steps to estimate E[Y_ij] and to estimate the marginal mean model parameters, we can infer that the marginal mean inferences would perform similarly well (or better) if any of the AIPW, collaborative tMLE, or interval censoring approaches for estimating the E[Y_ij] were used, as long as they perform well for estimating the E[Y_ij]. For HIV prevention trial data sets the interval-censoring approach is appealing given that real data sets tend to have interval-censored infection times; however future work would be needed to conjoin this approach with the AIPW or collaborative tMLE methodology. The simulations use 5000 iterations.

4.3 Simulation results

The simulated data are analyzed with the standard sandwich approach and the four small sample corrections, using both the assumed correlation structure matching and mismatching the true correlation structure (as AR-1 for both the truth and the methods or as AR-1 for the truth and exchangeable for the methods), yielding very similar results. We report results for the mismatched case, as in practice some degree of mis-specification is expected. For all methods, we evaluate finite-sample bias of the standard errors of β̂₁ and β̂₃ in model (2) (or the reduced model for β₁ in scenarios 1–2), as well as coverage probabilities of Wald-based 95% confidence intervals for β₁ and β₃. (We also explored a null reference t-distribution with J-p degrees of freedom with p the number of coefficients in the linear predictor, but it yielded overly conservative Wald tests.) In addition, we investigate power of the Wald tests to reject $H_0^0:{\beta }_1=0$ and to reject $H_0^1:{\beta }_3=0$.

Figure 3 and Web Figure 1 show that all standard error estimators accurately reflect the true variability in coefficient estimation for the setting with a large number of clusters (I=50), as expected. Also as expected, with a small number of clusters the uncorrected GEE standard error estimator is too small whereas all of the corrected estimators are more accurate. The mFG and FG d5 estimators appear to be the most accurate, closely tracking the sample standard errors calculated across the simulation runs, although in Web Figure 1 FG d5 appears to slightly overestimate the SEs. MBN, MD and FG tend to be close or slightly conservative.

Figure 3.

Median standard-GEE, bias-corrected and empirical standard error (SE) estimates of β̂₁ and β̂₃ using the stepped wedge and parallel designs in simulation scenarios 1–3 with cluster-step incidences satisfying the AR-1 correlation structure. The empirical SE estimate is computed as the sample standard deviation of the β̂ estimates. In scenarios 1–2, a reduced form of model (2) excluding the interaction term is considered for inference about β₁.

Figure 4 and Web Figure 2 show results on coverage probabilities. Recall that MBN, MD and mFG use the normal reference distribution, while FG d5 uses different t-distributions for β₁ and β₃. For MBN and MD, the combination of the conservative SE and the anti-conservative use of the normal distribution can lead to close to nominal coverage in many cases. However, considering AR-1 within-cluster correlation, the MBN method has slightly low coverage probabilities for β₃. In contrast to MBN and MD, the coverage of mFG tends to be anti-conservative because although its SE is close to the empirical SE, the use of the normal distribution leads to under-coverage. The best method in terms of guaranteeing coverage is the FG d5 method, giving at least nominal coverage even with only 10 clusters per group. The coverage for FG d5 can be overly large for some cases with 10 clusters.

Figure 4.

Coverage probabilities (CP) of 95% standard-GEE and small sample corrected Wald confidence intervals for β₁ and β₃ using the stepped wedge and parallel designs in simulation scenarios 1–3 with cluster-step incidences satisfying the AR-1 correlation structure. The horizontal band represents ±2 × Monte Carlo standard error. In scenarios 1–2, a reduced form of model (2) excluding the interaction term is considered for inference about β₁

Figure 5 and Web Figure 3 show that all of the corrected-GEE Wald tests have closer to nominal size than the standard Wald test. The apparent gain in power of some methods over others appears to be due primarily to the anti-conservative sizes, since the order of the sizes and the powers appears consistent across scenarios (with Standard having the highest size and power, and mFG having the next highest). Differences in power between the methods become negligible for larger numbers of clusters.

Figure 5.

Size and power of standard-GEE and small sample corrected Wald tests to reject the null hypotheses $H_0^0:{\beta }_1=0$ and $H_0^1:{\beta }_3=0$ at 5% significance level using the stepped wedge and parallel designs in simulation scenarios 1–3 with cluster-step incidences satisfying the AR-1 correlation structure. For readability of the estimated sizes, size and power are plotted on the log scale. The horizontal band represents ±2 × Monte Carlo standard error.

We observe that the underlying within-cluster correlation structure impacts relative efficiency and power comparing the SW versus parallel design. For example, independence of the Y_ij's leads to superiority of the parallel design in terms of efficiency and power for β₁ in scenario 2, whereas an AR-1 correlation leads to superiority of the SW design in the same setting. Overall, the small sample corrected methods have size closer to the nominal level, with the FG d5 being conservative or close to nominal, and MD or MBN being anti-conservative or close to nominal. Thus, the choice between FG d5 and either MD of MBN depends on the importance of guaranteeing coverage versus maximizing power. The Standard or mFG methods are not recommended unless the cluster size is large.

5 Discussion

This article considers the use of generalized linear models for inference on population-average parameters measuring total treatment efficacy over time in stepped wedge and parallel cluster randomized trials (CRTs). We focused on overcoming the challenge that CRTs typically have a small number of clusters, yet in this context the standard sandwich variance estimator for GEE is anti-conservative. We found that for this setting small-sample corrected GEE inferences allow GEE to provide Wald hypothesis testing procedures with closer to nominal size and confidence intervals with more correct coverage probabilities. In particular, our analytical and empirical evaluation suggests that FG d5 has close to nominal or conservative coverage, MD and MBN have close to nominal or anti-conservative coverage, and mFG/KC has generally anti-conservative coverage. Therefore, for settings where anti-conservative inference is strongly unacceptable, we recommend the FG d5 method, and, for settings where a slight inflation of the type I error rate is tolerable and maximizing power is at a premium, we recommend the MD or MBN method. Overall the spread between these three different methods is useful because it allows choices based on the context-dependent relative utility of maintaining type I error versus maximizing power.

A contribution of this work lies in the fact that most stepped wedge design analyses have used mixed models, yet marginal mean models may be preferred because they avoid the two drawbacks of mixed models that they rely on untestable assumptions about latent variable distributions and they require correctly specified error distributions for consistent estimation. Moreover, we discussed ways in which the approach can be combined modularly with methods to estimate the cluster-step disease incidences, for example allowing use of semiparametric efficient methods and allowing use of the methods that accommodate interval censoring of disease times. While GEE methods with finite-sample corrections have been evaluated for CRTs for parallel designs,^23,38,47 we compared the performance of the four small sample corrected methods for stepped wedge versus parallel designs. We found that the level of correlation of cluster-step disease incidences across steps affects their relative power, where greater correlation implies greater relative power of the stepped wedge design, likely due to accounting for within-cluster information as well as between-cluster information.