Novel Methods for the Analysis of Stepped Wedge Cluster Randomized Trials

Abstract

Stepped wedge cluster randomized trials (SW-CRTs) have become increasingly popular and are used for a variety of interventions and outcomes, often chosen for their feasibility advantages. SW-CRTs must account for time trends in the outcome because of the staggered rollout of the intervention. Robust inference procedures and non-parametric analysis methods have recently been proposed to handle such trends without requiring strong parametric modeling assumptions, but these are less powerful than model-based approaches. We propose several novel analysis methods that reduce reliance on modeling assumptions while preserving some of the increased power provided by the use of mixed effects models. In one method, we use the synthetic control approach to find the best matching clusters for a given intervention cluster. Another method makes use of within-cluster crossover information to construct an overall estimator. We also consider methods that combine these approaches to further improve power. We test these methods on simulated SW-CRTs, describing scenarios in which these methods have increased power compared to existing non-parametric methods while preserving nominal validity when mixed effects models are misspecified. We also demonstrate theoretical properties of these estimators with less restrictive assumptions than mixed effects models. Finally, we propose avenues for future research on the use of these methods; motivation for such research arises from their flexibility, which allows the identification of specific causal contrasts of interest, their robustness, and the potential for incorporating covariates to further increase power. Investigators conducting SW-CRTs might well consider such methods when common modeling assumptions may not hold

1. BACKGROUND

Cluster randomized trials (CRTs) have become a popular form of randomized trial, with many practical benefits, reflecting the necessity of implementing some interventions on clusters of individuals, and statistical benefits, such as accounting for interference between individuals.^1–3 The causal estimand of interest and the overall risk-benefit profile of the trial can also affect the choice to use cluster randomization.^4–6 While parallel-arm CRTs are the most common, stepped wedge CRTs (SW-CRTs) have also become more common, being used for a variety of interventions.^7–14 In SW-CRTs, each cluster begins in the control arm. At designated time points, a cluster or clusters “cross over” to the intervention arm and remain in that arm for the duration of the study. The order in which clusters cross over to the intervention is randomized.^15,16

SW-CRTs are especially valuable when the intervention cannot be implemented in a large number of clusters simultaneously due to practical constraints.^4,7,8,17 They can also be useful when the communities who will participate in the trial wish to ensure that all clusters receive the intervention before the end of the trial.^7,8,17,18 In particular, the design can be useful for assessing complex health interventions and for evaluating effectiveness of implementation.^19,20 There are, however, drawbacks to the design, and some of the benefits of the design may be achieved with parallel-arm CRT designs as well.²¹ Ethical arguments both for and against SW-CRTs have been made in various contexts, including arguments about the role of clinical equipoise.^7,22,23 And while the design may yield increased power over parallel-arm CRTs, this depends on both a large number of measurements over time and a statistically valid analysis method that controls for confounding of the treatment effect by time.^{16,20–22,24,25} Because fewer clusters are assigned to the intervention at the beginning of the trial, and more clusters are assigned to the intervention at later time points, the effect of time on the outcome must be accounted for in order to obtain unbiased or consistent treatment effects.^10,22,26 Additionally, SW-CRTs with a relatively small number of clusters can be underpowered to detect effects, at least without making strong modeling assumptions.²⁷

The most common method for analyzing SW-CRTs is the use of a linear or generalized linear mixed effects model. As described by Hussey and Hughes, this model can include a random intercept for each cluster and a fixed effect for time periods.¹⁵ This form of the model assumes that the additive effect for each time period is the same across clusters. A more general model proposed by Hooper et al. adds an independent random intercept for cluster-period;²⁸ however, this approach still assumes that the time trend does not vary systematically among clusters. In addition, both models require the specification of the distribution of these random intercepts. Misspecified random effects distributions can affect inference on the fixed effect estimators (i.e., the treatment effect estimator), although the effect on fixed effect estimates themselves is unclear and context-dependent.^27,29–34 Finally, for the relatively small number of clusters in many SW-CRTs, asymptotic inference based on the assumption of normally distributed random effects—as is frequently made for analysis using mixed effects models—can lead to inflated Type I Error and poor confidence interval coverage.^35,36

Various methods have been proposed to remedy these issues. One approach, proposed by Wang and De Gruttola and by Ji et al., uses permutation tests to ensure nominal Type I Error and accurate inference, even for small numbers of clusters, as long as the effect estimate is unbiased.^35,36 In the longitudinal context more generally, linear and generalized linear mixed effects models have been proposed that allow for flexible semi-parametric specification of the random effects distributions.^37–39 The operating characteristics of these different approaches to robust mixed effect model fitting have not been well-studied for SW-CRTs. Scott et al. have proposed the use of generalized estimating equations with finite-sample corrections to avoid the need to specify random effects distributions.⁴⁰ Thompson et al. recently proposed a non-parametric analysis method that uses within-period (“vertical”) comparisons.⁴¹ They propose conducting inference by permutation tests as well to ensure nominal Type I Error and confidence interval coverage. They demonstrate through simulation that this method has no or low bias and nominal Type I Error and coverage.⁴¹ Finally, Hughes et al. have proposed a robust inference method for SW-CRTs using vertical comparisons that gives a closed-form standard deviation estimate.⁴² However, both of these vertical methods can suffer from greatly reduced power compared to the parametric mixed effects models. Because SW-CRTs often have relatively few clusters, this can result in analyses that are highly underpowered to detect meaningful treatment effects.

In Section 2, we propose novel non-parametric methods to analyze SW-CRTs. In the first method, we, like Thompson et al., use within-period comparisons to avoid the problem of misspecification of time effects and cluster random intercept distributions. We incorporate the synthetic control procedure to match treated clusters with untreated clusters that are likely to be most similar. Synthetic controls are a relatively new but increasingly popular method for causal inference most common in the econometrics literature.^43–45 The approach is generally used when there is one treated cluster and a “donor pool” of untreated clusters, with outcome data both before and after the treatment began. The method finds the linear combination of untreated clusters that most closely matches the pre-treatment outcomes of the treated cluster. The causal effect estimate is then some contrast of the treated cluster’s post-treatment outcomes and that linear combination of the outcomes of the untreated clusters in the same period.⁴⁴ We use this approach, somewhat akin to matching or covariate adjustment in parallel-arm clinical trials, to improve the power of the analysis. The second method we propose uses the within-cluster between-period (“horizontal”) comparisons that are inherent in SW-CRT and other crossover designs to improve the power of non-parametric approaches.^22,27 This crossover method compares the between-period effect of clusters crossing over to that of clusters in the control arm in both periods (or in the intervention arm in both periods). We also propose two ways of combining these methods. In one, we use synthetic controls to find the best-matching clusters for the crossover approach. In the other, we form an ensemble estimator by averaging the estimators obtained from the synthetic control and crossover methods.

In Section 3, we compare by simulation the operating characteristics of these novel methods, the mixed effects models with both asymptotic and permutation-based inference, and the non-parametric within-period model, for both risk difference and odds ratio effect scales. We also apply these novel methods to a SW-CRT on the effects of diagnostic tests on tuberculosis outcomes reported by Trajman et al.,⁴⁶ and compare the results to those for existing methods. Finally, in Section 4, we discuss the implications of these results for those designing and analyzing SW-CRTs. We also propose future research directions to better understand the relative performance of the methods considered here, as well as to better understand in which settings a SW-CRT may or may not be a reasonable design.

2. METHODS

In this section, we propose several novel methods of analysis for SW-CRTs: a synthetic control-based method, a crossover-based method, a combination method, and an ensemble method. These methods have flexible weighting schemes that allow the method to be tailored to particular situations. These methods do not rely on any particular distribution of the outcome data and can be used to estimate any causal contrast of interest.

2.1. Setting and Notation

Consider a SW-CRT with I clusters with outcome measurements in each of J periods. Denote by Y_i,j the mean outcome for all K measured individuals in cluster i in period j (K can be fixed or vary by cluster-period). Let X_i,j denote the intervention status of cluster i in period j, with X_i,j = 1 indicating that the cluster is on intervention and X_i,j = 0 indicating that the cluster is on control. For each period j, let I_0,j = {i :X_i,j = X_i,j-1 = 0}, I_i,j= { i :X_i,j = 1, X_i,j-1, = 0}, and I_2,j= { i :X_i,j = X_i,j-1 = 1}, the set of clusters on control in both periods j and j − 1, crossing over in period j, and on intervention in both periods j and j− 1, respectively. Denote the number of clusters in each of these sets by n_0,j, n_1,j, and n_2,j, respectively. We assume that each cluster only crosses over once; once a cluster is on intervention, it remains so for the rest of the periods under study. We assume throughout that the order of crossover is determined randomly. For each cluster i, let j_i be the last period for which it is on control (define j_i = 0 if cluster i is on intervention in period 1); then j_i + 1 is the first period on intervention for cluster i. For any period j, denote by Y._j the expected value of the outcome (marginal across clusters) in period j in the absence of intervention. That is, Y._j = E[Y_i,j | X_i,j = 0] for any cluster i

Let g(y₁,y₂) be the contrast of interest. For example, for binary outcomes, the risk difference is given by g(y₁,y₂) = y₁ − y₂ and the log odds ratio is given by $g\left(y_1,y_2\right)=\text{log}\left(\frac{y_1/\left(1-y_1\right)}{y_2/\left(1-y_2\right)}\right)$. Although binary outcomes are more common in SW-CRTs,¹⁰ contrasts of continuous and count outcomes may be specified as well.

2.2. Existing methods for comparison

We compare the performance of these novel analysis methods with that of three current approaches for the analysis of SW-CRTs: two mixed effects model specifications (each with both asymptotic and exact inference) and the non-parametric within-period method.

First, we consider the commonly-used mixed effects model with a random intercept for cluster and fixed effects for time:

$$h\left(E\left[Y_{i,j}\right]\right)=\mu +{\alpha }_i+{\theta }_j+X_{i,j}\beta ,$$ (1)

where h is the link function, μ is the global mean under control in period 1, ${\alpha }_i\stackrel{iid}{~}N\left(0,{\tau }^2\right)$ and θ₁ = 0 for identifiability.¹⁵ Generalized linear mixed model theory can be used for asymptotic inference, and permutation tests (and associated confidence intervals) can be used for exact inference with this model.^35,36

Second, we consider the mixed effects model with an additional cluster-period random intercept:

$$h\left(E\left[Y_{i,j}\right]\right)=\mu +{\alpha }_i+{\theta }_j+{\eta }_{i,j}+X_{i,j}\beta ,$$ (2)

where h is the link function, μ is the global mean under control in period 1 ${\alpha }_i\stackrel{iid}{~}N\left(0,{\tau }^2\right)$, θ₁ = 0 for identifiability, ${\eta }_{i,j}\stackrel{iid}{~}N\left(0,{\nu }^2\right)$ and ${\eta }_{i,j}\perp {\alpha }_i$ for all i,j.²⁸ Inference can proceed on an asymptotic or exact basis as above.³⁵

Third, we consider the non-parametric within-period method. In this method, for each period j where there are clusters on control and on intervention, a period-specific effect estimate is calculated by comparing the mean outcome of clusters on intervention (i ∈ I_1,j ∪ I_2,j) to the mean outcome of clusters on control (i ∈ I_0,j):

$${\widehat{\beta }}_j=g\left(\frac{{\sum }_{i\in I_{1,j}\cup I_{2,j}}{Y}_{i,j}}{n_{1,j}+n_{2,j}},\frac{{\sum }_{i\in I_{0,j}}{Y}_{i,j}}{n_{0,j}}\right).$$ (3)

The period-specific effect estimates are combined using an inverse-variance weighted average to obtain an overall estimated intervention effect:

$$\widehat{\beta }=\sum _{j:0<n_{0,j}<I}\frac{w_j}{w}{\widehat{\beta }}_j,$$ (4)

where $w_j={\left[\left(\frac{\left(n_{0,j}-1\right)s_{0,j}^2+\left(n_{1,j}+n_{2,j}-1\right)s_{1,j}^2}{J-2}\right)\left(\frac{1}{n_{0,j}}+\frac{1}{n_{1,j}+n_{2j}}\right)\right]}^{-1}$, $w={\sum }_{j:0<n_{0,j}<I}{w}_j$, and $s_{0,j}^2$ and $s_{1,j}^2$ are the empirical variances of the Y_i,j values for clusters on control and on intervention, respectively, for period j.⁴¹ A schematic representation of this estimation method is given in Figure 1a. Exact inference can proceed using permutation tests.⁴¹

Figure 1.

Schematic Representations of Several Existing and Novel Analysis Methods for a SW-CRT with Seven Clusters, Eight Periods, and One Crossover Per Period. Dark Green Boxes Indicate Cluster-Periods on Intervention and White/Gray Boxes Indicate Cluster-Periods on Control.

2.3. Synthetic control method

Our first proposed method uses the synthetic control procedure developed by Abadie et al. to estimate the effect of treatment for each intervention cluster-period.⁴⁴ Similar to the non-parametric within-period method proposed by Thompson et al., this novel method constructs vertical comparisons and then uses a weighted average of these comparisons as an overall effect estimate.⁴¹

1. For each period j where there are clusters on control and on intervention, for each cluster i on intervention (i ∈ I_1,j ∪ I_2,j), we construct a synthetic control estimator Z_i,j, using the procedure outlined by Abadie et al.⁴⁴ The synthetic control for cluster i in period j is a weighted average of the outcomes of the clusters on control in period $j:Z_{i,j}={\sum }_{n=1}^{n_{0,j}}{v}_{i,j,n}{Y}_{m_n,j}$, where $m_1,\dots ,m_{n_{0,j}}$ are the clusters on control in period j. The weights, υ_i,j,n are selected by the synthetic control procedure to minimize the mean squared difference between the synthetic control for periods where cluster i was on control and the outcome for cluster i in that period subject to the constraints that the weights are nonnegative and ${\sum }_{n=1}^{n_{0,j}}{v}_{i,j,n}=1$ for all i, j That is, they minimize:

   $$\mathit{\text{MSP}}{E}_{i,j}=\sum _{j^{\prime }:X_{i,j^{\prime }}=0}{\left(Y_{i,j^{\prime }}-\sum _{n:X_{m_n,j}=0}{v}_{i,j,n}{Y}_{m_n,j^{\prime }}\right)}^2.$$ (5)

   See the proof of Theorem 1 in Appendix A for details. If specific cluster-level covariates are known, they can be included in estimation of the synthetic control as well.⁴⁴ When the synthetic control procedure does not converge or there are no pre-intervention periods for this cluster, the unweighted mean of the outcomes of clusters on control in period j is used as Z_ij. In these cases, the period-specific effect estimator is the same as that for the non-parametric within-period method described above, and so the properties of that estimator hold.
2. For each intervention cluster i for each period j where X_i,j = 1 and n_0,j ≥1, we construct an estimator:

   $${\widehat{\beta }}_{i,j}=g\left(Y_{i,j},Z_{i,j}\right)$$ (6)
3. We find an overall estimator via a weighted average of these cluster-period-specific estimators:

   $$\widehat{\beta }=\sum _{j:0<n_{0,j}<I}\sum _{i\in I_{1,j}\cup I_{2,j}}\frac{w_{i,j}}{w}{\widehat{\beta }}_{i,j},$$ (7)

   where w_i,j ≥ 0 and $w=\sum _{j:0<n_{0,j}<I}\sum _{i\in I_{1,j}\cup I_{2,j}}{w}_{i,j}$.

A schematic representation of this estimation method is given in Figure 1b.

2.3.1. Inferential procedure

A permutation test can be used for exact inference, as for mixed effects models and the non-parametric within-period method.^35,41 The standard permutation test approach is used: P random permutations of the crossover order are generated and an estimate of the treatment effect is obtained from each permutation using the estimation procedure described above. The observed estimate $\widehat{\beta }$ is compared to these estimates and the p-value for the null hypothesis of no effect of treatment is given by the proportion of the P estimates for which $|{\widehat{\beta }}^p|\ge |\widehat{\beta }|$. This approach matches inferential methods for synthetic control estimators, which rely on treatment of units and estimating placebo synthetic control estimators to derive the null distribution of the estimator.^44,45,47 To obtain confidence intervals, the permutation test can be inverted in the standard way.

2.3.2. Computation

This procedure is implemented in the R code included in the Supplemental Material. The latest version of the code can be found at https://github.com/leekshaffer/SW-CRT-analysis This implementation uses the synth function from the Synth package to obtain the synthetic control weights υ_i,j,n.⁴⁸ User-defined functions to implement this method can also be constructed in Stata using the Synth command to perform the synthetic control fits,⁴⁸ and the swpermute command to permute clusters in a manner that preserves exchangeability.⁴⁹ Implementations using either of these Synth commands rely on Synth for convergence. In R, Synth automatically runs with two starting values to improve the likelihood of convergence. However, for clusters that have extreme outcome values compared to the donor clusters, there is still no guarantee that the procedure will converge. This is most likely to occur when there are few donor clusters, i.e. later in the trial, or when the pattern of the target cluster is heterogeneous compared to the others.⁴⁸

2.3.3. Properties of the estimator

In a SW-CRT with a randomized order of crossover, the synthetic control estimator Z_i,j is an unbiased estimate of the expected outcome under control, Y._j, if the underlying cluster-level outcome distribution is symmetric around some global mean outcome vector across periods; see Theorem 1 in Appendix A. If the individual-level outcomes have cluster-conditional expectations that are symmetric around a global mean vector, the estimator is asymptotically unbiased as the number of subjects with measured outcomes per cluster increases; see Corollary 1. Thus, for any weights independent of the outcomes, the SC estimator using the risk difference is unbiased or asymptotically unbiased under these conditions if there is a common risk difference across cluster-periods. See Theorem 2 and Corollary 2. For a non-linear contrast function (e.g., risk ratio or odds ratio), the unbiasedness of Z_i,j does not guarantee unbiasedness of the effect estimate. Depending on the contrast and the assumed data-generating process, it may be possible to show consistency of this estimate. Further research is needed on the effect of applying a non-linear contrast function to cluster-level outcomes, specifically on targeting marginal or cluster-specific parameters. Note that all of the assumptions of Corollary 2 are satisfied under the mixed effects models described in Section 2.2 with an identity link function as long as the random effects are independent and identically distributed following a normal (or any other symmetric) distribution. Hypothesis tests using the permutation test method consider the sharp null hypothesis that the treatment has no effect in any cluster.⁵⁰ When the treatment effect varies by cluster—e.g., if there is a random cluster-by-treatment effect—the estimator will estimate some weighted average of the treatment effects and the permutation test may not guarantee nominal Type I Error and coverage rates, depending on the correlation of this variation with other parameters.

2.3.4. Selecting weights

The weights for combining the cluster-period-specific estimators, w_i,j, can be selected on the basis of two primary goals: (1) minimizing the variance of the overall estimator $\widehat{\beta }$ or (2) estimating a specific causal contrast when treatment effects may not be equal across clusters and time periods.

For the first goal, a natural approach is to follow the synthetic control literature on evaluating the accuracy of the synthetic control estimator or combining multiple synthetic control estimators by using the inverse of the mean squared prediction error (MSPE) values for each synthetic control estimator.^51–53 For cluster i in period j, the MSPE of the synthetic control fit is given by equation (5). In the SW-CRT setting, however, the MSPE values are not directly comparable as different synthetic control estimators have a different number of pre-intervention periods that contribute. By contrast, the MSPE values will be comparable for intervention clusters that begin treatment in the same period, as these clusters will always have the same number of pre-intervention periods, regardless of which of their intervention periods are being examined. We therefore propose to weight the ${\widehat{\beta }}_{i,j}$ values by the inverse-MSPE within each set of intervention clusters that cross over in the same period, and then weight across these sets equally. That is, for each (i,j) such that X_i,j= 1, set weights proportional to:

$$w_{i,j}=\frac{\mathit{\text{MSP}}{E}_{i,j}^{-1}}{{\sum }_{\left(i^{\prime },j^{\prime }\right):i^{\prime }\in I_{1,j},X_{i^{\prime },j^{\prime }}=1}\mathit{\text{MSP}}{E}_{i^{\prime },j^{\prime }}^{-1}},$$ (8)

where MSPE_i,j is the MSPE of the synthetic control estimation procedure that produces Z_i,j. There may be other considerations that affect the variances of the cluster-period summaries. Most notably, if cluster sizes vary by period or cluster, it may be desirable to consider this in selecting the weights. All of the properties of the estimator hold for varying cluster sizes, provided that cluster sizes are large enough to assure asymptotic symmetry of the cluster-period summary, when that property is necessary.

For the second goal of weighting, the weighting approach will depend on the causal estimand of interest. If, for example, investigators are only interested in the effect of intervention in the first period of its introduction to any cluster, they may select as weights:

$$w_{i,j}=\{\begin{array}{ll}1,\hfill & i\in I_{1,j}\hfill \\ 0,\hfill & \text{otherwise}\hfill \end{array},$$ (9)

that is, only using the ${\widehat{\beta }}_{i,j}$ estimates for the first period on intervention for each cluster. We do not present results on this approach here, but further research is needed to understand the causal estimands that may be of interest when the treatment effect cannot be assumed to be constant across clusters and periods. In this way, the weights also aid interpretability of the estimator, as it is clear which clusters and periods are considered and how much weight is given to each.

2.4. Crossover method

The second novel method seeks to improve on the power of the non-parametric within-period method by incorporating horizontal comparisons at the time of crossover. There is substantial literature on the value of within-subject analysis methods and methods combining within- and between-subject analyses for individual randomized crossover trials, especially in the absence of anticipation, lag, or carryover effects of treatment.^54–57 The method we propose for SW-CRTs compares the mean contrast between the last control period and the first intervention period for each cluster crossing over from one period to the next to the mean contrast in those same periods among clusters on control in both periods. Since standard mixed effects models give a large weight to horizontal comparisons,²⁷ the crossover approach may recover some of the power of mixed effects models while preserving the robustness of non-parametric estimation. The procedure is as follows:

1. For each cluster i and period j > 1, define D_i,j ≡ g(Y_i,j, Y_i,j-1) the contrast in outcomes in cluster i between consecutive periods. E.g., for a risk difference analysis, D_i,j = Y_i,j − Y_i,j-1, the difference in outcomes between consecutive periods.

2. For each period j > 1 with clusters on both intervention and control, estimate the treatment effect for period j by:

   $${\widehat{\beta }}_j=\sum _{i\in I_{1j}}\frac{D_{i,j}}{n_{1,j}}-\sum _{i\in I_{0,j}}\frac{D_{i,j}}{n_{0,j}}.$$ (10)

   If the treatment effect is assumed to be constant across time, an alternate estimator is given by:

   $${\tilde{\beta }}_j=\sum _{i\in I_{1,j}}\frac{D_{i,j}}{n_{1,j}}-\sum _{i\in I_{0,j}\cup I_{2,j}}\frac{D_{i,j}}{n_{0,j}+n_{2,j}}.$$ (11)

   This alternative compares the change in outcome for the clusters which cross over to the change for clusters which either remain on control in both periods or remain on intervention in both periods.
3. Construct an overall estimator with a weighted average of period-specific estimators:

   $$\widehat{\beta }=\sum _{j>1:0<n_{0,j},n_{1,j}<I}\frac{w_j}{w}{\widehat{\beta }}_j,\text{\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}or\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}}\tilde{\beta }=\sum _{j>1:0<n_{1,j}<I}\frac{w_j^{\prime }}{w^{\prime }}{\tilde{\beta }}_j,$$ (12)

   where $w_j,w_j^{\prime }\ge 0,w=\sum _{j>1:0<n_{0,j},n_{1,j}<I}{w}_j$ and $w^{\prime }=\sum _{j>1:0<n_{1,j}<I}{w}_j^{\prime }$.

A schematic representation of this estimation method is given in Figure 1c.

2.4.1. Inferential procedure

A permutation test can again be used for hypothesis testing and to obtain confidence intervals. The procedure is the same as the inferential procedure for the synthetic control estimator, detailed in Section 2.3.1.

2.4.2. Computation

This procedure is implemented in the R code included in the Supplemental Material. Again, a user-defined function to implement this method can also be constructed in Stata using the swpermute command to permute clusters in a manner that preserves exchangeability.⁴⁹

2.4.3. Properties of the estimator

For the risk difference, g(y₁,y₂) = y₁-y₂ any of these crossover estimates are unbiased estimates of the true risk difference β, under a randomized crossover order and the assumption of a constant β across clusters and periods. See Theorem 3 in Appendix A. The controls-only estimator $\widehat{\beta }$ is unbiased for the intervention effect in the first period on intervention if that effect is constant across clusters. See Corollary 3 in Appendix A. As for the synthetic control estimator, there may be settings where consistency can be shown for non-linear contrast functions, although unbiasedness is not guaranteed. Again, non-linear link functions applied to cluster-level outcomes target specific causal estimands and further research is needed on the consequences of targeting marginal rather than cluster-specific estimands.

2.4.4. Selecting weights

As for the synthetic control estimator, the weights can be selected either to minimize the variance of the overall estimator or to ensure proper estimation of a specific causal estimand. For the latter, again, this will depend on the specific estimand of interest, e.g., to match a target population of clusters.

To minimize the variance of the overall estimator, the weights may depend on the variance of the cluster-level outcome for each cluster-period. If all of these variances are assumed to be the same (i.e., all have the same subject-level variance and the cluster sizes do not vary by cluster or period), then the weight should depend only on the number of clusters in each treatment condition in that period. That is, we weight each estimator ${\widehat{\beta }}_j$ by $w_j={\left(\frac{1}{n_{0,j}}+\frac{1}{n_{1,j}}\right)}^{-1}$ the harmonic mean of the number of clusters used to estimate the consecutive-period control effect and the number of clusters used to estimate the crossover effect. For ${\tilde{\beta }}_j$ where the clusters which were on intervention in both periods j and j – 1 are used as control crossovers as well, we weight by $w_j^{\prime }={\left(\frac{1}{n_{0,j}+n_{2,j}}+\frac{1}{n_{1,j}}\right)}^{-1}$. Note that when the same number of clusters cross over at each time point, $w_j^{\prime }$ is constant across j, while w_j decreases as j increases.

2.5. Crossover-synthetic control method

A third potential method combines these two approaches by finding a synthetic control for the horizontal crossover contrast and comparing the intervention horizontal contrast to this synthetic control. This may combine the benefits of using horizontal comparisons with the benefits of synthetic control-based matching between clusters.

1. For each cluster i and period j > 1, define D_i,j ≡ g(Y_i,j, Y_i,j-1) the contrast in outcomes in cluster i between consecutive periods. E.g., for a risk difference analysis, D_i,j = Y_i,j − Y_i,j-1, the difference in outcomes between consecutive periods.

2. For each cluster $i\in I*\equiv \left\{i:j_i+1>1\cap n_{0,j_i+1}>0\right\}$ the set of clusters that begin intervention in a period after period 1 that has clusters on control, construct a synthetic control horizontal contrast estimator C_i, using the procedure outlined by Abadie et al.⁴⁴ For cluster i, the synthetic control is a weighted average of the horizontal contrasts of the clusters on control in both periods j_i and $j_i+1:C_i={\sum }_{n=1}^{n_{0,j_i+1}}{v}_{i,n}{D}_{m_n,j_i+1}$, where $m_1,\dots ,m_{n_0,j_i+1}$ are the clusters on control in both periods. The weights are selected by the synthetic control procedure to minimize the mean squared difference between the synthetic control for periods j′ ≤ 1 where cluster i ∈ I_0,j′ and the true horizontal contrast for cluster i in that period subject to the constraints that the weights are nonnegative and sum to one. When the synthetic control procedure does not converge or there are no pre-crossover consecutive period contrasts for this cluster, the unweighted mean of the values D_i′,ji+1 for i′ ∈ I_0,ji+1 is used as C_i

3. For each cluster i ∈ I*, we construct an estimator using its crossover effect:

   $${\widehat{\beta }}_i=D_{i,j_i+1}-C_i.$$ (13)
4. We find an overall estimator via a weighted average of these cluster-specific estimators:

   $$\widehat{\beta }=\sum _{i\in I*}\frac{w_i}{w}{\widehat{\beta }}_i,$$ (14)

   where $w={\sum }_{i\in I*}{w}_i$.

A schematic representation of this estimation method is given in Figure 1d. Note that this procedure is the same as that for the synthetic control method, but using D_i,j as the “outcomes” in place of Y_i,j.

2.5.1. Inferential procedure

The inferential procedure for the synthetic control estimator, detailed in Section 2.3.1, can again be used here for exact inference.

2.5.2. Computation

This procedure is implemented in the R code included in the Supplemental Material. This implementation uses the synth function from the Synth package to obtain the synthetic control weights v_i,n.⁴⁸

2.5.3. Selecting weights

As for the synthetic control estimator, a natural approach to minimize the variance of the overall estimator is to use weights inversely proportional to the MSPE of the synthetic control fits. Again, though, because the number of pre-crossover periods varies, these are only comparable among clusters which cross over in the same period. So we propose to weight the ${\widehat{\beta }}_i$ values by the inverse-MSPE within each set of intervention clusters that cross over in the same period, and then weight across these sets equally. That is, for each i, set:

$$w_i=\frac{\mathit{\text{MSP}}{E}_i^{-1}}{{\sum }_{i^{\prime }\in I_{1j_i+1}}\mathit{\text{MSP}}{E}_{i^{\prime }}^{-1}},$$ (15)

where MSPE_i is the MSPE of the synthetic control estimation procedure that produces Z_i.

2.6. Ensemble method

Finally, we consider an ensemble method that averages across the estimators of previously-described methods. For any unbiased and/or consistent estimators, a weighted average of those estimators with weights that do not depend on the data will also be unbiased/consistent. If the covariance of the estimators is small enough compared to the variances, it may also reduce the variance of the estimator. In Appendix B, we derive the variances and covariance of the non-parametric within-period and crossover estimators under a simplified data-generating process. We then demonstrate that in this setting, when the difference in the mean outcome between clusters is relatively small compared to the variability within clusters, a simple mean of the non-parametric within-period estimator and the crossover estimator has a lower variance than either estimator on its own.

No analytic formula is available for the variance of the synthetic control estimator, although we expect (and simulation results presented below suggest) the synthetic control estimator to have lower variance than the non-parametric within-period estimator when the synthetic control matching performs well. Since the synthetic control and non-parametric within-period estimators are both vertical methods of analysis, we consider here an ensemble estimator that is a simple mean of the synthetic control estimator and the crossover estimator. That is,

$${\widehat{\beta }}^{\mathit{\text{ENS}}}=\frac{1}{2}{\widehat{\beta }}^{\mathit{\text{SC}}}+\frac{1}{2}{\widehat{\beta }}^{\mathit{\text{CO}}},$$ (16)

where ${\widehat{\beta }}^{\mathit{\text{SC}}}$ is a synthetic control estimator and ${\widehat{\beta }}^{\mathit{\text{CO}}}$ is a crossover estimator.

Note that many other ensemble estimators could be constructed using different analysis methods and different weights. In addition, within-period ensembles may be constructed and then combined across periods (e.g., take the average of the SC estimators within each period j and average those with the CO estimator for period j, and then combine across periods to target a specific causal estimand). We use this simple version here to demonstrate the concept of the ensemble method and show its potential to improve power, but different ensembles will have different operating characteristics and may perform better or worse, relative both to one another and to other methods, depending on the setting.

2.6.1. Inferential procedure

The inferential procedure for the synthetic control estimator, detailed in Section 2.3.1, can again be used here for exact inference.

2.6.2. Computation

This procedure is implemented in the R code included in the Supplemental Material. Other ensemble methods can be constructed by altering the weights and estimators used; a generic function is provided for this purpose in the R code.

3. RESULTS

We compare the performance of these novel methods with the existing methods under two simulation settings: the first using the risk difference contrast, g(y₁,y₂) = y₁ – y₂, and the second using the log odds ratio contrast, $g\left(y_1,y_2\right)=\text{log\hspace{0.17em}}\left(\frac{y_1/\left(1-y_1\right)}{y_2/\left(1-y_2\right)}\right)$. As SW-CRTs most commonly have binary outcomes, we consider binary outcomes here; the methods, however, also work for continuous outcomes. Throughout we denote the methods considered as follows:

• MEM denotes the mixed-effects model defined in equation (1).

• CPI denotes the mixed-effects model with a cluster-period random effect defined in equation (2).

• NPWP denotes the non-parametric within-period method defined in equations (3) and (4).

• SC-1 denotes the synthetic control method defined in equations (6) and (7), with equal weights across cluster-period estimators.

• SC-2 denotes the synthetic control method with inverse-MSPE weights as defined in equation (8). In this case, there is only one cluster crossing over per period, so the estimators are weighted by inverse-MSPE within each target cluster, and then equally weighted across clusters.

• CO-1 denotes the crossover method defined in equations (10) and (12), using comparison data only from control clusters, with equal weights.

• CO-2 denotes the crossover method defined in equations (10) and (12), using comparison data only from control clusters, with weights proportional to the harmonic mean of the number of control and crossover clusters.

• CO-3 denotes the crossover method defined in equations (11) and (12), using comparison data from both control clusters and intervention clusters, with equal weights.

• COSC-1 denotes the crossover-synthetic control method defined in equations (13) and (14) with equal weights across cluster-specific estimators.

• COSC-2 denotes the crossover-synthetic control method with inverse-MSPE weights defined in equation (15).

• ENS denotes the ensemble method defined in equation (16), using a simple mean of SC-2 and CO-2.

All inference is based on exact permutation tests, except for asymptotic inference using the MEM and CPI models, which is denoted by MEM-a and CPI-a. All permutation tests were conducted with 500 randomly-sampled permutations of the crossover order.

3.1. Simulation 1: risk difference

3.1.1. Setting and parameters

We consider a setting where the risk difference is the contrast of interest. There are I = 7 clusters and J = 8 time periods, with one cluster beginning treatment in each of periods 2 through 8. At each cluster-period, K = 100 individuals are sampled. The data are generated from a mixed effects model similar to that in equation (2) with μ = 0.30 and τ = 0.06, with an identity link. We consider four scenarios:

1. Fixed time effects θ = θ₁ ≡ (0, 0.08, 0.18, 0.29, 0.30, 0.27, 0.20, 0.13) and no cluster-period effect (v = 0). The MEM model is correctly specified in this case.

2. Fixed time effects θ = θ₁ and cluster-period effect with v = 0.01. The CPI model is correctly specified in this case.

3. Equal probability of each cluster having either the time effects θ₁ or θ₂ ≡ (0, 0.02, 0.03, 0.07, 0.13, 0.19, 0.27, 0.3). No cluster-period effect (v = 0). Neither MEM nor CPI is correctly specified in this case.

4. Equal probability of each cluster having either the time effects θ₁or θ₂. Cluster-period effect with v = 0.01. Neither MEM nor CPI is correctly specified in this case.

Note that all scenarios satisfy the conditions of Corollaries 1 and 2 and Theorem 3 in Appendix A, so the SC-1 estimator is asymptotically unbiased and the CO estimators are unbiased. Since SC-2 does not have equal weights, it does not meet the conditions of Theorem 2 or Corollary 2, so we cannot guarantee it is asymptotically unbiased. For scenarios 1 and 2, the global mean vector is Y_.J = μ + θ₁ For scenarios 3 and 4, the global mean vector is $Y_{.J}=\mu +\frac{{\theta }_1+{\theta }_2}{2}$.

These scenarios are designed to show the performance of the methods under the commonly-assumed mixed effects models, and under scenarios that are slightly more complex and thus have misspecified MEM and CPI models. Since the assumption of common or known time effects distributions are so key to the common mixed effects models, we focus on settings where that assumption does not hold. The difference in the two time effect vectors is set to be greater than the standard deviation of the random effects, so that it cannot be captured by that parameter. While there is heterogeneity in the size of SW-CRTs, many real and simulated studies have used 3–12 randomized treatment initiation times (“sequences” or “waves”) with one or more clusters per time.^{12–14,26,35,41,46} We chose seven initiation times with one cluster each to enable the varying time trends to have an effect on the outcomes while ensuring power was low enough in some scenarios to show variation between the methods and ensuring feasible computation time for a large number of simulations. We show results here for 100 individuals per cluster but found that using a higher number of individuals did not substantially affect the relative performance of different methods.

For each scenario, 1,000 data sets were simulated for each of three treatment effects: β = −0.2, β = −0.1, and β = 0. We do not present the results for the strong treatment effect (β = −0.2) here, as they are very similar to those for the moderate treatment effect (β = −0.1), but with such high power (all methods except NPWP over 90% in all scenarios) that it is hard to distinguish differences. A representative plot of cluster outcomes for each of the four scenarios with no treatment effect is given in Figure 2. If the probability of outcome for any cluster-period was less than 0 or greater than 1, it was truncated to 0 or 1, respectively. The number of simulations per scenario was chosen so that, for methods with a true Type I Error of 0.05, the empirical Type I Error will be between 0.037 and 0.064 with 95% probability. Similarly, for methods with a true confidence interval coverage of 95%, the empirical coverage will be between 93.6% and 96.3% with 95% probability. Code to generate and analyze the simulated data is available in the Supplemental Materials.

Figure 2.

Sample Generated Data for Four Risk Difference Scenarios (Simulation 1) with No Treatment Effect. Each Line Represents the Simulated Cluster-Level Outcome for One Cluster over Eight Time Periods. Black Lines Represent Clusters with Time Effects θ = θ₁ and Gray Lines Represent Clusters with Time Effects θ = θ₂.

For each of the twelve scenarios, each data set was analyzed using the following methods: MEM, CPI, NPWP, SC-1, SC-2, CO-1, CO-2, CO-3, COSC, and ENS. Note that since only one cluster crosses over in each period, COSC-2 is equivalent to COSC-1; this is denoted COSC. The weights for SC-2 are calculated with inverse-MSPE weighting only within each intervention cluster but still differ from SC-1, which is equally weighted among all cluster-periods.

3.1.2. Simulation results

Figure 3 shows the mean effect estimate and 1/2-standard deviation of the effect estimates across the 1,000 simulations for each method for each scenario. The two subplots each show the scenarios for one treatment effect, with Scenario 1 at the top and Scenario 4 at the bottom of each plot. For all of the settings, all of the methods exhibit little overall bias, with the average estimate for each method within 0.005 of the true effect in each scenario. As expected given that all four scenarios meet the assumptions of Corollary 2 and Theorem 3, SC-1, CO-1, and CO-2 appear to be unbiased in the simulations. As noted by Thompson et al.,⁴¹ the nonparametric estimator NPWP must also be unbiased in all scenarios. Despite the misspecification of MEM and CPI in scenarios 3 and 4, they nonetheless result in unbiased estimators, albeit with wider empirical variance. And SC-2 appears unbiased in these simulations as well, despite its not meeting the conditions of Corollary 2. The variability of the effect estimates varies a great deal by method, with the MEM and CPI methods exhibiting the least variability when the time effects do not vary, and the CO and ENS methods exhibiting the least variability when the time effects do vary. Figure 4 shows the Type I Error (probability of finding a significant treatment effect when β = 0) for each analysis method under each scenario. All of the methods are close to the nominal Type I Error of 5% with the exception of asymptotic inference for the MEM and CPI methods when the time effects vary. All of the exact inference methods also achieve the nominal coverage in 95% confidence intervals, as shown in Figure 5. When the time effects do not vary, the MEM and CPI methods with asymptotic inference also achieve or nearly achieve the nominal coverage; when the time effects do vary, they both have less than 90% coverage. Even though MEM is misspecified in scenario 2, with fixed time effects but a non-zero random cluster-period effect, it still achieves nominal Type I Error and confidence interval coverage. This is likely due to the fact that the magnitude of the cluster-period effect is small compared to the treatment effect and random cluster effect. A model with a larger departure from the assumptions may lead to improper inference for this method.

Figure 3.

Mean Treatment Effect Estimates and 1/2-Standard Deviation of Estimates across 1,000 Simulations for Risk Difference Scenarios (Simulation 1) by Analysis Method.

Figure 4.

Type I Error Rate across 1,000 Simulations for Risk Difference Scenarios (Simulation 1) by Analysis Method.

Figure 5.

95% Confidence Interval Coverage Rate across 1,000 Simulations for Risk Difference Scenarios (Simulation 1) by Analysis Method.

Figure 6 shows the power (estimated probability of finding a significant treatment effect at the 5% significance level) for each analysis method under each scenario for the moderate treatment effect (β = −0.1). The asymptotic inference MEM and CPI results are not shown when the time effects vary as they have inflated Type I Error. The MEM and CPI (exact or asymptotic inference) methods have the highest power when the time effects do not vary. When the time effects do vary, the CO and ENS methods perform the best among the exact inference methods, followed by the COSC, SC, MEM, and CPI methods. The NPWP method has the least power. As expected with weights selected to reduce variance, CO-2 outperforms CO-1 and SC-2 outperforms SC-1. These differences, however, are smaller than the differences between classes of methods. These results are also shown in Table C1 in Appendix C.

Figure 6.

Power across 1,000 Simulations for Risk Difference Scenarios (Simulation 1) with Moderate Treatment Effect (β = −0.1) by Analysis Method.

3.2 |. Simulation 2: odds ratio

3.2.1 |. Setting and parameters

We consider now a setting where the odds ratio is the contrast of interest. There are again I = 7 clusters and J = 8 time periods, with one cluster beginning treatment in each of periods 2 through 8. At each cluster-period, K = 100 individuals are sampled. The data are generated from a mixed effects model similar to that in equation (2) with μ = logit(0.30) and τ = 0.1, with a logit link. We consider four scenarios:

1. Fixed time effects θ = θ₁ ≡ log(1, 1.43, 2.15, 3.36, 3.50, 3.09, 2.33, 1.76) and no cluster-period effect (v = 0). The MEM model is correctly specified in this case.

2. Fixed time effects θ = θ₁ and cluster-period effect with v = 0.01. The CPI model is correctly specified in this case.

3. Equal probability of each cluster having either the time effects θ₁ or θ₂ ≡ log(1, 1.10, 1.15, 1.37, 1.76, 2.24, 3.09, 3.50). No cluster-period effect (v = 0). Neither MEM nor CPI is correctly specified in this case.

4. Equal probability of each cluster having either the time effects θ₁ or θ₂. Cluster-period effect with ν = 0.01. Neither MEM nor CPI is correctly specified in this case.

For each scenario, 1,000 data sets were simulated for each of three treatment effects: β = log(0.50) ≈ −0.693, β = log(0.66) ≈ −0.416, and β = log(1) = 0. Again, we do not present the results for the strong treatment effect as they are very similar to those for the moderate treatment effect, but with such high power as to make comparisons difficult. These parameters were chosen to give similar outcome probabilities under control as in Simulation 1, but specified on the log-odds ratio scale. A representative plot of cluster outcomes for each of the four scenarios with no treatment effect is given in Figure 7. Code to generate and analyze the simulated data is available in the Supplemental Materials. For each of the twelve scenarios, each data set was analyzed using the same set of methods as in the previous section.

Figure 7.

Sample Generated Data for Four Odds Ratio Scenarios (Simulation 2) with No Treatment Effect. Each Line Represents the Simulated Cluster-Level Outcome for One Cluster over Eight Time Periods. Black Lines Represent Clusters with Time Effects θ = θ₁ and Gray Lines Represent Clusters with Time Effects θ = θ₂

3.2.2. Simulation results

The same set of results are shown as for the risk difference simulations, in Figures 8, 9, 10, and 11, and in Table C2 in Appendix C. For all of the settings, all of the methods exhibit little overall bias, with the average estimate for each method within 0.01 of the true effect in each scenario. Thus, even without theoretical proofs of unbiasedness, in these simulated settings, the methods appear to give unbiased estimates. As in the risk difference setting, all of the methods are close to the nominal Type I Error of 5% and nominal 95% confidence interval coverage with the exception of asymptotic inference for the MEM and CPI methods when the time effects vary. Figure 11 shows the power for each analysis method under each scenario for the moderate treatment effect (β= log(0.66) ≈ −0.416). The asymptotic inference MEM and CPI results are not shown when the time effects vary as they have inflated Type I Error. The MEM and CPI (exact or asymptotic inference) methods have the highest power when the time effects do not vary, but there is relatively little loss of power for the ENS, SC, and CO-3 methods. When the time effects do vary, the ENS method performs the best among the exact inference methods, followed by the CO-3 method, the SC and other CO methods, and then the COSC method. The NPWP and exact inference MEM and CPI methods have the least power.

Figure 8.

Mean Treatment Effect Estimates and 1/2-Standard Deviation of Estimates across 1,000 Simulations for Odds Ratio Scenarios (Simulation 2) by Analysis Method.

Figure 9.

Type I Error Rate across 1,000 Simulations for Odds Ratio Scenarios (Simulation 2) by Analysis Method.

Figure 10.

95% Confidence Interval Coverage Rate across 1,000 Simulations for Odds Ratio Scenarios (Simulation 2) by Analysis Method.

Figure 11.

Power across 1,000 Simulations for Odds Ratio Scenarios (Simulation 2) with Moderate Treatment Effect (β = log(0.66) ≈ −0.416) by Analysis Method.

These results are largely similar to those seen in Simulation 1. This suggests that the contrast of interest is less important to the relative performance of these methods than the underlying distribution of the data.

3.3. Variance and covariance of estimators

To assess the variability between methods for a given instance of analysis, we determined the pairwise correlation for each pair of methods across the simulated settings. Within each data-generating setting, we found the correlation between methods across all 1,000 simulations. As a representative example of these correlations, we take scenario 4, the scenario with the most complex data-generating process, under the null hypothesis of no treatment effect, for both Simulation 1 and Simulation 2. The correlations are displayed in the heat map shown in Figure 12 for the risk difference (Simulation 1) and in Figure 13 for the odds ratio (Simulation 2).

Figure 12.

Pairwise Correlations between Effect Estimates from Different Methods: Simulation 1 (Risk Difference), Scenario 4, No Treatment Effect.

Figure 13.

Pairwise Correlations between Effect Estimates from Different Methods: Simulation 2 (Odds Ratio), Scenario 4, No Treatment Effect.

These results indicate rather high correlations within classes of methods; that is, the mixed effects model methods are highly correlated with one another, the synthetic control methods are highly correlated with one another, and the crossover methods (including COSC) are highly correlated with one another. NPWP is correlated with the mixed effects model methods and the SC methods.

The least correlation occurs between the NPWP method and any of the CO-based methods, followed by the correlations between any mixed effects or SC method and the CO methods. This suggests that using an ensemble method combining an SC method and a CO method is indeed valuable here as the low correlation may lead to a covariance that is lower than the variance of either method individually. This corresponds with the increased power for the ENS method compared with SC-2 and CO-2 seen in the previous sections. Multiplying by the square root of the empirical variances gives the covariance; the ENS method has relatively low, equal covariance with the other methods. This suggests there is little to be gained in this setting by more complex ensemble methods.

3.4. Application to tuberculosis SW-CRT

We applied the methods discussed here to a SW-CRT that assessed the effect of a tuberculosis (TB) diagnostic test on reducing unsuccessful (non-cure) outcomes of adults on TB treatment.⁴⁶ Note that this is the same trial re-analyzed by Thompson et al. using the within-period methods they proposed.⁴¹

3.4.1. Trial description

In this study, Trajman et al. conducted a SW-CRT in fourteen laboratories in the Brazilian cities of Rio de Janeiro and Manaus. While in the control arm, the labs diagnosed TB using two-sample sputum smear microscopy; in the intervention arm, diagnosis and first-line evaluation of potential drug resistance was by a single sputum sample XpertMTB/RIF assay. Data were collected on individuals diagnosed with TB in eight months in 2012 in the clinics associated with these laboratories. In the first month, all labs were in the control arm. In each subsequent month, two labs were switched to the intervention arm. In the final month, all labs were in the intervention arm.⁵⁸

The outcome of interest was the proportion of unfavorable TB treatment outcomes, where unfavorable outcomes are defined as: loss to follow-up, TB-attributed death, death from other causes, change of diagnosis, transfer out (including to specialized clinics for management of drug-resistant TB or drug intolerance), and suspicion of drug resistance. In total, the trial analyzed the intervention and outcome status of 3,924 patients.⁴⁶

3.4.2. Goodness of fit of mixed effects models

Before analyzing these data using non-parametric approaches, we consider the goodness of fit of the mixed effects models. We fit both the MEM and CPI models, as usual assuming independent normally-distributed random effects. In this case, the CPI model yields nearly the same fitted values as the MEM model, so we consider only the MEM model from this point. A variety of methods have been proposed to assess the assumption of independent normally-distributed random effects.^59–65 We use several of these methods to assess the assumption in this case; details are in Appendix D. Some methods indicate a violation of the assumption and others do not, but caution should be exercised in interpreting these results as diagnostic tests may not be powerful or reliable for such a small number of clusters.⁶⁶ Because of the potential of model misspecification, we proceed with the non-parametric analyses.

3.4.3. Results

The primary analysis conducted by Trajman et al., which did not adjust for time effects, found a decrease in the number of events (unsuccessful outcomes) in the intervention arm compared to the control arm, although this decrease was not statistically significant at the 0.05 level.⁴⁶ Re-analyzing the data using the NPWP method, Thompson et al. found a statistically significant decrease on both the odds ratio and risk difference scales.⁴¹

We analyzed these data using all of the methods described here using both the risk difference and log odds ratio contrasts; all exact inference methods use 500 permutations. Note that the NPWP method corresponds to that used by Thompson et al. for the risk difference scale. There is a slight discrepancy on the odds ratio scale, since Thompson et al. estimate the within-period contrast by comparing the mean log odds among the intervention clusters to the mean log odds among the control clusters, while we estimate that contrast by applying the log odds ratio contrast to the mean cluster-level outcome among the intervention clusters and among the control clusters.⁴¹ In both cases, inference may differ slightly because of the stochasticity in the permutation-based inference. This stochasticity, as well as the difference of calculating under the alternative hypothesis rather than the null, can also lead to confidence intervals including the null when the hypothesis test rejects the null and vice versa. Also note that since there are two clusters which cross over at each time period, COSC-1 and COSC-2 yield different results. The results, reported on the risk difference scale and the odds ratio scale, are summarized in Table 1.

TABLE 1.

Results from SW-CRT of diagnostic method on rates of unfavorable TB treatment outcomes in Brazil, by analysis method

	Risk Difference			Odds Ratio
Method	Estimate	95% Conf. Int.	p	Estimate	95% Conf. Int.	p
MEM/CPI	−3.59%	(−8.9%, 1.4%)	0.126	0.835	(0.66, 1.07)	0.104
MEM/CPI-a	−3.59%	(−8.4%, 1.1%)	0.105	0.835	(0.66, 1.05)	0.091
NPWP	−4.83%	(−10.1%, 0.1%)	0.050	0.794	(0.61, 0.99)	0.046
SC-1	−7.28%	(−18.2%, 1.0%)	0.084	0.703	(0.44, 1.04)	0.066
SC-2	−8.29%	(−18.3%, 1.1%)	0.080	0.675	(0.43, 1.07)	0.082
CO-1	−7.34%	(−14.5%, 0.5%)	0.064	0.703	(0.49, 1.04)	0.046
CO-2	−6.97%	(−14.0%, 0.5%)	0.052	0.717	(0.50, 1.03)	0.054
CO-3	−7.00%	(−14.0%, 0.0%)	0.050	0.721	(0.51, 1.00)	0.036
COSC-1	−7.01%	(−15.5%, 1.1%)	0.078	0.728	(0.49, 1.10)	0.118
COSC-2	−5.12%	(−14.7%, 4.5%)	0.242	0.784	(0.50, 1.18)	0.222
ENS	−7.63%	(−15.0%, −0.6%)	0.036	0.696	(0.49, 0.95)	0.032

The novel methods identify a stronger treatment effect than do the model-based and NPWP methods. As Thompson et al. show, the NPWP method here places a large amount of weight on the contrast in the fifth period, which has a modest (−2.23%) effect.²⁷ This attenuates the effect compared to, for example, CO-1, which equally weights contrasts in different periods. It also, however, reduces the variance of the overall estimator, thus yielding a lower p-value for the NPWP method than the CO methods which use the control crossovers only. The COSC methods do not appear to give more precision than the CO methods, but yield similar effect estimates. On both scales, the ENS method yields the lowest p-values, as it detects a strong effect and has more precision than the other novel methods. All of the results suggest a protective effect of the intervention, with the novel methods detecting a larger effect but with more uncertainty, and the NPWP method estimating a narrower confidence interval of smaller effect sizes. This example clearly shows how the choice of analysis method can have a substantial impact on the estimation and inference made on a given data set, based on the assumptions of the methods and their operating characteristics in the specific data-generating setting.

4. DISCUSSION

These results demonstrate the potential of analytic methods for SW-CRTs that do not rely on parametric modeling of secular trends for validity. These methods achieve greater power than the purely vertical within-period method by using the history of outcomes within each cluster inherently collected in a SW-CRT to match the most similar clusters or by using horizontal, within-cluster information. In the simulation settings used here, when the mixed effects models were misspecified, an ensemble method that averaged the crossover method and the synthetic control method had the highest power to detect a true treatment effect, followed by the crossover method. Further research is needed to determine in which settings each of these methods is likely to perform the best, and which of the possible ensemble methods may perform the best in which settings. These results demonstrate that this simple ensemble method may in some settings perform better, but not uniformly; nor is it necessarily the most powerful ensemble method in any setting. The potential for incorporating measured covariates or stratified randomization into the SC method may also lead to increased power in some situations.

While these methods are valuable and in general rely on weaker assumptions than do mixed effects methods for unbiasedness, they are still not as powerful as parametric mixed effects methods when the modeling assumptions are met. Investigators and analysts must assess when assumptions are likely to be met; additional research is needed to ascertain when non-parametric methods are required to accommodate secular trends that may arise in particular research settings. Additionally, further work on using regression diagnostics to identify violations of modeling assumptions would be very valuable. Investigators should consider exact inference on parametric methods when the modeling assumptions of mixed effects methods are likely to nearly hold and the non-parametric methods when the secular trends are unknown or the modeling assumptions methods are likely to be strongly violated. Caution should be exercised regarding the SC methods as well when the underlying data distribution is unlikely to be symmetric or asymptotically symmetric.

As can be seen in variability in estimates across methods, with relatively few clusters, the estimation can be very sensitive to the analysis method and even to the selection of the weighting scheme. The performance of any method in one particular analysis of a trial may not reflect its overall operating characteristics. The specific settings where the estimators depend heavily on certain cluster-periods and the impact that has on operating characteristics deserve more scrutiny. Again, this is an area where methods that equally weight clusters or periods, including specifically constructed ensemble methods, may prove useful in mitigating high dependence on specific cluster-periods by certain methods.

The methods presented here also provide advantages in interpretability and flexibility. When the treatment effect is not constant across clusters or across time periods, the mixed effects model estimate for nonlinear link functions is a conditional parameter, and its interpretation can be unclear.⁶⁷ For linear link functions, the mixed effects model estimate is a weighted average intervention effect that depends on the form of the treatment effect, including any treatment-time interactions or random treatment-by-cluster effects.²⁶ With the non-parametric methods, using equal weighting across clusters and periods, the estimate is easily interpreted as an average treatment effect across cluster-periods in the study. Other causal effects can be estimated using weights chosen to match the target parameter, depending on the effect of interest and assumptions the investigators are willing to make about generalizability to a separate target population. More work is needed to determine how to select weights that maximize efficiency for specific causal parameters that may be of common interest. For instance, if the effect of time on the intervention effect is known or a parametric form can be assumed, there may be an efficiency-maximizing weighting scheme.

When treatment effects are not instantaneous—common in settings where treatment effects vary over time—methods must be modified. Throughout this article, we have assumed that the full effect of treatment occurs during the first period of treatment and that there are no anticipation effects prior to that point. In practice, it may be desirable to account for a lag in, or gradual onset of, treatment effects resulting from logistical complexity in reaching everyone in the cluster or for the effect to reach its full strength.^8,16 This can be incorporated into the SC methods by taking as the time of start of the intervention the time of completion of such a transition period. It can be incorporated into the CO methods by taking as the “crossover effect” the contrast between the first period after the transition and the last period prior to any anticipation effects. Achieving the same efficiency as would be achieved with a similar trial with no transition period may require more clusters or more time between successive cluster crossovers. All SW-CRT methods are sensitive to properly accounting for the transition period, but the CO methods are particularly sensitive because of their focus on the horizontal comparison. If the transition period length is unknown or likely to vary across clusters, the CO methods may not be appropriate.

The synthetic control method allows for additional flexibility and the potential for increased power and use in a wider variety of settings. As mentioned above, it can be useful when lagged treatment effects or time-varying treatment effects make a specific causal estimand more desirable as a target for inference. It also, as shown in the simulations here, can be a valuable part of an ensemble method that improves the power of an estimator. And for trials with more periods, or a longer pre-intervention history, the SC method itself may perform better. In general, it provides many of the advantages of the non-parametric within-period method while using a matching-like procedure to increase power. For the COSC method, the relatively poor performance in these simulation settings may stem from the fact that one period of history is lost by using the crossover estimator. With few periods, that can have a large effect on the power. Again, a longer pre-intervention history may improve the value of this method.

Additionally, more advanced techniques can be used to improve synthetic control matching and thus potentially improve the power of the SC and COSC methods. Synthetic controls can incorporate measured covariates to improve the matching.^44,68 Moreover, new synthetic control algorithms and methodologies may also be useful in improving the matching and designing efficiency-maximizing weighting schemes. These include Bayesian synthetic control approaches,^69,70 flexible non-parametric synthetic control,⁷¹ generalized synthetic control,⁷² and augmented synthetic control.⁷³ The SC method, potentially incorporating these approaches to improve the causal inference component, may also provide a path for analysis of non-randomized studies that mimic stepped wedge trials, as the synthetic control may address confounding of treatment initiation. Further work is needed in this area to determine whether the stepped wedge trial design can be used as a target trial for causal inference from observational studies.^74,75

More research, with specific simulation settings derived from representative trials in various domains, is necessary to determine the relative performance of these methods across a wide variety of settings. Various data-generating processes and assumptions about those processes—including specific non-normal random effects, different correlation structures, and treatment effects that vary by time or cluster—have been proposed in prior research on SW-CRTs.^26,27,36,76 Some of these may be more reasonable in some individual fields than in others, and so research to determine which methods are best suited to specific SW-CRT settings, considering the outcome, cluster, and intervention of interest, would be very valuable. In addition, future work should consider appropriate sample size and power calculation approaches for these new methods. While the MEM method in the scenario where it is correctly specified here gives empirical power that matches the power predicted by analytic methods,¹⁵ this power calculation is clearly inadequate for the other methods and data scenarios, as they all suffer reduced power compared to the correctly specified MEM. Analytic formulae for specific common data-generating processes and simulation-based explorations of the power of these approaches are necessary future steps to improve the usability of novel and existing methods in a wider variety of settings. Understandings of sample size and power will also contribute to future work on the optimal design of SW-CRTs to be analyzed using different methods.^77,78 For example, all of the novel methods presented here make use of information from the initial control period, while the NPWP method does not. This suggests that while the initial control period may be inefficient for existing methods,^77,78 it may in fact be an efficient use of resources for novel methods.

These methods increase the number of analysis options available to investigators conducting stepped wedge cluster randomized trials. The SC method provides a semi-parametric option that relies on weaker assumptions about the underlying data-generating process than mixed effects models, while increasing power compared to the NPWP method, and it can be improved with advanced methods or with additional pre-intervention data. The CO method provides a non-parametric option with greatly improved power, although it relies on a constant treatment effect that appears very soon after treatment initiation. Variations of these methods and ensemble methods can also be used to target specific causal parameters and improve power in certain circumstances. Careful consideration is still required, however, to determine which analysis method is most appropriate for each individual circumstance, and more work is needed to clarify how to make that determination a priori or in a systematic way. Moreover, careful selection of analysis method does not alleviate all of the drawbacks and concerns about SW-CRTs and, as mentioned above, does not ensure ideal performance of any single analysis. Investigators should continue to select the appropriate trial design for each study, taking into account analysis methods, the target estimand, and power considerations, along with issues of logistical feasibility, ethics, risk-benefit profiles, and generalizability.

APPENDIX

A. PROOFS OF THEOREMS

Theorem 1. Suppose that for each cluster i, denoting by j_i the last period for which cluster i is on control, E[(Y_i,1, Y_i,2 …,Y_i,j,i)] = (Y_.1, Y_.2, …, Y_.ji) ≡ Y_.ji and that the distribution of (Y_i,1, Y_i,2 … Y_i,j,i) is symmetric about Y_.ji. Suppose further that the cluster-level outcomes from two different clusters are uncorrelated conditional on the full vector of expected outcomes, Y_.j, and the treatment effect β. Then, for any cluster i* in any period j* where that cluster is on intervention (j*>j_i*), the synthetic control estimator Z_i*,j* is an unbiased estimate of the marginal (across clusters) expectation for an untreated cluster in period j*. That is, E[Z_i*,j*] = Y_.j*.

Proof. Consider a target cluster i* and period j* such that X_i*,j* = 1. Let m₁… ,m_n* index the n* ≡ n_0,j* clusters on control (“donor clusters”) in period j*. For any cluster i, define Y_i,ji* ≡(Y_i,1,…, Y_i,ji*)^T, where j_i* is the last period for which cluster i* is on control (and thus j_i* < j*). Denote the j_i × n* matrix of pre-intervention donor cluster outcomes by Y ≡ (Y_m1,_ji*,…, Y_mn*,_ji*). Construct a j_i* × n* matrix of pre-intervention target cluster outcomes by repeating the vector Y_i*,ji* n* times Y_i* ≡(Y_i*,_ji*,…, Y_i*,_ji*).

By definition of the synthetic control estimator, Z_i*,j* The vector of these weights is denoted ν_i*,j* and lies in the set:

$$\mathcal{V}\equiv \left\{v\in {ℝ}^n:\sum _{n=1}^{n*}{v}_n=1\cap 0\le v_n\le 1\forall 1\le n\le n*\right\}.$$

Note that for all $v\in \mathcal{V},Y_{i*}v=Y_{i*,j_{i*}}$ Then We can Write

$$v_{i*,j*}=\underset{v\in \mathcal{V}}{\text{arg\hspace{0.17em}min}}\Vert Y_{i*,j_{i*}}-Yv\Vert =\underset{v\in \mathcal{V}}{\text{arg\hspace{0.17em}min}}\Vert \left(Y_{i*}-Y\right)v\Vert \equiv \underset{v\in \mathcal{V}}{\text{arg\hspace{0.17em}min}}\Vert Y_{i*,j*}^{\mathit{\text{diff}}}v\Vert ,$$ (A1)

where the difference matrix is :

$$Y_{i*,j*}^{\mathit{\text{diff}}}\equiv Y_{i*}-Y=\left(\begin{array}{cccc}{Y}_{i*,1}-Y_{m_1,1}& Y_{i*,1}-Y_{m_2,1}& \dots & Y_{i*,1}-Y_{m_{n*},1}\\ ⋮& ⋮& \ddots & ⋮\\ Y_{i*,j_{i*}}-Y_{m_1,j_{i*}}& Y_{i*,j_{i*}}-Y_{m_2,j_{i*}}& \dots & Y_{i*,j_{i*}}-Y_{m_{n*},j_{i*}}\end{array}\right)=\left(Y_{i*,j_{i*}}-{\mathit{Y}}_{m_1,j_{i*}},\dots ,{\mathit{Y}}_{i*,j_{i*}}-{\mathit{Y}}_{m_{n*},j_{i*}}\right)$$

By the symmetry and independence assumptions, for any n = 1, … ,n*, the distribution of Y_i*,ji* are independent and both are symmetrically distributed with a common mean Y_,ji*. Thus, each column of $Y_{i*,j*}^{\mathit{\text{diff}}}$ is symmetrically distributed with expectation 0 and hence the matrix $Y_{i*,j*}^{\mathit{\text{diff}}}$ is symmetrically distributed with expectation 0.

Moreover, for any n = 1, … ,n*, the distribution of Y_mn,j* is only correlated with the nth column of $Y_{i*,j*}^{\mathit{\text{diff}}}$ Since $\left(Y_{m_n,1}-Y_{.1},\dots ,Y_{m_n,j*}-Y_{\cdot j*}\right)$ is symmetrically distributed about (0, …, 0), the distribution of $Y_{m_n,j*}-Y_{.j*}$ conditional on $Y_{m_n,j*}-Y_{.j*}=\left(a_1,\dots ,a_{j*}\right)$ for any constants a_j is equal to the distribution of $-\left(Y_{m_n,j*}-Y_{\cdot j*}\right)$ conditional on $Y_{m_n,j*}-Y_{\cdot j*}=\left(-a_1,\dots ,-a_{j*}\right)$ Hence:

$$E\left[Y_{m_{n,j*}}-Y_{.j*}|Y_{i*,j*}^{\mathit{\text{diff}}}=A\right]=-E\left[Y_{m_n,j*}-Y_{\cdot j*}|Y_{i*,j*}^{\mathit{\text{diff}}}=-A\right],$$ (A2)

for any difference matrix A

For any difference matrix A and any donor cluster m_n, then:

$$\begin{gathered} E\left[Y_{m_n,j*}|Y_{i*,j*}^{\mathit{\text{diff}}}\in \{-A,A\}\right]=P\left[Y_{i*,j*}^{\mathit{\text{diff}}}=A|Y_{i*,j*}^{\mathit{\text{diff}}}\in \{-A,A\}\right]E\left[Y_{m_n,j*}|Y_{i*,j*}^{\mathit{\text{diff}}}=A\right]+P\left[Y_{i*,j*}^{\mathit{\text{diff}}}=-A|Y_{i*,j*}^{\mathit{\text{diff}}}\in \{-A,A\}\right]E\left[Y_{m_n,j*}|Y_{i*,j*}^{\mathit{\text{diff}}}=-A\right] \\ =\frac{1}{2}E\left[Y_{m_n,j*}|Y_{i*,j*}^{\mathit{\text{diff}}}=A\right]+\frac{1}{2}E\left[Y_{m_n,j*}|Y_{i*,j*}^{\mathit{\text{diff}}}=-A\right],\text{\hspace{0.17em}by\hspace{0.17em}symmetry\hspace{0.17em}of\hspace{0.17em}}{Y}_{i*,j*}^{\mathit{\text{diff}}} \\ =Y_{\cdot j*}+\frac{1}{2}\left(E\left[Y_{m_n,j*}-Y_{\cdot j*}|Y_{i*,j*}^{\mathit{\text{diff}}}=A\right]+E\left[Y_{m_n,j*}-Y_{\cdot j*}|{\mathit{Y}}_{i*,j*}^{\mathit{\text{diff}}}=-A\right]\right)= \\ {Y}_{.j*}+\frac{1}{2}(0),\text{\hspace{0.17em}by\hspace{0.17em}equation\hspace{0.17em}(A2)} \\ =Y_{ij*}. \end{gathered}$$ (A3)

By equation (A1) and since $\Vert Y_{i*,j*}^{\mathit{\text{diff}}}v\Vert =\Vert -Y_{i*,j*}^{\mathit{\text{diff}}}v\Vert$ for all $v\in \mathcal{V},v_{i*,j*}$ is correlated with the outcome vector $\left(Y_{m_1,j*},Y_{m_2,j*},\dots ,Y_{m_{n*},j*}\right)$ only through the element wise absolute value of $Y_{i*,j*}^{\mathit{\text{diff}}}$. Hence, for any n =1,…,n*:

$$\begin{array}{l}E\left[Y_{m_n,j*}|v_{i*,j*}\right]=E\left[E\left[Y_{m_n,j*}|v_{i*,j*},Y_{i*,j*}^{\mathit{\text{diff}}}\in \{A,-A\}\right]|v_{i*,j*}\right]=E\left[E\left[Y_{m_n,j*}|Y_{i*,j*}^{\mathit{\text{diff}}}\in \{A,-A\}\right]|v_{i*,j*}\right]\\ =E\left[Y_{\cdot j*}|{\mathit{v}}_{i*,j*}\right],\text{\hspace{0.17em}by\hspace{0.17em}equation\hspace{0.17em}(A3)}\\ =Y_{i,j*},\text{\hspace{0.17em}\hspace{0.17em}since\hspace{0.17em}}{Y}_{\cdot j*}\text{\hspace{0.17em}is\hspace{0.17em}fixed.}\end{array}$$ (A4)

And thus, denoting by ν_i*,j*,n the nth component of the vector v_i*,j*:

$$\begin{array}{l}E\left[Z_{i*,j*}\right]=E\left[\sum _{n=1}^{n*}{v}_{i*,j*,n}{Y}_{m_n,j*}\right]=E\left[E\left[\sum _{n=1}^{n*}{v}_{i*,j*,n}{Y}_{m_n,j*}|v_{i*,j*}\right]\right]\\ =\sum _{n=1}^{n*}E\left[v_{i*,j*,n}E\left[Y_{m_n,j*}|v_{i*,j*}\right]\right]=\sum _{n=1}^{n*}E\left[v_{i*,j*,n}{Y}_{\cdot j*}\right],\text{\hspace{0.17em}by\hspace{0.17em}equation\hspace{0.17em}(A4)}\\ =Y_{\cdot j*}E\left[\sum _{n=1}^{n*}{v}_{i*,j*,n}\right]=Y_{\cdot j*}\cdot 1=Y_{\cdot j*},\text{\hspace{0.17em}as\hspace{0.17em}desired.}\end{array}$$ (A5)

Corollary 1. Suppose that for each control cluster-period (i,j), the individual outcomes Y_i,j,k are independent and identically distributed, conditional on the cluster and period, with expectation Y′_i,j and finite variance. Suppose further that the Y′_i,j values satisfy the conditions in Theorem 1; that is, they are symmetrically distributed about some common expectation vector Y._J and each cluster’s values are independent of the values from all other clusters. Then the synthetic control estimator Z_i*j* for any cluster i* in any period j* where that cluster is on intervention is an asymptotically (with respect to the number of individuals in each cluster) unbiased estimate of Y_.j*.

Proof. If the individual outcomes are independent and identically distributed with expectation Y′_i,j, and finite variance, then, by the Central Limit Theorem, the mean outcome Y_i,j, for each control cluster-period is asymptotically (with respect to the number of individuals in each cluster) normally distributed with expectation Y′_i,j and finite variance. Thus, for any cluster i, we can write the distribution of the vector of pre-intervention cluster-level outcomes, as:

$$\left(Y_{i,1},\dots ,Y_{i,j_i}\right)\stackrel{\mathcal{D}}{=}\left(Y_{i,1}^{\prime },\dots ,Y_{i,j_i}^{\prime }\right)+\left(B_{i,1},\dots ,B_{i,j_i}\right)+o_p(1),$$ (A6)

where $B_{i,j}~N\left(0,{\xi }_{i,j}^2\right)$ for some finite ${\xi }_{i,j},o_p(1)\stackrel{P}{\to }0$, and the B_i,j are mutually independent. Since the B_i,j are normally (and hence symmetrically) distributed, the limiting distribution of (Y_i,1,…,Y_i,j,i) is symmetric about (Y_1,…,Y_ji) by the assumption on Y′_i,j. Moreover, since the individual outcomes are independent conditional on the cluster-period mean and the cluster means are independent by assumption $\left(Y_{i,1},\dots ,Y_{i,j_i}\right)\perp \left(Y_{i^{\prime },1},\dots ,Y_{i^{\prime },j_{i^{\prime }}}\right)$ for any i ≠ i′.

Because of this asymptotic symmetry, for any target cluster-period (i*,j*) where X_i*,j*= 1, for any difference matrix A:

$$\underset{K\to \infty }{\text{lim}}P\left[Y_{i*,j*}^{\mathit{\text{diff}}}=A|Y_{i*,j*}^{\mathit{\text{diff}}}\in \{A,-A\}\right]=\underset{K\to \infty }{\text{lim}}P\left[Y_{i*,j*}^{\mathit{\text{diff}}}=-A|Y_{i*,j*}^{\mathit{\text{diff}}}\in \{A,-A\}\right],\text{\hspace{0.17em}and\hspace{0.17em}so\hspace{0.17em}}P\left[Y_{i*,j*}^{\mathit{\text{diff}}}=A|Y_{i*,j*}^{\mathit{\text{diff}}}\in \{A,-A\}\right]=1/2+o(1),$$ (A7)

where ${\text{lim}}_{K\to \infty }o(1)=0$. Additionally, by this symmetry, for any donor cluster m_n ∈ {m₁,…,m_n*}(defined as in Theorem 1):

$$E\left[Y_{m_n,j*}-Y_{.j}|Y_{i*,j*}^{\mathit{\text{diff}}}=A\right]=-E\left[Y_{m_n,j*}-Y_{\cdot j*}|Y_{i*,j*}^{\mathit{\text{diff}}}=-A\right]+o(1).$$ (A8)

Thus, for any difference matrix A:

$$\begin{array}{l}E\left[Y_{m_n,j*}|Y_{i*,j*}^{\mathit{\text{diff}}}\in \{A,-A\}\right]=P\left[Y_{i*,j*}^{\mathit{\text{diff}}}=A|Y_{i*,j*}^{\mathit{\text{diff}}}\in \{A,-\text{A}\}\right]E\left[Y_{m_n,j*}|Y_{i*,j*}^{\mathit{\text{diff}}}=A\right]+P\left[{\mathit{Y}}_{i*,j*}^{\mathit{\text{diff}}}=-A|Y_{i*,j*}^{\mathit{\text{diff}}}\in \{A,-A\}\right]E\left[Y_{m_n,j*}|Y_{i*,j*}^{\mathit{\text{diff}}}=-A\right]\\ =\left(\frac{1}{2}+o(1)\right)\left(E\left[Y_{m_n,j*}|{\mathit{Y}}_{i*,j*}^{\mathit{\text{diff}}}=A\right]+E\left[Y_{m_n,j*}|{\mathit{Y}}_{i*,j*}^{\mathit{\text{diff}}}=-A\right]\right),\text{\hspace{0.17em}by\hspace{0.17em}equation\hspace{0.17em}}(\text{A}7)\\ =\left(\frac{1}{2}+o(1)\right)\left(2Y_{\cdot j*}+E\left[Y_{m_n,j*}-Y_{\cdot j*}|{\mathit{Y}}_{i*,j*}^{\mathit{\text{diff}}}=\mathit{A}\right]+E\left[Y_{m_n,j*}-Y_{\cdot j*}|Y_{i*,j*}^{\mathit{\text{diff}}}=-A\right]\right)\\ =\left(\frac{1}{2}+o(1)\right)\left(2Y_{\cdot j*}+o(1)\right),\text{\hspace{0.17em}by\hspace{0.17em}equation\hspace{0.17em}}(\text{A}8)\\ =Y_{\cdot j*}+o(1),\text{\hspace{0.17em}by\hspace{0.17em}the\hspace{0.17em}properties\hspace{0.17em}of\hspace{0.17em}convergence.}\end{array}$$

And so for any difference matrix A for any n = 1, … ,n*:

$$E\left[Y_{m_n,j*}|{\mathit{Y}}_{i*,j*}^{\mathit{\text{diff}}}\in \{A,-A\}\right]=Y_{\cdot j*}+o(1).$$ (A9)

Following the steps in the proof of Theorem 1, and using the properties of convergence, then, for any n= 1, … ,n*:

$$E\left[Y_{m_n,j*}|v_{i*,j*}\right]=Y_{\cdot j*}+o(1),$$ (A10)

and thus, using that ${\text{lim}}_{K\to \infty }o(1)=0$:

$$\underset{K\to \infty }{\text{lim}}E\left[Z_{i*,j*}\right]=\underset{K\to \infty }{\text{lim}}E\left[\sum _{n=1}^{n*}{v}_{i*,j*,n}{Y}_{m_n,j*}\right]=\underset{K\to \infty }{\text{lim}}\left(Y_{\cdot j*}+o(1)\right)=Y_{\cdot j*}.$$ (A11)

Hence, for any cluster i*and period j* where X_i*,j*= 1, Z_i,j is an asymptotically unbiased estimate of Y_.j*.

Theorem 2. Assume that the assumptions of Theorem 1 are met and that for any intervention cluster-period (i,j), E[Yi,j]= Y_.j + β Then for the risk difference function, g(y_1,y₂) = y₁–y₂, the synthetic control estimator $\widehat{\beta }$ with weights w_i,j independent of the outcomes is an unbiased estimate of β

Proof. By Theorem 1, for any target cluster-period (i,j) such that X_i,j= 1, E[Z_i,j] = Y_.j (note that we have dropped the i*,j* notation for simplicity). Thus:

$$\begin{array}{l}E[\widehat{\beta }]=E\left[\sum _{(i,j):X_{i,j}=1}\frac{w_{i,j}}{{\sum }_{(i,j):X_{i,j}=1}{w}_{i,j}}{\widehat{\beta }}_{i,j}\right]=\sum _{(i,j):X_{ij}=1}E\left[\frac{w_{i,j}}{{\sum }_{(i,j):X_{i,j}=1}{w}_{i,j}}\left(Y_{i,j}-Z_{i,j}\right)\right]\\ =\sum _{(i,j):X_{ij}=1}\frac{w_{i,j}}{{\sum }_{(i,j):X_{i,j}=1}{w}_{i,j}}E\left[Y_{i,j}-Z_{i,j}\right],\text{\hspace{0.17em}since\hspace{0.17em}}{w}_{i,j}\perp ,Y_{i,j},Z_{i,j}\\ =\sum _{(i,j):X_{i,j}=1}\frac{w_{i,j}}{{\sum }_{(i,j):X_{i,j}=1}{w}_{i,j}}\left(Y_{\cdot j}+\beta -Y_{\cdot j}\right)=\frac{{\sum }_{(i,j):X_{i,j}=1}{w}_{i,j}}{{\sum }_{(i,j):X_{i,j}=1}{w}_{i,j}}\beta =\beta .\end{array}$$ (A12)

Corollary 2. Assume that the assumptions of Corollary 1 are met and that for any intervention cluster-period (i,j), E[Y_i,j]= Y_.j + β Then for the risk difference function, g(y_1,y₂) = y₁–y₂, the synthetic control estimator β with weights w_i,j independent of the outcomes is an asymptotically (with respect to the number of individuals in each cluster) unbiased estimate of β.

Proof. By Corollary 1, for any target cluster-period (i,j) such that X_i,j= 1, ${\text{lim}}_{K\to \infty }E\left[Z_{i,j}\right]=Y_{\cdot j}$ (again dropping the i*, j* notation). Thus:

$$\begin{array}{l}\underset{K\to \infty }{\text{lim}}E[\widehat{\beta }]=\underset{K\to \infty }{\text{lim}}E\left[\sum _{(i,j):X_{i,j}=1}\frac{w_{i,j}}{{\sum }_{(i,j):X_{i,j}=1}{w}_{i,j}}{\widehat{\beta }}_{i,j}\right]=\sum _{(i,j):X_{i,j}=1}\underset{K\to \infty }{\text{lim}}E\left[\frac{w_{i,j}}{{\sum }_{(i,j):X_{i,j}=1}{w}_{i,j}}\left(Y_{i,j}-Z_{i,j}\right)\right]\\ =\sum _{(i,j)}\sum _{:X_{i,j}=1}\frac{w_{i,j}}{{\sum }_{(i,j):X_{i,j}=1}{w}_{i,j}}\underset{K\to \infty }{\text{lim}}E\left[Y_{i,j}-Z_{i,j}\right],\text{ since }{w}_{i,j}\perp Y_{i,j},Z_{i,j},K\\ =\sum _{(i,j):X_{ij}=1}\frac{w_{i,j}}{{\sum }_{(i,j):X_{i,j}=1}{w}_{i,j}}\left(\underset{K\to \infty }{\text{lim}}E\left[Y_{i,j}\right]-Y_{.j}\right)=\sum _{(i,j):X_{i,j}=1}\frac{w_{i,j}}{{\sum }_{(i,j):X_{i,j}=1}{w}_{i,j}}\left(Y_{.j}+\beta -Y_{\cdot j}\right)\\ =\frac{{\sum }_{(i,j):X_{i,j}=1}{w}_{i,j}}{{\sum }_{(i,j):X_{i,j}=1}{w}_{i,j}}\beta =\beta .\end{array}$$ (A13)

Thus $X_{i*,j*}=1,Z_{i,j}$ is an asymptotically unbiased estimate of β.

Theorem 3. Assume that there is a constant risk difference β due to treatment across clusters and periods; that is, E[Y_i,j|Xi,j =1]=E[Yi,j | Xi,j = 0] + β for all i,j Then for any weights w_j that are independent of th e outcomes Y_i,j, the crossover estimators $\widehat{\beta }$ and $\tilde{\beta }$ using the risk difference function, g(y₁,y₂),= y₁ – y₂ are unbiased estimates of β. That is $E[\widehat{\beta }]=E[\tilde{\beta }]=\beta$

Proof. We denote by Y._j the expectation (marginal across clusters) of the outcome of any cluster on control in period j. By the assumptions, for all j > 1:

$$E\left[D_{i,j}|i\in I_{0,j}\right]=E\left[Y_{i,j}|X_{i,j}=0\right]-E\left[Y_{i,j-1}|X_{i,j-1}=0\right]=Y_{\cdot j}-Y_{\cdot j-1}.$$ (A14)

$$\begin{array}{l}E\left[D_{i,j}|i\in I_{1,j}\right]=E\left[Y_{i,j}|X_{i,j}=1\right]-E\left[Y_{i,j-1}|X_{i,j-1}=0\right]\\ =E\left[Y_{i,j}|X_{i,j}=1\right]-E\left[Y_{i,j}|X_{i,j}=0\right]+E\left[Y_{i,j}|X_{i,j}=0\right]-E\left[Y_{i,j-1}|X_{i,j-1}=0\right]\\ =\beta +Y_{\cdot j}-Y_{\cdot j-1}.\end{array}$$ (A15)

$$\begin{array}{l}E\left[D_{i,j}|i\in I_{2,j}\right]=E\left[Y_{i,j}|X_{i,j}=1\right]-E\left[Y_{i,j-1}|X_{i,j-1}=1\right]\\ =E\left[Y_{i,j}|X_{i,j}=1\right]-E\left[Y_{i,j}|X_{i,j}=0\right]+E\left[Y_{i,j}|X_{i,j}=0\right]-E\left[Y_{i,j-1}|X_{i,j-1}=0\right]+E\left[Y_{i,j-1}|X_{i,j-1}=0\right]-E\left[Y_{i,j-1}|X_{i,j-1}=1\right]\\ =\beta +Y_{\cdot j}-Y_{\cdot j-1}-\beta =Y_{\cdot j}-Y_{\cdot j-1}.\end{array}$$ (A16)

Define θ′_j = Y._j − Y._j-1 for all j > 1. Then:

$$E\left[{\widehat{\beta }}_j\right]=\sum _{i\in I_{1j}}\frac{1}{n_{1,j}}E\left[D_{i,j}|i\in I_{1,j}\right]-\sum _{i\in I_{0,j}}\frac{1}{n_{0,j}}E\left[D_{i,j}|i\in I_{0,j}\right]=\frac{n_{1,j}}{n_{1,j}}\left(\beta +{\theta }_j^{\prime }\right)-\frac{n_{0,j}}{n_{0,j}}\left({\theta }_j^{\prime }\right)=\beta .$$ (A17)

$$\begin{array}{l}E\left[{\tilde{\beta }}_j\right]=\sum _{i\in I_{1,j}}\frac{1}{n_{1,j}}E\left[D_{i,j}|i\in I_{1,j}\right]-\sum _{i\in I_{0,j}}\frac{1}{n_{0,j}+n_{2,j}}E\left[D_{i,j}|i\in I_{0,j}\right]-\sum _{i\in I_{2,j}}\frac{1}{n_{0,j}+n_{2,j}}E\left[D_{i,j}|i\in I_{2,j}\right]\\ =\frac{n_{1,j}}{n_{1,j}}\left(\beta +{\theta }_j^{\prime }\right)-\frac{n_{0,j}}{n_{0,j}+n_{2,j}}\left({\theta }_j^{\prime }\right)-\frac{n_{2,j}}{n_{0,j}+n_{2,j}}\left({\theta }_j^{\prime }\right)=\beta .\end{array}$$ (A18)

Now, for any weights w_jthat are independent of the outcomes:

$$\begin{array}{l}E[\widehat{\beta }]=E\left[\sum _{j>1:0<n_{0,j}{n}_{1,j}<I}\frac{w_j}{w}{\widehat{\beta }}_j\right]=\sum _{j>1:0<n_{0,j}{n}_{1,j}<I}\frac{w_j}{w}\beta ,\text{\hspace{0.17em}by\hspace{0.17em}equation\hspace{0.17em}}(\text{A}17)\\ =\frac{{\sum }_{j>1:0<n_{0,j},n_{1,j}<I}{w}_j}{w}\beta =\beta .\end{array}$$ (A19)

$$\begin{array}{l}E[\tilde{\beta }]=E\left[\sum _{j>1:0<n_{1,j}<I}\frac{w_j}{w}{\tilde{\beta }}_j\right]=\sum _{j>1:0<n_{1j}<I}\frac{w_j}{w}\beta ,\text{\hspace{0.17em}by\hspace{0.17em}equation\hspace{0.17em}(A18)}\\ =\frac{{\sum }_{j>1:0<n_{1,j}<I}{w}_j}{w}\beta =\beta .\end{array}$$ (A20)

So as desired, $E[\widehat{\beta }]=E[\tilde{\beta }]=\beta$

Corollary 3. Assume that there is a constant risk difference β due to treatment in the first period on treatment across clusters; that is E[Y_i,j| X_i,j = 1 ⋂ X_i,j-1 = 0] = E[Y_i,j| X_i,j = 0] +β for all i,j Then for any weights w,_j that are independent of the outcomes Y_i,j, the crossover estimator $\widehat{\beta }$ using the risk difference function, g(y₁,y₂),= y₁ – y₂, is an unbiased estimate of β. That is, $E[\widehat{\beta }]=\beta$

Proof. Again, we denote by Y._j the expectation (marginal across clusters) of the outcome of any cluster on control in period j. By the assumptions, for all j > 1:

$$E\left[D_{i,j}|i\in I_{0,j}\right]=E\left[Y_{i,j}|X_{i,j}=0\right]-E\left[Y_{i,j-1}|X_{i,j-1}=0\right]=Y_{\cdot j}-Y_{\cdot j-1}.$$ (A21)

$$\begin{array}{l}E\left[D_{i,j}|i\in I_{1,j}\right]=E\left[Y_{i,j}|X_{i,j}=1\right]-E\left[Y_{i,j-1}|X_{i,j-1}=0\right]\\ =E\left[Y_{i,j}|X_{i,j}=1\right]-E\left[Y_{i,j}|X_{i,j}=0\right]+E\left[Y_{i,j}|X_{i,j}=0\right]-E\left[Y_{i,j-1}|X_{i,j-1}=0\right]\\ =\beta +Y_{\cdot j}-Y_{\cdot j-1}.\end{array}$$ (A22)

Define θ′_j = Y._j Y._j-1 for all j > 1. Then:

$$\begin{array}{l}E\left[{\widehat{\beta }}_j\right]=\sum _{i\in I_{1,j}}\frac{1}{n_{1,j}}E\left[D_{i,j}|i\in I_{1,j}\right]-\sum _{i\in I_{0,j}}\frac{1}{n_{0,j}}E\left[D_{i,j}|i\in I_{0,j}\right]\\ =\frac{n_{1,j}}{n_{1,j}}\left(\beta +{\theta }_j^{\prime }\right)-\frac{n_{0,j}}{n_{0,j}}\left({\theta }_j^{\prime }\right)=\beta .\end{array}$$ (A23)

Now, for any weights w_j that are independent of the outcomes, by equation (A23):

$$E[\widehat{\beta }]=E\left[\sum _{j>1:0<n_{0,j},n_{1,j}<I}\frac{w_j}{w}{\widehat{\beta }}_j\right]=\sum _{j>1:0<n_{0,j},n_{1,j}<I}\frac{w_j}{w}\beta =\frac{{\sum }_{j>1:0<n_{0,j}{n}_{1,j}<I}{w}_j}{w}\beta =\beta .$$ (A24)

So as desired, $E[\widehat{\beta }]=\beta$.

Remark 1. Since $E[\widehat{\beta }]$ depends only on E[D_i,j| i ∈ I_1,j] and E[D_i,j| i ∈ I_0,j] it requires only the weaker assumption of Corollary 3 to be unbiased, while $E[\widehat{\beta }]$ requires the stronger assumption given in Theorem 3.

Remark 2. Specifically, equal weighting and the weights w_j and w′_j given in Section 2.4.4 are independent of the outcomes Y_i,j and thus result in unbiased estimates if the other conditions of Theorem 3 are met.