Sample size calculators for planning stepped-wedge cluster randomized trials: a review and comparison

Summary

Recent years have seen a surge of interest in stepped-wedge cluster randomized trials (SW-CRTs). SW-CRTs include several design variations and methodology is rapidly developing. Accordingly, a variety of power and sample size calculation software for SW-CRTs has been developed. However, each calculator may support only a selected set of design features and may not be appropriate for all scenarios. Currently, there is no resource to assist researchers in selecting the most appropriate calculator for planning their trials. In this paper, we review and classify 18 existing calculators that can be implemented in major platforms, such as R, SAS, Stata, Microsoft Excel, PASS and nQuery. After reviewing the main sample size considerations for SW-CRTs, we summarize the features supported by the available calculators, including the types of designs, outcomes, correlation structures and treatment effects; whether incomplete designs, cluster-size variation or secular trends are accommodated; and the analytical approach used. We then discuss in more detail four main calculators and identify their strengths and limitations. We illustrate how to use these four calculators to compute power for two real SW-CRTs with a continuous and binary outcome and compare the results. We show that the choice of calculator can make a substantial difference in the calculated power and explain these differences. Finally, we make recommendations for implementing sample size or power calculations using the available calculators. An R Shiny app is available for users to select the calculator that meets their requirements (https://douyang.shinyapps.io/swcrtcalculator/).

Key Messages.

• Stepped-wedge cluster randomized trials (SW-CRTs) are an attractive alternative to parallel-arm cluster randomized trials in some scenarios and have been increasing in popularity. A variety of calculators that can help researchers calculate power and sample size for SW-CRTs have been developed.

• Available calculators differ in the features they can support, including the types of designs, outcomes, correlation structures and treatment effects that can be specified; ability to accommodate incomplete designs, cluster-size variation or secular trends; and the type of analytical approach used; available calculators can also differ substantially in the calculated power in some scenarios.

• We summarize and compare the features of 18 available calculators to calculate power and sample size for SW-CRTs and describe a Shiny app to help researchers identify the calculators that support the desired features for a particular planned trial.

• We advise users on how to choose the most appropriate calculator based on their requirements and what to prioritize when no calculator can accommodate all their needs.

Introduction

In recent years, the stepped-wedge cluster randomized trial (SW-CRT) design has been increasing in popularity.^1–4 In SW-CRTs (Figure 1a), each cluster begins in the control condition. At different time points, clusters are randomized to cross over to the intervention condition until by the end of the trial, all clusters have received the intervention. In a complete SW-CRT design, measurements are taken from each cluster in each period. The SW-CRT can help facilitate recruitment when stakeholders perceive there are benefits to being exposed to the intervention at some point during the trial.^5,6 The design is also attractive when the goal is to implement the intervention in all clusters but there are inadequate resources to do so in all clusters simultaneously.^5,6 Despite their potential advantages, the use of stepped-wedge (SW) designs must be appropriately justified and researchers must be aware of the potential risks of bias.^7,8

Figure 1.

(a) Stepped-wedge design with five sequences and six periods (steps). (b) Stepped-wedge design with transition periods. (c) ‘Staircase’ stepped-wedge design.

Methods for planning SW-CRTs are rapidly evolving and a variety of sample size calculation methods and tools are now available. Applied statisticians are faced with a bewildering array of choices and there is no centralized discussion of the functionalities of different calculators. We aim to provide a comprehensive overview of the trial design features that can be accommodated by available power and sample size calculators for SW-CRTs, illustrate how to use the main calculators and provide a guide to allow users to select the most appropriate calculators for their studies.

The rest of the article is organized as follows. We start with an overview of SW-CRTs, including major features to consider in sample size calculation. We then provide a detailed review of four main calculators including their strengths and limitations. Using two example trials, we then illustrate how to use these calculators to obtain power for each of the trials and compare the results across the four calculators. We conclude by summarizing our findings and making recommendations for selecting the most appropriate calculator in practice.

Main factors to consider when planning a SW-CRT

Types of SW designs based on sampling structure

There are three main types of SW-CRTs, depending on the sampling structure.⁹ In a cross-sectional design, outcomes are measured on different participants over time so each participant contributes outcomes to either the control or intervention condition but not both. In the closed cohort design, the same participants are measured repeatedly over time so each participant contributes outcomes under both the control and intervention conditions. The open cohort is a variation on the closed cohort in which participants may join or leave the cohort during the trial and thus at least some participants contribute outcomes under both control and intervention conditions.¹⁰ The majority of work on sample size calculation for SW-CRTs has focused on cross-sectional or closed cohort designs; open cohort designs have received relatively little development except for the work by Kasza et al.¹⁰ Recent work has started to consider cross-sectional designs arising from the common scenario in which participants are recruited continuously in time (but exposed for only a short period) as a special class of design requiring different correlation structures, called ‘continuous-time decay’, and different analytical approaches.^9,11 Further work is required but some methodology for continuous-time decay has already been developed by Grantham et al.¹² that includes an R Shiny web app to determine power for some scenarios, assuming individual measurement times are evenly spaced within each interval. In practice, cross-sectional designs are often considered to be approximations of continuous recruitment short exposure designs and, depending on the assumed correlation structure, sample size calculators based on discrete sampling may yield identical calculations.

Incomplete designs

Standard SW-CRTs collect data in each cluster period. Incomplete SW-CRTs are designs that do not collect data in all cluster periods.^13,14Figure 1b illustrates a design in which no data are collected during the period immediately prior to the intervention. These ‘transition periods’ allow extra time to implement the intervention or for the intervention to start exerting its effect. Each sequence can have one or more transition periods.¹⁴Figure 1c illustrates a design in which each cluster has the same number of control and intervention periods, but not all clusters enter and exit at the same time. This type of design is called a staircase design.¹⁵

Correlation structures and statistical models for SW-CRTs

In a SW-CRT, the intervention effect is partially confounded with the time effect (i.e. secular trend) by design; therefore, the analytical model must always account for time to obtain unbiased inferences. It is further well known that CRTs must account for the correlation among multiple responses from different individuals within the same cluster. In a post-test-only parallel-arm CRT, the degree of clustering is usually measured using a single intra-cluster correlation coefficient (ICC), defined as the ratio of the between-cluster variance and the sum of the between-cluster and within-cluster variances.¹⁶ In multi-period CRTs, such as SW-CRTs, correlation structures may be more complex as outcomes are measured in different periods. Nevertheless, the initial statistical model for SW-CRTs developed by Hussey and Hughes was a random intercepts model that assumed a single exchangeable correlation, i.e. the correlation between any two individuals from the same cluster is the same regardless of their distance apart in time.²

Building upon earlier work on multi-period parallel-arm CRTs,¹⁷ Hooper et al.¹⁸ and separately Girling and Hemming¹⁹ introduced a conditional (i.e. mixed-effects) model (referred to as the Hooper/Girling model) that allows the within-period ICC (wp-ICC) (i.e. the correlation between two individuals from the same cluster and same period) to differ from the between-period ICC (bp-ICC) (i.e. the correlation between two individuals from the same cluster but different periods). We would typically expect the bp-ICC to be less than the wp-ICC. The ratio of the bp-ICC and wp-ICC is called the Cluster Autocorrelation Coefficient (CAC), which is typically <1.^18,20 Li²¹ referred to this as a nested exchangeable correlation structure. In a cohort design, an additional correlation is needed to account for repeated measures on the same individual over time. In the Hooper/Girling model, this correlation was introduced through the addition of a patient-level random effect and quantified as an Individual Autocorrelation Coefficient (IAC), which implies a constant correlation in the repeated measures on the same individual over time.¹⁸ Li²¹ referred to this as a block exchangeable correlation structure.

Intuitively, within a cluster, the responses from individuals between two consecutive periods are usually more correlated than those separated by a longer interval. Therefore, assuming a constant bp-ICC may not be realistic, Kasza et al.²² proposed an extension to the Hooper/Girling model that allows the bp-ICC to decay exponentially. In this case, the CAC refers to the rate of exponential decay per period. In the case of a cohort design, individual-level correlation is also required and there are two possibilities: either a constant individual-level correlation over time (specified through an IAC) or allowing individual-level correlations to decay exponentially (referred to as the proportional decay structure).²¹

As an alternative to mixed-effects models, Li et al.²³ proposed a marginal model based on generalized estimating equations (GEE) and corresponding power and sample size calculation methodology allowing for both cross-sectional and closed cohort SW-CRT designs. In a cohort setting, it is worth noting that in a marginal model, the correlation in repeated measures on the same individual is modelled directly through the within-individual ICC (wi-ICC) and is not identical to the IAC referred to above in the context of a mixed-effects model. A comprehensive overview of statistical models available for SW-CRTs can be found in Li et al.²⁴

In addition to the above correlation structures that are based on time (period), correlation structures can also be allowed to depend on exposure. This is referred to as cluster-treatment effect heterogeneity and is accomplished by including a random cluster by treatment effect in the analytical model.^25,26

Different ways to specify treatment effects

The most common way to model a treatment effect in a SW-CRT is as a time-averaged effect (step/immediate change),² which implies an immediate effect that is sustained. With this specification, the average response across the intervention periods is compared with that across the control periods (Figure 2a). In some situations, the intervention will not be fully implemented in the period immediately after crossing over or may have a delayed effect. This can be handled by specifying a fractional treatment indicator.^2,25 For example, using prior knowledge, one may specify that the treatment will only be 50% effective during the first intervention period and code the treatment indicator as 0.5 in that period. When the treatment effect is anticipated to be gradual across multiple periods, one can assign fractional values to consecutive periods (assuming the fractions are known) (Figure 2b). More generally, rather than estimating a single treatment effect, one may allow the treatment effect to vary in an unrestricted way according to the number of periods of exposure to the intervention. This is referred to as general time-on-treatment effects^24,25; it requires that an adequate number of clusters are available given the desired number of treatment effects. A more parsimonious version is to assume a constant (linear) change with increasing duration of exposure (called a linear time-on-treatment effect).²⁴ Linear time-on-treatment can also be viewed as a special case of fractional treatment indicators but where the fractions are equally spaced and determined by the maximum exposure time (Figure 2c).

Figure 2.

(a) Design matrix for a stepped-wedge design with time-averaged treatment effect. (b) Design matrix for a stepped-wedge design with delayed/gradual treatment effect and known fractions. (c) Design matrix for a stepped-wedge design with linear time-on-treatment effects (treatment increases linearly with duration of exposure).

Analytical approximation for non-continuous outcomes

For non-continuous (binary and count) outcomes, a common approach to sample size calculation is to use the normal approximation.^2,27 However, this approach has several issues. First, it requires the specification of an absolute difference even if relative measures are of interest. Second, the variance of the estimated treatment effect for non-continuous outcomes theoretically depends on the assumed treatment effect as well as on the size of any time effects, whereas under the normal approximation it depends on neither. Several recent methods have shifted away from normal approximation approaches towards methods based on maximum-likelihood estimation (MLE)²⁸ of the generalized linear mixed-model (GLMM) or marginal models using GEE.^23,29 These methods allow users to specify odds ratios and risk ratios directly in their sample size calculation procedures. Moreover, they allow the variance to depend on the assumed treatment effect and thus require the effect of secular trends to be incorporated in the power calculation. Although the analytical model should always include time effects to account for potential secular trends, power calculations for continuous outcomes are not affected by the magnitude of the secular trends.

Table 1 summarizes these and other features that researchers need to consider when designing a SW-CRT.

Table 1.

Common features that users need to consider or specify when designing a stepped-wedge cluster randomized trial

Variable	Definition or explanation
Type of primary outcome	Continuous, binary, count
Details of the design
Number of sequences	Number of rows in design matrix
Number of periods	Determines number of columns in design matrix (when time is modelled as a categorical variable)
Cluster size per period	Average number of participants per cluster per period
Type of design
Cross-sectional	Different individuals in each period
Closed cohort	The same individuals repeatedly measured over time
Open cohort	Mixture of different or same individuals
Correlation structures
Cross-sectional designs
Exchangeable	Assume a constant within-cluster correlation over time
Nested exchangeable	Allow the between-period ICC to be different from the within-period ICC (often specified using the CAC, which is the ratio of the between-period to the within-period ICC)
Exponential decay	Specify a within-period ICC and a CAC measuring the exponential rate of decay in the between-period ICC per period
Cohort designs
Exchangeable	As for cross-sectional design; in addition, assume a constant within-individual correlation over time
Block exchangeable	As for cross-sectional design; in addition, assume a constant within-individual correlation over time
Proportional decay	As for cross-sectional design; in addition, specify an exponential decay in the within-individual correlation over time
Type of treatment effect
Average (immediate) effect	Intervention leads to step change (immediate change)
Delayed or gradual effect	Allows gradual treatment effect or partial exposure to the intervention
Linear time-on-treatment effect	Allows the treatment to increase linearly with duration of exposure
General time-on-treatment effect	Allows the treatment effect to vary in an unrestricted way, according to the number of periods of exposure to the intervention
Link function (for binary outcomes)	Identity (Risk Difference), Log (Relative Risk), Logit (Odds Ratio)
Equal or Unequal allocation	Same or different number of clusters per sequence; calculators that allow users to upload a user-defined design matrix may allow users to explore power implications of different allocations
Varying or fixed cluster sizes	Allowing cluster (or cluster-period) sizes to vary across clusters can affect power. Often specified in terms of a coefficient of cluster-size variation
Incomplete design	Data are not collected from some clusters in some periods either to allow for a transition period or to ease the data collection burden (see Figure 1). Calculators that allow users to upload or define their own design matrix provide more flexibility over specifying incomplete designs
Analytic model	Mixed-effects regression or marginal model (generalized estimating equations)
Different methods to specify correlation	Different calculators may require different inputs to measure correlation. The most common approach is to specify anticipated intra-cluster correlation coefficients. An alternative is to specify the anticipated variance or standard deviation of the random-effects distributions. Finally, the coefficient of outcome variation can sometimes be specified
Cluster-treatment heterogeneity	Users may allow treatment effects to vary randomly across clusters (which can reduce power)
Multilevel clustering	More than two levels of clustering during each time period, e.g. patients may be clustered within providers who are clustered within primary care practices
Hybrid design	The design contains a mixture of SW-CRT and parallel CRT (see Girling et al.19 for details)
Continuous or categorical time effects	Users may model secular trends using fixed categorical or fixed continuous effects. Whether linear or categorical time effects are incorporated in complete stepped-wedge designs has no influence on continuous outcomes30
Magnitude of time effect	Users may specify the expected secular trend, which, in the case of a non-continuous outcome, can affect power

ICC, intra-cluster correlation coefficient; CAC, Cluster Autocorrelation Coefficient; SW-CRT, stepped-wedge cluster randomized trial; CRT, cluster randomized trial.

Summary of available sample size and power calculators

To date, 18 power and sample size calculators for SW-CRTs have been developed for implementation in R, online via an R Shiny app, or in SAS, Stata, Excel, PASS³¹ and nQuery.³² A standalone calculator maintained by the National Institutes of Health has also recently become available.³³ A summary of the functionality of the calculators with respect to the major features described in Table 1 is presented in Supplementary File 1, Tables S1–S6 (available as Supplementary data at IJE online) organized by software platform. Each calculator is also classified based on how correlation structures need to be specified (i.e. in terms of coefficient of variation (CV) of outcomes, ICCs or variance components of random effects) and whether the calculator allows for unequal cluster-period sizes, unequal allocation of clusters to sequences and whether the user can specify their own design matrix. Note that any calculator that can support a user-specified design matrix can typically also be used for other multiple-period designs (such as repeated parallel-arm or multiple-period cluster crossovers) as the sample size methodology underlying all these designs is similar. We also indicate whether cluster-treatment effect heterogeneity, continuous time effects (as opposed to only categorical), subclusters (i.e. more than two levels of clusters within each period) and hybrid designs (i.e. a mixture of parallel and SW-CRTs) can be accommodated.¹⁹ A Shiny app is available (https://douyang.shinyapps.io/swcrtcalculator/) in which users can obtain a list of calculators that accommodate their selected design features. Among available calculators, we now review in more detail four main packages that offer many of the available features.

SHINYCRT by Hemming et al.

Overview

• A menu-based web app (RShiny app) that calculates power and sample size for SW-CRTs under linear mixed-effects models.²⁷

• Type of design and outcome: cross-sectional and closed cohort designs with continuous, binary and count outcomes using the normal approximation method.

• Correlation structures: exchangeable (users need to specify a common wp-ICC); block or nested exchangeable (specify the wp-ICC and CAC as well as an IAC for cohort designs); exponential decay (specify the wp-ICC and CAC for the decay per period, as well as a constant IAC for cohort designs). Exponential decay is referred to as ‘discrete time decay’ in this app.

Important features and strengths

SHINYCRT automatically displays power or sample size curves. Users can either provide sample sizes per cluster period to calculate the power or specify a target power to obtain the required sample sizes. In addition to the specified ‘base’ ICC values, users can conduct a sensitivity analysis by specifying the upper and lower bounds of the wp-ICC. The plot will automatically include results for lower and higher CAC values (80% and 120% of the specified CAC) as a sensitivity analysis.

Users can upload their own design matrices to accommodate unequal numbers of clusters per sequence. Uploading a design matrix also activates additional features including an option for accommodating cluster-treatment heterogeneity. The design matrix may contain empty cells to allow for incomplete designs or cells with fractions to accommodate delayed or linear time-on-treatment effects (although the design matrix will be displayed as integers). This calculator can also accommodate unequal cluster sizes by specifying a CV for cluster-size variation. Instead of a z-test, users can request a t-test with degrees of freedom equal to the number of cluster periods minus the number of periods minus one, although the t-test is not available for exponential decay or when uploading a user-specified design matrix. An attractive feature of this app is that in addition to SW-CRT, menu options include other multiple-period designs, e.g. parallel-arm and multiple-period cluster crossover designs.

Limitations

The SHINYCRT web app has multiple features and allows substantial flexibility in specifying a design, but also has some limitations. First, methods for binary and count outcomes use a normal approximation; thus, the magnitude of time effects on power cannot be accommodated. Second, only identity link functions are allowed; thus, when the research requires log or logit links, the sample size calculation procedure may not be compatible with the analytical approach. Third, in the case of closed cohort designs, the app only allows constant individual-level correlation, even though proportional decay may be more appropriate. Fourth, the t-distribution approximation was still under development at the time of submission. Finally, the RShiny app platform limits the number of hours the app can run on the server. When the quota has been reached in a particular month, users will have to find an alternative way to access the calculator.

SWDPWR by Chen et al.

Overview

• A calculator running in R, RShiny and SAS that calculates power and sample size for SW-CRTs using GEE or GLMM with MLE.³⁴

• Type of design and outcome: cross-sectional and closed cohort designs with continuous and binary outcomes under identity, log and logit link functions.

• Correlation structures: block or nested exchangeable correlation structures (including exchangeable as a special case); wp-ICC and bp-ICC must be specified for cross-sectional designs (specified as equal to obtain exchangeable correlation); for cohort designs, wi-ICC must also be specified.

Important features and strengths

The most significant feature of this calculator is the availability of GEE and MLE to obtain more accurate sample size estimates for binary outcomes. When the research requires relative measures, the implementation of log and logit link functions allows users to specify relative treatment effects directly. When assuming no secular trend, users can specify the target difference by either (i) specifying the expected response in the control condition at the beginning of the study and the expected response in the intervention condition at the end or (ii) replacing the expected response in the intervention condition at the end of the study with the effect size. To accommodate a secular trend, users can specify the expected response in the control condition at the beginning and end of the study (i.e. assuming the intervention has not been introduced) in addition to the target difference. Users can upload a design matrix and accommodate hybrid designs and designs with different numbers of clusters per sequence.

Limitations

The SWDPWR calculator has some limitations. First, it does not allow for exponential or proportional decay and cannot accommodate count outcomes. Second, with the conditional model, a maximum cluster-period size of 150 is allowed for binary outcomes and the calculator is further limited to cross-sectional designs with exchangeable correlation. If larger cluster-period sizes, cohort designs or more complex correlation structures are desired, GEE must be used instead. Third, when specifying no time effects, i.e. the same expected response in the control condition at the beginning and end, the time variable will be removed from the underlying model used for calculation, which is incompatible with recommendations to always include time effects in the analytical model. Fourth, there is currently no option for choosing a t-test when the number of clusters is small. Finally, incomplete designs (including transition periods), cluster-treatment heterogeneity, unequal cluster sizes and delayed or time-on-treatment effects are not available.

SWCRTDESIGN by Hughes et al.

Overview

• A calculator running in R and RShiny app that calculates power and sample size for SW-CRTs under a linear mixed-effects model.³⁵

• Type of design and outcome: cross-sectional designs with continuous and binary outcomes under an identity link function. For non-continuous outcomes, a new method that includes log and logit link functions and that can provide more accurate estimates than MLE even with a small number of clusters is being planned as a future addition.³⁶

• Correlation structure: exchangeable and nested exchangeable correlation structures. Users can specify the wp-ICC and CAC or standard deviations for random intercepts, random treatment effects across clusters and random cluster by time effects; the correlation between random intercepts and random treatment effects can also be specified (most other calculators assume independence).

Important features and strengths

The most important contribution of this package is to allow the intervention effect to vary across clusters by including a random treatment effect.²⁵ To activate this feature, users must input standard deviations of the anticipated random effects (rather than ICCs). In the accompanying Shiny app, delayed or linear time-on-treatment effects can be accommodated by allowing fractional treatment indicators; however, general time-on-treatment effects are not available. Designs must be specified via the function ‘swDns’. Users can specify the number of clusters changing from control to intervention at each step. An incomplete design can be constructed by specifying a cluster-period size of 0 in select periods.

Limitations

SWCRTDESIGN has several limitations. First, calculations for binary outcomes are only available using the normal approximation and for count outcomes only via simulation. Second, it cannot be used for cohort designs. Third, exponential or proportional decay cannot be accommodated. Fourth, there is no option for choosing a t-distribution when the number of clusters is small. Fifth, compared with other packages, specifying incomplete designs is less straightforward: a complete design matrix must be specified using ‘swDns’ followed by specification of cluster-period sizes. Finally, delayed treatment effects are only available in the Shiny app version so far, but relevant code for simulation-based power calculation can be performed by modifying the code provided in Voldal et al.³⁵

CRTFASTGEEPWR by Zhang and Preisser

Overview

• An SAS macro that calculates power for SW-CRT based on GEE analysis of a marginal model.^37,38

• Type of design and outcome: cross-sectional or closed cohort designs with continuous, binary and count outcomes and identity, log and logit link functions.

• Correlation structure: nested or block exchangeable (including exchangeable as a special case) and exponential or proportional decay. Under proportional decay, the calculator requires the bp-ICC and wi-ICC to decay at the same rate.

Important features and strengths

Users can specify either time-averaged or ‘incremental’ intervention effects, the latter using fractional treatment values for linear time-on-treatment effects. Additionally, time effects can be specified as either categorical or linear (continuous). Both intervention and time effects are specified as regression parameters on the scale of the link function. The model with categorical time effects implies an expected proportion/rate under the control condition for each time period, whereas specification of linear time effects requires the user to specify both a fixed intercept and slope that determine marginal means for the outcome under the control condition.

Users are required to enter the number of clusters per sequence, which may vary. A required design pattern matrix for sequence-periods uses 0 for control periods, 1 for intervention periods and 2 for empty cells (for incomplete designs). Users may determine power for the normal approximation or from a t-distribution with choice of degrees of freedom (the number of clusters minus two or the number of clusters minus the number of parameters in the model). Unequal cluster-period sizes can be accommodated; however, users must enter the cluster-period sizes rather than a single CV.

Limitations

There are few limitations to this calculator. First, cluster-treatment effect heterogeneity cannot be accommodated. Second, for binary and count outcomes, the magnitude of secular trends is relevant but the calculator only considers these parameters on the link function scale, so the user will need to use inverse link functions to convert proportions and rates in case of a non-identity link.²³

Other calculators

The above four calculators may cover most scenarios of interest; features of other available calculators are summarized in the tables and available in our app and only briefly highlighted here. For Stata users, POWERSWGEE³⁹ might offer the most flexibility and has similar functionalities as CRTFASTGEEPWR. Notably, when specifying the proportional decay structure, it allows bp-ICC and individual-level correlation to decay at different rates. However, POWERSWGEE currently cannot accommodate incomplete designs. A few calculators in our review accommodate special features, e.g. designs with subclusters⁴⁰; a calculator is also available to determine allocation-based power.^41,42 The choice of calculators that can deal with open cohort designs is limited^10,43; however, a reviewer pointed out that by multiplying IAC by one minus the churn rate (the rate that participants leave the trial), calculators that support closed cohort designs can also be used for open cohort designs. This observation will hold for continuous outcomes under a constant individual-level correlation. Conversely, calculators for open cohort designs can also support closed cohort and cross-sectional designs by setting the churn rate to 0 or 1.

During peer review, a new standalone calculator (SWGRT³³) became available and was added to our manuscript. SWGRT can support continuous and binary outcomes, all three types of designs and available correlation structures, and is accompanied by excellent documentation. A notable feature of this calculator is that it can incorporate the effect of covariate adjustments in the design. It also allows an ICC obtained from a block exchangeable model to be converted to a discrete time decay ICC.¹⁰ However, it is based on the normal approximation for binary outcomes and cannot deal with incomplete designs, unequal allocation and user-specified design matrices. Finally, it only allows the calculation of the required number of clusters per sequence for a pre-specified treatment effect and a minimum of two clusters per sequence is required.

Examples

In this section, we illustrate power calculations using each of the four packages for two published trials: Girls on the Go!⁴⁴ and the Washington State Expedited Partner Therapy (EPT) trial.⁴⁵ We show how the sample size parameters should be specified in each package and compare the obtained power values across the four packages. We make comparisons under exchangeable, nested or block exchangeable and exponential or proportional decay structures. We were able to obtain realistic parameter values under these correlation structures using real data from both trials. The code used to implement the calculations is presented in Sections 1 and 2 of Supplementary File 2 (available as Supplementary data at IJE online) or RPubs (https://rpubs.com/derek6561/swcrtcalculator).

‘Girls on the Go!’ trial

‘Girls on the Go!’ is a closed cohort SW-CRT designed to determine whether a 10-week programme that incorporates interactive and practical learning would help to increase self-esteem, self-confidence and body image among school girls aged 10–16 years in Melbourne, Victoria, Australia.⁴⁴ The trial was designed with three sequences and four periods with two clusters per sequence (a total of six clusters) and 10 participants per cluster, i.e. a total sample size of 60 participants. There were no transition periods. The primary outcome was continuous: a measure of self-esteem on a scale from 0 to 40. The sample size parameter values including ICC estimates under the three different correlation structures from the CLOUDbank data set, which is a repository of ICC values under different correlation structures, are summarized in Table 2.⁴⁶ Due to the small number of clusters, we illustrate power with both a z-test and a t-test. SWCRTDESIGN was excluded from this example as it cannot be used for cohort designs. Power calculations when using SWDPWR and CRTFASTGEEPWR are shown both with and without time effects. (This does not contradict an earlier statement that the magnitude of time effects is irrelevant for continuous outcomes but was included to illustrate how SWDPWR handles time effects compared with other packages.)

Table 2.

Sample size calculation parameter values used for the ‘Girls on the Go’ trial (continuous outcome)

Sample size parameter	Assumed value
Number of sequences	3
Number of periods	4
Number of clusters (equal allocation)	6
Cluster size per period	10
Standard deviation	1
Mean difference (treatment effect)	0.39
Type of design	Closed cohort
Exchangeable correlation
Within-period ICC	0.005
IAC or within-individual ICCa	0.65
Block exchangeable correlation
Within-period ICC	0.01
CAC	0.223
IAC or within-individual ICCa	0.65
Proportional decay
Within-period ICC	0.025
CAC (per period)	0.689
IAC or within-individual ICCa	0.65
Varying cluster sizes	No
Transition periods	No
Type of model	Linear mixed-effects model or GEE
Type of treatment effect	Step change (time-averaged effect)
Cluster-treatment heterogeneity	No
Type of time effect	Fixed categorical
Magnitude of time effects	Not applicableb
Significance level	Two-sided, 5%

ICC, intra-cluster correlation coefficient; IAC, Individual Autocorrelation Coefficient; CAC, Cluster Autocorrelation Coefficient; GEE, generalized estimating equations.

^aSHINYCRT requires IAC; SWDPWR and CRTFASTGEEPWR require within-individual ICC. We used the same value for both parameters.

^b Assumed 0.1 in SWDPWR where specification of time effects is necessary to avoid inflating power; changing the value of assumed time effects will not change power as the outcome is continuous

The power when using the three main calculators under these assumptions is presented in Table 3. Not surprisingly, the power did not differ much whether GLMM or GEE was used. Whereas power was substantially different under different correlation structures, power was similar across the different calculators when using the normal distribution. Under the no-time-effect scenario however, SWDPWR yields substantially higher power due to fixed time effect parameters being excluded from the model, leaving fewer fixed parameters to estimate. Therefore, to approximate the results of CRTFASTGEEPWR assuming a null secular trend (where the analytical model includes time effects), users must specify a small secular trend in SWDPWR. As expected, the magnitude of time effects in the linear model is irrelevant to CRTFASTGEEPWR. Note that power from CRTFASTGEEPWR is based on an exponentially decaying wi-ICC, whereas SHINYCRT assumes a constant IAC; nevertheless, the power obtained was comparable. When using the t-distribution, calculated power in SHINYCRT and CRTFASTGEEPWR were substantially different (76% vs 50%) due to the degrees of freedom implemented being different (19 in SHINYCRT vs 4 in CRTFASTGEEPWR, i.e. the number of clusters minus two).

Table 3.

Power calculated under the main software packages for the ‘Girls on the Go’ trial

	SHINYCRTb	SWDPWRc	SWDPWRc	CRTFASTGEEPWRd	CRTFASTGEEPWRd
		No time effecte	With time effect (0.1)	No time effect	With time effect (0.1)
Power using z-testa
Exchangeable correlation	0.796	0.997	0.794	0.794	0.794
Block exchangeable correlation	0.720	0.991	0.724	0.724	0.724
Proportional decay	0.713 (exponential decay with fixed IAC)	N/A	N/A	0.708 (the same exponential decay in between-period and within-individual ICC)	0.708 (the same exponential decay in between-period and within-individual ICC)
Power using t-testf
	DF = 19g			DF = 4
Exchangeable correlation	0.751	N/A	N/A	0.501	0.501
Block exchangeable correlation	0.671	N/A	N/A	0.418	0.418
Proportional decay	N/A	N/A	N/A	0.401	0.401

IAC, Individual Autocorrelation Coefficient; ICC, intra-cluster correlation coefficient; DF, degrees of freedom.

^aSWCRTDESIGN does not support cohort designs.

^bLinear mixed-effects model used.

^cGeneralized estimating equations (GEE) and mixed-effects model produced the same results.

^dGEE method for marginal model.

^eBy default, coefficients for time are excluded from the underlying model when no time effect is specified.

^fSHINYCRT and CRTFASTGEEPWR used different DF in t-test.

^gThis is a manual correction for the DF used in SHINYCRT based on its documentation; the actual implementation is under development.

Community EPT trial

The Washington State EPT trial is a cross-sectional SW-CRT designed to evaluate the impact of promoting the use of EPT on the decrease of Chlamydia test positivity among women.⁴⁵ The trial was designed with four sequences and five periods. Each sequence has 6, 6, 6 and 4 clusters, respectively (a total of 22 clusters). The average cluster-period size was 305 participants, i.e. a total sample size of 33 550 participants. There were no transition periods. The primary outcome was binary: prevalence of Chlamydia positivity. The sample size parameter values including ICC estimates under the three different correlation structures, from reanalysis of the real EPT trial data,^47,48 are summarized in Table 4. Since the magnitude of any assumed time effects will impact the power for binary outcomes, we assessed power assuming (i) no time effect, (ii) a small time effect equal to half the size of the treatment effect and (iii) a moderate time effect equal to the treatment effect.

Table 4.

Sample size calculation parameter values used for the Expedited Partner Therapy trial (binary outcome)

Sample size parameter	Assumed value
Number of sequences	4
Number of periods	5
Number of clusters	22 (6, 6, 6, 4 for each sequence)
Cluster size per period	305
Baseline rate	0.076
Mean (absolute) difference (treatment effect)	−0.014
Type of design	Cross-sectional
Exchangeable correlation
Within-period ICC	0.007
Nested exchangeable correlation
Within-period ICC	0.007
CAC	0.50
Exponential decay
Within-period ICC	0.007
CAC (per period)	0.70
Varying cluster sizes	No
Transition periods	No
Type of model	Linear mixed-effects regression or GEE
Type of treatment effect	Step change (time-averaged effect)
Cluster-treatment heterogeneity	No
Type of time effect	Fixed categorical
Magnitude of time effect	None, small (−0.007) or large (−0.014)
	A time effect of, say, −0.007 means that the outcome is expected to decrease, even in the absence of any intervention, from 0.076 in the first period to 0.069 in the last period
Significance level	Two-sided, 5%

ICC, intra-cluster correlation coefficient; CAC, Cluster Autocorrelation Coefficient; GEE, generalized estimating equations.

The power when using the four main calculators is presented in Table 5. The power was dramatically different under different assumed correlation structures. The power did not appear to differ based on whether GLMM or GEE was used, but when assuming no time effects, SWDPWR gave substantially different results to the other packages, as explained earlier. When the time effects were not null, the power from SWDPWR and CRTFASTGEEPWR was similar but substantially different from the power under the normal approximation in SHINYCRT and SWCRTDESIGN (since the normal approximation does not rely on the magnitude of time effects). The differences tend to increase when the time effects increase.

Table 5.

Power calculated under the main software packages for the Expedited Partner Therapy trial

	SHINYCRTa	SWDPWRb			SWCRTDESIGNa	CRTFASTGEEPWRb
		No time effectc	Small time effect (−0.007)	Large time effect (−0.014)		No time effect	Small time effect (−0.007)	Large time effect (−0.014)
Exchangeable correlation	0.806	0.995	0.819	0.837	0.803	0.803	0.819	0.837
Nested exchangeable correlation	0.543	0.904	0.560	0.580	0.540	0.542	0.560	0.580
Exponential decay	0.570	N/A	N/A	N/A	N/A	0.568	0.586	0.607

^aNormal approximation using mixed-effects model. Magnitude of time effects cannot be accommodated.

^bGeneralized estimating equations (GEE) was used for calculation. With cluster-period size of >150, only GEE can be used in SWDPWR.

^cBy default, coefficients for time are excluded from the underlying model when no time effect is specified.

The logit link can also be specified for the mean model in SWDPWR and CRTFASTGEEPWR (see Supplementary File 2, Section 3, available as Supplementary data at IJE online). Both calculators require users to specify the effect size on the log scale and the ICC of the binary outcomes on the proportions scale. Whereas SWDPWR requires the time effect to be specified as proportions, CRTFASTGEEPWR requires all marginal mean model parameters (i.e. period and intervention effects) to be specified on the link function scale.

Summary and recommendations

There are several key considerations when designing a SW-CRT and choosing a power and sample size calculator. For a continuous or non-continuous outcome when the normal approximation is appropriate (i.e. where there is a relatively large number of clusters and/or the anticipated proportion is not too close to 0 or 1) and when anticipated secular trends are small, SHINYCRT is potentially the best choice for most users. It is free to access and provides an intuitive interface. It can accommodate continuous, binary and count outcomes as well as cross-sectional and closed cohort designs under the three types of correlation structures. It allows for incomplete designs and cluster-treatment heterogeneity. Furthermore, it automatically implements a sensitivity analysis for the assumed correlation parameters. However, for non-continuous outcomes in which relative effects are of interest or when extreme baseline rates/proportions or large time effects are expected, CRTFASTGEEPWR seems to be the better choice. This SAS macro allows for cross-sectional and closed cohort designs, incorporates all correlation structures considered herein and can accommodate incomplete designs, fractional treatment effects and both categorical and linear time effects. However, non-SAS users or users who do not want to use GEE may use SWDPWR instead as long as they have a complete design with a continuous or binary outcome (exchangeable correlation only) and do not require decaying correlation structures or delayed treatment effects. When using SWDPWR, users should incorporate non-zero time effects (even for continuous outcomes), otherwise the calculation will be based on a model that excludes the time effect coefficients and this could yield spuriously high power.

Although users may select the calculator most convenient to them facilitated by our Shiny app, a good rule of thumb when not all requirements can be met is to prioritize those factors that have the most impact on power in their scenario. As our examples illustrate, an important consideration is the assumed correlation structure: assuming an overly simple correlation structure (e.g. exchangeable) when a more complex correlation structure (e.g. block exchangeable or exponential decay) is appropriate can lead to power being overestimated.⁴⁹ Ideally, ICC values should be informed by analysing existing data (e.g. historical or routinely collected data) under a similar setting over a similar study duration. A practical approach is to select the best-fitting correlation structure based on information criteria, i.e. AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion).⁵⁰ Regardless of the availability of prior data, it is always good practice to check the sensitivity of the calculations by assuming a range of plausible ICC values. When there are no prior data to inform the choice of correlation structure, we would recommend using a nested or block exchangeable correlation for binary outcomes and exponential or proportional decay for continuous outcomes. For non-continuous outcomes, the type of approximation used is also critical. For non-continuous outcomes, users must consider whether strong time effects are anticipated and choose a calculator that can accommodate such effects. As shown in our example, with binary outcomes, the normal approximation works poorly if the baseline proportion is away from 0.5 and the time effect is not null. A similar observation was made by Zhou et al.²⁸ Finally, for continuous outcomes with a small number of clusters, using the t-test is generally advised, but users must be aware that different degree-of-freedom options are implemented in different calculators and the best choice for degree of freedom remains unclear.

Ethics approval

Not applicable.

Data availability

There are no new data associated with this article.

Supplementary data

Supplementary data are available at IJE online.

Author contributions

Y.O. led the writing of the manuscript, developed the idea with M.T., extracted the software information, created tables and figures, and developed the Shiny app. M.T. conceived of the ideas, helped with data extraction and co-led the writing. F.L. participated in the development of the idea and helped with data extraction. J.S.P. critically reviewed the methods and ideas. All authors contributed to writing, drafting and editing the manuscript.

Funding

M.T. and F.L. are supported by the National Institute of Aging (NIA) of the National Institutes of Health (NIH) under Award Number U54AG063546, which funds NIA Imbedded Pragmatic Alzheimer's Disease and AD-Related Dementias Clinical Trials Collaboratory (NIA IMPACT Collaboratory). Y.O. is funded by the Ontario SPOR SUPPORT Unit, which is supported by the Canadian Institutes of Health Research and the Province of Ontario. J.S.P. and F.L. are supported through Patient-Centered Outcomes Research Institute^® (PCORI^® Award ME-2019C1-16196). The statements presented in this article are solely the responsibility of the authors and do not necessarily represent the views of the NIH nor PCORI^®, its Board of Governors or Methodology Committee.