Rank-based estimators of global treatment effects for cluster randomized trials with multiple endpoints of different scales

Abstract

Cluster randomized trials commonly employ multiple endpoints. When a single summary of treatment effects across endpoints is of primary interest, global methods represent a common analysis strategy. However, specification of the required joint distribution is non-trivial, particularly when the endpoints have different scales. We develop rank-based interval estimators for a global treatment effect referred to here as the “global win probability,” or the mean of multiple Wilcoxon Mann-Whitney probabilities, and interpreted as the probability that a treatment individual responds better than a control individual on average. Using endpoint-specific ranks among the combined sample and within each arm, each individual-level observation is converted to a “win fraction” which quantifies the proportion of wins experienced over every observation in the comparison arm. An individual’s multiple observations are then replaced with a single “global win fraction” by averaging win fractions across endpoints. A linear mixed model is applied directly to the global win fractions to obtain point, variance, and interval estimates adjusted for clustering. Simulation demonstrates our approach performs well concerning confidence interval coverage and type I error, and methods are easily implemented using standard software. A case study using public data is provided with corresponding R and SAS code.

1. Introduction

Complex diseases with complex interventions demand complex trials. Cluster randomized trials, or cluster trials, can simplify the delivery of complex interventions and mitigate the risk of contamination by randomly allocating groups of individuals, or “clusters,” rather than individuals to intervention arms. However, their design and analysis is complicated by correlation among individuals within the same cluster, or “intracluster correlation.”¹ Correlation structures are complicated even further when multiple endpoints are of primary interest. In addition to multiple sources of intracluster correlation which may differ by endpoint, endpoint correlations within-subject must also be captured. Parametric methods exist to model multiple endpoints, but are limited to the same scale, e.g., all binary² or all continuous with normality assumed³.

In this paper, we focus on the scenario where a single summary of the treatment effect across multiple endpoints of different scales, or a “global treatment effect,” is of primary interest. This scenario may arise when there is a lack of consensus on which single best endpoint to use, or when multiple endpoints are necessary to characterize disease burden at a single time point, risk-benefit trade-off, or even longitudinal disease course.^4,5 Global tests represent a common analytic approach in this scenario, and assess overarching hypotheses about all endpoints simultaneously using a single test statistic constructed from their joint distribution. By testing endpoints jointly rather than separately, multiplicity is not a concern, and power is often greater than the multiple corresponding univariate tests.⁶

Motivated by a need to obtain a single probability statement regarding multiple disparate endpoints among few individually randomized subjects, O’Brien⁷ developed three 1-df global tests of a directed alternative: the ordinary least squares (OLS) test, the generalized least squares (GLS) test, and the nonparametric rank-sum test. All three tests construct the global treatment effect as the sum (or mean) of the endpoint-specific effects, but differ in their standardization and weighting schemes. Each of the tests is actually equivalent to: (1) a univariate test of the global treatment effect; (2) a combination of the univariate, endpoint-specific test statistics;⁸ and notably, (3) a two-sample t-test of a composite endpoint constructed as the within-subject mean of standardized responses.⁹ Thus, if an appropriate standardization can be identified and the corresponding composite is relevant, univariate tests may be applied directly, avoiding explicit specification of complex correlation structures.

The “parametric” OLS and GLS tests apply Z-standardization to each endpoint under the assumption that they are multivariate normal but this assumption is often untenable, particularly as multiple endpoints are commonly ordinal or differ in scale.¹⁰ O’Brien’s “nonparametric” rank-sum test instead replaces responses with their rank in the pooled sample.⁷ Ranks are then summed across endpoints within-subject, yielding composite “rank-sums.” For large sample sizes, correlation between rank-sums is sufficiently weak to permit application of the Central Limit Theorem. Thus, the nonparametric rank-sum test is constructed as a t-test for the mean difference in rank-sums, here the global treatment effect. Of the three tests, O’Brien promoted the rank-sum test for general use, as it suffers from little loss in efficiency when parametric assumptions are met and provides gains in power otherwise.^7,11 While developed with one-sided alternatives in mind, the nonparametric rank-sum test is also applicable to two-sided alternatives.^12,13

The nonparametric rank-sum test suffers from three main drawbacks. First, hypothesis tests and p-values, while informative in their own right, are no longer sufficient. Trials are encouraged to report point estimates and their uncertainty, preferably in the form of confidence intervals.¹⁴ Second, the mean difference in rank-sums is not an interpretable measure of group differences. As we demonstrate in what follows, it may be re-expressed in terms of “the win probability.” The win probability is a nonparametric treatment effect defined as the probability that a randomly selected treatment individual responds better than, or “wins over,” a randomly selected control individual. Some readers may recognize this quantity as the Wilcoxon Mann-Whitney statistic. We use this terminology to align with related “win statistics” currently gaining traction among trialists, though we do not adopt “prioritized evaluation.”^4,15,16 Third, there are currently no feasible extensions to the cluster randomized setting. Zou & Zou demonstrated that the family of O’Brien-Wei-Lachin methods could be constructed using win fraction methods in individually randomized trials.¹⁷

Zou¹⁸ developed rank-based confidence interval estimators for a single win probability within cluster randomized trials by transforming individual-level responses into “win fractions.” Win fractions are proportions that summarize the wins experienced by an individual when compared to all others within the comparator arm. For example, a win fraction of 0.75 suggests that a (treatment) individual responded better than, or “won over,” 75% of those in the comparator (control) arm. Zou¹⁸ demonstrated that unbiased and consistent estimators of the win probability could be recovered from a linear mixed model for the win fractions with a random cluster intercept. Simulation studies demonstrated that resulting confidence intervals perform well, maintaining nominal coverage probabilities and type I error rates. Perhaps most importantly, these methods can be easily implemented using standard software and place win-based treatment effects within a regression framework, permitting easy adjustment for baseline responses for example.¹⁹

This paper presents an easily implemented, widely applicable, and interpretable solution to the otherwise complex analysis of cluster randomized trials with multiple endpoints of ordinal or different scales by building upon the ideas of O’Brien’s nonparametric rank-sum test⁷, win fraction methods for individually randomized trials,^17,19–21 and the mixed model estimators of Zou¹⁸. Using ranks, a “global win fraction” is first constructed for each individual within the cluster trial as the within-subject mean of their endpoint-specific win fractions. A univariate linear mixed model is then applied directly to the global win fractions to obtain point and variance estimates for the global win probability adjusted for clustering effects. While hypothesis testing methods are provided, emphasis is placed on confidence interval estimation. The developed methods are simple yet flexible enough to handle multiple endpoints with differing properties such as type, scale, or priority, avoid explicit specification of complex correlation structures, can be implemented using existing software, yield meaningful treatment effects, and have the potential to increase power – a particular concern of cluster trials.

The rest of the paper is organized as follows. Section 2 defines notation, introduces the global win probability for multiple endpoints, and demonstrates translation to alternative win measures and the mean difference in rank-sums. In Section 3, simulation studies demonstrate the ability of the developed methods to maintain confidence interval coverage and the type I error rate while providing high power. In Section 4, a case study using publicly available data from the SHARE²² cluster trial exemplifies the application of the global win probability to two endpoints with different types and priority. Corresponding SAS and R code is provided in the Appendix to assist with implementation. The paper closes with a discussion, including a sketch of how sample size may be calculated.

2. Methods

2.1. Notation, context, and assumptions

Consider a parallel two-arm cluster randomized trial allocating $C_0$ and $C_1$ clusters to the control ($i=0$) and treatment ($i=1$) arms, respectively. Let $c=1,2,\dots ,C_i$ index the clusters randomized to the $i^{th}$ intervention arm and $j=1,2,\dots ,n_{ic}$ index the $n_{ic}$ individuals within the $c^{th}$ cluster in the $i^{th}$ arm. All $n_{ic}$ individuals receive the intervention allocated to their cluster. Then, $C=C_0+C_1$ is the total number of clusters randomized by the trial, $N_i=\sum _{c=1}^{C_i}{n}_{ic}$ is the total number of individuals within the $i^{th}$ intervention arm, and $N=N_0+N_1$ is the total number of individuals within the trial. Due to randomization, clusters are independent within and between arms.

Suppose $k=1,2,\dots ,K$ endpoints are recorded for each individual. Let $X_{icjk}$ denote the $k^{th}$ response of the $j^{th}$ individual within the $c^{th}$ cluster in the $i^{th}$ arm. Assume all individuals are completely observed so that ($N\times K$) total responses are recorded, and that cluster sizes are non-informative, so that responses within a cluster are independent of its size. The $X_{icjk}$ are not independent within the $c^{th}$ cluster in the $i^{th}$ arm, but are assumed to be identically distributed according to non-degenerate distribution function $F_{ik}$. To accommodate discrete endpoints, $F_{ik}$ is defined as the “normalized distribution function,” $F_{ik}(x)=0.5\left[F_{ik}(x{)}^{-}+F_{ik}^{+}(x)\right]$ where $F_{ik}^{-}(x)=\mathrm{P}\mathrm{r}\left(X_{icjk}<x\right)$ and $F_{ik}^{+}(x)=\mathrm{P}\mathrm{r}\left(X_{icjk}\le x\right)$ are the left- and right-continuous distribution functions, respectively. Finally, assume endpoints are at least ordinal in nature so any two responses can be ordered, and without loss of generalizability, greater responses correspond to better health.

2.2. Global win probability, and related measures

Following Zou,¹⁸ we consider the individual-level win probability for a single endpoint, hereon referred to simply as the win probability, which compares individual-level responses between arms rather than cluster-level summaries. For the $k^{th}$ endpoint, the win probability takes the form,

$${\theta }_k=\mathrm{P}\mathrm{r}(\underset{\mathrm{“}\mathrm{W}\mathrm{i}\mathrm{n}\mathrm{”}}{\underbrace{X_{1cjk}>X_{0cjk}}})+0.5\mathrm{P}\mathrm{r}(\underset{\mathrm{“}\mathrm{T}\mathrm{i}\mathrm{e}\mathrm{”}}{\underbrace{X_{1cjk}=X_{0cjk}}}).$$ (1)

The win probability is rooted in the Wilcoxon rank-sum or Mann-Whitney U test statistics and has had over a dozen unique names including the area under the receiver operating characteristic curve, concordance or the c-index, the probabilistic index, and the common language effect size.²⁰ It also shares relationships with commonly encountered effect measures. Of particular note, when the $k^{th}$ endpoint is normally distributed, ${\theta }_k$ is a one-to-one function of the standardized mean difference or Cohen’s effect size $\delta$, with ${\theta }_k=\mathrm{\Phi }(\delta /\sqrt{2})$ where $\mathrm{\Phi }(\cdot )$ is the standard normal distribution function. Cohen’s qualitative benchmarks are then easily transferable to the win probability, with ${\theta }_k=0.5,0.56,0.64,\text{and}0.71$ corresponding to null, small, moderate, and large effects, respectively.²³

When $K$ endpoints are of joint interest, we propose that the average of the $K$ corresponding win probabilities serves as the global treatment effect. Formally, the “global win probability” is defined as,

$$\theta =\frac{\sum _{k=1}^K{\omega }_k{\theta }_k}{\sum _{k=1}^K{\omega }_k}$$ (2)

where ${\omega }_k$ represents the optional, fixed contribution weight of the $k^{th}$ endpoint. Weights need not be specified to use our method, and in what follows, we focus primarily on equal contribution weights, ${\omega }_k=1/K$ for all $k$, so $\theta =\sum _{k=1}^K{\theta }_k/K$. The global win probability may be formally interpreted as the probability that a randomly selected individual from a treatment cluster responds no worse than a randomly selected individual from a control cluster with respect to the $K$ endpoints, on average.

2.2.1. Global win difference

Rather than the global win probability, one may consider the global win difference. For a single endpoint, the win difference has also been referred to as Somer’s $d$, the Mann-Whitney difference,²⁴ and the proportion in favor of treatment.²⁵ The win difference is commonly used for methodology development as it avoids explicit consideration of tie probabilities, $\mathrm{P}\mathrm{r}\left(X_{1cjk}=\right.\left.X_{0cjk}\right)$. For example, the global win difference was employed by Huang et al when developing improvements of the nonparametric rank-sum test,^12,13 and by Lachin when developing multivariate distribution-free hypothesis testing methods.²⁴

First, note that the $k^{th}$ win probability presented within Equation (1) is defined with respect to a treatment win, $\left(X_{1cjk}>X_{0cjk}\right)$. The probabilities of a treatment win, loss, and tie sum to unity,

$$\mathrm{P}\mathrm{r}\left(X_{1cjk}>X_{0cjk}\right)+\mathrm{P}\mathrm{r}\left(X_{1cjk}<X_{0cjk}\right)+\mathrm{P}\mathrm{r}\left(X_{1cjk}=X_{0cjk}\right)=1.$$ (3)

Since the win probability distributes ties equally between arms and a loss for treatment is a win for control, $\left(X_{0cjk}>X_{1cjk}\right)$, it follows from Equation (3) that the $k^{th}$ win probability defined with respect to a control win, or the “control win probability,” is the complement of the treatment win probability, $1-{\theta }_k$.

The $k^{th}$ win difference is defined as the difference between the treatment and control win probabilities,

$${\mathrm{\Delta }}_k=\mathrm{P}\mathrm{r}\left(X_{1cjk}>X_{0cjk}\right)-\mathrm{P}\mathrm{r}\left(X_{1cjk}<X_{0cjk}\right)$$ (4)

or equivalently, ${\mathrm{\Delta }}_k={\theta }_k-\left(1-{\theta }_k\right)$. ${\mathrm{\Delta }}_k$ represents a generalization of the risk difference for binary endpoints to all endpoint types, and ranges between −1 and +1 with ${\mathrm{\Delta }}_k=0$ suggesting no treatment effect and ${\mathrm{\Delta }}_k>0$ treatment benefit relative to the control. With $K$ endpoints, the “global win difference” may be obtained in a similar fashion to Equation (2) as $\mathrm{\Delta }=\sum _k{\omega }_k{\mathrm{\Delta }}_k/\sum _k{\omega }_k$. Thus, estimators provided in Sections 2.4 and 2.5 may be translated to the global win difference by applying the transformations $\stackrel{\mathrm{ˆ}}{\mathrm{\Delta }}=2\hat{\theta }-1$ and $\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}(\stackrel{\mathrm{ˆ}}{\mathrm{\Delta }})=4\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}(\hat{\theta })$.

2.2.2. Global win odds

Agresti proposed the generalized odds ratio, commonly referred to as “Agresti’s $\alpha$,” as a treatment effect for ordinal endpoints.²⁶ For the $k^{th}$ endpoint, ${\alpha }_k$ is equal to the ratio of the probability of a treatment win to a control win,

$${\alpha }_k=\frac{\mathrm{P}\mathrm{r}\left(X_{1cjk}>X_{0cjk}\right)}{\mathrm{P}\mathrm{r}\left(X_{1cjk}<X_{0cjk}\right)}$$

and is equivalent to the odds ratio when the $k^{th}$ endpoint is binary. Pocock et al popularized the use of Agresti’s $\alpha$ as a measure of effect size for prioritized time-to-event composites within cardiovascular trials,²⁷ referring to it as the “win ratio.” For a single continuous survival endpoint, ${\alpha }_k$ is equivalent to the inverse of the hazard ratio when the proportional hazards assumption holds. However, ${\alpha }_k$ excludes the probability of ties which can be both informative and substantial for discrete endpoints.

Dong et al²⁸ and Brunner et al²⁹ instead advocate for the use of the “win odds” which incorporates ties and is equal to the ratio of the treatment and control win probabilities. For the $k^{th}$ endpoint,

$${\lambda }_k=\frac{\mathrm{P}\mathrm{r}\left(X_{1cjk}>X_{0cjk}\right)+0.5\mathrm{P}\mathrm{r}\left(X_{1cjk}=X_{0cjk}\right)}{\mathrm{P}\mathrm{r}\left(X_{1cjk}<X_{0cjk}\right)+0.5\mathrm{P}\mathrm{r}\left(X_{1cjk}=X_{0cjk}\right)}$$

or equivalently, ${\lambda }_k={\theta }_k/\left(1-{\theta }_k\right)$. When the $k^{th}$ endpoint is continuous, $\mathrm{P}\mathrm{r}\left(X_{1cjk}=\right.\left.X_{0cjk}\right)=0$ so that ${\lambda }_k={\alpha }_k$. With $K$ endpoints, we define the “global win odds” as $\lambda =\theta /(1-\theta )$ where $\theta$ is defined per (2). $\lambda$ ranges between 0 and $+\mathrm{\infty }$ with $\lambda =1$ indicating no treatment effect and $\lambda >1$ treatment benefit relative to the control. Estimators provided in Sections 2.4 and 2.5 may be translated to the global win odds through application of the $\delta$-method, with $\hat{\lambda }=\hat{\theta }/(1-\hat{\theta })$ and $\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}(\mathrm{l}\mathrm{n}\hat{\lambda })=\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}(\hat{\theta })/[\hat{\theta }(1-\hat{\theta }){]}^2$.

2.3. Global win fractions

Point and variance estimation for associated measures such as the Wilcoxon Mann-Whitney U-test statistic or win difference have traditionally relied on the construction and comparison of all $N_1{N}_0$ pairs consisting of one treatment and one control response.^25,30 However, the construction of all “pairwise comparisons” is computationally expensive, even for moderate sample sizes.³¹ We focus instead on novel estimators which transform each individual’s observed response into a proportion summarizing their wins and ties experienced, referred to as a “win fraction.”^17–21 In Appendix A, we show that win fractions are asymptotically uncorrelated between arm and cluster, permitting the application of conventional methods, though not within cluster, requiring adjustment for intracluster correlation.

Formally, the $k^{th}$ win fraction for the $j^{th}$ individual in the $c^{th}$ treatment cluster is provided by,

$$Y_{1cjk}=\frac{1}{N_0}\sum _{c^{\prime }=1}^{C_0}\sum _{j^{\prime }=1}^{n_{0c^{\prime }}}H\left(X_{1cjk}-X_{0c^{\prime }{j}^{\prime }k}\right)$$ (5)

where $H\left(X_{1cjk}-X_{0c^{\prime }{j}^{\prime }k}\right)$ is the Heaviside function taking the value +1 when $\left(X_{1cjk}>X_{0c^{\prime }{j}^{\prime }k}\right)$ or a “win” occurs, +0.5 when $\left(X_{1cjk}=X_{0c^{\prime }{j}^{\prime }k}\right)$ or a “tie” occurs, and +0 when $\left(X_{1cjk}<X_{0c^{\prime }{j}^{\prime }k}\right)$ or a “loss” occurs. In other words, win fractions are equal to the within-subject mean of the ($N-N_i$) pairwise comparisons involving $X_{icjk}$. Of particular note, $Y_{1cjk}$ may also be expressed as ${\hat{F}}_{0k}\left(X_{1cjk}\right)$, where ${\hat{F}}_{ik}(x)$ is the empirical cumulative distribution function (ECDF) of the $k^{th}$ endpoint in the $i^{th}$ arm. That is, $Y_{1cjk}$ is the percentile that treatment observation $X_{1cjk}$ occupies among all control observations.¹⁸ Similarly, for the $j^{th}$ individual in the $c^{th}$ control cluster,

$$Y_{0cjk}=\frac{1}{N_1}\sum _{c^{\prime }=1}^{C_1}\sum _{j^{\prime }=1}^{n_{1c^{\prime }}}H\left(X_{0cjk}-X_{1c^{\prime }{j}^{\prime }k}\right)$$ (6)

or $Y_{0cjk}={\hat{F}}_1\left(X_{0cjk}\right)$, the percentile that $X_{0cjk}$ occupies among all treatment observations.

As a result of this relationship with the ECDF, win fractions may be conveniently expressed in terms of ranks. With ties, “ranks” refers to “midranks” or the average of the tied positions. Following Hoeffding,³² define two ranks for individual-level treatment response $X_{1cjk}$: (1) the “overall rank” among all $N$ responses in the trial,

$$R_{1cjk}=0.5+\sum _{c^{\prime }=1}^{C_1}\sum _{j^{\prime }=1}^{n_{1c^{\prime }}}H\left(X_{1cjk}-X_{1c^{\prime }{j}^{\prime }k}\right)+\sum _{c^{\prime }=1}^{C_0}\sum _{j^{\prime }=1}^{n_{0c^{\prime }}}H\left(X_{1cjk}-X_{0c^{\prime }{j}^{\prime }k}\right)$$

or equivalently, $R_{1cjk}=0.5+N_1{\hat{F}}_1\left(X_{1cjk}\right)+N_0{\hat{F}}_0\left(X_{1cjk}\right)$, and (2) the “group-specific rank” among the $N_1$ responses in the treatment arm,

$$G_{1cjk}=0.5+\sum _{c^{\prime }=1}^{C_1}\sum _{j^{\prime }=1}^{n_{1c}}H\left(X_{1cjk}-X_{1c^{\prime }{j}^{\prime }k}\right)$$

or equivalently, $G_{1cjk}=0.5+N_1{\hat{F}}_1\left(X_{1cjk}\right)$. These ranks can be defined analogously for control observations. It then follows from Equations (5) and (6) that the win fractions may be expressed generally as,

$$Y_{icjk}=\frac{R_{icjk}-G_{icjk}}{N-N_i}.$$ (7)

The rank-based form of the win fractions simplifies and speeds up calculation significantly as all $N_1{N}_0$ pairwise comparisons do not need to be constructed, rather only three sets of ranks.³¹

To estimate the global win probability, a single “global win fraction” is constructed for each individual as the (weighted) mean of their $K$ endpoint-specific win fractions,

$${\bar{Y}}_{icj.}=\frac{\sum _{k=1}^K{\omega }_k{Y}_{icjk}}{\sum _{k=1}^K{\omega }_k}.$$ (8)

With equal contribution weights, Equation (8) reduces to ${\bar{Y}}_{icj.}=\sum _{k=1}^K{Y}_{icjk}/K$, or the simple within-subject mean of the $K$ win fractions.

2.4. Mixed model estimators

Zou¹⁸ demonstrated that an unbiased estimator of a single win probability ${\theta }_k$ can be obtained as the mean treatment win fraction, ${\hat{\theta }}_k={\bar{Y}}_{1\cdot \cdot k}$, or since a win for control is a loss for treatment, one minus the mean control win fraction, ${\hat{\theta }}_k=1-{\bar{Y}}_{0\cdot \cdot k}$. It was also proved that asymptotically, ${\hat{\theta }}_k~𝒩({\theta }_k,\mathrm{V}\mathrm{a}\mathrm{r}({\hat{\theta }}_k))$ where $\mathrm{V}\mathrm{a}\mathrm{r}({\hat{\theta }}_k)\approx \mathrm{V}\mathrm{a}\mathrm{r}\left({\bar{Y}}_{1\cdot \cdot k}\right)+\mathrm{V}\mathrm{a}\mathrm{r}\left({\bar{Y}}_{0\cdot \cdot k}\right)$. Two estimators of the ${\bar{Y}}_{i\cdot \cdot k}$ and a total of three corresponding variance estimators, $\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}\left({\bar{Y}}_{i\cdot \cdot k}\right)$, were investigated.

Here, we construct a single transformed response for each individual within the cluster randomized trial, i.e., a global win fraction. In what follows, we obtain point and variance estimators of the global win probability by applying these univariate methods, the mixed model estimators specifically, directly to the global win fractions.

2.4.1. Application to global win fractions

An unbiased estimator of the global win probability $\theta$ and a consistent estimator of its variance can be obtained by applying the following linear mixed model to the global win fractions,

$${\bar{Y}}_{icj.}={\beta }_0+{\beta }_1{\mathrm{A}\mathrm{r}\mathrm{m}}_i+{\alpha }_{ic}+{\epsilon }_{icj}$$ (9)

where ${\mathrm{A}\mathrm{r}\mathrm{m}}_i=\mathbb{I}(i=1)$ is an indicator equal to 1 if individual $icj$ belongs to a treatment cluster and 0 if control, ${\alpha }_{ic}\stackrel{i.i.d.}{~}𝒩\left(0,{\sigma }_{\alpha }^2\right)$ represents the random intercept of the $i{c}^{th}$ cluster, ${\epsilon }_{icj}\stackrel{i.i.d.}{~}𝒩\left(0,{\sigma }_{\epsilon }^2\right)$ represents the residual of the $ic{j}^{th}$ win fraction, and ${\alpha }_{ic}$ and ${\epsilon }_{icj}$ are assumed to be independent.

From these definitions it follows that ${\hat{\beta }}_1=\left({\bar{Y}}_{1\dots }-{\bar{Y}}_{0\dots }\right)=2\hat{\theta }-1$ and an estimator of the global win probability can be recovered from the fitted model as $\hat{\theta }=0.5({\hat{\beta }}_1+1)$. Since the two intervention arms are independent, $\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}({\hat{\beta }}_1)=\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}\left({\bar{Y}}_1\dots \right)+\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}\left({\bar{Y}}_0\dots \right)$ and thus $\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}(\hat{\theta })=\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}({\hat{\beta }}_1)$. An estimate of the intracluster correlation of the global win fractions is also be obtained as $\hat{\rho }={\hat{\sigma }}_{\alpha }^2/\left({\hat{\sigma }}_{\alpha }^2+{\hat{\sigma }}_{\epsilon }^2\right)$. Coefficient estimators take the form of feasible generalized least squares (FGLS) estimators, which are well-known to be unbiased and asymptotically normal.³³ Regardless of alignment between (9) and the underlying data generation process, ${\hat{\beta }}_1$ is always consistent for the mean difference or average treatment effect, and under equal allocation, the model-based variance is always asymptotically correct.³⁴

2.4.2. Relationship with two-sample $\mathit{U}$-statistics

When all clusters feature the same number of individuals so that $n_{ic}=n$ for all $i$ and $c$, and endpoints are equally weighted so ${\omega }_k=1/K$, the mixed model estimator reduces to the simple mean of the global win fractions in the $i^{th}$ arm,

$${\bar{Y}}_{i\dots }=\frac{1}{N_i}\sum _{c=1}^{C_i}\sum _{j=1}^n{\bar{Y}}_{icj.}.$$ (10)

Expansion of Equation (10) according to the global win fraction definitions within Section 2.3 provides,

$$\hat{\theta }=\frac{1}{K}\sum _{k=1}^K\left\{\frac{1}{N_1{N}_0}\sum _{c=1}^{C_1}\sum _{c^{\prime }=1}^{C_0}\sum _{j=1}^{n_{1c}}\sum _{j^{\prime }=1}^{n_{0c^{\prime }}}H\left(X_{1cjk}-X_{0c^{\prime }{j}^{\prime }k}\right)\right\}$$ (11)

suggesting that $\hat{\theta }$ is equivalent to the simple mean of $K$ clustered, two-sample $\mathrm{U}$-statistics when individuals are equally weighted. Comparing the form of the treatment win fraction in Equation (5) to Equation (11), it becomes clear that each treatment win fraction can be conceptualized as a two-sample U-statistic where $N_1=1$. The same is true for control win fractions with $N_0=1$.

Obuchowski³⁵ developed the same estimator (11) for a single endpoint ($K=1$) by extending DeLong et al’s AUC estimators,³⁶ or equivalently Sen’s structural component estimators,³⁷ to the clustered setting. Brunner et al³⁸ established the unbiasedness, consistency, and multivariate normality of multiple win probabilities with independent observations, and Rubarth et al³⁹ established a similar result for clustered factorial designs. From these results, in addition to those of Zou,¹⁸ it follows for cluster randomized trials that the global win probability, equal to the mean of the multiple win probabilities, is asymptotically $\hat{\theta }~𝒩(\theta ,\mathrm{V}\mathrm{a}\mathrm{r}(\hat{\theta }))$.

2.4.3. Equivalence to mean difference in rank-sums

Consider the simplified scenario from the previous subsection. From the rank-based form of the win fractions presented previously in Equation (7) and the fact that the sum of the group-specific ranks $\sum _c\sum _j{G}_{1cjk}=N_1\left(N_1+1\right)/2$, with some algebra it follows that,

$$\hat{\theta }=\frac{1}{N_0}\left[\frac{{\bar{R}}_{1\cdots }}{K}-\frac{\left(N_1+1\right)}{2}\right]$$

where $R_{1cj}.=\sum _k{R}_{1cjk}$ is the rank-sum for the $c{j}^{th}$ treatment individual and ${\bar{R}}_{1\dots }=\sum _c\sum _j{R}_{1cj}/N_1$ is the mean treatment rank-sum. Similarly for the control arm,

$$1-\hat{\theta }=\frac{1}{N_1}\left[\frac{{\bar{R}}_{0\cdots }}{K}-\frac{\left(N_0+1\right)}{2}\right].$$

Thus, the mean difference in rank-sums is a linear transformation of the global win probability estimator with

$${\bar{R}}_{1\dots }-{\bar{R}}_{0\dots }=NK(\hat{\theta }-0.5)$$

and $\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}\left({\bar{R}}_{1\dots }-{\bar{R}}_{0\dots }\right)=N^2{K}^2\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}(\hat{\theta })$.

2.5. Interval estimators and hypothesis tests

A large-sample $(1-\alpha )\times 100\mathrm{\%}$ confidence interval for the global win probability is provided by,

$$\hat{\theta }\mp z_{\alpha /2}\sqrt{\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}(\hat{\theta })}$$

where $z_{\alpha /2}$ is the upper $\alpha /2$ quantile of $𝒩(\mathrm{0,1})$. For smaller samples, $z_{\alpha /2}$ may be substituted with the corresponding Student’s $t$ critical value with df degrees of freedom, $t_{\alpha /2,\mathrm{d}\mathrm{f}}$. Commonly, $\mathrm{d}\mathrm{f}=C-2$ where $C$ is the total number of clusters, but there is no unique way of specifying the degrees of freedom of a mixed model.¹ The corresponding test of the null hypothesis of no global treatment effect, $H_0:\theta =0.5$, can be assessed using the test statistic,

$$T=\frac{\hat{\theta }-0.5}{\sqrt{\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}(\hat{\theta })}}$$

which is distributed according to $𝒩(\mathrm{0,1})$ for large samples or approximately $t_{\mathrm{d}\mathrm{f}}$ for small samples. Using results from Section 2.4.3, the nonparametric rank-sum test is analogous to a test of the global win probability with $H_0:\theta =0.5$ as,

$$T=\frac{\hat{\theta }-0.5}{\sqrt{\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}(\hat{\theta })}}=\frac{{\bar{R}}_{1\dots }-{\bar{R}}_0\dots }{\sqrt{\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}\left({\bar{R}}_{1\dots }-{\bar{R}}_{0\dots }\right)}}.$$

A logit transformation may also be applied to improve behaviour for a small number of clusters or extreme values of $\hat{\theta }$. The lower and upper bounds of the large-sample $(1-\alpha )\times 100\mathrm{\%}$ logit interval, $\left[L_{\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}},U_{\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}}\right]$, are obtained respectively as,

$$L_{\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}}=\frac{\mathrm{e}\mathrm{x}\mathrm{p}(\ell )}{1+\mathrm{e}\mathrm{x}\mathrm{p}(\ell )},U_{\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}}=\frac{\mathrm{e}\mathrm{x}\mathrm{p}(u)}{1+\mathrm{e}\mathrm{x}\mathrm{p}(u)}$$

where

$$\ell ,u=\mathrm{l}\mathrm{n}\frac{\hat{\theta }}{1-\hat{\theta }}\mp z_{\alpha /2}\frac{\sqrt{\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}(\hat{\theta })}}{\hat{\theta }(1-\hat{\theta })}.$$

The null hypothesis of no treatment effect, $H_0:\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}(\theta )=0$, can also be assessed using the test statistic,

$$T_{\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}}=\frac{\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}(\hat{\theta })}{\hat{\theta }(1-\hat{\theta })/\sqrt{\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}(\hat{\theta })}}$$

which is distributed according to $𝒩(0,1)$ for large samples.

3. Simulation studies

3.1. Objectives and evaluation criteria

Our simulation study assesses the performance of the proposed interval estimators for the global win probability and their corresponding hypothesis tests of $H_0:\theta =0.5$ across a range of cluster trial designs. Performance metrics of interest are the coverage probability, balance of left and right tail error rates, type I error rates, and power. Emphasis is placed on the evaluation of interval estimators as their performance provides a combined summary of the quality of the proposed point and variance estimators.

Empirical coverage probability (ECP) is estimated as the proportion of confidence intervals containing the true global win probability $\theta$, while left (right) tail error rates are estimated as the proportion of lower (upper) bounds greater than (less than) $\theta$. The tail error ratio (TER), defined as the ratio of the left tail error rate to the right, is reported with a TER of 1 suggesting balance. Empirical type I error rates and power are estimated as the proportion of confidence intervals excluding the null value of 0.5, or the empirical rejection rate (ERR), when $\theta$ is null and non-null, respectively.

Reported ECPs are considered acceptable if they fall within approximately two standard errors of the specified nominal rate. Specifically, 95% confidence intervals are desired, and each scenario is replicated $B=\mathrm{5,000}$ times so that the acceptable range of the empirical coverage probability is $0.95\mp 1.96\sqrt{0.95(0.05)/5000}$ or 94.4% to 95.6%. The corresponding acceptable range for the empirical type I error rate is 0.044 to 0.056.

3.2. Scenarios and data generation

Our simulation study employed a factorial design, evaluating 1,536 scenarios in total. R code for this simulation study, and results for all scenarios, can be publicly accessed at https://github.com/statisticelle/sim-gwinp.

We consider both balanced and unbalanced allocation, so that the proportion of clusters allocated to treatment (versus control) is either $r=0.5$ or $r=0.7$, and $C=10,20,30and50$ total clusters.

Sample size parameters were selected to correspond to the number of clusters randomized and number of individuals per cluster reported by several reviews of cluster trials. Among cluster trials in primary care, Eldridge et al reported a median of 34 clusters randomized with a median cluster size of 32 individuals (IQR = 9 to 82).⁴⁰ Kahan et al reported a median of 25 clusters (IQR = 15 to 44) with a median cluster size of 34 individuals (IQR = 14 to 94).⁴¹ Ivers et al reported a median of 21 clusters (IQR = 12 to 52) with a median cluster size of 34 individuals (IQR = 13 to 89).⁴² Since analytic results in Appendix A suggest win fractions are uncorrelated between cluster and arm for large $N_0,N_1$, we consider $n_{ic}=10\text{and}30$.

As discussed in Section 2.4.2, the mixed model point estimator is equivalent to the unweighted mean of the global win fractions when cluster sizes are equal or fixed. Thus, two scenarios are considered: (i) a reference case with fixed cluster sizes so that $n_{ic}=n=10$ or 30 for all $i$ and $c$; and (ii) a realistic case with random cluster sizes, with an average of approximately $\bar{n}=10$ or 30 individuals per cluster. Random cluster sizes were generated according to a truncated log-normal distribution, with a lower bound of 3 and an upper bound of $4\bar{n}$. The mean of this distribution was selected to approximate $\bar{n}$, and variance chosen to approximate a coefficient of variation equal to 0.65, based on cluster trials of UK general practices.⁴³

Our simulation study considers only $K=2$ endpoints. Since our methods are particularly advantageous for ordinal endpoints lacking meaningful units or endpoints with different scales,²¹ we let $X_{0cj1}~\mathrm{B}\mathrm{i}\mathrm{n}\mathrm{o}\mathrm{m}\mathrm{i}\mathrm{a}\mathrm{l}\left(4,{\pi }_{01}\right)$ and $X_{0cj2}~\mathrm{B}\mathrm{i}\mathrm{n}\mathrm{o}\mathrm{m}\mathrm{i}\mathrm{a}\mathrm{l}\left(6,{\pi }_{02}\right)$ to reflect commonly encountered 5- and 7-point Likert scales. ${\pi }_{01}={\pi }_{02}=0.5$, to reflect symmetric distributions, or 0.3 to reflect skewed distributions. Parameters within the treatment arm were then chosen so that $\theta =0.5,0.56,0.64,0.71$, or to yield null, small, medium, and large effect sizes (based on relationships with Cohen’s effect size when endpoints are normal).²³ Endpoint effects ${\theta }_k$ could be homogeneous such that ${\theta }_1={\theta }_2=\theta$ or heterogeneous such that ${\theta }_1\ne {\theta }_2$. Heterogeneous effects were set to ${\theta }_1=(\theta -0.105)$ and ${\theta }_2=(\theta +0.105)$ so that a “large” difference existed between the two effects. The true global win probability is $\theta =\left({\theta }_1+{\theta }_2\right)/2$.

Let $\mathrm{\Omega }$ represent the $2\times 2$ within-subject correlation matrix,

$$\mathrm{\Omega }=\left[\begin{array}{cc}1& {\omega }_{12}\\ {\omega }_{12}& 1\end{array}\right]$$

with off-diagonal entries ${\omega }_{12}={\omega }_{21}$ equal to the pairwise correlation of the two endpoints on their original scale, i.e., $\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\left(X_{icj1},X_{icj2}\right)$. The pairwise correlation was varied from weak to strong, with ${\omega }_{12}=0.2,0.5,\text{and}0.8$.

Let $\mathrm{\Phi }$ represent the $2\times 2$ matrix of intracluster correlations,

$$\mathrm{\Phi }=\left[\begin{array}{ll}{\varphi }_{11}& {\varphi }_{12}\\ {\varphi }_{12}& {\varphi }_{22}\end{array}\right]$$

with diagonal entries ${\varphi }_{kk}$ equal to the ICC of the $k^{th}$ endpoint, i.e., $\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\left(X_{icjk},X_{ic{j}^{\prime }k}\right)$ where $j\ne j^{\prime }$, and off-diagonal entries ${\varphi }_{12}={\varphi }_{21}$ equal to the cross-subject cross-endpoint within-cluster correlation, i.e., $\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\left(X_{icjk},X_{ic{j}^{\prime }{k}^{\prime }}\right)$ where $j\ne j^{\prime }$ and $k^{\prime }\ne k$. Values of the intracluster correlation are generally small. A review of estimated intracluster correlations within primary care trials reported that 99% of ICCs were less than 0.10.⁴⁴ Focusing on heterogeneous endpoint ICCs, the ICCs were set to values of ${\varphi }_{11}=0.1,{\varphi }_{22}=0.05$, and ${\varphi }_{12}=0.025$.

The within-cluster correlation matrix is then provided by ${\mathbf{R}}_{ic}=\mathbf{I}\otimes (\mathrm{\Omega }-\mathrm{\Phi })+\mathbf{J}\otimes \mathrm{\Phi }$ where $\mathbf{I}$ is the $n_{ic}\times n_{ic}$ identity matrix, $\mathbf{J}$ is the $n_{ic}\times n_{ic}$ matrix of ones, and $\otimes$ denotes the Kronecker product. In other words, ${\mathbf{R}}_{ic}$ is a block diagonal matrix with $\mathrm{\Omega }$ repeated on the diagonal $n_{ic}$ times and each off-diagonal entry equal to $\mathrm{\Phi }$. For a cluster with three individuals, i.e., $n_{ic}=3$,

$${\mathbf{R}}_{ic}=\left[\begin{array}{ccc}\begin{array}{cc}1& {\omega }_{12}\\ {\omega }_{12}& 1\end{array}& \begin{array}{cc}{\varphi }_{11}& {\varphi }_{12}\\ {\varphi }_{12}& {\varphi }_{22}\end{array}& \begin{array}{cc}{\varphi }_{11}& {\varphi }_{12}\\ {\varphi }_{12}& {\varphi }_{22}\end{array}\\ \begin{array}{cc}{\varphi }_{11}& {\varphi }_{12}\\ {\varphi }_{12}& {\varphi }_{22}\end{array}& \begin{array}{cc}1& {\omega }_{12}\\ {\omega }_{12}& 1\end{array}& \begin{array}{cc}{\varphi }_{11}& {\varphi }_{12}\\ {\varphi }_{12}& {\varphi }_{22}\end{array}\\ \begin{array}{cc}{\varphi }_{11}& {\varphi }_{12}\\ {\varphi }_{12}& {\varphi }_{22}\end{array}& \begin{array}{cc}{\varphi }_{11}& {\varphi }_{12}\\ {\varphi }_{12}& {\varphi }_{22}\end{array}& \begin{array}{cc}1& {\omega }_{12}\\ {\omega }_{12}& 1\end{array}\end{array}\right].$$

The components of ${\mathbf{R}}_{ic}$ were assumed to be the same for both arms, and clusters are independent within- and between-arms. This correlation structure is commonly referred to as the “block exchangeable correlation model,” and has also been studied within the context of cluster trials with multiple continuous co-primary endpoints,⁴⁵ longitudinal and cross-over designs,⁴⁶ and stepped wedge designs.⁴⁷

Individual-level bivariate ordinal responses were generated independently by arm and cluster using the “mean mapping algorithm”.⁴⁸ In essence, realizations from a bivariate normal distribution with mean zero and intermediate correlation matrix ${\tilde{\mathbf{R}}}_{ic}$ are generated. Quantiles of the univariate standard normal distribution are then used to convert the normal variates into ordinal variates with the desired marginal distributions $F_{ik}$ and correlation matrix ${\mathbf{R}}_{ic}$. Root-finding algorithms are required to identify the intermediate correlation or entries of ${\tilde{\mathbf{R}}}_{ic}$ which yield the desired correlation upon discretization.

3.3. Results

Due to large number of scenarios examined, numerical summaries are provided here for only the most extreme scenario. For identity and logit back-transformed interval estimators, respectively, Tables 1 and 2 summarize performance with unequal allocation of clusters, skewed endpoint distributions, and large effect heterogeneity. Power was slightly greater for the least extreme scenario of equal allocation, symmetric endpoints, and no heterogeneity; otherwise performance was similar to that of Table 1 and 2. Empirical coverage probability (ECP) is also graphically summarized across all scenarios for the logit back-transformed interval estimator in Figure 1, suggesting coverage is similar regardless of allocation ratio, effect heterogeneity, and endpoint skew. Similar trends were observed for the identity back-transformed interval estimator.

Table 1.

Identity interval estimator: Simulation results for increasing numbers of clusters $C$, endpoint correlation ${\omega }_{12}$, and global win probabilities $\theta$ under unequal allocation of clusters ($C_1=0.7\mathrm{*}C$), skewed endpoint distributions (${\pi }_{0k}={\pi }_{1k}=0.3$), and large heterogeneity in endpoint effects (${\theta }_k=\theta \mp 0.105$).

			Fixed cluster sizes						Random cluster sizes
Parameters			$n=10$			$n=30$			$\overline{n}=10$			$\overline{n}=30$
$C$	${\omega }_{12}$	$\theta$	ECP	TER	ERR	ECP	TER	ERR	ECP	TER	ERR	ECP	TER	ERR
10	0.2	0.50	95.8	1.00	4.2	94.2	0.87	5.8	95.8	1.00	4.2	94.3	0.90	5.7
		0.56	95.8	1.10	12.8	94.7	1.12	19.6	95.3	1.35	14.6	94.4	0.87	18.2
		0.64	95.5	1.37	55.6	95.1	1.45	75.7	95.6	1.44	54.8	94.8	1.26	72.3
		0.71	95.9	3.10	91.8	95.0	1.50	98.5	94.7	1.52	88.8	94.5	1.89	97.1
	0.5	0.50	96.6	0.89	3.4	95.2	0.75	4.8	95.6	0.91	4.4	94.3	0.87	5.7
		0.56	96.1	1.29	11.3	95.3	1.24	18.5	95.6	1.10	12.8	94.7	1.12	18.5
		0.64	96.1	1.71	50.5	94.8	1.36	71.8	95.6	2.00	50.0	94.9	1.43	69.9
		0.71	96.3	2.80	88.0	95.2	1.82	97.6	95.0	2.27	86.2	94.5	1.75	96.4
	0.8	0.50	96.6	0.84	3.4	95.4	0.84	4.6	96.0	1.11	4.0	94.7	0.83	5.3
		0.56	96.3	0.95	10.6	95.2	1.47	18.5	95.6	1.25	12.1	94.8	1.36	17.7
		0.64	96.0	1.67	45.7	95.3	1.47	69.0	96.1	1.79	44.8	94.3	1.24	66.0
		0.71	96.5	2.18	82.8	96.0	1.56	97.1	95.8	2.42	82.1	94.4	1.89	95.0
20	0.2	0.50	94.8	0.86	5.2	95.3	0.92	4.7	94.8	0.96	5.2	94.5	0.87	5.5
		0.56	95.3	1.19	27.3	94.8	0.89	39.7	94.6	1.00	26.9	94.5	1.29	39.1
		0.64	95.3	1.24	89.7	95.0	1.33	98.1	94.4	1.20	88.4	94.6	1.45	97.3
		0.71	94.4	1.95	99.9	94.8	1.60	100	94.9	1.55	99.8	94.5	1.62	100
	0.5	0.50	94.7	0.89	5.3	95.0	0.92	5.0	94.0	1.00	6.0	94.5	0.77	5.5
		0.56	94.9	1.13	24.9	94.9	1.43	38.4	94.7	0.86	23.8	93.3	1.16	35.9
		0.64	95.4	1.47	85.6	94.8	1.32	97.2	95.3	2.00	85.0	94.3	1.38	96.0
		0.71	95.7	1.87	99.7	94.3	1.85	100	94.8	1.89	99.5	94.7	1.65	100
	0.8	0.50	95.6	1.20	4.4	94.7	0.96	5.3	94.8	0.79	5.2	94.0	0.90	6.0
		0.56	94.9	1.04	22.0	94.9	1.04	34.8	94.3	1.19	22.1	94.8	0.79	33.3
		0.64	95.5	1.37	81.5	94.8	1.36	96.1	95.3	1.76	80.9	94.3	1.11	95.0
		0.71	95.6	1.59	99.3	94.6	1.50	100	94.9	1.68	99.0	94.0	1.61	100
30	0.2	0.50	95.0	1.00	5.0	94.7	0.96	5.3	94.8	0.79	5.2	94.2	1.07	5.8
		0.56	95.0	1.08	39.6	95.2	1.09	56.1	94.5	0.90	38.1	94.4	1.24	54.7
		0.64	94.7	1.52	98.0	94.8	1.17	99.9	94.2	1.28	97.2	94.6	1.00	99.8
		0.71	95.3	1.24	100	95.7	1.53	100	94.5	1.80	100	94.5	1.43	100
	0.5	0.50	95.3	1.00	4.7	95.1	0.75	4.9	94.7	1.17	5.3	94.7	0.86	5.3
		0.56	95.2	1.14	35.4	94.9	1.04	52.8	94.8	0.93	34.8	94.3	1.00	49.5
		0.64	94.9	1.43	96.7	95.3	1.24	99.7	94.3	1.48	95.9	94.5	1.33	99.7
		0.71	95.7	1.87	100	95.6	1.44	100	94.7	1.52	100	94.4	1.55	100
	0.8	0.50	95.1	0.96	4.9	95.4	0.92	4.6	94.8	1.00	5.2	94.6	0.93	5.4
		0.56	95.3	0.96	30.9	94.8	1.08	51.5	94.8	1.08	31.5	94.2	1.23	48.7
		0.64	94.9	1.27	94.7	95.2	1.14	99.6	95.0	1.17	94.0	93.9	1.10	99.3
		0.71	94.7	1.94	99.9	95.7	1.69	100	94.5	1.62	100	94.3	1.71	100
50	0.2	0.50	95.1	1.13	4.9	94.7	0.93	5.3	94.3	1.11	5.7	94.0	1.00	6.0
		0.56	94.8	1.08	60.3	94.7	0.93	77.7	94.4	1.07	59.0	94.5	0.83	75.5
		0.64	94.5	1.12	100	94.8	1.00	100	94.5	1.00	99.9	94.6	1.08	100
		0.71	95.3	1.82	100	94.8	1.48	100	94.8	1.26	100	95.2	1.58	100
	0.5	0.50	95.2	0.96	4.8	95.5	1.25	4.5	94.7	0.71	5.3	94.6	1.04	5.4
		0.56	95.7	1.10	54.9	95.1	1.18	76.6	94.5	1.35	54.3	94.2	0.87	71.8
		0.64	95.0	1.27	99.9	95.2	1.40	100	94.7	1.30	99.9	94.7	1.21	100
		0.71	94.3	1.67	100	94.8	1.26	100	95.0	1.38	100	94.9	1.13	100
	0.8	0.50	95.3	0.96	4.7	94.6	0.80	5.4	94.9	1.13	5.1	93.6	0.85	6.4
		0.56	94.9	1.32	50.7	94.9	0.96	73.4	94.8	0.96	50.3	94.3	0.90	68.3
		0.64	95.3	1.47	99.6	95.1	1.27	100	94.8	1.26	99.6	94.3	1.48	100
		0.71	95.4	1.88	100	95.1	1.72	100	95.0	1.78	100	94.8	1.65	100

Table 2.

Logit back-transformed interval estimator; Simulation results for increasing numbers of clusters $C$, endpoint correlation ${\omega }_{12}$, and global win probabilities $\theta$ under unequal allocation of clusters ($C_1=0.7\mathrm{*}C$), skewed endpoint distributions (${\pi }_{0k}={\pi }_{1k}=0.3$), and large heterogeneity in endpoint effects (${\theta }_k=\theta \mp 0.105$).

			Fixed cluster sizes						Random cluster sizes
Parameters			$n=10$			$n=30$			$\overline{n}=10$			$\overline{n}=30$
$C$	${\omega }_{12}$	$\theta$	ECP	TER	ERR	ECP	TER	ERR	ECP	TER	ERR	ECP	TER	ERR
10	0.2	0.50	96.4	1.12	3.6	94.6	0.80	5.4	96.4	1.00	3.6	94.8	0.86	5.2
		0.56	96.6	0.84	11.2	95.2	0.92	18.2	96.1	1.05	12.8	94.9	0.70	16.5
		0.64	96.2	0.65	51.5	95.6	0.91	72.9	96.4	0.80	50.7	95.3	0.88	68.9
		0.71	96.6	1.12	88.7	95.6	0.83	97.8	95.4	0.70	85.1	95.2	1.09	96.2
	0.5	0.50	97.0	1.00	3.0	95.5	0.73	4.5	96.4	0.80	3.6	94.8	0.86	5.2
		0.56	96.7	1.06	9.9	95.6	1.10	17.1	96.3	0.90	10.7	95.1	0.92	17.0
		0.64	96.8	0.88	45.6	95.5	0.88	68.2	96.6	1.12	44.5	95.4	0.92	66.8
		0.71	97.3	1.08	84.4	95.7	1.05	96.9	96.6	0.89	81.9	95.1	0.88	95.4
	0.8	0.50	97.3	0.80	2.7	95.8	0.75	4.2	97.0	1.00	3.0	95.3	0.81	4.7
		0.56	96.9	0.68	8.4	95.7	1.26	16.9	96.4	0.89	9.9	95.4	1.09	16.1
		0.64	97.0	0.71	40.5	96.0	0.90	65.4	97.1	0.75	39.7	94.9	0.89	62.1
		0.71	97.3	0.59	78.1	96.6	0.70	95.8	96.8	0.60	76.7	95.1	0.78	93.6
20	0.2	0.50	95.4	0.84	4.6	95.6	0.91	4.4	95.1	1.00	4.9	94.7	0.83	5.3
		0.56	95.7	0.95	25.9	94.9	0.79	38.7	95.0	0.82	25.7	94.8	1.13	37.6
		0.64	95.8	0.68	88.7	95.5	0.96	97.9	94.7	0.86	87.4	94.8	1.08	96.9
		0.71	95.1	1.00	99.9	95.2	0.92	100	95.5	0.84	99.7	94.9	1.13	100
	0.5	0.50	95.2	0.92	4.8	95.2	0.88	4.8	94.6	1.00	5.4	94.7	0.77	5.3
		0.56	95.3	0.96	23.2	95.3	1.24	36.9	95.1	0.75	22.6	93.6	1.03	34.9
		0.64	96.0	1.00	84.1	94.9	0.96	97.0	95.8	1.21	83.1	94.7	1.04	95.5
		0.71	96.3	0.95	99.5	94.7	1.12	100	95.4	1.00	99.4	95.1	0.96	100
	0.8	0.50	96.0	1.22	4.0	95.0	1.00	5.0	95.2	0.78	4.8	94.3	0.97	5.7
		0.56	95.5	0.80	20.6	95.1	0.96	33.8	94.9	0.96	20.7	95.0	0.72	32.1
		0.64	96.2	0.76	79.9	95.2	0.96	95.6	96.1	1.05	79.2	94.7	0.83	94.4
		0.71	95.9	0.71	99.2	95.0	0.88	100	95.6	0.83	98.8	94.4	0.87	100
30	0.2	0.50	95.2	1.00	4.8	94.9	0.93	5.1	95.0	0.79	5.0	94.3	1.07	5.7
		0.56	95.3	0.88	38.7	95.4	1.00	55.4	94.8	0.79	37.0	94.7	1.08	54.0
		0.64	95.1	1.13	97.8	95.0	0.88	99.8	94.5	0.90	97.1	94.7	0.77	99.8
		0.71	95.7	0.72	100	95.9	1.11	100	95.0	1.08	100	94.9	0.89	100
	0.5	0.50	95.5	1.05	4.5	95.3	0.78	4.7	95.1	1.13	4.9	94.8	0.86	5.2
		0.56	95.5	1.00	34.2	95.1	0.96	52.1	95.0	0.79	33.4	94.4	0.90	48.8
		0.64	95.1	0.96	96.4	95.6	0.91	99.7	94.7	1.04	95.5	94.8	1.00	99.6
		0.71	96.1	0.95	100	95.6	1.00	100	95.2	0.78	100	94.8	1.08	100
	0.8	0.50	95.3	0.96	4.7	95.6	0.96	4.4	95.2	0.92	4.8	94.7	0.96	5.3
		0.56	95.6	0.83	30.0	95.0	1.00	50.7	95.2	0.92	30.5	94.3	1.07	47.5
		0.64	95.3	0.96	93.9	95.5	0.88	99.6	95.0	0.85	93.6	94.1	0.90	99.3
		0.71	95.3	1.04	99.9	96.1	1.05	100	95.1	0.96	100	94.7	1.12	100
50	0.2	0.50	95.3	1.14	4.7	94.9	0.96	5.1	94.4	1.15	5.6	94.1	0.97	5.9
		0.56	95.0	0.96	59.9	94.7	0.89	77.4	94.6	1.00	58.3	94.7	0.71	75.2
		0.64	94.7	0.96	100	94.9	0.79	100	94.7	0.77	99.9	94.6	0.93	100
		0.71	95.6	1.25	100	95.2	1.09	100	94.9	0.76	100	95.4	1.24	100
	0.5	0.50	95.4	1.00	4.6	95.5	1.20	4.5	94.9	0.70	5.1	94.7	1.00	5.3
		0.56	95.8	0.95	54.2	95.2	1.09	76.4	94.7	1.21	53.5	94.3	0.81	71.5
		0.64	95.1	0.88	99.8	95.2	1.18	100	95.0	0.92	99.9	94.9	0.96	100
		0.71	95.0	1.00	100	95.1	0.88	100	95.2	0.85	100	94.8	0.79	100
	0.8	0.50	95.6	0.91	4.4	94.7	0.77	5.3	95.2	1.09	4.8	93.8	0.85	6.2
		0.56	95.3	1.09	49.7	95.0	0.92	73.0	95.1	0.88	49.6	94.3	0.84	67.9
		0.64	95.5	1.00	99.6	95.3	0.96	100	95.1	1.04	99.5	94.8	1.17	100
		0.71	95.7	1.05	100	95.3	1.24	100	95.5	1.00	100	94.9	1.13	100

ECP: Empirical Coverage Probability, TER: Tail Error Ratio, ERR: Empirical Rejection Rate.

Figure 1.

Visual summary of empirical coverage probability (ECP) of logit back-transformed interval estimator across all 1,536 simulation scenarios and values of global win probability $\theta$. Grey band represents the acceptable range of ECP from 94.4% to 95.6%. The black, smoothed curves were fit via GAM. Jitter applied on x-axis for presentation.

Empirical coverage probability (ECP) generally fell within the acceptable range of 94.4% to 95.6% and the type I error rate within 4.4% to 5.6%. The only exceptions were scenarios featuring the fewest number of clusters with the smallest cluster size ($C=10,n=10$), which resulted in conservative coverage (> 95%) and type I error rates (< 5%). Desired coverage and type I error rates were consistently obtained for increasing numbers of clusters and larger cluster sizes $n=30$. This is likely a result of the large-sample nature of our developed methodology (see Appendix A), and note the closely related DeLong variance estimator³⁶ is known to be conservative for small sample sizes.⁴⁹ Performance was also negatively impacted by random cluster sizes, as should be expected^33,43.

The tail error ratio (TER) was almost always greater than 1 for the identity interval estimator, suggesting that the left tail error rate has a tendency to exceed the right tail error rate. Discrepancies between tail errors appeared to increase as the global win probability $\theta$ increased or the number of clusters $C$ or cluster size $n$ decreased. When $n=30$, use of the logit back-transformed interval estimator improved balance of the tail errors considerably while producing desirable ECPs and ERRs. When $n=10$, logit back-transformation also improved TERs, but appeared to exacerbate the conservative nature of our method when $C=10$, improving as $C$ increased.

Increasing the pairwise correlation of the endpoints appeared to decrease the power to detect a global treatment effect, as should be expected. However, our method appeared generally powerful. When $n=30$, an empirical rejection rate (power) of approximately 70% to detect a “moderate” global effect $(\theta =0.64)$ was achieved with as few as $C=10$ total clusters, while 80% power to detect a “small” global effect ($\theta =0.56$) required approximately $C=50$ clusters.

4. Case study: SHARE

4.1. Motivation

The Sexual Health and Relationships: Safe, Happy and Responsible (SHARE) trial aimed to determine whether an experimental sexual education curriculum reduced unsafe sex among students when compared to the existing curriculum.²² Schools in Scotland were randomized as clusters to be trained according to the experimental SHARE curriculum or not. This case study uses a subset of the trial data originally accessed from the Harvard Dataverse, with a link provided in the Appendix alongside reproducible R and SAS code.

Two endpoints are considered here: (1) knowledge, an ordinal endpoint ranging from −8 to 8, or least sexual health knowledge to most; and (2) activity, a binary endpoint taking the value 1 if the student was sexually active during follow-up and 0 otherwise. Suppose investigators wish to estimate the global treatment effect, assigning weights ${\omega }_1=0.7$ to knowledge and ${\omega }_2=0.3$ to activity, respectively. Traditional composite approaches may dichotomize knowledge to combine it with activity, potentially resulting in ceiling or floor effects, or ignore differences in priority.

4.2. Descriptives by endpoint

We consider $C_0=12$ and $C_1=13$ clusters assigned to conventional and experimental curriculum, respectively. $N=\mathrm{4,823}$ total students ($N_0=\mathrm{2,474},N_1=\mathrm{2,349}$) students with a completely observed bivariate response were retained for analysis, resulting in an average of 193 students per cluster ($\mathrm{S}\mathrm{D}=73$). Table 3 provides a summary of the observed endpoints, their corresponding win fractions, and correlation. The win fraction ICCs of knowledge and activity were both 0.028, and the corresponding observed ICCs were 0.031 and 0.028.

Table 3.

Summary of observed knowledge and activity scores and win fractions for N₀ = 2,474 and N₁ = 2,349 individuals receiving conventional and experimental curriculum within SHARE.

	Endpoint	Mean (SD)		Correlation
		Conventional	Experimental	Knowledge	Activity
Obs. scores	Knowledge	4.10 (2.36)	4.75 (2.28)	1.00	0.08
	Activity	0.27 (0.44)	0.27 (0.44)	0.08	1.00
Win fractions	Knowledge	0.42 (0.28)	0.58 (0.28)	1.00	−0.08
	Activity	0.50 (0.22)	0.50 (0.22)	−0.08	1.00

4.3. Estimates and interpretation

The use of SAS PROC MIXED to estimate the two-level mixed model presented in Equation (9) provides ${\hat{\beta }}_1=0.104$ with $\widehat{\mathrm{S}\mathrm{E}}({\hat{\beta }}_1)=0.017$ on $\mathrm{d}\mathrm{f}=23$ degrees of freedom, ${\hat{\sigma }}_{\alpha }\approx 0.039$ and ${\hat{\sigma }}_{\epsilon }\approx 0.199$. The corresponding weighted global win probability estimate is $\hat{\theta }=1.104/2=0.552$ with $\widehat{\mathrm{S}\mathrm{E}}(\hat{\theta })=0.017$ and a logit back-transformed 95% confidence interval of 0.517 to 0.587. The estimated ICC of the global win fractions was 0.037.

Formally, the estimated probability that a student receiving the experimental curriculum responds better than a student receiving the existing curriculum is 55.2% (95% CI: 0.517 to 0.587), on average, with respect to the two weighted endpoints of sexual knowledge and activity. Since the null value ${\theta }_0=0.5$ is excluded from the 95% confidence interval, there is statistically significant evidence at the $\alpha =0.05$ level to reject the null hypothesis of no weighted global treatment effect. For reference, if equal weights are assigned to each endpoint, the estimated global win probability is 0.537 (95% CI: 0.506 to 0.568).

Alternatively, the estimated global win difference is $\stackrel{\mathrm{ˆ}}{\mathrm{\Delta }}=2(0.552)-1=0.104(\widehat{\mathrm{S}\mathrm{E}}(\widehat{\mathrm{\Delta }})=2(0.017)=0.034;95\mathrm{\%}\mathrm{C}\mathrm{I}:0.037$ to 0.171), suggesting that the probability of winning on treatment is approximately 10.4 basis points greater than that on control. The estimated win odds are $\hat{\lambda }=0.552/(1-0.552)=1.23(\widehat{\mathrm{S}\mathrm{E}}(\mathrm{l}\mathrm{n}\hat{\lambda })=0.017/(0.552\times 0.448)=0.069$; 95%CI: 1.076 to 1.410), suggesting that the treatment win probability is approximately 23% greater than the control win probability.

5. Discussion

In this paper, we address the challenges of analyzing cluster randomized trials with multiple endpoints different scales by developing interval estimation and hypothesis testing methods for a nonparametric global treatment effect, referred to here as the global win probability. The global win probability directly quantifies the objective of most trials which is to determine if patients receiving treatment have better overall health outcomes than those on control or the current standard of care. The global win probability only compares the ordering of responses, rather than their size or difference, and is applicable to any set of endpoints that can be ranked including binary, ordinal, count, and continuous endpoints. Estimation is also robust to monotonic transformation, unlike mean-based methods which may yield differing or even conflicting results pre- and post-transformation.

The developed methods are simple as they bypass the need to consider complex correlation structures between-endpoint and within-cluster, and accessible as they may be implemented using standard statistical software. To estimate the global win probability, a single rank-based global win fraction is constructed for each individual within the cluster trial as the within-subject mean of their endpoint-specific win fractions. Unlike alternative composite measures like the rank-sum, global win fractions are interpretable at the individual-level as the average proportion of responses within the comparator arm exceeded by a given individual. Weights may also be incorporated within construction of the global win fractions to reflect differences in endpoint utility or priority, though are not required. The mixed model estimation framework previously introduced by Zou¹⁸ for a single win probability is then easily applied to the univariate global win fractions to obtain point, variance, and interval estimators of the global win probability adjusted for intracluster correlation. Simulation suggests interval estimators respect nominal coverage and type I error rates across a range of cluster randomized designs.

The flexible mixed model framework employed also permits the consideration of more complex models. For example, Zou et al recently detailed how win fraction regression methods could be used to analyze individually randomized pre-post designs in a fashion analogous to ANCOVA.²¹ A similar approach could be taken here by, e.g., regressing global win fractions at follow-up on global win fractions at baseline, potentially leading to additional boosts in power.⁵⁰ Investigation into additional baseline adjustment strategies, e.g., for precision or stratification, represents an area of future work. Multivariate linear mixed models for the win fractions would also permit the construction of a $K$-df test, use of more complex covariance structures, or assessment of global treatment effects over time.

Another advantage of using the mixed model framework is that the form of sample size estimators is relatively simple. The sample size for a corresponding individually randomized trial can be scaled by a design effect, i.e., $\mathrm{D}\mathrm{E}=1+(n-1)\rho$. Sample size formulas for a single win probability within individually randomized trials were recently provided by Zou et al.¹⁹ Relationships between endpoint distributions or alternative measures and the win probabilitiy are well-known, with a nice summary provided by Rahlfs and Zimmerman²³, for example. Thus, parametric treatment effect estimates reported in the literature may be transformed into endpoint-specific win probabilities and averaged to obtain an approximate global win probability for sample size estimation. Standard errors may also be obtained, see Shu & Zou for example.⁵¹ However, further investigation into the true relationship between the observed and win fraction ICCs is needed. Zou et al²¹ suggested use of the observed pre-post correlation for sample size. Using the largest, conservative, endpoint ICC estimate may be a reasonable strategy, for now.

If desired, the global win probability can also be directly transformed into other popular, alternative effect measures encountered in medicine. This includes related win measures such as the win difference or win odds,^25,29 or more traditional effects such as the risk difference or standardized mean difference.²³ The global win probability is inspired by the nonparametric rank-sum test introduced by O’Brien⁷, and as demonstrated here, the mean difference in rank-sums is a linear transformation of global win probability estimators. Zou & Zou¹⁷ proposed methods for estimating the global win probability in individually randomized trials with missing data. Expansion of their methods to the current context represents another area of future work.

The inability of some composites to reflect differences in endpoint priority is a commonly cited concern.⁵² Endpoint priority may differ when, e.g., endpoint utility, importance, or severity differs. Hence, we have expanded our presentation of the global win probability to permit contribution weights for each endpoint. However, these weights are not necessary to apply our methods. Furthermore, while relationships with other win-based effects are discussed for a single endpoint, their conceptual papers^25,27 evaluate multiple prioritized endpoints in a fundamentally different way using generalized pairwise comparisons (GPC). Explicit priority ordering of the endpoints must be specified to use GPC approaches, unlike our method. As pointed out by Rauch et al,⁵² estimates based on GPC may be challenging to interpret.

A. Asymptotic correlation of clustered win fractions

Perhaps surprisingly, win fractions are asymptotically uncorrelated between cluster and arm, but not within cluster due to intracluster correlation. Here, we provide a sketch proof examining the asymptotic covariance, and hence correlation, between win fractions in these three situations: within cluster, within arm between cluster, and between arm. In the following sketch it is important to remember that individuals within the same cluster are identically distributed and share an exchangeable correlation structure, and individuals in different clusters are independent as a result of cluster randomization.

For simplicity, we consider only a single endpoint so $K=1$, but extension to the global win fraction is straightforward. Let $Y_{1cjk}=Y_{1cj}$ represent the win fraction of the $j^{th}$ individual in the $c^{th}$ treatment cluster ($j=1,\dots ,n_{1c};c=1,\dots ,C_1$). Modifying the notation of the control responses and win fractions, let $Y_{0k\ell }$ represent the win fraction of the ${\ell }^{th}$ individual in the $k^{th}$ control cluster and $X_{0k\ell }$ the $k{\ell }^{th}$ control response ($\ell =1,\dots ,n_{0k};k=1,\dots ,C_0$). Finally, let $\varphi \left(X_{1cj},X_{0k\ell }\right)=H\left(X_{1cj}-X_{0k\ell }\right)$ represent the Heaviside scoring function.

Recall there are $n_{1c}$ responses $X_{1cj}$ within the $c^{th}$ treatment cluster, and $n_{0k}$ responses $X_{0k\ell }$ within the $k^{th}$ control cluster. There are a total of $N_1=\sum _c^{C_1}{n}_{1c}$ treatment responses and $N_0=\sum _k^{C_0}{n}_{0k}$ control responses. Each treatment response is compared to all $N_0$ control responses when constructing a win fraction. Similarly, each control response is compared to all $N_1$ treatment responses.

A. 1. Within cluster

The covariance of two win fractions within the same cluster is defined as,

$$N_0^2\mathrm{C}\mathrm{o}\mathrm{v}\left(Y_{1cj},Y_{1c{j}^{\prime }}\right)=\mathrm{C}\mathrm{o}\mathrm{v}\left\{\sum _{k=1}^{C_0}\sum _{\ell =1}^{n_{0c^{\prime }}}\varphi \left(X_{1cj},X_{0k\ell }\right),\sum _{k=1}^{C_0}\sum _{\ell =1}^{n_{0c^{\prime }}}\varphi \left(X_{1c{j}^{\prime }},X_{0k\ell }\right)\right\}.$$

The first arguments of the $\varphi \left(X_{1cj},\cdot \right)$ and $\varphi \left(X_{1c{j}^{\prime }},\cdot \right)$ will be correlated for all $N_0^2$ terms due to intracluster correlation. The second arguments $\varphi \left(\cdot ,X_{0k\ell }\right)$ will be correlated only when they belong to the same control cluster $k$. See the top panel of Figure 2 for a visualization of the corresponding covariance structure. Counting like terms, we have:

1. Same $X_1$ cluster, same $X_0$ response: $\varphi \left(X_{1cj},X_{0k\ell }\right)$ will be correlated with respect to $X_1$ and perfectly correlated with respect to $X_0$ for $N_0$ terms of the form $\varphi \left(X_{1c{j}^{\prime }},X_{0k\ell }\right)$.

2. Same $X_1$ cluster, same $X_0$ cluster: $\varphi \left(X_{1cj},X_{0k\ell }\right)$ will be correlated with respect to both $X_1$ and $X_0$ for $\left(\sum _k^{C_0}{n}_{0k}^2-N_0\right)$ terms of the form $\varphi \left(X_{1c{j}^{\prime }},X_{0k{\ell }^{\prime }}\right)$.

3. Same $X_1$ cluster, different $X_0$ cluster: $\varphi \left(X_{1cj},X_{0k\ell }\right)$ will be correlated with respect to only $X_1$ for $\left(N_0^2-\sum _k^{C_0}{n}_{0k}^2\right)$ terms of the form $\varphi \left(X_{1c{j}^{\prime }},X_{0k^{\prime }{\ell }^{\prime }}\right)$.

Figure 2.

From top to bottom: covariance pattern of kernel terms (a) within the same arm and cluster; (b) within the same arm but different clusters; and (c) in different arms and clusters. Darkest shading: covariation in $X_1$ and $X_0$ terms; lighter: covariation in $X_1$ only; lightest: covariation in $X_0$ only; none: no covariation. Striping indicates perfect covariation in $X_1$ and/or $X_0$.

Putting this together,

$$\begin{gathered} N_0^2\mathrm{C}\mathrm{o}\mathrm{v}\left(Y_{1cj},Y_{1c{j}^{\prime }}\right)=N_0\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{1c{j}^{\prime }},X_{0k\ell }\right)\right\} \\ +\left(\sum _{k=1}^{C_0}{n}_{0k}^2-N_0\right)\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{1c{j}^{\prime }},X_{0k{\ell }^{\prime }}\right)\right\} \\ +\left(N_0^2-\sum _{k=1}^{C_0}{n}_{0k}^2\right)\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{1c{j}^{\prime }},X_{0k^{\prime }{\ell }^{\prime }}\right)\right\}. \end{gathered}$$ (12)

and because $N_0^2={\left(\sum _{k=1}^{C_0}{n}_{0k}\right)}^2$ we have:

$$\begin{gathered} \mathrm{C}\mathrm{o}\mathrm{v}\left(Y_{1cj},Y_{1c{j}^{\prime }}\right)=\frac{1}{N_0}\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{1c{j}^{\prime }},X_{0k\ell }\right)\right\} \\ +\left(\frac{\sum _{k=1}^{C_0}{n}_{0k}^2}{{\left(\sum _{k=1}^{C_0}{n}_{0k}\right)}^2}-\frac{1}{N_0}\right)\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{1c{j}^{\prime }},X_{0k{\ell }^{\prime }}\right)\right\} \\ +\left(1-\frac{\sum _{k=1}^{C_0}{n}_{0k}^2}{{\left(\sum _{k=1}^{C_0}{n}_{0k}\right)}^2}\right)\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{1c{j}^{\prime }},X_{0k^{\prime }{\ell }^{\prime }}\right)\right\}. \end{gathered}$$

Because $n_{0k}>1$, we have ${\left(\sum _k{n}_{0k}\right)}^2>\sum _k{n}_{0k}^2$. Thus, as the number of control individuals increases towards infinity ($N_0\to \mathrm{\infty }$),

$$\mathrm{C}\mathrm{o}\mathrm{v}\left(Y_{1cj},Y_{1c{j}^{\prime }}\right)\to \mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{1c{j}^{\prime }},X_{0k^{\prime }{\ell }^{\prime }}\right)\right\}$$

so the covariance of win fractions within the same arm is dependent only on their intracluster correlation for large samples.

A.2. Within arm, between cluster

The covariance of two win fractions within the same arm but different clusters is defined as,

$$N_0^2\mathrm{C}\mathrm{o}\mathrm{v}\left(Y_{1cj},Y_{1c^{\prime }{j}^{\prime }}\right)=\mathrm{C}\mathrm{o}\mathrm{v}\left\{\sum _{k=1}^{C_0}\sum _{\ell =1}^{n_{0k^{\prime }}}\varphi \left(X_{1cj},X_{0k\ell }\right),\sum _{k=1}^{C_0}\sum _{\ell =1}^{n_{0k^{\prime }}}\varphi \left(X_{1c^{\prime }{j}^{\prime }},X_{0k\ell }\right)\right\}.$$

The first arguments of the $\varphi \left(X_{1cj},\cdot \right)$ and $\varphi \left(X_{1c^{\prime }{j}^{\prime }},\cdot \right)$ will be uncorrelated for all $N_0^2$ terms. The second arguments $\varphi \left(\cdot ,X_{0k\ell }\right)$ will be correlated only when they belong to the same control cluster $k$. See the middle panel of Figure 2 for a visualization of the corresponding covariance structure. Counting like terms, we have:

1. Different $X_1$ cluster, same $X_0$ response: $\varphi \left(X_{1cj},X_{0k\ell }\right)$ will be perfectly correlated with respect to $X_0$ only for $N_0$ terms of the form $\varphi \left(X_{1c^{\prime }{j}^{\prime }},X_{0k\ell }\right)$.

2. Different $X_1$ cluster, same $X_0$ cluster: $\varphi \left(X_{1cj},X_{0k\ell }\right)$ will be correlated with respect to only $X_0$ for $\left(\sum _k^{C_0}{n}_{0k}^2-N_0\right)$ terms of the form $\varphi \left(X_{1c^{\prime }{j}^{\prime }},X_{0k{\ell }^{\prime }}\right)$.

3. Different $X_1$ cluster, different $X_0$ cluster: $\varphi \left(X_{1cj},X_{0k\ell }\right)$ will be uncorrelated for $(N_0^2-\sum _k^{C_0}{n}_{0k}^2)$ terms of the form $\varphi \left(X_{1c^{\prime }{j}^{\prime }},X_{0k^{\prime }{\ell }^{\prime }}\right)$.

Putting this together,

$$\begin{gathered} \mathrm{C}\mathrm{o}\mathrm{v}\left(Y_{1cj},Y_{1c^{\prime }{j}^{\prime }}\right)=\frac{1}{N_0}\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{1c^{\prime }{j}^{\prime }},X_{0k\ell }\right)\right\} \\ +\left(\frac{\sum _{k=1}^{C_0}{n}_{0k}^2}{{\left(\sum _{k=1}^{C_0}{n}_{0k}\right)}^2}-\frac{1}{N_0}\right)\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{1c^{\prime }{j}^{\prime }},X_{0k{\ell }^{\prime }}\right)\right\} \end{gathered}$$

so $\mathrm{C}\mathrm{o}\mathrm{v}\left(Y_{1cj},Y_{1c^{\prime }{j}^{\prime }}\right)\to 0$ as $N_0\to \mathrm{\infty }$, and win fractions within the same arm but different clusters are asymptotically uncorrelated.

A.3. Between arm

The covariance of two win fractions within different arms is defined as,

$$N_0{N}_1\mathrm{C}\mathrm{o}\mathrm{v}\left(Y_{1cj},Y_{0k\ell }\right)=\mathrm{C}\mathrm{o}\mathrm{v}\left\{\sum _{k=1}^{C_0}\sum _{\ell =1}^{n_{0k^{\prime }}}\varphi \left(X_{1cj},X_{0k\ell }\right),\sum _{c^{\prime }=1}^{C_0}\sum _{j^{\prime }=1}^{n_{1c^{\prime }}}\varphi \left(X_{0k\ell },X_{1c^{\prime }{j}^{\prime }}\right)\right\}.$$

See the bottom panel of Figure 2 for a visualization of the corresponding covariance structure.

Counting like terms, we have:

$$\begin{gathered} N_0{N}_1\mathrm{C}\mathrm{o}\mathrm{v}\left(Y_{1cj},Y_{0k\ell }\right)=\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{0k\ell },X_{1cj}\right)\right\} \\ +\left(n_{0k}-1\right)\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k{\ell }^{\prime }}\right),\varphi \left(X_{0k\ell },X_{1cj}\right)\right\} \\ +\left(n_{1c}-1\right)\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{0k\ell },X_{1c{j}^{\prime }}\right)\right\} \\ +\left(n_{1c}-1\right)\left(n_{0k}-1\right)\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k{\ell }^{\prime }}\right),\varphi \left(X_{0k\ell },X_{1c{j}^{\prime }}\right)\right\} \\ +\left(N_1-n_{1c}\right)\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{0k\ell },X_{1c^{\prime }{j}^{\prime }}\right)\right\} \\ +\left(N_0-n_{0k}\right)\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k^{\prime }{\ell }^{\prime }}\right),\varphi \left(X_{0k\ell },X_{1cj}\right)\right\} \\ +\left(n_{1c}-1\right)\left(N_0-n_{0k}\right)\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k^{\prime }{\ell }^{\prime }}\right),\varphi \left(X_{0k\ell },X_{1c{j}^{\prime }}\right)\right\} \\ +\left(n_{0k}-1\right)\left(N_1-n_{1c}\right)\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k{\ell }^{\prime }}\right),\varphi \left(X_{0k\ell },X_{1c^{\prime }{j}^{\prime }}\right)\right\} \\ +\left(N_1-n_{1c}\right)\left(N_0-n_{0k}\right)\times 0 \end{gathered}$$

and once again, we have that $\mathrm{C}\mathrm{o}\mathrm{v}\left(Y_{1cj},Y_{0k\ell }\right)\to 0$ as $N_0,N_1\to \mathrm{\infty }$.

A.4. Variance of a treatment win fraction

For two win fractions within different clusters in the same arm, for example, recall the relationship:

$$\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\left(Y_{1cj},Y_{1c^{\prime }{j}^{\prime }}\right)=\frac{\mathrm{C}\mathrm{o}\mathrm{v}\left(Y_{1cj},Y_{1c^{\prime }{j}^{\prime }}\right)}{\sqrt{\mathrm{V}\mathrm{a}\mathrm{r}\left(Y_{1cj}\right)\mathrm{V}\mathrm{a}\mathrm{r}\left(Y_{1c^{\prime }{j}^{\prime }}\right)}}$$

In the previous sections, we have shown $\mathrm{C}\mathrm{o}\mathrm{v}\left(Y_{1cj},Y_{1c^{\prime }{j}^{\prime }}\right)\to 0$ as $N_0\to \mathrm{\infty }$. If $\mathrm{V}\mathrm{a}\mathrm{r}\left(Y_{1cj}\right)$ is asymptotically finite and non-zero, it thus follows that $\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\left(Y_{1cj},Y_{1c^{\prime }{j}^{\prime }}\right)\to 0$ as $N_0\to \mathrm{\infty }$. Here, we explore the asymptotic nature of $\mathrm{V}\mathrm{a}\mathrm{r}\left(Y_{1cj}\right)$.

The variance of a single win fraction is defined as,

$$N_0^2\mathrm{V}\mathrm{a}\mathrm{r}\left(Y_{1cj}\right)=\mathrm{C}\mathrm{o}\mathrm{v}\left\{\sum _{k=1}^{C_0}\sum _{\ell =1}^{n_{0k^{\prime }}}\varphi \left(X_{1cj},X_{0k\ell }\right),\sum _{k^{\prime }=1}^{C_0}\sum _{{\ell }^{\prime }=1}^{n_{0k^{\prime }}}\varphi \left(X_{1cj},X_{0k^{\prime }{\ell }^{\prime }}\right)\right\}.$$

The first arguments of the $\varphi \left(X_{1cj},\cdot \right)$ will be perfectly correlated for all $N_0^2$ terms. The second arguments $\varphi \left(\cdot ,X_{0k\ell }\right)$ will be correlated only when they belong to the same control cluster $k$. Counting like terms:

1. Same $X_1$ response, same $X_0$ response: $\varphi \left(X_{1cj},X_{0k\ell }\right)$ will be perfectly correlated with respect to $X_1$ and $X_0$ for $N_0$ terms of the form $\varphi \left(X_{1cj},X_{0k\ell }\right)$.

2. Same $X_1$ response, same $X_0$ cluster: $\varphi \left(X_{1cj},X_{0k\ell }\right)$ will be perfectly correlated in $X_1$ and correlated with $X_0$ for $\left(\sum _k^{C_0}{n}_{0k}^2-N_0\right)$ terms of the form $\varphi \left(X_{1cj},X_{0k{\ell }^{\prime }}\right)$.

3. Same $X_1$ response, different $X_0$ cluster: $\varphi \left(X_{1cj},X_{0k\ell }\right)$ will be perfectly correlated in $X_1$ and uncorrelated in $X_0$ for ($N_0^2-\sum _k^{C_0}{n}_{0k}^2$) terms of the form $\varphi \left(X_{1cj},X_{0k^{\prime }{\ell }^{\prime }}\right)$.

Putting this together, we have:

$$\begin{gathered} \mathrm{V}\mathrm{a}\mathrm{r}\left(Y_{1cj}\right)=\frac{1}{N_0}\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{1cj},X_{0k\ell }\right)\right\} \\ +\left(\frac{\sum _{k=1}^{C_0}{n}_{0k}^2}{{\left(\sum _{k=1}^{C_0}{n}_{0k}\right)}^2}-\frac{1}{N_0}\right)\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{1cj},X_{0k{\ell }^{\prime }}\right)\right\} \\ +\left(1-\frac{\sum _{k=1}^{C_0}{n}_{0k}^2}{{\left(\sum _{k=1}^{C_0}{n}_{0k}\right)}^2}\right)\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{1cj},X_{0k^{\prime }{\ell }^{\prime }}\right)\right\}. \end{gathered}$$

As $N_0\to \mathrm{\infty }$,

$$\mathrm{V}\mathrm{a}\mathrm{r}\left(Y_{1cj}\right)\to \mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{1cj},X_{0k^{\prime }{\ell }^{\prime }}\right)\right\}$$

so the variance of a treatment win fraction approaches a finite, non-zero function of the variance of the treatment response $X_{1cj}$ only. It follows that as $N_0\to \mathrm{\infty },\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\left(Y_{1cj},Y_{1c^{\prime }{j}^{\prime }}\right)\to 0$. Similar arguments can be used to show that as $N_0\to \mathrm{\infty },\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\left(Y_{1cj},Y_{0k\ell }\right)\to 0$ and for two win fractions within the same cluster,

$$\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\left(Y_{1cj},Y_{1c{j}^{\prime }}\right)\to \frac{\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{1c{j}^{\prime }},X_{0k^{\prime }{\ell }^{\prime }}\right)\right\}}{\mathrm{C}\mathrm{o}\mathrm{v}\left\{\varphi \left(X_{1cj},X_{0k\ell }\right),\varphi \left(X_{1cj},X_{0k^{\prime }{\ell }^{\prime }}\right)\right\}}.$$

B. SHARE

Data from the SHARE trial can be downloaded from the Harvard Dataverse in tab-delimited format (share.tab) at https://dataverse.harvard.edu/dataverse/crt.

B.1. R code

library(dplyr) # Data manipulation 
library(nlme) # Mixed models
# IMPORT: SHARE data 
share_raw <- read.table(‘share.tab’, sep=‘\t’, header=T)
# EXTRACT: identifiers and outcomes (Note: school=cluster) 
share <- share_raw %>%
select(arm, school, idno, ‘knowledge’=kscore, ‘active’=debut) %>% 
na.omit()
# EXTRACT: Number of students in each arm
N0 <- nrow(share[share$arm == 0,]) # Conventional
N1 <- nrow(share[share$arm == 1,]) # Experimental
# CONSTRUCT: overall and group ranks for each endpoint 
# Note: increased knowledge = good, increased active = bad 
ranks <- share %>% 
mutate(R1 = rank(knowledge), R2 = rank(-active)) %>% 
group_by(arm) %>% 
mutate(G1 = rank(knowledge), G2 = rank(-active))
# Endpoint weights 
w1 = 0.7; w2 = 0.3
# CONSTRUCT: endpoint and global win fractions 
winf <- ranks %>%
mutate(Y1 = ifelse(arm == 0, (R1-G1)/N1, (R1-G1)/N0), 
Y2 = ifelse(arm == 0, (R2-G2)/N1, (R2-G2)/N0),
YG = w1 * Y1+ w2* Y2)
# FIT: linear mixed model for global win fractions 
modG <- lme(YG ˜ arm, random = ˜1 | school, data=winf)
# EXTRACT: Fixed effect estimates, their variance, and df 
modG_fest <- fixef(modG) 
modG_fvar <- vcov(modG) 
modG_fdf <- modG$fixDF$X # Note: Arm df = C-2 by default
# EXTRACT: Random effect variance components 
modG_rvar <- matrix(as.numeric(VarCorr(modG)), ncol=2)
# CONSTRUCT: Global win probability point, variance, and ICC est 
estG <- (modG_fest[2] + 1)/2 
seG <- sqrt(modG_fvar[2,2]) 
iccG <- modG_rvar[1,1] / (modG_rvar[1,1] + modG_rvar[2,1])
# CONSTRUCT: Global win probability interval estimates 
alpha <− 0.05; t <- qt(1-alpha/2, modG_fdf[2]) 
untransform_ci <- estG + c(−1, 1) * t * seG 
logit_lu <- log(estG/(1-estG)) + c(−1, 1) * t * seG/(estG*(1-estG)) 
logit_ci <- exp(logit_lu) / (1 + exp(logit_lu))
# REPORT: Estimates 
report_point <- paste0(‘Est. global win probability = ‘, estG,
‘ (Est. SE = ‘, seG, ‘, df = ‘, modG_fdf[2], ‘)’)
report_icc <- paste0(‘Est. global ICC = ‘, iccG) 
report_uci <- paste0(‘95% untransformed confidence interval = (‘, 
untransform_ci[1], ‘, ‘, untransform_ci[2], ‘)’)
report_lci <- paste0(‘95% logit confidence interval = (‘, 
logit_ci[1], ‘, ‘, logit_ci[2], ‘)’)
cat(paste(report_point, report_uci, report_lci, report_icc, sep=‘\n’))

B.2. SAS code

PROC IMPORT DATAFILE=“share.tab” OUT=share DBMS=DLM REPLACE;
DELIMITER=‘09’x; RUN;
DATA share; SET share; 
knowledge = kscore;
* Reverse code active since higher => worse; 
active = −1 * debut;
* Exclude individuals with incomplete responses;
IF (knowledge ne .) and (active ne .) THEN OUTPUT;
KEEP arm school idno knowledge active; 
RUN;
* Sort data by Arm (0 = Conventional, 1 = Experimental);
PROC SORT DATA=share; BY arm; RUN;
* Obtain number of individuals in each arm;
PROC FREQ DATA=share NOPRINT;
TABLE arm / OUT=SampleSize(DROP=percent); RUN;
* Reverse Arm labels for win fraction denominator;
DATA Denominator; SET SampleSize; arm = 1 - arm; RUN;
* Sort for merge;
PROC SORT DATA=Denominator; BY arm; RUN;
* Overall (mid)ranks for each endpoint;
PROC RANK DATA=share OUT=O_ranks TIES=mean;
VAR knowledge active; RANKS O1 O2;
RUN;
* Group (mid)ranks for each endpoint;
PROC RANK DATA=share OUT=G_ranks TIES=mean; BY arm;
VAR knowledge active; RANKS G1 G2; 
RUN;
* Calculate endpoint and global win fractions;
DATA WinF; MERGE O_ranks G_ranks Denominator; BY arm;
Y1 = (O1-G1)/COUNT;
Y2 = (O2-G2)/COUNT;
YG = 0.7*Y1 + 0.3*Y2; 
RUN;
* Fit linear mixed model for global win fractions;
PROC MIXED DATA=WinF NOITPRINT NOCLPRINT;
CLASS arm school idno / REF=first;
MODEL YG = arm / solution;
RANDOM intercept / SUBJECT=school(arm);
ODS OUTPUT SolutionF = FixEff(KEEP=Arm Estimate StdErr DF);
ODS OUTPUT CovParms = CovParms; 
RUN;
* Extract estimates;
DATA VarInt VarRes; SET CovParms;
IF CovParm=‘Intercept’ THEN OUTPUT VarInt;
IF CovParm=‘Residual’ THEN OUTPUT VarRes;
DROP CovParm Subject; 
RUN;
DATA FixEff; SET FixEff; Beta1 = Estimate; 
IF Arm = 1 THEN OUTPUT; RUN;
DATA GlobalEstimates;
MERGE FixEff
VarInt(RENAME=(Estimate=VarAlpha))
VarRes(RENAME=(Estimate=VarEps));
EstG = (Beta1 + 1)/2; SeG = StdErr; DfG = DF; 
t = tinv(1–0.05/2, DF);
* Untransformed 95% confidence interval;
L1 = EstG - t * SeG; U1 = EstG + t * SeG;
* Logit 95% confidence interval;
L2 = log(EstG/(1-EstG)) - t * SeG / (EstG * (1-EstG));
U2 = log(EstG/(1-EstG)) + t * SeG / (EstG * (1-EstG));
L2 = exp(L2) / (1 + exp(L2)); U2 = exp(U2) / (1 + exp(U2));
* Intracluster correlation of global win fractions;
IccG = VarAlpha / (VarAlpha + VarEps);
KEEP EstG SeG DfG L1 U1 L2 U2 IccG;
RUN;
PROC PRINT DATA=GlobalEstimates; Run;