Asymptotic theory of rerandomization in treatment–control experiments

Significance

Rerandomization refers to experimental designs that enforce covariate balance. This paper studies the asymptotic properties of the difference-in-means estimator under rerandomization, based on the randomness of the treatment assignment without imposing any parametric modeling assumptions on the covariates or outcome. The non-Gaussian asymptotic distribution allows for constructing large-sample confidence intervals for the average treatment effect and demonstrates the advantages of rerandomization over complete randomization.

Abstract

Although complete randomization ensures covariate balance on average, the chance of observing significant differences between treatment and control covariate distributions increases with many covariates. Rerandomization discards randomizations that do not satisfy a predetermined covariate balance criterion, generally resulting in better covariate balance and more precise estimates of causal effects. Previous theory has derived finite sample theory for rerandomization under the assumptions of equal treatment group sizes, Gaussian covariate and outcome distributions, or additive causal effects, but not for the general sampling distribution of the difference-in-means estimator for the average causal effect. We develop asymptotic theory for rerandomization without these assumptions, which reveals a non-Gaussian asymptotic distribution for this estimator, specifically a linear combination of a Gaussian random variable and truncated Gaussian random variables. This distribution follows because rerandomization affects only the projection of potential outcomes onto the covariate space but does not affect the corresponding orthogonal residuals. We demonstrate that, compared with complete randomization, rerandomization reduces the asymptotic quantile ranges of the difference-in-means estimator. Moreover, our work constructs accurate large-sample confidence intervals for the average causal effect.

Ever since Fisher’s (1–3) seminal work, randomized experiments have become the “gold standard” for drawing causal inferences. Complete randomization balances the covariate distributions between treatment groups in expectation, thereby ensuring the existence of unbiased estimators of average causal effects. Covariate imbalance, however, often occurs in specific randomized experiments, as recognized by Fisher (2) and later researchers (e.g., refs. 4–9). The standard approach advocated by Fisher (3), stratification or blocking, ensures balance with a few discrete covariates (e.g., refs. 10–12).

When a randomized allocation is unbalanced, it is reasonable to discard that allocation and redraw another one until a certain predetermined covariate balance criterion is satisfied. This is rerandomization, an experimental design hinted at by Fisher (cf. ref. 13, p. 88) and Cox (14, 15) and formally proposed by Rubin (16) and Morgan and Rubin (17). Morgan and Rubin (17) showed that the difference-in-means estimator is generally unbiased for the average causal effect under rerandomization with equal-sized treatment groups and obtained the sampling variance of this estimator under additional assumptions of Gaussian covariate and outcome distributions and additive causal effects. When rerandomization is applied but these assumptions do not hold, statistical inference becomes more challenging, because the theory that is justified by the central limit theorem under complete randomization (18, 19) no longer generally holds. Some applied researchers believe that “the only analysis that we can be completely confident in is a permutation test or rerandomization test” (ref. 7, p. 219). However, permutation tests based on randomization require sharp null hypotheses that imply all missing potential outcomes are known.

Analogous to the repeated sampling properties for complete randomization (11, 20), we evaluate the repeated sampling properties of the difference-in-means estimator when rerandomization is used, where all potential outcomes and covariates are regarded as fixed quantities and all randomness arises solely from the random treatment assignments. The geometry of rerandomization reveals non-Gaussian asymptotic distributions, which serve as the foundation for constructing large-sample confidence intervals for average causal effects. Furthermore, we compare the lengths of quantile ranges of the asymptotic distributions of the difference-in-means estimator under rerandomization and complete randomization, extending Morgan and Rubin’s (17, 21) comparison of their sampling variances.

Framework, Notation, and Basic Results

Covariate Imbalance and Rerandomization.

Inferring the causal effect of some binary treatment on an outcome $Y$ is of central interest in many studies. We consider an experiment with $n$ units, with $n_1$ assigned to treatment and $n_0$ assigned to control, $n=n_1+n_0$. Before conducting the experiment, we collect $K$ covariates with values ${\mathit{X}}_i={\left(X_{1i},X_{2i},\dots ,X_{Ki}\right)}^{\prime }$ for the $i$th unit, which can possibly include transformations of basic covariates and their interactions. Let $Z_i$ be the treatment indicator for unit $i$ ($Z_i=1$ if the active treatment level; $Z_i=0$ if the control level) and $\mathit{Z}={\left(Z_1,Z_2,\dots ,Z_n\right)}^{\prime }$ be the treatment assignment vector with $n_1\equiv {\sum }_{i=1}^n{Z}_i$ and $n_0\equiv {\sum }_{i=1}^n\left(1-Z_i\right)$. In a completely randomized experiment (CRE), $n_1$ and $n_0$ are fixed, and the distribution of $\mathit{Z}$ is such that each possible value, $\mathit{z}={\left(z_1,\dots ,z_n\right)}^{\prime }$, of $\mathit{Z}$ has probability $n_1!n_0!/n!.$ The difference-in-means vector of the covariates between treatment and control groups is ${\widehat{\mathit{\tau }}}_{\mathit{X}}=n_1^{-1}{\sum }_{i=1}^n{Z}_i{\mathit{X}}_i-n_0^{-1}{\sum }_{i=1}^n\left(1-Z_i\right){\mathit{X}}_i.$ Although on average ${\widehat{\mathit{\tau }}}_{\mathit{X}}$ has mean zero over all $n!/\left(n_1!n_0!\right)$ randomizations, for any realized value of $\mathit{Z}$, imbalancedness in covariate distributions between treatment groups often occurs. For 10 uncorrelated covariates, with probability $40\%$, at least one of the absolute values of the $t$ statistics comparing covariate means will be larger than 1.96, the 0.975 quantile of the standard Gaussian distribution (17).

When covariate imbalance arises in a drawn allocation, it is reasonable to discard that allocation and draw another until some a priori covariate balance criterion is satisfied. This is rerandomization, an intuitive experimental design tool apparently personally advocated by R. A. Fisher (discussion in ref. 16) and formally discussed by Morgan and Rubin (17).

In general, rerandomization entails the following steps: (i) Collect covariate data; (ii) specify a balance criterion to determine whether a randomization is acceptable or not; (iii) randomize the units to treatment and control groups; (iv) if the balance criterion is satisfied, proceed to step v, and otherwise, return to step iii; (v) conduct the experiment using the final randomization obtained in step iv; and (vi) analyze the data, taking into account the rerandomization used in steps ii–iv.

Although the balance criterion in step ii can be general, Morgan and Rubin (17) suggested using the Mahalanobis distance between covariate means in treatment and control groups, and they (21) suggested considering tiers of covariates according to their presumed importance in predicting the outcomes in this experiment. We discuss these two types of rerandomization in detail and apposite statistical inference after these rerandomizations as implied by step vi. We then extend the theory to general rerandomizations in SI Appendix, section A1.

Potential Outcomes and Definitions of Finite Population Quantities.

We use the potential outcomes framework to define causal effects and let $Y_i\left(1\right)$ and $Y_i\left(0\right)$ denote the potential outcomes of unit $i$ under active treatment and control, respectively. On the difference scale, the individual causal effect for unit $i$ is ${\tau }_i=Y_i\left(1\right)-Y_i\left(0\right)$, and the average causal effect for the finite population of $n$ units is $\tau ={\sum }_{i=1}^n{\tau }_i/n$. Let $Ȳ\left(z\right)={\sum }_{i=1}^n{Y}_i\left(z\right)/n$ be the finite population average of potential outcomes under treatment arm $z$ and $\overline{\mathit{X}}={\sum }_{i=1}^n{\mathit{X}}_i/n$ be the finite population average of covariates. Let $S_{Y\left(z\right)}^2={\sum }_{i=1}^n{\left\{Y_i\left(z\right)-Ȳ\left(z\right)\right\}}^2/\left(n-1\right)$ be the finite population variance (with divisor $n-1$) of the potential outcomes under treatment arm $z$, $S_{\tau }^2={\sum }_{i=1}^n{\left({\tau }_i-\tau \right)}^2/\left(n-1\right)$ be the finite population variance of the individual causal effects, $S_{Y\left(z\right),\mathit{X}}=S_{\mathit{X},Y\left(z\right)}^{\prime }={\sum }_{i=1}^n\left\{Y_i\left(z\right)-Ȳ\left(z\right)\right\}{\left({\mathit{X}}_i-\overline{\mathit{X}}\right)}^{\prime }/\left(n-1\right)$ be the finite population covariance between potential outcomes and covariates, and ${\mathit{S}}_{\mathit{X}}^2={\sum }_{i=1}^n\left({\mathit{X}}_i-\overline{\mathit{X}}\right){\left({\mathit{X}}_i-\overline{\mathit{X}}\right)}^{\prime }/\left(n-1\right)$ be the finite population covariance matrix of covariates. These fixed quantities depend on $n$ implicitly, but do not depend on the randomization or rerandomization scheme.

Repeated Sampling Inference in a CRE.

The observed outcome for unit $i$ is $Y_i=Z_i{Y}_i\left(1\right)+\left(1-Z_i\right)Y_i\left(0\right)$, a function of treatment assignment and potential outcomes. In a CRE, Neyman (20) showed that, for estimating $\tau$, the difference-in-means estimator $\widehat{\tau }=n_1^{-1}{\sum }_{i=1}^n{Z}_i{Y}_i-n_0^{-1}{\sum }_{i=1}^n\left(1-Z_i\right)Y_i$ is unbiased (the expectation of $\widehat{\tau }$ over all possible randomizations is $\tau$) and obtained its sampling variance over all randomizations for constructing a large-sample confidence interval for $\tau$. However, Neyman’s (20) interval is not accurate if rerandomization is used.

Let $r_1=n_1/n$ and $r_0=n_0/n$ be the proportions of units receiving treatment and control. According to the finite population central limit theorem (19), under some regularity conditions, the large-$n$ sampling distribution, over all randomizations, of $\sqrt{n}\left(\widehat{\tau }-\tau ,{\widehat{\mathit{\tau }}}_{\mathit{X}}^{\prime }\right)$ is Gaussian with mean zero and covariance matrix

$$\begin{array}{ccc}\hfill & \hfill & \mathit{V}=\left(\begin{array}{cc}{V}_{\tau \tau }\hfill & \hfill {\mathit{V}}_{\tau \mathit{x}}\\ {\mathit{V}}_{\mathit{x}\tau }\hfill & \hfill {\mathit{V}}_{\mathit{x}\mathit{x}}\end{array}\right)\hfill \\ \hfill & \hfill & \hspace{1em}=\left(\begin{array}{cc}{r}_1^{-1}{S}_{Y\left(1\right)}^2+r_0^{-1}{S}_{Y\left(0\right)}^2-S_{\tau }^2\hfill & \hfill r_1^{-1}{\mathit{S}}_{Y\left(1\right),\mathit{X}}+r_0^{-1}{\mathit{S}}_{Y\left(0\right),\hspace{0.17em}\mathit{X}}\hfill \\ r_1^{-1}{\mathit{S}}_{\mathit{X},Y\left(1\right)}+r_0^{-1}{\mathit{S}}_{\mathit{X},Y\left(0\right)}\hfill & \hfill {\left(r_1{r}_0\right)}^{-1}{\mathit{S}}_{\mathit{X}}^2\hfill \end{array}\right).\hfill \end{array}$$

We are conducting randomization-based inference, where all of the covariates and potential outcomes are fixed numbers, and randomness comes solely from the treatment assignment. We embed $n$ units into an infinite sequence of finite populations with increasing sizes, and a sufficient condition for the asymptotic Gaussianity of $\sqrt{n}\left(\widehat{\tau }-\tau ,{\widehat{\mathit{\tau }}}_{\mathit{X}}^{\prime }\right)$ is as follows (19).

Condition 1:

As $n\to \infty$, for $z=\mathrm{0,1}$, (i) $r_z$, the proportion of units under treatment arm $z$, has positive limits; (ii) the finite population variances and covariances $S_{Y\left(z\right)}^2,S_{\tau }^2,{\mathit{S}}_{\mathit{X}}^2$ and $S_{\mathit{X},Y\left(z\right)}$ have finite limiting values, and the limit of ${\mathit{S}}_{\mathit{X}}^2$ is nonsingular; and (iii) $\underset{1\le i\le n}{max}|Y_i\left(z\right)-Ȳ\left(z\right){|}^2/n\to 0$ and $\underset{1\le i\le n}{max}\Vert {\mathit{X}}_i-{\overline{\mathit{X}}\Vert }_2^2/n\to 0.$

We introduce the notation $\stackrel{.}{\sim }$ for two sequences of random vectors converging weakly to the same distribution. Therefore, under the CRE and Condition 1, $\sqrt{n}\left(\widehat{\tau }-\tau ,{\widehat{\mathit{\tau }}}_{\mathit{X}}^{\prime }\right)\dot{\sim }\left(A,\mathit{B}\prime \right),$ where $\left(A,\mathit{B}\prime \right)$ is a random vector from $\mathcal{N}\left(\mathbf{0},\mathit{V}\right)$ (19).

Rerandomization Using the Mahalanobis Distance

Mahalanobis Distance.

The Mahalanobis distance between the covariate means in treatment and control groups is

$$M={\widehat{\mathit{\tau }}}_{\mathit{X}}^{\prime }{\left\{\text{Var}\left({\widehat{\mathit{\tau }}}_{\mathit{X}}\right)\right\}}^{-1}{\widehat{\mathit{\tau }}}_{\mathit{X}}={\left(\sqrt{n}{\widehat{\mathit{\tau }}}_{\mathit{X}}\right)}^{\prime }\hspace{0.17em}{\mathit{V}}_{\mathit{x}\mathit{x}}^{-1}\left(\sqrt{n}{\widehat{\mathit{\tau }}}_{\mathit{X}}\right),$$

where ${\mathit{V}}_{\mathit{x}\mathit{x}}={\left(r_1{r}_0\right)}^{-1}{\mathit{S}}_{\mathit{X}}^2$ is a fixed and known $K\times K$ matrix in our setting. A rerandomization scheme proposed by Morgan and Rubin (17) accepts only those randomizations with the Mahalanobis distance less than or equal to $a$, a prespecified threshold. Let $\mathcal{M}$ denote the event that a treatment assignment $\mathit{Z}$ is accepted; that is, $M\le a$. Below we use rerandomization using the Mahalanobis distance (ReM) to denote rerandomization using this criterion, which, as a design, depends on both the covariates and the threshold $a$.

When we allow transformations and interactions of $\mathit{X}$, ReM can incorporate a wide class of rerandomization schemes. For small sample sizes, there may not exist any randomization satisfying some balance criterion. However, according to the finite population central limit theorem (19), the acceptance probability of a randomization is asymptotically $p_a=P\left({\chi }_K^2\le a\right)$. Therefore, for relatively large sample sizes, there usually exist many randomizations satisfying the balance criterion. In practice, we want to choose the asymptotic acceptance probability to be small, e.g., $p_a=0.001$. We comment on this issue in Discussion.

Multiple Correlation Between $\widehat{\tau }$ and ${\widehat{\mathit{\tau }}}_{\mathit{X}}$.

The sampling distribution of $\widehat{\tau }$ under ReM depends on the squared multiple correlation between $\widehat{\tau }$ and ${\widehat{\mathit{\tau }}}_{\mathit{X}}$ under the CRE, which is also the proportion of the variance of $\widehat{\tau }$ explained by ${\widehat{\mathit{\tau }}}_{\mathit{X}}$ in linear projection: $R^2=\text{Cov}\left(\widehat{\tau },{\widehat{\mathit{\tau }}}_{\mathit{X}}\right)\text{Var}{\left({\widehat{\mathit{\tau }}}_{\mathit{X}}\right)}^{-1}\text{Cov}\left({\widehat{\mathit{\tau }}}_{\mathit{X}},\widehat{\tau }\right)/\text{Var}\left(\widehat{\tau }\right)={\mathit{V}}_{\tau \mathit{x}}{\mathit{V}}_{\mathit{x}\mathit{x}}^{-1}{\mathit{V}}_{\mathit{x}\tau }/{\mathit{V}}_{\tau \tau }$. Define the variance of the linear projection of $Y\left(z\right)$ on $\mathit{X}$ as $S_{Y\left(z\right)\mid \mathit{X}}^2={\mathit{S}}_{Y\left(z\right),\hspace{0.17em}\mathit{X}}{\left({\mathit{S}}_{\mathit{X}}^2\right)}^{-1}{\mathit{S}}_{\mathit{X},Y\left(z\right)}$ for $z=\mathrm{0,1}$. We similarly define $S_{\tau \mid \mathit{X}}^2$, the variance of the linear projection of $\tau$ on $\mathit{X}$.

Proposition 1.

$R^2$ can be expressed in terms of the variances of the potential outcomes and of their projections on $\mathit{X}$:

$$R^2=\frac{r_1^{-1}{S}_{Y\left(1\right)\mid \mathit{X}}^2+r_0^{-1}{S}_{Y\left(0\right)\mid \mathit{X}}^2-S_{\tau \mid \mathit{X}}^2}{r_1^{-1}{S}_{Y\left(1\right)}^2+r_0^{-1}{S}_{Y\left(0\right)}^2-S_{\tau }^2}.$$

When the causal effect is additive, $S_{\tau }^2=0$, ${\mathit{S}}_{\tau ,\mathit{X}}=0$, and ${\mathit{S}}_{Y\left(1\right),\mathit{X}}={\mathit{S}}_{Y\left(0\right),\mathit{X}}$, and then $R^2=S_{Y\left(0\right)\mid \mathit{X}}^2/S_{Y\left(0\right)}^2$ is the squared multiple correlation between $\mathit{X}$ and $Y\left(0\right)$.

Asymptotic Sampling Distribution of $\widehat{\tau }$ Under ReM.

Simply stated, $\sqrt{n}\left(\widehat{\tau }-\tau \right)$ has two parts: the part unrelated to the covariates, which we call ${\epsilon }_0$, and is thus unaffected by rerandomization, and the other part related to the covariates, which we call $L_{K,a}$, and is thus affected by rerandomization. Therefore, the asymptotic distribution of $\widehat{\tau }$ is a linear combination of two independent random variables: ${\epsilon }_0\sim \mathcal{N}\left(\mathrm{0,1}\right)$ is a standard Gaussian random variable, and $L_{K,a}\sim D_1\mid {\mathit{D}}^{\prime }\mathit{D}\le a$, where $\mathit{D}={\left(D_1,\dots ,D_K\right)}^{\prime }\sim \mathcal{N}\left(\mathbf{0},{\mathit{I}}_K\right)$.

Theorem 1.

Under ReM and Condition 1,

$$\sqrt{n}\left(\widehat{\tau }-\tau \right)\mid \mathcal{M}\dot{\sim }\hspace{0.17em}\sqrt{V_{\tau \tau }}\left(\sqrt{1-R^2}{\cdot \epsilon }_0+\sqrt{R^2}\cdot L_{K,a}\right),$$ [1]

where ${\epsilon }_0$ is independent of $L_{K,a}$.

The coefficients of the linear combination are functions of $R^2$, which measures the association between the potential outcomes and the covariates. When $R^2=0$, the right-hand side of [1] becomes a Gaussian random variable, the same as the asymptotic distribution of $\sqrt{n}\left(\widehat{\tau }-\tau \right)$ in Repeated Sampling Inference in a CRE. Importantly, the definition of $R^2$ is based on linear projections instead of linear models of the potential outcomes. Our asymptotic theory is based on the distribution of the randomization without imposing any modeling assumptions on the potential outcomes.

Representation of the Asymptotic Distribution Under ReM.

The asymptotic distribution in [1] involves a random variable $L_{K,a}$ that does not appear in standard statistical problems. The spherical symmetry of the standard Gaussian vector allows us to represent $L_{K,a}$ using some known distributions, which allows for easy simulation of $L_{K,a}.$

Let ${\chi }_{K,a}^2\sim {\chi }_K^2\mid {\chi }_K^2\le a$ be a truncated ${\chi }^2$ random variable, $U_K$ be the first coordinate of the uniform random vector over the $\left(K-1\right)$-dimensional unit sphere, $S$ be a random sign taking $\pm 1$ with probability $1/2$, and ${\beta }_K\sim \text{Beta}\left(1/2,\left(K-1\right)/2\right)$ be a Beta random variable degenerating to point mass at 1 when $K=1$.

Proposition 2.

$L_{K,a}$ can be represented as

$$L_{K,a}\sim D_1\mid {\mathit{D}}^{\prime }\mathit{D}\le a\sim {\chi }_{K,a}{U}_K\sim {\chi }_{K,a}S\sqrt{{\beta }_K},$$ [2]

where $\left({\chi }_{K,a},U_K\right)$ are mutually independent, and $\left({\chi }_{K,a},S,{\beta }_K\right)$ are jointly independent. $L_{K,a}$ is symmetric and unimodal around zero, with $\text{Var}\left(L_{K,a}\right)=v_{K,a}=P\left({\chi }_{K+2}^2\le a\right)/P\left({\chi }_K^2\le a\right)<1$.

Because both ${\epsilon }_0$ and $L_{K,a}$ are symmetric and both are unimodal at zero, their linear combination is also symmetric and unimodal at zero according to Wintner’s (22) theorem. The same is true for the asymptotic distribution of $\sqrt{n}\left(\widehat{\tau }-\tau \right)$ in [1]. Moreover, the unimodal property plays an important role in the conservativeness of confidence intervals, discussed shortly for ReM. The representation in [2] allows for easy simulation of $L_{K,a}$, as well as the asymptotic distribution of $\sqrt{n}\left(\widehat{\tau }-\tau \right)$ in [1], which is relevant for statistical inference discussed later.

In SI Appendix, section A2, we give more detailed explanations regarding the geometry and the shape of the asymptotic distribution in [1].

Asymptotic Unbiasedness, Sampling Variance, and Quantile Ranges.

Theorem 1 characterizes the asymptotic behavior of $\widehat{\tau }$ over ReM, which immediately implies the following conclusions.

First, the asymptotic distribution in [1] is symmetric around 0, implying that $\widehat{\tau }$ is asymptotically unbiased for $\tau$. Let ${\mathbb{E}}_{\text{a}}\left(\cdot \right)$ denote the expectation of the asymptotic sampling distribution of a sequence of random vectors.

Corollary 1.

Under ReM and Condition 1, ${\mathbb{E}}_{\text{a}}\left\{\sqrt{n}\left(\widehat{\tau }-\tau \right)|\mathcal{M}\right\}=0.$

Morgan and Rubin (17) gave a counterexample showing that, in an experiment with unequal treatment group sizes, $\widehat{\tau }$ can be biased for $\tau$ over ReM. Our result confirms a conjecture in ref. 21 that the bias is often small with large samples. Corollary 1 extends their theorem 2.1 (17) and ensures the asymptotic unbiasedness of $\widehat{\tau }$ for experiments with any ratio of $n_1/n_0$.

Covariates, whether observed or unobserved, are variables unaffected by the treatments. Therefore, the average causal effect on any covariate is 0, and Corollary 1 implies that any covariate asymptotically has the same means under treatment and control.

Furthermore, from Proposition 2 and Theorem 1, we can calculate the asymptotic sampling variances of ${\widehat{\mathit{\tau }}}_{\mathit{X}}$ and $\widehat{\tau }$ and the percentage reductions in asymptotic sampling variances (PRIASV) under ReM compared with the CRE. Recalling that $v_{K,a}=P\left({\chi }_{K+2}^2\le a\right)/P\left({\chi }_K^2\le a\right)$, we summarize the results in Corollary 2.

Corollary 2.

Under ReM and Condition 1, the asymptotic sampling covariance of $\sqrt{n}{\widehat{\mathit{\tau }}}_{\mathit{X}}$ is $v_{K,a}{\mathit{V}}_{\mathit{x}\mathit{x}},$ and the PRIASV of any component of $\sqrt{n}{\widehat{\mathit{\tau }}}_{\mathit{X}}$ is $1-v_{K,a}.$ The asymptotic sampling variance of $\sqrt{n}\left(\widehat{\tau }-\tau \right)$ is $V_{\tau \tau }\left\{1-\left(1-v_{K,a}\right)R^2\right\},$ and the PRIASV of $\sqrt{n}\left(\widehat{\tau }-\tau \right)$ is $\left(1-v_{K,a}\right)R^2.$

Rigorously, the asymptotic sampling covariance and variance of ${\widehat{\mathit{\tau }}}_{\mathit{X}}$ and $\widehat{\tau }$ should be the limits of $v_{K,a}{\mathit{V}}_{\mathit{x}\mathit{x}}$ and $V_{\tau \tau }\left\{1-\left(1-v_{K,a}\right)R^2\right\}$ in the sequence of finite populations. However, for descriptive convenience, we omit these limit signs when discussing the expectation and covariance of asymptotic sampling distributions. When $a$ is close to 0, that is, when the asymptotic acceptance probability is small, the asymptotic sampling variance $V_{\tau \tau }\left\{1-\left(1-v_{K,a}\right)R^2\right\}$ reduces to $V_{\tau \tau }\left(1-R^2\right)$, which is identical to the asymptotic sampling variance of the regression-adjusted estimator under the CRE (18). Therefore, rerandomization accomplishes covariate adjustment in the design stage, whereas regression accomplishes covariate adjustment in the analysis stage.

When the causal effect is additive, $R^2$ equals the finite population squared multiple correlation between $\mathit{X}$ and $Y\left(0\right)$. Therefore, Corollary 2 is an asymptotic extension of theorem 3.2 in Morgan and Rubin (17).

Under ReM, in addition to the sampling variance reduction result concerning $\widehat{\tau }$ in Corollary 2, we consider the reduction in the length of the $\left(1-\alpha \right)$ quantile range of $\widehat{\tau }$ compared with that under the CRE. We choose the length of the $\left(1-\alpha \right)$ quantile range, because of its connection to constructing confidence intervals as discussed shortly.

Let $z_{\xi }$ be the $\xi$th quantile of a standard Gaussian distribution. Let ${\nu }_{\xi }\left(R^2,p_a,K\right)$ be the $\xi$th quantile of $\sqrt{1-R^2}{\cdot \epsilon }_0+\sqrt{R^2}\cdot L_{K,a}$, with ${\nu }_{\xi }\left(0,p_a,K\right)=z_{\xi }$. Because $p_a$ and $K$ are usually known by design, we write ${\nu }_{\xi }\left(R^2,p_a,K\right)$ as ${\nu }_{\xi }\left(R^2\right)$ for notational simplicity. Under ReM, the $\left(1-\alpha \right)$ quantile range of the asymptotic distribution of $\sqrt{n}\left(\widehat{\tau }-\tau \right)$ is

$${\text{QR}}_{\alpha }\left(V_{\tau \tau },R^2\right)=\left[{\nu }_{\alpha /2}\left(R^2\right)\sqrt{V_{\tau \tau }},\hspace{0.17em}\hspace{0.17em}{\nu }_{1-\alpha /2}\left(R^2\right)\sqrt{V_{\tau \tau }}\right],$$ [3]

and the corresponding quantile range under the CRE is

$${\text{QR}}_{\alpha }\left(V_{\tau \tau },0\right)=\left[z_{\alpha /2}\sqrt{V_{\tau \tau }},\hspace{0.17em}\hspace{0.17em}{z}_{1-\alpha /2}\sqrt{V_{\tau \tau }}\right].$$ [4]

Theorem 2.

Under Condition 1, the length of the $\left(1-\alpha \right)$ quantile range of the asymptotic sampling distribution of $\sqrt{n}\left(\widehat{\tau }-\tau \right)$ under ReM is less than or equal to that under the CRE, with the difference nondecreasing in $R^2$ and nonincreasing in $p_a$ and $K$.

Sampling Variance Estimation and Confidence Intervals.

Asymptotic sampling variance and quantile ranges for $\widehat{\tau }$ depend on $V_{\tau \tau }$ and $R^2$, which are determined by the covariances among potential outcomes and covariates. To obtain a sampling variance estimator and to construct an asymptotic confidence interval for $\tau$, we need to estimate these variances and covariances. Let $s_{Y\left(z\right)}^2$, $s_{Y\left(z\right)\mid \mathit{X}}^2$, and ${\mathit{s}}_{Y\left(z\right),\mathit{X}}$ be the sample variance of $Y$, sample variance of linear projection of $Y$ on $\mathit{X}$, and sample covariance between $Y$ and $\mathit{X}$ in treatment arm $z.$ We show in SI Appendix, section A4 that under ReM they are consistent for their population analogues. Therefore, we can then estimate $S_{\tau \mid \mathit{X}}^2$ by

$$s_{\tau \mid \mathit{X}}^2=\left({\mathit{s}}_{Y\left(1\right),\hspace{0.17em}\mathit{X}}-{\mathit{s}}_{Y\left(0\right),\hspace{0.17em}\mathit{X}}\right){\left({\mathit{S}}_{\mathit{X}}^2\right)}^{-1}\left({\mathit{s}}_{\mathit{X},Y\left(1\right)}-{\mathit{s}}_{\mathit{X},Y\left(0\right)}\right)$$

and $V_{\tau \tau }$ by (23)

$${\widehat{V}}_{\tau \tau }=r_1^{-1}{s}_{Y\left(1\right)}^2+r_0^{-1}{s}_{Y\left(0\right)}^2-s_{\tau \mid \mathit{X}}^2.$$

We can then estimate $R^2$ by

$${\widehat{R}}^2={\widehat{V}}_{\tau \tau }^{-1}\left\{r_1^{-1}{s}_{Y\left(1\right)\mid \mathit{X}}^2+r_0^{-1}{s}_{Y\left(0\right)\mid \mathit{X}}^2-s_{\tau \mid \mathit{X}}^2\right\}.$$ [5]

We set ${\widehat{R}}^2$ to be 0 if the estimator in [5] is negative.

According to Corollary 2, we can estimate the asymptotic sampling variance of $\widehat{\tau }$ by ${\widehat{V}}_{\tau \tau }\left\{1-\left(1-v_{K,a}\right){\widehat{R}}^2\right\}/n,$ and according to [3], we can construct a large sample $\left(1-\alpha \right)$ confidence interval for $\tau$ using $\widehat{\tau }-{\text{QR}}_{\alpha }\left({\widehat{V}}_{\tau \tau },{\widehat{R}}^2\right)/\sqrt{n}.$ Not surprisingly, similar to Neyman’s (20) analysis of the CRE, unless the residual from the linear projection of the individual causal effect on the covariates is constant, the above sampling variance estimator and the associated confidence interval are both asymptotically conservative, in the sense that the probability limit of the variance estimator is larger than or equal to the actual sampling variance, and the limit of coverage probability of the confidence interval is larger than or equal to $\left(1-\alpha \right)$.

Moreover, the sampling variance estimator is smaller than Neyman’s (20) sampling variance estimator for the CRE, and the confidence interval is shorter than Neyman’s (20) confidence interval for the CRE. Therefore, if we conduct ReM in the design stage but analyze data as in the CRE, the consequential sampling variance estimator and confidence intervals will be overly conservative.

The above results are all intuitive, and we present the algebraic details for the proofs of these results in SI Appendix, section A4. Interestingly, as shown in SI Appendix, section A4, we do not need conditions beyond Condition 1 to ensure the asymptotic properties of the sampling variance estimator and the confidence intervals.

Rerandomization with Tiers of Covariates

Mahalanobis Distance with Tiers of Covariates.

When covariates are thought to have different levels of importance for the outcomes, Morgan and Rubin (21) proposed rerandomization using the Mahalanobis distance with differing criteria for different tiers of covariates. We partition the covariates into $T$ tiers indexed by $t=1,\dots ,T$ with decreasing importance, with $k_t$ covariates in tier $t$. Let ${\mathit{X}}_i=\left({\mathit{X}}_i\left[1\right],\dots ,{\mathit{X}}_i\left[T\right]\right)$, where ${\mathit{X}}_i\left[t\right]$ denotes the covariates in tier $t$. Define ${\mathit{X}}_i\left[\overline{t}\right]=\left({\mathit{X}}_i\left[1\right],\dots ,{\mathit{X}}_i\left[t\right]\right)$, the covariates in the first $t$ tiers. Following Morgan and Rubin (21), let ${\mathit{S}}_{\mathit{X}\left[\overline{t-1}\right]}^2$ be the finite population covariance matrix of the covariates in the first $t-1$ tiers and ${\mathit{S}}_{\mathit{X}\left[t\right],\mathit{X}\left[\overline{t-1}\right]}$ be the finite population covariance between $\mathit{X}\left[t\right]$ and $\mathit{X}\left[\overline{t-1}\right]$. We first apply a block-wise Gram–Schmidt orthogonalization to the covariates to create the orthogonalized covariates. Let ${\mathit{E}}_i\left[1\right]={\mathit{X}}_i\left[1\right]$, and for $2\le t\le T$, let

$${\mathit{E}}_i\left[t\right]={\mathit{X}}_i\left[t\right]-{\mathit{S}}_{\mathit{X}\left[t\right],\mathit{X}\left[\overline{t-1}\right]}{\left({\mathit{S}}_{\mathit{X}\left[\overline{t-1}\right]}^2\right)}^{-1}\hspace{0.17em}{\mathit{X}}_i\left[\overline{t-1}\right],$$

where ${\mathit{E}}_i\left[t\right]$ is the residual of the projection of the covariates ${\mathit{X}}_i\left[t\right]$ in tier $t$ onto the space spanned by the covariates in previous tiers. Let ${\mathit{E}}_i=\left({\mathit{E}}_i\left[1\right],\dots ,{\mathit{E}}_i\left[T\right]\right)$. Let ${\widehat{\mathit{\tau }}}_{\mathit{E}\left[t\right]}$ be the difference-in-means vector of ${\mathit{E}}_i\left[t\right]$ between treatment and control groups and ${\mathit{S}}_{\mathit{E}\left[t\right]}^2$ be the finite population covariance matrix of ${\mathit{E}}_i\left[t\right]$. The Mahalanobis distance in tier $t$ is $M_t=\left(n_1{n}_0\right)/n\cdot {\widehat{\mathit{\tau }}}_{\mathit{E}\left[t\right]}^{\prime }{\left({\mathit{S}}_{\mathit{E}\left[t\right]}^2\right)}^{-1}{\widehat{\mathit{\tau }}}_{\mathit{E}\left[t\right]},$ and rerandomization using the Mahalanobis distance with tiers of covariates (ReMT) accepts those treatment assignments with $M_t\le a_t$, where the $a_t$s are predetermined constants $\left(1\le t\le T\right)$. If $T=1$, then ReMT is simply ReM. Let $\mathcal{T}$ denote the event that a treatment assignment $\mathit{Z}$ is accepted under ReMT. The theory below extends Morgan and Rubin (21), using the concepts from our Rerandomization Using the Mahalanobis Distance section.

Multiple Correlation Between $\widehat{\tau }$ and ${\widehat{\mathit{\tau }}}_{\mathit{E}\left[t\right]}$.

Let ${\rho }_t^2$ be the squared multiple correlation between $\widehat{\tau }$ and the difference-in-means vector of the orthogonalized covariates in tier $t\hspace{0.17em}{\widehat{\mathit{\tau }}}_{\mathit{E}\left[t\right]}$: ${\rho }_t^2=\text{Cov}\left(\widehat{\tau },{\widehat{\mathit{\tau }}}_{\mathit{E}\left[t\right]}\right)\text{Var}{\left({\widehat{\mathit{\tau }}}_{\mathit{E}\left[t\right]}\right)}^{-1}\text{Cov}\left({\widehat{\mathit{\tau }}}_{\mathit{E}\left[t\right]},\widehat{\tau }\right)/\text{Var}\left(\widehat{\tau }\right).$ Define the finite population variance of the projection of $Y\left(z\right)$ on $\mathit{E}\left[t\right]$ as $S_{Y\left(z\right)\mid \mathit{E}\left[t\right]}^2={\mathit{S}}_{Y\left(z\right),\mathit{E}\left[t\right]}{\left({\mathit{S}}_{\mathit{E}\left[t\right]}^2\right)}^{-1}{\mathit{S}}_{\mathit{E}\left[t\right],Y\left(z\right)}$ for $z=\mathrm{0,1}$, where ${\mathit{S}}_{Y\left(z\right),\mathit{E}\left[t\right]}$ is the finite population covariance between potential outcomes and orthogonalized covariates in tier $t$. We can similarly define $S_{\tau \mid \mathit{E}\left[t\right]}^2$. According to Proposition 1,

$${\rho }_t^2=\frac{r_1^{-1}{S}_{Y\left(1\right)\mid \mathit{E}\left[t\right]}^2+r_0^{-1}{S}_{Y\left(0\right)\mid \mathit{E}\left[t\right]}^2-S_{\tau \mid \mathit{E}\left[t\right]}^2}{r_1^{-1}{S}_{Y\left(1\right)}^2+r_0^{-1}{S}_{Y\left(0\right)}^2-S_{\tau }^2},\hspace{1em}\left(1\le t\le T\right).$$

When the causal effect is additive, ${\rho }_t^2=S_{Y\left(0\right)\mid \mathit{E}\left[t\right]}^2/S_{Y\left(0\right)}^2$ reduces to the squared multiple correlation between $Y\left(0\right)$ and $\mathit{E}\left[t\right]$. For descriptive simplicity, we introduce ${\rho }_{T+1}^2=1-{\sum }_{t=1}^T{\rho }_t^2=1-R^2$ for later discussion.

Asymptotic Sampling Distribution of $\widehat{\tau }$ Under ReMT.

Intuitively, $\sqrt{n}\left(\widehat{\tau }-\tau \right)$ can be decomposed into $\left(T+1\right)$ parts: the part unrelated to covariates and the $T$ projections onto the spaces spanned by the orthogonalized covariates in $T$ tiers. Due to their construction, these $\left(T+1\right)$ parts are orthogonal to each other, and the constraint for balance in tier $t$ affects only the $t$th projection.

As earlier, let ${\epsilon }_0\sim \mathcal{N}\left(\mathrm{0,1}\right)$, and extending earlier notation using the subscript $t$, let $L_{k_t,a_t}\sim D_{t1}\mid {\mathit{D}}_t^{\prime }{\mathit{D}}_t\le a_t$, where ${\mathit{D}}_t=\left(D_{t1},\dots ,D_{t{k}_t}\right)\sim \mathcal{N}\left(\mathbf{0},{\mathit{I}}_{k_t}\right)$ for $1\le t\le T$.

Theorem 3.

Under ReMT and Condition 1,

$$\sqrt{n}\left(\widehat{\tau }-\tau \right)\mid \mathcal{T}\hspace{0.17em}\dot{\sim }\hspace{0.17em}\sqrt{V_{\tau \tau }}\left({\rho }_{T+1}{\cdot \epsilon }_0+\sum _{t=1}^T{\rho }_t\cdot L_{k_t,a_t}\right),$$ [6]

where $\left({\epsilon }_0,L_{k_1,a_1},\dots ,L_{k_T,a_T}\right)$ are jointly independent.

In [6], ${\epsilon }_0$ is the part of $\sqrt{n}\left(\widehat{\tau }-\tau \right)$ that is unrelated to the covariates, and $L_{k_t,a_t}$ is the part related to the orthogonalized covariates ${\mathit{E}}_i\left[t\right]$ in tier $t$. According to Proposition 2, the distribution in Theorem 3 is easy to simulate.

Asymptotic Unbiasedness, Sampling Variance, and Quantile Ranges.

Theorem 3 characterizes the asymptotic behavior of $\sqrt{n}\left(\widehat{\tau }-\tau \right)$ under ReMT, which extends Morgan and Rubin (21) as follows.

First, the asymptotic distribution in [6] is symmetric around 0, implying that $\widehat{\tau }$ is asymptotically unbiased for $\tau$. Therefore, all observed or unobserved covariates have asymptotically zero difference in means.

Corollary 3.

Under ReMT and Condition 1, ${\mathbb{E}}_{\text{a}}\left\{\sqrt{n}\left(\widehat{\tau }-\tau \right)\mid \mathcal{T}\right\}=0.$

The asymptotic sampling covariance of ${\widehat{\mathit{\tau }}}_{\mathit{X}}$ under ReMT has a complicated but conceptually obvious form, and we give it in SI Appendix, section A3. Below we present only the PRIASV of $\widehat{\tau }$; the PRIASVs for covariates are special cases of the same corollary because covariates are formally “outcomes” unaffected by the treatment. Recall that $v_{k_t,a_t}=P\left({\chi }_{k_t+2}^2\le a_t\right)/P\left({\chi }_{k_t}^2\le a_t\right)$.

Corollary 4.

Under ReMT and Condition 1, the asymptotic sampling variance of $\sqrt{n}\left(\widehat{\tau }-\tau \right)$ is $V_{\tau \tau }\left\{1-{\sum }_{t=1}^T\left(1-v_{k_t,a_t}\right){\rho }_t^2\right\}$, and the PRIASV of $\sqrt{n}\left(\widehat{\tau }-\tau \right)$ is ${\sum }_{t=1}^T\left(1-v_{k_t,a_t}\right){\rho }_t^2.$

When the causal effect is additive, ${\rho }_t^2$ becomes the squared multiple correlation between $\mathit{E}\left[t\right]$ and $Y\left(0\right)$. Therefore, Corollary 4 is an asymptotic extension of Morgan and Rubin’s theorem 4.2 (21). When the thresholds $a_t$s are close to zero, the asymptotic sampling variance $V_{\tau \tau }\left\{1-{\sum }_{t=1}^T\left(1-v_{k_t,a_t}\right){\rho }_t^2\right\}$ reduces to $V_{\tau \tau }\left(1-{\sum }_{t=1}^T{\rho }_t^2\right)=V_{\tau \tau }\left(1-R^2\right)$, which is identical to that of the regression-adjusted estimator under the CRE (18).

We now compare the quantile range under ReMT to that under the CRE. Let ${\nu }_{\xi }\left({\rho }_1^2,{\rho }_2^2,\dots ,{\rho }_T^2\right)$ be the $\xi$th quantile of ${\rho }_{T+1}{\epsilon }_0+{\sum }_{t=1}^T{\rho }_t{L}_{k_t,a_t}.$ Although ${\nu }_{\xi }\left({\rho }_1^2,{\rho }_2^2,\dots ,{\rho }_T^2\right)$ depends also on $p_{a_t}$ and $k_t$ ($1\le t\le K$), we suppress the dependence to avoid notational clutter. The $\left(1-\alpha \right)$ quantile range of the asymptotic distribution of $\sqrt{n}\left(\widehat{\tau }-\tau \right)$ under ReMT is

$$\begin{array}{ll}\hfill & {\text{QR}}_{\alpha }\left(V_{\tau \tau },{\rho }_1^2,\dots ,{\rho }_T^2\right)\hfill \\ \hfill & =\left[{\nu }_{\alpha /2}\left({\rho }_1^2,\dots ,{\rho }_T^2\right)\sqrt{V_{\tau \tau }},\hspace{0.17em}\hspace{0.17em}{\nu }_{1-\alpha /2}\left({\rho }_1^2,\dots ,{\rho }_T^2\right)\sqrt{V_{\tau \tau }}\right].\hfill \end{array}$$ [7]

The stronger the squared correlation is between the outcome and the orthogonalized covariates in tier $t$, the more reduction in quantile range when using ReMT rather than the CRE. The following Theorem 4 is intuitive.

Theorem 4.

Under Condition 1, the $\left(1-\alpha \right)$ quantile range of the asymptotic distribution of $\sqrt{n}\left(\widehat{\tau }-\tau \right)$ under ReMT is less than, or equal to, the range under the CRE, and the reduction in length is nondecreasing in ${\rho }_t^2$ and nonincreasing in $p_{a_t}$ and $k_t$, for all $1\le t\le T$.

Sampling Variance Estimation and Confidence Interval.

We can estimate $V_{\tau \tau }$ and ${\rho }_t^2\hspace{0.17em}\left(1\le t\le T\right)$ in the same way as in ReM, and we estimate ${\rho }_{T+1}^2$ by $1-{\widehat{R}}^2.$ In practice, we set ${\widehat{\rho }}_t^2\hspace{0.17em}\left(1\le t\le T\right)$ to 0 when it is negative due to sampling variability and standardize their sum to ${\widehat{R}}^2.$ According to Corollary 4 and [7], we can estimate the sampling variance of $\widehat{\tau }$ and construct confidence intervals for $\tau$ by replacing the unknown quantities with their point estimates. The sampling variance estimator is smaller than Neyman’s (20) sampling variance estimator for the CRE, and the confidence interval is shorter than Neyman’s (20) confidence interval for the CRE; both are asymptotically conservative in general, and only when the residual from the linear projection of the individual causal effect on the covariates is constant, are they asymptotically exact. Therefore, analyzing data from ReMT as if they arose from a CRE, the resulting sampling variance estimator and confidence intervals are overly conservative. These intuitive statements appear to require lengthy proofs, which are relegated to SI Appendix, section A4.

An Education Example with Tiers of Covariates

We illustrate our theory using the data from the Student Achievement and Retention Project (24), a randomized evaluation of academic services and incentives at one of the satellite campuses of a large Canadian university. A treatment group of 150 students was offered an array of support services and substantial cash awards for meeting a target first-year grade-point average (GPA), and a control group of many more (1,006) students received only standard university support services.

To illustrate the benefit of rerandomization, we use the 15 covariates listed in Table 1 and exclude students with missing values, resulting in $n_1=118$ and $n_0=856$. To make the simulation relevant to the real data, we fix unknown parameters based on some simple model fitting: We fit a linear regression of the observed first-year GPA on the treatment indicator, all covariates and their interactions, and use the fitted model to generate all potential outcomes eschewing additivity. Note that the generating models for the potential outcomes are nonlinear in the basic covariates. To make the data-generating process realistic, we simulate eight pseudosets of potential outcomes, using the fitted model with different choices for the variance of the residuals. The error terms for $Y\left(1\right)$ and $Y\left(0\right)$ are independent, and therefore conditional on the covariates, and the potential outcomes are simulated as uncorrelated, but they have a positive correlation marginally. The final potential outcomes are all truncated to lie on $\left[\mathrm{0,4}\right]$, mimicking the value of the GPA. We choose different variances of residuals such that the values of $R^2$ for the eight simulated datasets are located approximately evenly within interval $\left[\mathrm{0,0.5}\right]$. One choice for the variance of residuals is the one estimated from the fitted linear model, and the corresponding $R^2$ is about 0.23.

Table 1.

Three tiers of covariates

Tier	Covariates
Tier 1	High-school GPA
Tier 2	Whether lives at home, gender, age
	Whether rarely puts off studying for tests
Tier 3	Whether mother/father is a college graduate
	Whether mother/father is a high-school graduate
	Whether never puts off studying for tests
	Whether wants more than a bachelor degree
	Whether intends to finish in 4 y
	Whether plans to work while in school
	Whether at the first choice school, mother tongue

Table 1 partitions the 15 covariates into three tiers. We choose $a_t$ such that $P\left({\chi }_{k_t}^2\le a_t\right)={\left(0.001\right)}^{1/3}=0.1$ for $t=\mathrm{1,2,3}$. We simulate data under ReMT and obtain the confidence intervals based on our asymptotic theory for ReMT and Neyman’s (20) results for the CRE. Fig. 1, Left shows the empirical coverage probabilities of our and Neyman’s (20) confidence intervals, showing that Neyman’s (20) CRE confidence intervals are very conservative.

Fig. 1.

Eight datasets simulated based on the Student Achievement and Retention Project. Left shows the empirical coverage probabilities of our and Neyman’s (20) $95\%$ confidence intervals under ReMT, and Right shows the percentage reductions of average lengths of confidence intervals and quantile ranges comparing ReMT with a CRE.

To evaluate the performance of ReMT compared with a CRE, we compare the average length of Neyman’s (20) confidence interval under a CRE with the confidence interval under ReMT. From Fig. 1, Right, the percentage reduction in average lengths of the 95% confidence intervals under ReMT compared with Neyman’s (20) under a CRE is nondecreasing in $R^2$. We also compare the empirical $95\%$ quantile range of $\widehat{\tau }$ under ReMT and a CRE: The percentage reductions in the lengths of quantile ranges are close to the percentage reductions for average lengths of confidence intervals. When $R^2$ is close to that of the real dataset (i.e., 0.23), the percentage increase in the effective sample size, that is, the sample size needed with a CRE for $\widehat{\tau }$ to achieve the same $95\%$ quantile range under ReMT, is about $24\%$. When $R^2$ is about twice as large as with the real data (i.e., 0.5), the percentage increase in the effective sample size increases to $80\%$.

Discussion

Our theory suggests that choosing a small $p_a$ will lead to more precise difference in means in general. However, we do not suggest choosing $p_a$ to be too small, such as accepting only those assignments with the smallest Mahalanobis distance. The extreme rerandomization choosing an allocation that balances observed covariates as well as possible has an undesirable consequence. Because it is deterministic, randomization distributions are degenerate, rendering randomization inference impossible. Even if we randomize over all allocations satisfying the best allocation, randomization inference has little power due to very few possible allocations. How to choose $p_a$ remains an open problem.

Materials and Methods

We did not conduct the experiment, and we are analyzing secondary data without any personal identifying information. As such, this study is exempt from human subjects review. The original experiments underwent human subjects review in Canada (24).