Exact Confidence Intervals in the Presence of Interference

Abstract

For two-stage randomized experiments assuming partial interference, exact confidence intervals are proposed for treatment effects on a binary outcome. Empirical studies demonstrate the new intervals have narrower width than previously proposed exact intervals based on the Hoeffding inequality.

1. Introduction

In a randomized experiment, it is commonly assumed that an individual only has two potential outcomes: an outcome on control, and an outcome on treatment. That an individual has only two potential outcomes assumes no interference (Cox, 1958) between individuals, i.e., an individual’s potential outcomes are unaffected by the treatment assignment of any other individual in the study. There are many settings where this assumption of no interference is clearly violated (Hong and Raudenbush, 2006; Sobel, 2006; Rosenbaum, 2007).

Partial interference holds when individuals can be partitioned into groups such that there is no interference between individuals in different groups. In settings where partial interference holds, two-stage randomized experiments have been suggested as a study design for drawing inference about treatment (i.e., causal) effects. Two-stage randomized experiments proceed by (i) randomizing groups to treatment strategies and (ii) randomizing individuals within groups to different treatments based on the treatment strategy assigned to their group in stage (i). Two-stage randomized experiments are found in many fields of study, e.g., infectious diseases (Baird et al., 2012), medicine (Borm et al., 2005), economics (Duflo and Saez, 2003), and political science (Ichino and Schündeln, 2012; Sinclair et al., 2012). Building upon ideas in Halloran et al. (1991), Hudgens and Halloran (2008) defined and derived unbiased estimators for the direct, indirect, total, and overall effects of treatment in a two-stage randomized experiment assuming partial interference. Liu and Hudgens (2014) showed that Wald-type confidence intervals based on these estimators perform well when the number of groups is large; however, often the number of groups may not be large enough. For example, Moulton et al. (2001) describe a group-randomized vaccine trial involving approximately 9,000 individuals but only 38 groups. Tchetgen Tchetgen and VanderWeele (2012), henceforth TV, proposed exact confidence intervals using the Hoeffding inequality for these four effects in a two-stage randomized experiment with partial interference. Unfortunately, as will be shown below, the TV intervals can be very wide and conservative.

In this paper, we propose different exact confidence intervals based on inverting exact hypothesis tests that tend to be less conservative than TV. The remainder of the paper is organized as follows. In §2, treatment effects in the presence of interference are defined and existing inferential results are reviewed. In §3, the assumption of stratified interference is presented and bounds are derived for the causal effects under this assumption. In §4 the proposed new exact confidence intervals are described by inverting certain permutation tests. In §5 a simulation study is conducted comparing the TV, asymptotic, and new exact confidence intervals. §6 concludes with a discussion. An R package is available implementing the proposed confidence intervals.

2. Preliminaries

2.1. Estimands

Consider a finite population of N individuals partitioned into k groups with n_i individuals in group i for i = 1, …, k. Assume partial interference, i.e., there is no interference between individuals in different groups. Consider a two-stage randomized experiment wherein h of k groups are assigned to strategy α₁ and k–h are assigned to α₀ in the first stage, where strategy α_s specifies that $m_i^s$ of n_i individuals will receive treatment. For example, strategy α₀ might entail assigning (approximately) 1/3 of individuals within a group to treatment whereas strategy α₁ might entail assigning (approximately) 2/3 of individuals within a group to treatment (see TV for further discussion about different types of treatment allocation strategies). Let S_i = 1 if group i is randomized to α₁ and 0 otherwise so that Pr[S_i = 1] = h/k. In the second stage, individuals will be randomized to treatment conditional on group assignment in the first stage. Let Z_ij = 1 if individual j in group i is assigned treatment and 0 otherwise. Let Z_i = (Z_i1, …, Z_ini) be the random vector of treatment assignments for group i taking on values $z_i\in \mathcal{R}(n_i,m_i^s)$, the set of all vectors of length n_i composed of $m_i^s$ elements equal to 1 and $n_i-m_i^s$ elements equal to 0. Additionally, let Z_i(j) denote the random vector of treatment assignments in group i excluding individual j taking on values $z_{i(j)}\in \mathcal{R}(n_i-1,m_i^s-z_{ij})$.

Let y_ij(z_i) be the binary potential outcome for individual j in group i when group i receives treatment vector z_i. A randomization inference framework is adopted wherein potential outcomes are fixed features of the finite population of N individuals and only treatment assignments S and Z are random (as in Sobel (2006); Rosenbaum (2007); Hudgens and Halloran (2008)). Define the average potential outcome for individual j in group i on treatment z = 0, 1 under strategy α_s as

$${\overline{y}}_{ij}(z;{\alpha }_s)=\sum _{\omega \in \mathcal{R}(n_i-1,m_i^s-z)}{y}_{ij}(z_{ij}=z,z_{i(j)}=\omega )Pr(Z_{(i)j}=\omega \mid Z_{ij}=z;S_i=s)$$ (1)

where $Pr(Z_{i(j)}=\omega \mid Z_{ij}=z;S_i=s)={\left(\begin{array}{c}{n}_i-1\\ m_i^s-z\end{array}\right)}^{-1}$. Henceforth, let ${\mathrm{\sum }}_i={\mathrm{\sum }}_{i=1}^k$ and ${\mathrm{\sum }}_j={\mathrm{\sum }}_{j=1}^{n_i}$. For treatment z under strategy α_s define the group average potential outcome as ${\overline{y}}_i(z;{\alpha }_s)\equiv n_i^{-1}{\mathrm{\sum }}_j{\overline{y}}_{ij}(z;{\alpha }_s)$, and the population average potential outcome as ȳ(z; α_s) ≡ k⁻¹ Σ_i ȳ_i(z; α_s). Define the average potential outcome for individual j in group i under strategy α_s as

$${\overline{y}}_{ij}({\alpha }_s)\equiv \sum _{\omega \in \mathcal{R}(n_i,m_i^s)}{y}_{ij}(z_i=\omega )Pr(Z_i=\omega ;S_i=s),$$ (2)

the group average potential outcome as ${\overline{y}}_i({\alpha }_s)\equiv n_i^{-1}{\mathrm{\sum }}_j{\overline{y}}_{ij}({\alpha }_s)$, and the population average potential outcome as $\overline{y}({\alpha }_s)\equiv k_i^{-1}{\mathrm{\sum }}_j{\overline{y}}_i({\alpha }_s)$. Define the direct effect of treatment for strategy α_s as DE(α_s) = ȳ(0; α_s) − ȳ(1; α_s), the indirect effect of α₀ versus α₁ as IE(α₀, α₁) = ȳ(0; α₀) − ȳ(0; α₁), the total effect as TE(α₀, α₁) = ȳ(0; α₀) − ȳ(1; α₁), and the overall effect of α₀ versus α₁ as OE(α₀, α₁) = ȳ(α₀) − ȳ(α₁); see Hudgens and Halloran (2008) and TV for additional discussion regarding these effects.

2.2. Existing Inferential Results

Hudgens and Halloran (2008) derived unbiased estimators for all population average potential outcomes, and thus for the four causal effects. Noting that Pr[S_i = s] and Pr[Z_ij = z|S_i = s] are known by design, the estimator

$$\widehat{y}(z;{\alpha }_s)=k^{-1}\sum _i\frac{1\{S_i=s\}{\widehat{y}}_i(z;{\alpha }_s)}{Pr[S_i=s]}$$ (3)

where ${\overline{y}}_i(z;{\alpha }_s)=n_i^{-1}{\mathrm{\sum }}_{j}1\{Z_{ij}=z\}{y}_{ij}(Z_{ij})/Pr[Z_{ij}=z\mid S_i=s]$ is unbiased for ȳ(z; α_s). Additionally, the estimator

$$\widehat{y}({\alpha }_s)=k^{-1}\sum _i\frac{1(S_i=s)n_i^{-1}{\mathrm{\sum }}_j{y}_{ij}(Z_{ij})}{Pr[S_i=s]}$$ (4)

is unbiased for ȳ(α_s). Unbiased estimators for the effects of interest follow immediately: $\widehat{DE}({\alpha }_s)=\widehat{y}(0;{\alpha }_s)-\widehat{y}(1;{\alpha }_s),\widehat{IE}({\alpha }_0,{\alpha }_1)=\widehat{y}(0;{\alpha }_0)-\widehat{y}(0;{\alpha }_1),\widehat{TE}({\alpha }_0,{\alpha }_1)=\widehat{y}(0;{\alpha }_0)-\widehat{y}(1;{\alpha }_1)$, and $\widehat{OE}({\alpha }_0,{\alpha }_1)=\widehat{y}({\alpha }_0)-\widehat{y}({\alpha }_1)$.

TV proposed exact confidence intervals based on the Hoeffding inequality for the effects of interest in a two-stage randomized experiment where partial interference is assumed. In particular, for any γ ∈ {0, 1}, $\widehat{DE}({\alpha }_s)\pm {\epsilon }_D^{\ast }(\gamma ,{\alpha }_s,q_s,k)$ is a 1 − γ exact confidence interval for DE(α_s) where ${\epsilon }_D^{\ast }(\gamma ,{\alpha }_s,q_s,k)$ is given in equation (17) of TV for s = 0, 1. Additionally, $\widehat{IE}({\alpha }_0,{\alpha }_1)\pm {\epsilon }^{\ast }(\gamma ,{\alpha }_0,q_0,{\alpha }_1,q_1,k),\widehat{TE}({\alpha }_0,{\alpha }_1)\pm {\epsilon }^{\ast }(\gamma ,{\alpha }_0,q_0,{\alpha }_1,q_1,k)$, and $\widehat{OE}({\alpha }_0,{\alpha }_1)\pm {\epsilon }^{\ast }(\gamma ,{\alpha }_0,q_0,{\alpha }_1,q_1,k)$ are all 1 − γ exact confidence intervals for their target parameters where ε^*(γ, α₀, q₀, α₁, q₁, k) is given in Theorem 3 of TV.

Liu and Hudgens (2014) examined conditions under which Wald-type intervals $\widehat{DE}({\alpha }_s)\pm z_{(1-\gamma /2)}{\{\widehat{\mathit{var}}(\widehat{DE}({\alpha }_s))\}}^{1/2}$ and Chebyshev-type intervals $\widehat{DE}({\alpha }_s)\pm {\{\widehat{\mathit{var}}(\widehat{DE}({\alpha }_s))/\gamma \}}^{1/2}$ are valid, large sample confidence intervals for DE(α_s), where z_(1−γ/2) is the 1 − γ/2 quantile for the standard normal distribution and $\widehat{\mathit{var}}(\widehat{DE}({\alpha }_s))$ is an estimator of the variance of $\widehat{DE}({\alpha }_s)$ for s = 0, 1. They also considered Wald and Chebyshev-type confidence intervals for the indirect, total, and overall effects.

3. Bounds Under Stratified Interference

Exact randomization based inference about the four effects is challenging without further assumptions as the experiment reveals only N of the ${\mathrm{\sum }}_i{\mathrm{\sum }}_j\{\left(\begin{array}{c}{n}_i\\ m_i^0\end{array}\right)+\left(\begin{array}{c}{n}_i\\ m_i^1\end{array}\right)\}$ total potential outcomes. One such additional assumption is stratified interference (Hudgens and Halloran, 2008), which assumes that individual j in group i has the same potential outcome when assigned control or treatment as long as a fixed number of other individuals in group i are assigned treatment, i.e.,

$$y_{ij}(z_i)=y_{ij}(z_i^{\prime })\text{for}\text{all}{z}_i,z_i^{\prime }\in \mathcal{R}(n_i,m_i^s)\text{such}\text{that}{z}_{ij}=z_{ij}^{\prime }.$$ (5)

Under (5), individual j in group i only has four potential outcomes, which we denote by y_ij(z; α_s) for z, s = 0, 1, so that the experiment reveals the observed outcome Y_ij = Σ_z,s=0,1 1{Z_ij = z; S_i = s}y_ij(z; α_s) for each individual and thus N of the 4N total potential outcomes. Furthermore, (5) implies that ȳ_ij(z; α_s) = y_ij(z; α_s), and that ${\overline{y}}_{ij}({\alpha }_s)=w_i^s{y}_{ij}(1;{\alpha }_s)+(1-w_i^s)y_{ij}(0;{\alpha }_s)\equiv y_{ij}({\alpha }_s)$ where $w_i^s=Pr[Z_{ij}=1\mid S_i=s]=m_i^s/n_i$.

Under (5), the observed data form bounded sets for all effects contained in the interval [−1, 1]. The bounded sets have widths less than two where here and in the sequel the width of a set is defined to be the difference between its maximum and minimum values. Consider $DE({\alpha }_0)=k^{-1}{\mathrm{\sum }}_i{n}_i^{-1}{\mathrm{\sum }}_j\{y_{ij}(0;{\alpha }_0)-y_{ij}(1;{\alpha }_0)\}$ for illustration. For the Σ_i Σ_j(1 − S_i)(1 − Z_ij) individuals with S_i = Z_ij = 0, y_ij(0; α₀) is revealed; however, for the N − Σ_i Σ_j(1 − S_i)(1 − Z_ij) individuals with S_i = 1 or Z_ij = 1, y_ij(0; α₀) is missing and only known to be 0 or 1. Let y⃗(z; α_s) be the N-dimensional vector of potential outcomes for treatment z under strategy α_s. Under (5), a lower bound for DE(α₀) is found by filling in all missing potential outcomes in y⃗(0; α₀) as 0 and all missing potential outcomes in y⃗(1; α₀) as 1. An upper bound for DE(α₀) is found by filling in all missing potential outcomes in y⃗(0; α₀) as 1 and all missing potential outcomes in y⃗(1; α₀) as 0. Simple algebra shows that width of the bounded set for DE(α₀) is equal to 2 − (k − h)/k. The width of this bounded set approaches 1 as (k − h)/k → 1, i.e., as more groups are randomized to α₀.

Similar logic leads to bounds for the other effects. The width of the bounded set for DE(α₁) is equal to 2 − h/k which approaches 1 as h/k → 1. The width of the bounded set for IE(α₀, α₁) is equal to $2-k^{-1}{\mathrm{\sum }}_i{n}_i^{-1}{\mathrm{\sum }}_j(1-Z_{ij})$ which approaches 1 as the proportion of individuals assigned Z_ij = 0 approaches 1. The width of the bounded set for TE(α₀, α₁) is equal to $2-k^{-1}{\mathrm{\sum }}_i{n}_i^{-1}\{(1-S_i){\mathrm{\sum }}_j(1-Z_{ij})+S_i{\mathrm{\sum }}_j{Z}_{ij}\}$ which approaches 1 as the proportion of individuals with S_i = Z_ij = 0 or S_i = Z_ij = 1 approaches 1. Lower and upper bounds for OE(α₀, α₁) can be derived similarly but the corresponding width does not have a simple closed form.

4. EIT Confidence Intervals

In addition to leading to unbiased estimators and bounds, the observed data can be used to form 1 − γ confidence sets for the four effects. The confidence sets are formed by inverting hypothesis tests about the potential outcomes that define the effect of interest. This section is divided into two parts: §4.1 outlines how the confidence sets are formed and §4.2 presents a computationally feasible algorithm for constructing an interval that contains the exact confidence set. Henceforth this interval is referred to as the exact inverted test (EIT).

4.1. An Exact Confidence Set

The methods to follow can be generalized to any effect, so consider DE(α₀). Inference about DE(α₀) concerns the vectors y⃗(0; α₀) and y⃗(1; α₀), which are partially revealed by the experiment. A hypothesis about these vectors is considered sharp if it completely fills in the potential outcomes not revealed by the experiment. A sharp null H₀ : y⃗(0; α₀) = y⃗⁰(0; α₀), y⃗(1; α₀) = y⃗⁰(1; α₀) maps to a value of DE(α₀), which we denote DE⁰(α₀). Only sharp null hypotheses that are compatible with the observed data need to be tested as other sharp nulls can be rejected with zero probability of making a type I error. Thus for each sharp null to be tested, the implied null value DE⁰(α₀) will be a member of the bounded set derived in §3. There are B₁ = 2^{Σ_i(1−S_i)n_i}4^Σ_iS_in_i sharp null hypotheses to test, as individuals with S_i = 0 have only one missing potential outcome with two possible values {0, 1}, and individuals with S_i = 1 have two missing potential outcomes with four possible values {0, 1} × {0, 1}.

After filling in the missing potential outcomes under H₀, the null distribution of the test statistic $\widehat{DE}({\alpha }_0)$ can be found by computing the statistic, denoted by ${\widehat{DE}}_c({\alpha }_0)$, for each of the c = 1, …, C₁ possible experiments under H₀, where $C_1={\mathrm{\sum }}_{S\in \mathcal{S}}{\mathrm{\Pi }}_{i=1}^k{\left(\begin{array}{c}{n}_i\\ m_i^1\end{array}\right)}^{S_i}{\left(\begin{array}{c}{n}_i\\ m_i^0\end{array}\right)}^{(1-S_i)}$ and is the set of all possible values of the vector S such that $\mid \mathcal{S}\mid =\left(\begin{array}{c}k\\ h\end{array}\right)$. A two-sided p-value to test H₀ is given by $p_0={\mathrm{\sum }}_{c=1}^{C_1}1\{\mid {\widehat{DE}}_c({\alpha }_0)-{DE}^0({\alpha }_0)\mid \ge \mid \widehat{DE}({\alpha }_0)-{DE}^0({\alpha }_0)\mid \}/C_1$. If p₀ < γ, H₀ is rejected. Note p₀ is a function of the null hypothesis vectors y⃗⁰(0; α₀) and y⃗⁰(1; α₀). Let p(DE⁰(α₀)) denote the set of all p₀ which are functions of compatible vectors y⃗⁰(0; α₀) and y⃗⁰(1; α₀) that map to DE⁰(α₀). A 1 − γ confidence set for DE(α₀) is {DE⁰(α₀) : max{p(DE⁰(α₀))} ≥ γ}. P-values, and thus confidence sets, can be found in an analogous manner for the other effects.

4.2. A Computationally Feasible Algorithm

Finding the exact confidence set for DE(α₀) described above entails testing B₁ hypotheses, where each hypothesis test involves C₁ randomizations. As N becomes large, the computational time necessary to perform B₁×C₁ operations grows exponentially. For illustration of the problem, consider two examples in which h = 1 of k = 1 groups are assigned α₀, in which $m_1^0=10$ of n₁ = 20 individuals are randomized to treatment such that B₁ = 2²⁰ and $C_1=\left(\begin{array}{c}20\\ 10\end{array}\right)=184,756$. Suppose there are two cases of observed data: (a) 5 of 10 unexposed experienced an event, and 5 of 10 exposed experienced an event, and (b) 8 of 10 unexposed experienced an event and 2 of 10 exposed experienced an event. Figure 1 displays a plot of DE⁰(α₀) versus p(DE⁰(α₀)) for both examples. The bounded set and 95% exact confidence set for DE⁰(α₀) are, respectively, {−0.5, −0.45, …, 0.45, 0.5} and {−0.35, −0.3, …, 0.3, 0.35} in (a) and {−0.2, −0.15, …, 0.75, 0.8} and {0.15, 0.2, …, 0.75, 0.8} in (b).

Figure 1.

Plot of DE(α₀) versus p(DE(α₀)) for examples (a) and (b) as outlined in §4.2.

A computationally feasible algorithm is given below for approximating the confidence sets. The algorithm entails testing a targeted random sample of B₂ of the B₁ total sharp null hypotheses, and computing p-values for each sampled sharp null based on a random sample of C₂ of the C₁ possible randomizations. The set of computed p-values are then used to approximate the confidence set endpoints using local linear interpolation. For intuition underlying the interpolation step, consider the piecewise linear function that connects the maximum p-values for each compatible value of DE(α₀) in Figure 1. Finding the x-coordinates for the intersection points of this function and a horizontal line at γ will conservatively approximate the lower and upper 1 − γ confidence limits for DE(α₀). This suggests the following targeted, local linear interpolation algorithm for estimating the lower bound of a confidence set for DE(α₀). An analogous algorithm can be used to target the upper limit of the confidence set for DE(α₀).

Let $\widehat{DE}{({\alpha }_0)}_l$ denote the lower bound for DE(α₀), and ŷ(z; α₀)_l and ŷ(z; α₀)_u denote the lower and upper bounds, respectively, for ȳ(z; α₀).

1. Test the unique sharp null about y⃗(0; α₀) and y⃗(1; α₀) that maps to $\widehat{DE}{({\alpha }_0)}_l$. If the corresponding p-value p₀ ≥ γ, let $\widehat{DE}{({\alpha }_0)}_l$ be the lower limit of the confidence set and do not proceed. Otherwise, let $l=\widehat{DE}({\alpha }_0)$ and let p_l = 1 − 1/B₂. Let $\mathcal{L}=\{\widehat{DE}{({\alpha }_0)}_l\}$ and = {p₀}.

2. Fill in the missingness in y⃗(0; α₀) with samples from a Bernoulli distribution with mean $f(\{\widehat{y}{(0;{\alpha }_0)}_l+\widehat{y}{(1;{\alpha }_0)}_u+q_{p_l}(\widehat{DE}{({\alpha }_0)}_l,l)\}/2)$ and fill in the missingness in y⃗(1; α₀) with samples from a Bernoulli distribution with mean $f(\{\widehat{y}{(0;{\alpha }_0)}_l+\widehat{y}{(1;{\alpha }_0)}_u-q_{pl}(\widehat{DE}{({\alpha }_0)}_l,l)\}/2)$ where q_p(a, b) = (1 − p)a + pb, and f(x) = x if 0 ≤ x ≤ 1, f(x) = 0 if x < 0, and f(x) = 1 if x > 1.

3. If the sampled sharp null maps to a value ${DE}^0({\alpha }_0)\in [\widehat{DE}{({\alpha }_0)}_l,l]$, add DE⁰(α₀) to the set , add the corresponding p₀ to , and if p₀ ≥ γ then update l to equal DE⁰(α₀). Otherwise, do not compute a p-value corresponding to the sampled sharp null and let p_l = p_l − 1/B₂.

   Repeat Steps 2 and 3 B₂/2 − 1 times.

Let t be the function from to that maps each p-value p₀ in to the null value of DE⁰(α₀) in which corresponds to the sharp null hypothesis which generated p₀. Let = {max{p ∈ : t(p) = l} : l ∈ }. Let r₁ = min{r ∈ : r ≥ γ} and let r₂ = max{r ∈ : r < γ}. Let l_i = t(r_i) for i = 1, 2. The lower limit of the confidence set l^* is found by local linear interpolation by finding the x-coordinate for the point at which a line drawn from (l₂, r₂) to (l₁, r₁) intersects a horizontal line at γ, i.e., l^* = l₂+(γ−r₂)(l₂−l₁)/(r₂−r₁). The upper limit u^* is found analogously. As B₂ → B₁ and C₂ → C₁, the interval [l^*, u^*] will contain the exact confidence set described in §4.1 with probability approaching 1.

The algorithms for approximating confidence sets for IE(α₀, α₁) and TE(α₀, α₁) are analogous. For OE(α₀, α₁) the algorithm is modified slightly as it involves all four vectors y⃗(z; α_s), z, s = 0, 1. Let ŷ(α_s)_l and y⃗(α_s)_u be the lower and upper limits, respectively, for ȳ(α_s) under (5). If p₀ < γ for OE(α₀, α₁)_l, set $l=\widehat{OE}({\alpha }_0,{\alpha }_1)$ and fill in the missingness in y⃗(0; α₀) and y⃗(1; α₀) with samples from a Bernoulli distribution with mean $f(\{\widehat{y}{({\alpha }_0)}_l+\widehat{y}{({\alpha }_1)}_u+q_{p_l}(\widehat{OE}{({\alpha }_0,{\alpha }_1)}_l,l)\}/2)$ where p_l = 1 − 1/B₂. A p-value is computed if $O{E}^0({\alpha }_0,{\alpha }_1)\in [\widehat{OE}{({\alpha }_0,{\alpha }_1)}_l,l]$ and if not p_l = p_l − 1/B₂. If p₀ ≥ γ for OE(α₀, α₁)_l, l is set to equal OE⁰(α₀, α₁). The upper endpoint can be approximated using an analogous approach.

The R package interferenceCI is available on CRAN (Rigdon, 2015) for computing EIT confidence intervals via this algorithm for the four effects assuming stratified interference when the outcome is binary. The Wald, Chebyshev, and TV intervals are also computed in the package.

5. Comparisons Via Simulation

A simulation study was carried out to compare the asymptotic, TV, and EIT confidence intervals. The simulation proceeded as follows for fixed values of α₀, α₁, DE(α₀), DE(α₁), IE(α₀, α₁), k, n_i = n for i = 1, …, k such that N = kn:

• 0Potential outcomes were generated by first fixing the vectors y⃗(z; α_s) for z, s = 0, 1 to be length N vectors of all 0s. Group membership was assigned by letting elements n(i − 1) + 1, …, ni of each vector belong to group i = 1, …, k. Then, N(0.5+DE(α₀)/2) elements in y⃗(0; α₀) were randomly set to equal 1 and N(0.5−DE(α₀)/2) elements in y⃗(1; α₀) were randomly set to equal 1. Then, N(0.5 + DE(α₀)/2 − IE(α₀, α₁)) elements in y⃗(0; α₁) were randomly set to equal 1. Finally, N(0.5 + DE(α₀)/2 − IE(α₀, α₁) − DE(α₁)) elements in y⃗(1; α₁) were randomly set to equal 1.
• 1Observed data were generated by (i) randomly assigning h of k groups to strategy α₁ and (ii) randomly assigning $m_i^s={\alpha }_{s}n$ of n individuals per group to treatment for s = 0, 1. Observed outcomes followed based on these treatment assignments and the potential outcomes from step 0.
• 2For each effect, 95% confidence intervals were computed using the observed data generated in step 1.
• 3Steps 1–2 were repeated 1000 times.

In the simulation we let k = n = 10 or k = n = 20 with h = k/2, $m_i^0=0.3n$ under α₀, $m_i^1=0.6n$ under α₁, DE(α₀) = 0.95, DE(α₁) = 0.3, and IE(α₀, α₁) = 0.5 (such that TE(α₀, α₁) = 0.8 and OE(α₀, α₁) = 0.395). In the targeted sampling algorithm, B₂ = C₂ = 100 such that B₂/B₁ and C₂/C₁ were less than 10⁻²⁰ for all effects. Table 1 displays average widths and coverages for Wald, EIT, Chebyshev, and TV. Wald and Chebyshev fail to achieve nominal coverage for DE(α₀) when k = n = 10 and Wald additionally fails to cover for DE(α₀) when k = n = 20 and for IE(α₀, α₁) and TE(α₀, α₁) when k = n = 10. As guaranteed by their respective constructions, EIT and TV achieve nominal coverage for all setups; however, EIT has narrower width than TV in all setups. In fact, EIT is an order of magnitude narrower than TV in three instances: DE(α₀), TE(α₀, α₁), and OE(α₀, α₁) when k = n = 20.

Table 1.

Empirical width [coverage] of Wald (W), exact inverted test (EIT), Chebyshev (C), and TV 95% CIs for simulation study discussed in §5.

	n	k	DE(α0)	DE(α1)	IE(α0, α1)	TE(α0, α1)	OE(α0, α1)
W	10	10	0.13 [0.84]	0.51 [0.96]	0.39 [0.93]	0.30 [0.93]	0.24 [0.94]
	20	20	0.09 [0.89]	0.26 [0.96]	0.21 [0.95]	0.14 [0.98]	0.11 [0.97]
EIT	10	10	0.28 [0.98]	0.52 [0.98]	0.47 [0.99]	0.31 [0.98]	0.36 [1.00]
	20	20	0.12 [0.98]	0.27 [0.98]	0.24 [0.98]	0.14 [0.98]	0.18 [1.00]
C	10	10	0.22 [0.84]	1.15 [1.00]	0.84 [1.00]	0.54 [1.00]	0.54 [1.00]
	20	20	0.15 [0.99]	0.59 [1.00]	0.49 [1.00]	0.32 [1.00]	0.26 [1.00]
TV	10	10	1.95 [1.00]	2.00 [1.00]	2.00 [1.00]	2.00 [1.00]	2.00 [1.00]
	20	20	1.41 [1.00]	2.00 [1.00]	1.86 [1.00]	1.56 [1.00]	1.96 [1.00]

6. Discussion

In this paper new exact confidence intervals have been proposed for causal effects in the presence of partial interference. The new intervals are constructed by inverting permutation based hypothesis tests. These intervals do not rely on any parametric assumptions and require no assumptions about random sampling from a larger population. The confidence intervals are exact in the sense that the probability of containing the true treatment effects is at least the nominal level. As there may be many vectors of potential outcomes that map to one value of the causal estimand, a computationally feasible algorithm was proposed in §4.2 to approximate the exact confidence intervals. Empirical studies demonstrate the new exact intervals have narrower width than previously proposed exact intervals based on the Hoeffding inequality. Nonetheless, the empirical coverage of the proposed intervals still tends to exceed the nominal level, suggesting one possible future avenue of research would be to develop alternative intervals which are less conservative and narrower but maintain nominal coverage.