Response to comment on “Randomization inference for treatment effects on a binary outcome”

Abstract

We thank Professor Yasutaka Chiba [1] for commenting on Rigdon and Hudgens (RH) [2]. Chiba [1] described a certain exact confidence interval reported in RH as “somewhat unnatural.” Chiba also presented an alternative approach to constructing confidence intervals [3]. In this response, we (i) provide a simple explanation why the confidence interval in RH appeared “unnatural,” and (ii) explain the relationship between the RH [2] and Chiba [3] confidence intervals. Essentially the two approaches are equivalent, except RH entails inverting one two-sided test whereas Chiba inverts two one-sided tests. We present a more computationally efficient method (RLH) for computing the RH intervals based on Chiba’s principal stratification formulation of the problem. We also propose a third method based on Blaker [4] which inverts a single two-sided test but forms a confidence interval that is at least as narrow as inverting two one-sided tests. Simulation results show the RLH intervals tend to be as narrow or narrower than the Chiba and Blaker intervals on average.

1. Introduction

We would like to thank Professor Yasutaka Chiba [1] for his comments on Rigdon and Hudgens (RH) [2]. In this response, we compare Chiba’s approach to constructing confidence intervals (CIs), as described in [3], with the approach in RH. First, we explain that despite the different parameterizations, both approaches test the same null hypotheses for the average causal effect of treatment, using the same randomization distributions to carry out the permutation tests. The difference between the two methods lies in the choice of whether to invert and combine two one-sided tests as Chiba proposed, or to invert a single two-sided test as in RH.

Next, we present a more computationally efficient method (RLH) for computing the RH intervals based on Chiba’s principal stratification formulation of the problem. We also propose a third method based on Blaker [4] which inverts a single two-sided test but forms a CI that is at least as narrow as Chiba’s method. Our simulation results show the RLH intervals tend to be narrowest on average among the three methods, but the differences between the methods are reduced either as the magnitude of the average causal effect increases or as the population size increases.

Finally, we provide a simple explanation why a certain CI reported in RH was described by Chiba as “somewhat unnatural” [1]. As the original formulation of the RH approach could be computationally intensive, Monte Carlo sampling was used to approximate p-values when carrying out the hypothesis tests. The interval reported in RH appeared “unnatural” because of the Monte Carlo approximation. Using the RLH approach without Monte Carlo sampling, we obtain the same interval as Chiba [1, 3].

2. Methods

Suppose m of n individuals are randomized to either treatment or control. Assume m is fixed by experimental design. Let Z_j = 1 if individual j is assigned treatment and Z_j = 0 otherwise. Let the binary outcome of interest be coded as either Y_j = 1 or Y_j = 0. Assume prior to randomization, each individual has two potential outcomes: y_j(1) if treatment is assigned and y_j(0) if control is assigned. Consequently, the treatment assignment reveals the observed outcome Y_j = Z_jy_j(1) + (1 − Z_j)y_j(0). The parameter of interest is the average causal effect of treatment $\tau ={\sum }_{j=1}^n{\delta }_j/n$, where δ_j = y_j(1) − y_j(0). The data observed from the experiment can be summarized using the notation in Table 1. The observed difference in proportions T = a/(a + b) − c/(c + d) is an unbiased estimator of τ [5].

Table 1.

Observed data from experiment

	Outcome
Randomization Assignment	Y = 1	Y = 0	Total
Z = 1	a	b	a + b = m
Z = 0	c	d	c + d = n − m
Total	a + c	b + d	n

2.1. Exact Confidence Interval of Rigdon and Hudgens (2015)

Two exact CIs for τ were proposed in [2]. The first combines two prediction intervals for attributable effects [6], while the second entails inverting a permutation test. Simulations presented in [2] showed the permutation-based CIs tend to be narrower than the attributable effects-based CIs. Only the permutation-based CI will be considered below.

The observed data reveal one of the two potential outcomes for each individual. Since the missing potential outcome is either 0 or 1 for a binary outcome, there are 2ⁿ possible unique values for the vector δ = (δ₁, …, δ_n) given the observed data. Furthermore, there are n + 1 unique values of τ that are compatible with the observed data and comprise a set of width one, where the width of a set is defined as the difference between the maximum and minimum elements of the set [2, §1]. A confidence set for τ can be constructed by conducting hypothesis tests about δ as follows.

To test the null hypothesis $H_0:\delta ={\delta }^0=({\delta }_1^0,\dots ,{\delta }_n^0)$, a test statistic can be chosen, its distribution under H₀ computed, and a measure of extremeness of the observed data defined [7, §4.1]. A natural choice for the test statistic is T. The sampling distribution of T under H₀ can be determined exactly by computing T for each of the $C=\left(\begin{array}{c}\hfill n\hfill \\ \hfill m\hfill \end{array}\right)$ possible randomizations for {Z_j : j = 1, …, n} because all potential outcomes are known under the sharp null H₀. For randomizations c = 1, …, C, let t^c denote the value of T under H₀. Each randomization occurs with probability 1/C, so a two-sided p-value can be defined as ${\sum }_{c=1}^{C}1\{|t^c-{\tau }^0|\ge |t^{\mathit{\text{obs}}}-{\tau }^0|\}/C$ where t^obs is the value of T for the observed data and ${\tau }^0={\sum }_{j=1}^n{\delta }_j^0/n$. The subset of δ⁰ vectors where the p-value is greater than or equal to α forms a 100(1 − α)% confidence set for δ. The τ⁰ values corresponding to the δ⁰ vectors in this confidence set for δ then form a 100(1 − α)% confidence set for τ.

However, one need not test all 2ⁿ hypotheses for H₀ : δ = δ⁰ to obtain an exact confidence set for τ. The set of possible δ⁰ values can be partitioned into (a + 1)(b + 1)(c + 1)(d + 1) subsets such that all δ⁰ in the same subset yield the same p-value. It is thus only necessary to test one δ⁰ in each of these subsets [2, §3]; this approach was proposed in RH. When the total number of possible treatment assignments C becomes large, RH suggested approximating the p-value for each hypothesis test with a Monte Carlo random sample of nperm assignments instead of evaluating all C possibilities.

2.2. Exact Confidence Interval of Chiba (2015)

Chiba [3] proposed an exact CI for τ using a principal stratification approach. Let n_st denote the number of individuals with {y_j(1) = s, y_j(0) = t} for s, t = 0, 1, such that τ = (n₁₀ − n₀₁)/n. Chiba considered null hypotheses about the parameter n = (n₁₁, n₁₀, n₀₁, n₀₀) that are compatible with the observed data, i.e., values of n that satisfy the conditions:

$$\begin{array}{ccccc}\hfill {\sum }_{s=0}^1{\sum }_{t=0}^1{n}_{\mathit{\text{st}}}=n,& \hfill n_{11}\le a+c,& \hfill n_{10}\le a+d,& \hfill n_{01}\le b+c,& \hfill n_{00}\le b+d,\\ \hfill & \hfill n_{11}+n_{10}\le n-b,& \hfill n_{11}+n_{01}\le n-d,& \hfill n_{00}+n_{10}\le n-c,& \hfill n_{00}+n_{01}\le n-a.\end{array}$$ (1)

Let n_st,z denote the number of individuals with {y_j(1) = s, y_j(0) = t} and Z_j = z for s, t = 0, 1. For m fixed by design, the conditional probability of (n_11,1, n_10,1, n_01,1, n_00,1) given n is:

$$p_{\mathbf{n}}^{*}(n_{11,1},n_{10,1},n_{01,1},n_{00,1})=\prod _{s=0}^1\prod _{t=0}^1\left(\begin{array}{c}\hfill n_{\mathit{\text{st}}}\hfill \\ \hfill n_{\mathit{\text{st}},1}\hfill \end{array}\right)/\left(\begin{array}{c}\hfill n\hfill \\ \hfill m\hfill \end{array}\right).$$ (2)

Chiba proposed the following conditional exact p-value to test a null hypothesis for a given value of n:

$$P_f(t_{\mathit{\text{obs}}};\mathbf{n})=\sum _{i=0}^{n_{11}}\sum _{j=0}^{n_{10}}\sum _{k=0}^{n_{01}}\sum _{l=0}^{n_{00}}f(T,t_{\mathit{\text{obs}}})p_{\mathbf{n}}^{*}(i,j,k,l)1\{i+j+k+l=m\},$$ (3)

where 1{B} = 1 if B is true and 0 otherwise, and f is a function of T = (i + j)/m − {(n₁₁ − i) + (n₀₁ − k)} /(n − m) and t_obs. Unlike the RH method described above, Chiba’s approach does not explicitly compute a test statistic for all $\left(\begin{array}{c}\hfill n\hfill \\ \hfill m\hfill \end{array}\right)$ possible randomization assignments, thus potentially obviating the need for Monte Carlo approximation of the p-values P_f (t_obs; n). Chiba’s 100(1 − α)% CI [3, §1] entails inverting two one-sided tests and equals [L₁, U₁] where:

$$L_1=\text{min }\{\frac{n_{10}-n_{01}}{n}:P_{f_{L,1}}(t_{\mathit{\text{obs}}};\mathbf{n})\ge \alpha /2\},U_1=\text{max }\{\frac{n_{10}-n_{01}}{n}:P_{f_{U,1}}(t_{\mathit{\text{obs}}};\mathbf{n})\ge \alpha /2\},$$ (4)

and f_L,1 = 1{(T − t_obs) ≥ 0} and f_U,1 = 1{(T − t_obs) ≤ 0}.

2.3. Comparison of RH and Chiba

In general, the approach of inverting and combining two separate one-sided tests is termed the tail method [8]. An alternative to Chiba’s tail method CI is to invert a single two-sided test. A 100(1 − α)% CI analogous to (4) that inverts a single two-sided test is [L₂, U₂], where:

$$L_2=\text{min }\{\frac{n_{10}-n_{01}}{n}:P_{f_2}(t_{\mathit{\text{obs}}};\mathbf{n})\ge \alpha \},U_2=\text{max }\{\frac{n_{10}-n_{01}}{n}:P_{f_2}(t_{\mathit{\text{obs}}};\mathbf{n})\ge \alpha \},$$ (5)

and f₂ = 1{|T − (n₁₀ − n₀₁)/n| ≥ |t_obs − (n₁₀ − n₀₁)/n|}. The interval [L₂, U₂] in (5) is equivalent to the permutation-based interval RH in [2]. Note in the finite population where n is fixed, one may determine an element n_st in n if given the other three, for example n₀₀ = n − n₁₁ − n₁₀ − n₀₁. The set of possible values for the parameter n that satisfy (1) thus lies in a three-dimensional space [9]. Therefore, under the formulation in (5), one need only test O(n³) hypotheses for n, as compared to the O(n⁴) hypotheses under the RH formulation. Moreover, the requirement for Monte Carlo approximations of the p-values is also potentially avoided. We will henceforth refer to this modified approach as RLH. Li and Deng [10] describe a more computationally efficient approach to construct these intervals which requires only O(n²) hypothesis tests, but the resulting CIs may potentially be wider.

2.4. Blaker Confidence Interval

An alternative approach that inverts a single two-sided test but forms an interval necessarily contained within the interval using the tail method was proposed by Blaker [4]. Let the minimum one-sided tail probability of T be denoted as γ(T, n) = min {P_fL,1 (T; n), P_fU,1 (T; n)}, and let f_γ = 1{γ(T, n) ≤ γ(t_obs, n)}. A 100(1 − α)% CI is [L_γ, U_γ], henceforth referred to as a Blaker interval, where:

$$L_{\gamma }=\text{min }\{\frac{n_{10}-n_{01}}{n}:P_{f_{\gamma }}(t_{\mathit{\text{obs}}};\mathbf{n})\ge \alpha \},U_{\gamma }=\text{max }\{\frac{n_{10}-n_{01}}{n}:P_{f_{\gamma }}(t_{\mathit{\text{obs}}};\mathbf{n})\ge \alpha \}.$$ (6)

The Chiba, RLH and Blaker intervals can be computed using version 1.3 of the R [11] package RI2by2 [12].

3. Results

3.1. Simulation Study

We compared the Chiba interval (4), the RLH interval (5), and the Blaker interval (6) in a simulation study. The simulations were carried out as described in scenarios (i) and (ii) of [2], and the results are summarized in Tables 2 and 3 respectively. The CIs using all three methods had coverage greater than the nominal level. For the scenarios where exactly 50% were assigned to treatment, all three methods returned identical intervals. For the scenarios where either 30% or 70% were assigned to treatment, the RLH intervals were as narrow or narrower than both the Blaker and Chiba intervals on average for smaller values of n. The Blaker intervals tended to be slightly narrower than the Chiba intervals. The differences between the methods were reduced when either n or the magnitude of the true average causal effect τ increased.

Table 2.

Simulation results under scenario (i) in [2]. Table entries give the average empirical width [coverage] of 95% CIs, where τ is the true average treatment effect, % treatment is the percent of n total individuals assigned to treatment in each experiment, Chiba CIs are given by (4), RLH CIs by (5), and Blaker CIs by (6).

		30% treatment			50% treatment			70% treatment
n	Method	τ = 0.2	τ = 0.5	τ = 0.95	τ = 0.2	τ = 0.5	τ = 0.95	τ = 0.2	τ = 0.5	τ = 0.95
20	Chiba	0.76[1.00]	0.72[1.00]	0.39[1.00]	0.72[1.00]	0.69[1.00]	0.30[1.00]	0.76[1.00]	0.72[1.00]	0.39[1.00]
	Blaker	0.73[1.00]	0.68[1.00]	0.39[1.00]	0.72[1.00]	0.69[1.00]	0.30[1.00]	0.73[1.00]	0.67[1.00]	0.39[1.00]
	RLH	0.71[1.00]	0.66[1.00]	0.39[1.00]	0.72[1.00]	0.69[1.00]	0.30[1.00]	0.71[1.00]	0.66[1.00]	0.39[1.00]
40	Chiba	0.58[0.99]	0.51[0.99]	0.25[1.00]	0.54[1.00]	0.47[1.00]	0.17[1.00]	0.58[0.99]	0.51[0.99]	0.25[1.00]
	Blaker	0.57[0.99]	0.49[0.99]	0.23[1.00]	0.54[1.00]	0.47[1.00]	0.17[1.00]	0.57[0.99]	0.49[0.99]	0.23[1.00]
	RLH	0.56[0.99]	0.49[0.99]	0.24[1.00]	0.54[1.00]	0.47[1.00]	0.17[1.00]	0.56[0.99]	0.49[0.99]	0.24[1.00]
60	Chiba	0.48[0.99]	0.41[0.99]	0.18[1.00]	0.45[1.00]	0.38[1.00]	0.13[1.00]	0.48[0.99]	0.41[0.99]	0.18[1.00]
	Blaker	0.48[0.99]	0.40[0.99]	0.17[1.00]	0.45[1.00]	0.38[1.00]	0.13[1.00]	0.48[0.99]	0.41[0.99]	0.17[1.00]
	RLH	0.47[0.99]	0.40[0.99]	0.18[1.00]	0.45[1.00]	0.38[1.00]	0.13[1.00]	0.47[0.99]	0.40[0.99]	0.18[1.00]
100	Chiba	0.39[0.99]	0.32[1.00]	0.13[1.00]	0.36[1.00]	0.29[1.00]	0.10[1.00]	0.39[0.99]	0.32[1.00]	0.13[1.00]
	Blaker	0.38[0.99]	0.31[1.00]	0.12[1.00]	0.36[1.00]	0.29[1.00]	0.10[1.00]	0.38[0.99]	0.32[1.00]	0.12[0.99]
	RLH	0.38[0.99]	0.31[1.00]	0.13[1.00]	0.36[1.00]	0.29[1.00]	0.10[1.00]	0.38[0.99]	0.31[1.00]	0.13[0.99]

Table 3.

Simulation results under scenario (ii) in [2]. Table entries give the average empirical width [coverage] of 95% CIs, where γ is the degree of additivity, % treatment is the percent of n total individuals assigned to treatment in each experiment, Chiba CIs are given by (4), RLH CIs by (5), and Blaker CIs by (6).

		30% treatment			50% treatment			70% treatment
n	Method	γ = 0.2	γ = 0.8	γ = 1	γ = 0.2	γ = 0.8	γ = 1	γ = 0.2	γ = 0.8	γ = 1
20	Chiba	0.77[1.00]	0.76[0.99]	0.76[0.98]	0.72[1.00]	0.72[1.00]	0.72[0.99]	0.77[1.00]	0.76[0.99]	0.76[0.98]
	Blaker	0.74[1.00]	0.73[0.98]	0.73[0.97]	0.72[1.00]	0.72[1.00]	0.72[0.99]	0.73[1.00]	0.73[0.99]	0.73[0.97]
	RLH	0.71[1.00]	0.71[0.98]	0.71[0.97]	0.72[1.00]	0.72[1.00]	0.72[0.99]	0.71[1.00]	0.71[0.99]	0.71[0.97]
40	Chiba	0.58[1.00]	0.58[1.00]	0.58[0.98]	0.55[1.00]	0.55[0.99]	0.55[0.98]	0.58[1.00]	0.58[0.98]	0.58[0.97]
	Blaker	0.57[1.00]	0.57[0.99]	0.57[0.96]	0.55[1.00]	0.55[0.99]	0.55[0.97]	0.57[1.00]	0.57[0.97]	0.57[0.95]
	RLH	0.57[1.00]	0.56[0.99]	0.56[0.97]	0.55[1.00]	0.55[0.99]	0.55[0.98]	0.57[1.00]	0.56[0.97]	0.56[0.96]
60	Chiba	0.49[1.00]	0.49[0.98]	0.49[0.97]	0.46[1.00]	0.46[0.99]	0.46[0.97]	0.49[1.00]	0.49[0.98]	0.49[0.97]
	Blaker	0.48[1.00]	0.48[0.98]	0.48[0.96]	0.46[1.00]	0.46[0.99]	0.46[0.97]	0.48[1.00]	0.48[0.98]	0.48[0.97]
	RLH	0.48[1.00]	0.48[0.98]	0.48[0.96]	0.46[1.00]	0.46[0.99]	0.46[0.97]	0.48[1.00]	0.48[0.98]	0.48[0.97]
100	Chiba	0.39[1.00]	0.39[0.98]	0.39[0.97]	0.36[1.00]	0.36[0.98]	0.36[0.96]	0.39[1.00]	0.39[0.98]	0.39[0.97]
	Blaker	0.39[1.00]	0.39[0.97]	0.39[0.96]	0.36[1.00]	0.36[0.98]	0.36[0.96]	0.39[1.00]	0.39[0.98]	0.39[0.96]
	RLH	0.39[1.00]	0.39[0.97]	0.39[0.96]	0.36[1.00]	0.36[0.98]	0.36[0.96]	0.39[1.00]	0.39[0.98]	0.39[0.96]

3.2. Counterexamples

While the RLH intervals tend to be narrower than the Chiba intervals, it is possible that the Chiba intervals are narrower for particular data sets. For example, suppose a = 11, b = 1, c = 7, d = 21. Then both the Chiba and Blaker 95% CIs equal [0.400, 0.775] whereas the RLH interval equals [0.375, 0.775].

The Blaker CI may also be narrower than both the Chiba and RLH CIs for particular data sets. For example, suppose a = 7, b = 5, c = 1, d = 27. For these data the Blaker 95% CI equals [0.275, 0.750], whereas the RLH interval equals [0.250, 0.750] and the Chiba interval equals [0.250, 0.775].

3.3. Application

In the vaccine adherence example presented in §4.3 of [2], n = 96 injection drug users were randomized to a monetary incentive group or an outreach group. Of the m = 48 individuals in the monetary incentive group, a = 33 were adherent, and of the n − m = 48 in the outreach group, c = 11 were adherent. The estimated average causal effect was t^obs = 33/48 − 11/48 ≈ 0.46. The RH approach entails testing (33 + 1)(15 + 1)(11 + 1)(37 + 1) = 248064 hypotheses to obtain an exact CI. Given the large number of possible treatment assignments $C=\left(\begin{array}{c}\hfill 96\hfill \\ \hfill 48\hfill \end{array}\right)\approx 6\times {10}^{27}$, RH approximated the p-value for each hypothesis test with nperm=100 re-randomizations. The resulting 95% CI in [2] was [0.28, 0.64], which Chiba [1] described as “somewhat unnatural.” However, because only 100 re-randomizations were used, the resulting CI in RH was heavily dependent on the random seed, with different random seeds producing different intervals. Using RLH without Monte Carlo sampling, the interval equals [0.28125, 0.59375], which is the same as Chiba’s interval [1, 3]. The Blaker interval also equals [0.28125, 0.59375].