Using a Satisficing Model of Experimenter Decision-Making to Guide Finite-Sample Inference for Compromised Experiments

Summary

This paper presents a simple decision-theoretic economic approach for analyzing social experiments with compromised random assignment protocols that are only partially documented. We model administratively constrained experimenters who satisfice in seeking covariate balance. We develop design-based small-sample hypothesis tests that use worst-case (least favorable) randomization null distributions. Our approach accommodates a variety of compromised experiments, including imperfectly documented re-randomization designs. To make our analysis concrete, we focus much of our discussion on the influential Perry Preschool Project. We reexamine previous estimates of program effectiveness using our methods. The choice of how to model reassignment vitally affects inference.

1. INTRODUCTION

This paper develops a finite-sample, design-based approach for analyzing data from compromised social experiments using a satisficing model of experimenter behavior. Compromises can take many forms, including exchanges or transfers of subjects across the experimental groups based on post-randomization considerations that are not fully documented. For specificity, we motivate our approach drawing on the influential Perry Preschool Project, an experimental high-quality preschool program targeted toward disadvantaged African-American children in the 1960s.¹

Previous studies of the Perry program report substantial treatment effects on numerous outcomes.² These studies have greatly influenced discussions about the benefits of early childhood programs.³ However, critics of the Perry program question the validity of these conclusions. They point to the small sample size of the experiment—just over a hundred observations. They also mention incomplete knowledge of, and compromises in, the randomization protocol used to form control and treatment groups. Problems with attrition and non-response are also cited. Previous research (Heckman et al., 2010a; Heckman et al., 2020) addresses some of these concerns.⁴ We offer an alternative approach that models experimenter decision-making in conducting the experiment.

The Perry randomization protocol was a multi-stage process. Its main compromised feature is shared by many randomized controlled trials: undocumented re-randomization. This involves reassignment of treatment status after initial random assignment in order to improve balance between experimental groups with respect to baseline covariates, but without a pre-specified, fully documented reassignment plan.

This practice occurs often. Bruhn and McKenzie (2009) survey 25 leading researchers using randomized experiments and report a typical response:

“[Experimenters] regressed variables like education on assignment to treatment, and then re-did the assignment if these coefficients were ‘too big.’”

Some 52% admit to “subjectively deciding whether to redraw” and 15% admit to “using a statistical rule to decide whether to redraw” the treatment assignment vector in at least one of the experiments they conducted.⁵ The authors conclude that

“this reveals common use of methods to improve baseline balance, including several rerandomization methods not discussed in print.”

The approach developed in this paper applies to experiments conducted in such a subjective and incompletely documented manner. If rerandomization criteria are specified and adhered to before carrying out final treatment assignment, there exist simpler methods for conducting valid inference.⁶ We supplement the literature by considering the case where the reassignment rule is only partially documented. We build on and complement the analysis of Heckman et al. (2020) with an explicit model of experimenter behavior.

We model experimenters as decision-makers who satisfice in seeking to achieve covariate balance with a “suitable” metric. Implicit decision rules underlie all covariate balancing procedures. The decision-makers forming the experimental groups do not necessarily have a precise rule in mind and satisfice in the sense of Simon (1955). Even if experimenters have a specific rule in mind, it may not be carefully documented.

This paper proceeds in the following way. Section 2 illustrates the class of problems addressed in this paper by reexamining the reassignment protocols of an influential compromised small-sample social experiment, to which we apply our methods both here and in a more extensive analysis Heckman (2020). Section 3 presents a satisficing model of experimenter behavior consistent with the available information on it from published and informal accounts. We partially identify the set of randomization protocols consistent with our model. We consider the generality of our approach by discussing the class of experiments to which our model applies. In Section 4, we first discuss hypotheses of interest and conventional testing procedures used in the literature. We then construct worst-case randomization tests using stochastic approximations of least favorable randomization null distributions. We also compare our approach with that of Heckman et al. (2010a) and Section 5 presents our test statistics and uses our methodology to reexamine the inference reported by Heckman et al. (2020). Section 6 concludes.

2. THE MOTIVATING PROBLEM

To give specificity to our analysis we draw on a prototypical social experiment, the Perry Preschool Project, which was conducted in the early 1960s. The original sample for the experiment consisted of 128 children. Five of these children were dropped from the study due to extraneous reasons.⁷ Starting at age 3, treatment in the following two years included preschool for 2.5 hours per day on weekdays during the academic year. The program also offered 1.5-hour weekly home visits by the Perry teachers to promote parental engagement with the child.⁸ For more details on the background and eligibility criteria of the program, see Heckman et al. (2010a) and Appendix A.

2.1. Randomization Protocol

Understanding the randomization protocol is essential for constructing valid frequentist inference for any experiment. As Bruhn and McKenzie (2009) emphasize, many experimental studies in economics do not report the complete set of rules (e.g., balancing criteria) used to form experimental samples. They conduct hypothesis tests that ignore the randomization protocols actually used. In analyzing the Perry data, this issue is salient. Reports vary about the procedure used and the exact rules followed in creating experimental samples. We discuss the various descriptions of the randomization protocols. While the core descriptions of the procedure followed are broadly consistent across texts, some of the details provided are vague and inconsistent, even those by the same authors. We account for this ambiguity in designing and interpreting our hypothesis tests. While the details are Perry-specific, the general principles involved are not.

Before the initiation of the randomization procedure by the Perry staffers in each of the last four Perry cohorts, any younger siblings of participants enrolled in previous waves are separated from children of freshly recruited families, whom we term “singletons” Schweinhart et al. (1985); Schweinhart (2013). As Schweinhart et al. (1985) explain,

“[A]ny siblings [are] assigned to the same group [either treatment or control] as their older siblings in order to maintain the independence of the groups.”

By construction this does not apply to the very first cohort.

The singletons from new families are then randomized into the two experimental groups as follows. Weikart et al. (1978) detail the second step of the randomization protocol:

“First, all [singletons] are rank-ordered according to Stanford–Binet [IQ] scores. Next, they are sorted (odd / even) into two groups.”

Singletons are then divided into two groups, one comprising those with even IQ ranks and another with odd IQ ranks. The latter group has one additional person if the singletons are odd in number; otherwise, the sizes of the two groups are equal.

In the third step, children are exchanged between the two groups to balance the vector of means of an index of socioeconomic status (SES), the proportions of boys and girls, and the proportion of children with working mothers, in addition to mean IQ Weikart et al. (1964); Schweinhart et al. (1993). The exact balancing criteria and the number of exchanges are not specified, and the exchanges are not necessarily restricted to those between consecutively ranked IQ pairs,⁹ as is sometimes assumed, e.g., in Heckman et al. (2020). After the first three steps, there are two undesignated groups that differ in number by at most one, and the two groups are balanced with respect to mean IQ, mean SES, percentage of boys, and the proportion of children with working mothers, in a manner acceptable to the staffers, using balancing rules that are undocumented.

All sources agree that in the fourth step a toss of a fair coin decides assignment of the two groups to treatment and control conditions. The fifth step concerns children with working mothers who are placed in the treatment group after the fourth step. In the fifth step, some of these children are transferred to the control group.¹⁰ Although there is no consistent account of the number of transfers, the sources describe the fifth step as involving one-way transfers of some children of working mothers from the treatment group to the control group.¹¹ Weikart et al. (1978) provide reasons for the transfers: “no funds were available [to provide all working mothers with logistical support, and] special arrangements could not always be made.” We interpret this statement as implying that special arrangements could be made for at least some working mothers to enable their children to attend preschool and participate in home visits if placed in the treatment group. The constraints facing program administrators in doing so likely vary across cohorts. We assume that the Perry staffers are impartial as to which working mothers get special arrangements.

Table 1 summarizes the randomization protocol. The main sources of ambiguity are in boldface: (a) the undocumented balancing criteria and rules used to satisfactorily balance the two undesignated groups with respect to the mean levels of baseline variables in the third step; and (b) the nature of constraints on the provision of special home visitation arrangements for children of working mothers in the fifth step.

Table 1.

Schematic of the Actual Randomization Protocol

1. Recruit participants and separate any younger siblings of participants enrolled in previous waves from singletons (children of freshly recruited families)

↓

2. Rank singletons by IQ and split into two groups based on whether the rank is even or odd

↓

3. Exchange singletons between the two groups to satisfactorily balance the mean levels of a vector of IQ, SES, gender, and mother's working status

↓

4. Toss a fair coin to determine which of the two groups becomes the initial treatment group

↓

5. Transfer some children of working mothers from the treatment group to the control group impartially if special arrangements for home visits can be made for only a limited number

↓

6. Assign any eligible younger siblings to the same group as their enrolled older siblings

3. MODELING THE RANDOMIZATION PROTOCOL AND BOUNDING THE UNKNOWN PARAMETERS

Since no precise description of the full Perry randomization protocol exists, we do not know who was exchanged in the third step and who was transferred in the fifth step, making a standard bounding analysis intractable. To address this problem, we assume that experimenters satisfice¹² in seeking “balance” in the baseline covariate means of treatment and control groups, while facing capacity constraints on special home visits for children of working mothers.

Using this model, we bound the level of covariate balance deemed acceptable by the experimenters at the end of the first three stages of the protocol. We also bound the number of possible transfers at the fifth stage of the assignment procedure. Our model and the identified bounds are used to construct worst-case randomization tests using least favorable null distributions (for treatment effects). While the details differ, the approach readily generalizes to the class of compromised re-randomization designs discussed by Bruhn and McKenzie (2009).

3.1. Formalizing the Randomization Protocol

We first model the Perry randomization protocol and later discuss its generalizability. Let ${\mathcal{S}}_c$ be the set of unique identifiers of participants in cohort¹³ c ∈ {0, 1, 2, 3, 4} with no elder siblings already enrolled in the Perry Preschool Project. The cardinality of the set of singletons is $\left|{\mathcal{S}}_c\right|$.¹⁴ The participants in the set ${\mathcal{S}}_c$ are ranked according to their IQs by the Perry staffers, using an undocumented method to break any ties. The participants with odd and even ranks are then split into two undesignated groups, with $\lceil \left|{\mathcal{S}}_c\right|/2\rceil$ and $\lfloor |{\mathcal{S}}_c\mid /2\rfloor$ members, respectively.¹⁵ Staffers exchange participants between the two groups until the mean levels of four variables (Stanford–Binet IQ, index of socioeconomic status, gender, and mother’s working status) are balanced to their satisfaction.¹⁶ The exact metric the staffers used to determine satisfactory covariate balance is not documented.

We assume that they use Hotelling’s two-sample t-squared statistic ${\tau }_c^2$, which is equivalent to the Mahalanobis distance metric often used in matching.¹⁷ However, for each cohort’s initial groups (partially identified in Section 3.2), the Hotelling statistic and raw mean differences do not correspond to their possible minimum values and are sometimes far away from them.¹⁸ Thus, it appears in terms of this model that program officials were satisficing rather than optimizing (minimizing covariate imbalance) in constructing the two groups.

This process results in a partition $\left({\mathcal{A}}_c^{*},{\mathcal{B}}_c^{*}\right)$ of the set ${\mathcal{S}}_c$ chosen uniformly from

$${\mathbb{U}}_c\left({\delta }_c\right)=\left\{(\mathcal{A},\mathcal{B}):\mathcal{A}\subset {\mathcal{S}}_c,\mathcal{B}={\mathcal{S}}_c\backslash \mathcal{A},\left|\mathcal{A}\right|=\lceil \left|{\mathcal{S}}_c\right|/2\rceil ,{\tau }_c^2(\mathcal{A},\mathcal{B})⩽{\delta }_c\right\},$$ (3.1)

where δ_c is a satisficing threshold that captures how stringent or lax the Perry staffers were in trying to balance the mean levels of the two groups.¹⁹ Note that the above set is invariant to the choice of any strictly increasing transformation of the Hotelling statistic and the corresponding satisficing threshold. Define $D_{i,c}^{(0)}$ as an indicator of whether participant $i\in {\mathcal{S}}_c$ belongs to ${\mathcal{A}}_c^{*}$. In other words, $D_{i,c}^{(0)}=\mathbb{I}\left\{i\in {\mathcal{A}}_c^{*}\right\}$.

In the next stage, the Perry staffers flip a fair coin to determine whether ${\mathcal{A}}_c^{*}$ or ${\mathcal{B}}_c^{*}$ becomes the preliminary treatment group. Let Q_c be an indicator of whether the coin flip results in a head. If Q_c = 1, then ${\mathcal{B}}_c^{*}$ becomes the treatment group. If Q_c = 0, then ${\mathcal{A}}_c^{*}$ becomes the treatment group. Let $D_{i,c}^{(1)}$ denote membership in the preliminary treatment group. Thus

$$D_{i,c}^{(1)}=Q_c(1-D_{i,c}^{(0)})+\left(1-Q_c\right)D_{i,c}^{(0)}.$$ (3.2)

In the next step, some children of working mothers initially placed in the treatment group are transferred to the control group.²⁰ To model this process, we introduce additional notation. Define M_i as an indicator of whether participant i’s mother was working at baseline. Cohorts 0 and 1 were both randomized in the fall of 1962, while each of the remaining cohorts were randomized in successive years from 1963 through 1965. For cohorts 2 and higher, i.e., for c ∈ {2, 3, 4}, let m_c be the number of children of working mothers initially placed in the treatment group: $m_c={\sum }_{i\in {\mathcal{S}}_c}{M}_i{D}_{i,c}^{(1)}$. For the entry cohorts, let m_0,1 be the number of children of working mothers initially placed in the treatment group for cohorts 0 and 1, that is, $m_{0,1}={\sum }_{c\in \{0,1\}}{\sum }_{i\in {\mathcal{S}}_c}{M}_i{D}_{i,c}^{(1)}$.

Define η_c as a parameter indicating the maximum number of children of working mothers in cohort c ∈ {2, 3, 4} for whom special arrangements could be made to enable special home visits.²¹ We define η_0,1 to be the parameter indicating the maximum number of children of working mothers in the pooled cohorts 0 and 1 for whom special home visitation arrangements could be made, averting their transfer to the control group if placed in the initial treatment group.²²

Special arrangements are made for min(η_0,1, m_0,1) children of working mothers in the entry cohorts and for min(η_c, m_c) such children in each cohort c ∈ {2, 3, 4} to enable special home visits, as opposed to weekday home visits for children of non-working mothers. If there are any remaining children with working mothers in the initial treatment group, they are transferred to the control group, potentially increasing covariate imbalance.²³ We assume that the Perry staffers impartially choose (with equal probability) the children for whom the special accommodations are made.²⁴ To formalize this assumption, let V_i,c be a binary indicator for whether the participant $i\in {\mathcal{S}}_c$ was placed the initial treatment group, had a working mother, and remained in the treatment group through special accommodations for home visits. The vector $\left(V_{i,c}:i\in {\mathcal{S}}_c,M_i{D}_{i,c}^{(1)}=1\right)$ is assumed to be drawn uniformly from the set $\{v\in {\{0,1\}}^{m_c}:\Vert v{\Vert }_1=\min \left({\eta }_c,m_c\right)\}$ for all c ∈ {2, 3, 4}. Since the two entry cohorts face a common capacity constraint with respect to special home visitation accommodations, the vector $\left(V_{i,c}:i\in {\mathcal{S}}_0\cup {\mathcal{S}}_1,M_i{D}_{i,c}^{(1)}=1\right)$ is assumed to be drawn uniformly from the set $\left\{v\in {\{0,1\}}^{m_{0,1}}:\Vert v{\Vert }_1=\min \left({\eta }_{0,1},m_{0,1}\right)\right\}$. In addition, V_i,c = 0 for a participant $i\in {\mathcal{S}}_c$ if $M_i{D}_{i,c}^{(1)}=0$ for all c ∈ {0, 1, 2, 3, 4}.²⁵ In this notation, the participant’s final treatment status $D_{i,c}^{(2)}$ is given by

$$D_{i,c}^{(2)}=M_i{D}_{i,c}^{(1)}{V}_{i,c}+(1-M_i{D}_{i,c}^{(1)})D_{i,c}^{(1)}.$$ (3.3)

Any Perry subjects with identifiers not in ${\bigcup }_{c=0}^4{\mathcal{S}}_c$ receive the same treatment status as their elder siblings already enrolled in the Perry study. Thus, the final treatment status D_i of the i-th subject is given by $D_i=D_{i,c}^{(2)}$ if $i\in {\bigcup }_c{\mathcal{S}}_c$. Otherwise, if participant i is not from a freshly recruited family, the assignment is given by D_i = D_h, where the h-th subject is the i-th subject’s eldest sibling enrolled in the Perry study, if $i\in \mathcal{I}\backslash {\bigcup }_c{\mathcal{S}}_c$, where $\mathcal{I}$ is the set of all Perry subjects.

3.2. Partially Identifying Satisficing Thresholds and Capacity Constraints

Using the Perry data, we now demonstrate how we can partially identify the satisficing thresholds δ_c and the special home visitation capacity constraints η_c using the last three cohorts as examples. We then present a general framework for partially identifying these parameters.

Example 1: Wave 2 (A Case with One Transfer in the Last Stage)

Wave 2	Di= 0	Di = 1	Total
Mi = 0	9	7	16
Mi = 1	3	3	6
Total	12	10	22

This example discusses the steps for bounding the parameters δ₂ and η₂ in wave 2. Shown above is a contingency table of mother’s working status M_i and final treatment status D_i for participants $i\in {\mathcal{S}}_2$ in cohort 2 with no elder siblings already enrolled in the Perry study. There are 22 such participants in total. Since there are an even number of participants, each of the initial two undesignated groups (as well as the initial treatment and control groups in the next stage) would have been $\lceil \left|{\mathcal{S}}_2\right|/2\rceil =\lfloor \left|{\mathcal{S}}_2\right|/2\rfloor =11$ in size. However, we observe only 10 members in the final treatment group but 12 members in the final control group. This implies that there must have been one transfer from the initial treatment group to the control group. Thus, one of the 3 children of working mothers in the final control group was in the initial treatment group. However, we do not know exactly which one of these children was transferred, so there are 3 possibilities for the initial treatment group. Let ${\tau }_{2,1}^2$, ${\tau }_{2,2}^2$, ${\tau }_{2,3}^2$ be the Hotelling two-sample statistics for these three possibilities. One of these Hotelling statistics was the actual level of covariate imbalance between the initial treatment and control groups, and this level of imbalance is assumed to be within the satisficing threshold δ₂ of the Perry staffers (by construction). Thus, ${\delta }_2⩾\min \left\{{\tau }_{2,1}^2,{\tau }_{2,2}^2,{\tau }_{2,3}^2\right\}$. In addition, m₂ = 4, since there must have been 4 children of working mothers in the initial treatment group, consisting of the 3 participants who remain in the final treatment group and the 1 participant who was transferred to the control group. Since 3 of the initial 4 participants remained in the final treatment group, min(η₂, m₂) = min(η₂, 4) = 3, implying that η₂ = 3, the only solution that satisfies the equality. We next present two other examples.

Example 2: Wave 3 (A Case with 1 or 2 Transfers in the Last Stage)

Wave 3	Di = 0	Di= 1	Total
Mi = 0	7	9	16
Mi = 1	5	0	5
Total	12	9	21

In this example, we show a contingency table of M_i and D_i for the 21 participants $i\in {\mathcal{S}}_3$ in cohort 3. The sizes of the larger and smaller undesignated groups would have been $\lceil \left|{\mathcal{S}}_3\right|/2\rceil =11$ and $\lfloor \left|{\mathcal{S}}_3\right|/2\rfloor =10$, respectively. However, either of these two groups could have been the initial treatment group. Since there are 12 members in the final control group and 9 in the final treatment group, there are two possible cases: if the initial treatment group had 10 members, there would have been 10 – 9 = 1 transfer; but if it had 11 members, there would have been 11 − 9 = 2 transfers. Since the number of transfers involving children of working mothers is either 1 or 2, the number of possibilities for the initial treatment group is $\left(\begin{array}{l}5\hfill \\ 1\hfill \end{array}\right)+\left(\begin{array}{c}5\\ 2\end{array}\right)=5+10=15$, as all the 5 children of working mothers in this cohort are in the control group. Let ${\tau }_{3,1}^2,\dots ,{\tau }_{3,15}^2$ be the Hotelling statistics for those 15 possibilities. Then, ${\delta }_3⩾\min \left\{{\tau }_{3,1}^2,\dots ,{\tau }_{3,15}^2\right\}$. In addition, m₃ ∈ {1, 2}, since m₃ is the sum of the number of transfers (either 1 or 2) and the number of remaining children in the final treatment group (0 in this cohort). As no working mother remained in the treatment group, min(η₃, m₃) = 0, implying that η₃ = 0, which is the only number consistent with this equality. Thus, the Perry staffers were unable to provide special home visitation accommodations for any of the participants in this cohort.

Example 3: Wave 4 (A Case with No Transfers in the Last Stage)

Wave 4	Di = 0	Di = 1	Total
Mi = 0	5	10	15
Mi = 1	4	0	4
Total	9	10	19

In this example, we show a contingency table of M_i and D_i for the 19 participants $i\in {\mathcal{S}}_4$ in cohort 4. The sizes of the larger and smaller undesignated groups would have been $\lceil \left|{\mathcal{S}}_3\right|/2\rceil =10$ and $\lfloor \left|{\mathcal{S}}_3\right|/2\rfloor =9$. These coincide with the final sizes of the treatment and control groups, respectively. Accordingly, we can conclude that the observed final treatment group was indeed the initial treatment group for this cohort. Otherwise, the control group would have had at least 10 members. Let ${\tau }_{4,1}^2$ be the Hotelling statistic for the observed partition of ${\mathcal{S}}_4$ based on the final treatment status. Then, ${\delta }_4⩾{\tau }_{4,1}^2$. In addition, note that there are no children of working mothers in the final treatment group, which was also the initial treatment group, and so m₄ = 0. Since min(η₄, m₄) = min(η₄, 0) = 0 and there are 4 members with working mothers in total, it follows that η₄ ∈ {0, 1, 2, 3, 4}, because any of these values satisfies the equality. Thus, the observed data for cohort 4 is not helpful in bounding η₄.

Partial Identification of the Satisficing Thresholds and Capacity Constraints in General

We now present a general characterization of how to partially identify the satisficing thresholds and capacity constraints on special home visits.

Wave c	Di = 0	Di = 1	Total
Mi = 0	ω 0,0	ω 0,1	ω 0,*
Mi = 1	ω 1,0	ω 1,1	ω 1,*
Total	ω *,0	ω *,1	|Sc|

The above contingency table shows that there are ω_m,d participants with (M_i, D_i) = (m, d) ∈ {0, 1}² among the set of participants ${\mathcal{S}}_c$ in cohort c.²⁶ The total number of children with non-working mothers is ω_{0,* =} ω_0,0 + ω_0,1 and that of working mothers is ω_1,* = ω_1,0 + ω_1,1. The total number of participants in the final control group is ω_{*,0 =} ω_0,0 + ω_1,0 and that in the final treatment group is ω_*,1 = ω_0,1 + ω_1,1. The partial identification of the satisficing thresholds and capacity constraints would vary depending on whether $\left|{\mathcal{S}}_c\right|$ is even or odd and also depending on whether ${\omega }_{*,1}=\lceil \left|{\mathcal{S}}_c\right|/2\rceil$ or ${\omega }_{*,1}<\lceil \left|{\mathcal{S}}_c\right|/2\rceil$. We discuss each of these cases separately.

First, consider the case where $\left|{\mathcal{S}}_c\right|$ is even or odd and ${\omega }_{*,1}=\lceil \left|{\mathcal{S}}_c\right|/2\rceil$. In this case, since the size of the final treatment group remains the same as that of the initial treatment group, there must have been no transfers of children with working mothers from the treatment group to the control group. Since the final treatment group is the same as the initial one, we can bound the satisficing threshold as follows: ${\delta }_c⩾{\tau }_{c,1}^2$, where ${\tau }_{c,1}^2$ is the Hotelling statistic for the partition of ${\mathcal{S}}_c$ based on the final treatment status. In addition, since there are no transfers, the number of children of working mothers in the initial treatment group m_c equals ω_1,1. Since min(η_c, ω_1,1) = ω_1,1, it follows that η_c ∈ {ω_1,1,…,ω_1,*}, i.e., the number of slots available for special home visits must be at least the number ω_1,1 observed in the data.

Second, consider the case where $\left|{\mathcal{S}}_c\right|$ is even and ${\omega }_{*,1}=\lceil \left|{\mathcal{S}}_c\right|/2\rceil$. As in Example 1, in this case it is clear that the number of transfers in the final stage must have been ${\chi }_c=\lceil \left|{\mathcal{S}}_c\right|/2\rceil -{\omega }_{*,1}$, which is a positive number. The χ_c transferred children must be among the ω_1,0 members with working mothers in the final control group. Thus, there are $\left(\begin{array}{c}{\omega }_{1,0}\\ {\chi }_c\end{array}\right)$ possibilities for the initial treatment group. Let ${\vartheta }_c^{\delta }$ be the set containing the Hotelling statistics for those possibilities. Then, ${\delta }_c⩾\min {\vartheta }_c^{\delta }$. In addition, there must have been m_c = ω_1,1 + χ_c children with working mothers in the initial treatment group. It remains to determine which values of η_c are consistent with the equality min(η_c, ω_1,1 + χ_c) = ω_1,1. Since χ_c > 0, it follows that η_c = ω_1,1.

Third, consider the case where $\left|{\mathcal{S}}_c\right|$ is odd and ${\omega }_{*,1}<\lceil \left|{\mathcal{S}}_c\right|/2\rceil .$. As in Example 2, in this case there are two possibilities for the number χ_c of transfers in the final stage. Specifically, ${\chi }_c\in \left\{\lfloor \left|{\mathcal{S}}_c\right|/2\rfloor -{\omega }_{*,1},\lceil \left|{\mathcal{S}}_c\right|/2\rceil -{\omega }_{*,1}\right\}$. These χ_c transferred children must be among the ω_1,0 members with working mothers in the final control group. Thus, there are $\left(\begin{array}{c}{\omega }_{1,0}\\ \lfloor \left|{\mathcal{S}}_c\right|/2\rfloor -{\omega }_{*,1}\end{array}\right)+\left(\begin{array}{c}{\omega }_{1,0}\\ \lfloor \left|{\mathcal{S}}_c\right|/2\rfloor -{\omega }_{*,1}\end{array}\right)$ possibilities for the initial treatment group. Let ${\vartheta }_c^{\delta }$ be the set containing the Hotelling statistics for those possibilities. Then, ${\delta }_c⩾\min {\vartheta }_c^{\delta }$. The number m_c of children with working mothers initially assigned treatment is either equal to ${\omega }_{1,1}+\lfloor \left|{\mathcal{S}}_c\right|/2\rfloor -{\omega }_{*,1}$ or equal to ${\omega }_{1,1}+\lceil \left|{\mathcal{S}}_c\right|/2\rceil -{\omega }_{*,1}$. Let ${\vartheta }_c^{\delta }$ be the set of values of η_c consistent with the equality min(η_c, m_c) = ω_1,1. If $m_c={\omega }_{1,1}+\lceil \left|{\mathcal{S}}_c\right|/2\rceil -{\omega }_{*,1}$, then η_c = ω_1,1, since $\lceil \left|{\mathcal{S}}_c\right|/2\rceil >{\omega }_{*,1}$. However, if $m_c={\omega }_{1,1}+\lfloor \left|{\mathcal{S}}_c\right|/2\rfloor -{\omega }_{*,1}$, there are two sub-cases: if $\lfloor \left|{\mathcal{S}}_c\right|/2\rfloor >{\omega }_{*,1}$, then η_c = ω_1,1; but if $\lfloor \left|{\mathcal{S}}_c\right|/2\rfloor ={\omega }_{*,1}$, then η_c ∈ {ω_1,1 …, ω_1,*}. Therefore, the special home visiting slots can be partially identified as follows: ${\eta }_c\in {\vartheta }_c^{\eta }$, where ${\vartheta }_c^{\eta }=\left\{{\omega }_{1,1}\dots ,{\omega }_{1,*}\right\}$ if $\lfloor \left|{\mathcal{S}}_c\right|/2\rfloor ={\omega }_{*,1}$, and ${\vartheta }_c^{\eta }=\left\{{\omega }_{1,1}\right\}$ if $\lfloor \left|{\mathcal{S}}_c\right|/2\rfloor >{\omega }_{*,1}$.

This general characterization of the partial identification of satisficing thresholds δ_c applies to all cohorts c ∈ {0, 1, 2, 3, 4} but that of the special home visiting capacity constraints η_c applies only to cohorts c ∈ {2, 3, 4}. However, similar reasoning can be used to partially identify the capacity constraint η_0,1 for pooled cohorts 0 and 1.²⁷

3.3. Applicability of Our Approach to Other Compromised Experiments

Our approach can be applied to many of the studies that Bruhn and McKenzie (2009) criticize, especially experiments using undocumented re-randomization. All of these experiments have the feature that some criterion determines “satisfactory balance.” For example, Bruhn and McKenzie (2009) quote a survey response that says, “[experimenters] regressed variables like education on assignment to treatment, and then re-did the assignment if these coefficients were ‘too big.’” With appropriate modifications, our model of satisficing thresholds directly applies to experiments conducted in such a subjective and incompletely documented manner. Suitable adjustments include replacing Hotelling’s statistic in our model with studentized regression coefficients (selected by pretesting or otherwise) or other metrics actually used to measure covariate imbalance between the treatment and control groups. Our methods for partially identifying the underlying randomization rules can be used when the subjective satisficing thresholds are not documented. Even though we only use one balancing criterion (Hotelling’s statistic) for dimensionality reduction in our definition of ${\mathbb{U}}_c(\cdot )$, it can be trivially modified to accommodate multiple balancing criteria. In addition, if the experiment has strata instead of cohorts, the c’s in our model would correspond to strata.

If an experiment does not have transfers after forming the intermediate treatment and control groups, then there are no capacity constraints, i.e., the η_c’s play no role. However, in some social experiments, post-randomization transfer of some participants from the control to the treatment group can occur if additional funding for the intervention becomes available. For example, wait-list control groups are used in some clinical studies. While this is the reverse of what occurred in the Perry experiment, our model (with appropriate modifications) can be readily applied. Overall, our approach can be adapted to analyze a variety of compromised experiments across multiple disciplines.

4. HYPOTHESES OF INTEREST AND INFERENCE

The conventional way to analyze randomized experiments is to posit a null hypothesis that the average effect of treatment is zero and to proceed testing it with large-sample methods using asymptotic or bootstrap distributions. Given the relatively small size of many experimental samples, reliance on large sample methods can be problematic.²⁸

In some settings permutation tests can be used to test the null hypothesis that the outcomes in the control group have the same distribution as those in the treatment group without relying on large-sample theory. Permutation tests exploit the property that treatment and control labels within the same strata are exchangeable under the null hypothesis of a common outcome distribution. If randomization of the treatment status did not involve explicit stratification on baseline covariates, permutation tests need to make restrictive assumptions on the strata within which treatment and control labels are exchangeable. This approach is used by Heckman et al. (2010a).²⁹ They assume that conditioning on covariates solves the problem of post-random assignment reallocation but without any explicit model for why it is effective in doing so.³⁰

This paper uses knowledge of the randomization protocol to draw inferences about treatment effects. Once a precise null hypothesis is specified, we can determine the distribution of estimates generated by the randomization scheme and assess the statistical significance of the observed treatment effects.

In this section, we first formulate our hypotheses of interest. We then discuss conventional inferential procedures. Finally, we introduce worst-case (least favorable) randomization tests and discuss how to conduct them using stochastic approximations, and then we compare our methods with alternative approaches for inference with imperfect randomization.

4.1. Hypotheses of Interest

Let Y¹ be the treated outcome, Y⁰ be the untreated outcome, Z represent background variables, and F be their joint distribution at the population level. The conventional approach tests the hypothesis ${\mathcal{H}}_{\mathcal{C}}$ of equality of means, i.e.,

$${\mathcal{H}}_{\mathcal{C}}:{\mathbb{E}}_F\left[Y^1-Y^0\right]=0,$$ (4.4)

assuming that the realizations $\left(Y_i^1,Y_i^0,Z_i\right)$ of those variables for individual i are distributed according to F for all $i\in \mathcal{P}$, where $\mathcal{P}$ is the set of experimental subjects. Because each participant in our sample is assigned to either the treatment group or the control group, we only observe either $Y_i^1$ or $Y_i^0$ for each $i\in \mathcal{P}$. The hypothesis ${\mathcal{H}}_{\mathcal{C}}$ is traditionally tested by applying large-sample methods to the observed data ${\left(Y_i,D_i,Z_i\right)}_{i\in \mathcal{P}}$, where D_i is the treatment status, $Y_i=D_i{Y}_i^1+\left(1-D_i\right)Y_i^0$, and Z_i is the vector of pre-program covariates. It is of interest to conduct tests about the missing counterfactual outcomes within our sample, even though tests of population-level parameters are more commonly employed.

Instead of appealing to some hypothetical long-run sampling experiment to conduct inference, we seek knowledge of the sample in hand. One hypothesis of interest is whether the average treatment effect within the sample is zero, i.e.,

$${\mathcal{H}}_{\mathcal{N}}:\frac{1}{N_{\mathcal{P}}}\sum _{i\in \mathcal{P}}\left(Y_i^1-Y_i^0\right)=0,$$ (4.5)

³¹ where $N_{\mathcal{P}}=|\mathcal{P}|$. A special case of ${\mathcal{H}}_{\mathcal{N}}$ is the sharp null hypothesis of no treatment effects whatsoever for each participant:

$${\mathcal{H}}_{\mathcal{F}}:{\tau }_i\equiv Y_i^1-Y_i^0=0$$ (4.6)

for all $i\in \mathcal{P}$,³² Fisher’s (1935; 1925) null hypothesis. It involves a joint test of zero individual treatment effects and is trivially equivalent to ${\mathcal{H}}_{\mathcal{N}}$ if there is no heterogeneity in the treatment effects. The advantage of Fisher’s hypothesis ${\mathcal{H}}_{\mathcal{F}}$ is that it provides a testable model in which all the counterfactual outcomes are specified.³³ Such hypothesis testing can be conducted using our knowledge of the randomization protocol without relying on large-sample theory. With all the counterfactual outcomes specified, we can learn about the randomization distribution of treatment effects, and we can gauge the extent to which the observed data can be rationalized using the specified null model.³⁴

Hypothesis ${\mathcal{H}}_{\mathcal{N}}$ nests the sharp null hypothesis ${\mathcal{H}}_{\mathcal{F}}$. In general there are many configurations of the individual treatment effects that are all consistent with ${\mathcal{H}}_{\mathcal{N}}$. Thus, to test ${\mathcal{H}}_{\mathcal{N}}$ using only limited knowledge of the randomization protocol, we would need to test each one of all the sharp null hypotheses like ${\mathcal{H}}_{\mathcal{F}}$ that imply ${\mathcal{H}}_{\mathcal{N}}$.³⁵ However, a non-rejection of ${\mathcal{H}}_{\mathcal{F}}$ implies non-rejection of ${\mathcal{H}}_{\mathcal{N}}$, and so testing other sharp null hypotheses may not be necessary if we are unable to reject ${\mathcal{H}}_{\mathcal{F}}$. Of course, a rejection of ${\mathcal{H}}_{\mathcal{F}}$ would not imply a rejection of ${\mathcal{H}}_{\mathcal{N}}$. The latter is a very conservative criterion. We next discuss conventional hypothesis testing procedures.

4.2. Conventional Hypothesis Testing Procedures

For tests of population-level parameters such as ${\mathcal{H}}_{\mathcal{C}}$ in equation (4.4), the most commonly reported measure of statistical significance is the asymptotic p-value. For completely randomized experiments, it can be interpreted as the p-value based on a large-sample approximation of the distribution of an estimator, say difference-in-means, over all possible randomizations under the null hypothesis ${\mathcal{H}}_{\mathcal{N}}$ Neyman (1923). Li et al. (2018) derive an asymptotic theory of the difference-in-means estimator in experiments involving rerandomization with a pre-specified balancing rule using the Mahalanobis distance, for which the asymptotic distribution of the estimator is a linear combination of normal and truncated normal variables. Resampling methods are also widely used to quantify statistical uncertainty. For example, the bootstrap standard error is reported in many research papers with an associated bootstrap p-value.

Permutation tests are often used when researchers are interested in testing whether treatment and control groups have a common outcome distribution without relying on large-sample theory. Such tests rely on the property that the treatment and control labels are exchangable within each stratum of the experiment under the null hypothesis of a common distribution. In their permutation tests, Heckman et al. (2010a) use strata defined by wave, gender, and indicator for above-median socioeconomic status, assuming that experimental labels within each stratum are exchangeable. To compare their permutation procedures with the methods developed in this paper, we use a simplified version of their permutation tests using block permutations within cohorts of eldest participant-siblings (whose treatment statuses determine that of their younger participant-siblings).

In the Perry context, Heckman et al. (2020) develop an extension of permutation tests to account for imperfect randomization. In this paper, we offer an alternative design-based approach to conduct inference for a broader class of compromised experiments. We first present our approach and then compare it with theirs.

4.3. Worst-Case Randomization Tests

This paper advocates and uses worst-case approximate randomization tests to analyze the Perry data. Fisher’s sharp null hypothesis ${\mathcal{H}}_{\mathcal{F}}$ specifies all the counterfactual outcomes, which are imputed according to the hypothesis using the observed data. If we knew the exact randomization protocol of the Perry experiment, we could measure where the observed test statistic falls along its exact randomization distribution, i.e., the distribution of the test statistic over all possible treatment status vectors that could have been hypothetically generated by the randomization protocol. The more extreme the observed test statistic falls along the null distribution, the more incompatible the observed data would be with the sharp null hypothesis. However, for Perry and many other social experiments, the exact randomization protocol is unknown: even in our stylized model of the randomization protocol, the satisficing thresholds and capacity constraints are only partially identified. To account for this ambiguity, we could in theory conduct randomization tests³⁶ over the set of all possible randomization protocols. Thus, we could conduct the worst-case randomization test using the least favorable distribution among all the possible randomization distributions. This results in the following worst-case p-value that serves as an upper bound for the true randomization p-value:

$$p_w=\underset{{\gamma }^{*}\in \mathrm{\Xi }}{\sup }{\mathbb{P}}_{{\mathrm{\Lambda }}_{\gamma }*}\left\{T\left({\tilde{D}}_{{\gamma }^{*}}\right)⩾T(D)\right\},$$ (4.7)

where Ξ is the partially identified set for γ = (δ₀,…, δ₄, η_0,1, η₂, η₃, η₄), the vector of true values of parameters (satisficing thresholds and capacity constraints), ${\mathbb{P}}_{{\mathrm{\Lambda }}_{\gamma }*}$ represents probability under the probability space Λ_γ* of randomizations generated by the protocol parameterized by γ*, ${\tilde{D}}_{{\gamma }^{*}}$ represents a random treatment status vector defined on the probability space Λ_γ*, D denotes the observed treatment status vector, and T(·) is the chosen test statistic such that T(·) maps a treatment status vector to a real number measuring the magnitude of the outcome difference between the treatment and control groups. Since the sharp null hypothesis specifies counterfactual outcomes, the data ${\left(Y_i^0,Y_i^1,Z_i\right)}_{i\in \mathcal{P}}$ are fixed according to ${\mathcal{H}}_{\mathcal{F}}$, and the only random variation in the above construction comes from the randomization protocol. The sample space Ω_γ* of the uniform probability space Λ_γ*, on which the random treatment status vector ${\tilde{D}}_{{\gamma }^{*}}$ is defined, is given by

$${\mathrm{\Omega }}_{{\gamma }^{*}}=\left(\underset{c=0}{\overset{4}{\times }}{\mathbb{U}}_c\left({\delta }_c^{*}\right)\right)\times {\mathrm{\Omega }}_{Q,V_{{\gamma }^{*}}},$$ (4.8)

where $\mathbb{U}\left({\delta }_0^{*},\dots ,{\delta }_4^{*}\right)\equiv {\times }_{c=0}^4{\mathbb{U}}_c\left({\delta }_c^{*}\right)$ is the Cartesian product of the sets of admissible partitions of ${\mathcal{S}}_c$ (in the initial step of the protocol) across all cohorts c ∈ {0, …, 4}, and ${\mathrm{\Omega }}_{Q,V_{{\gamma }^{*}}}$ is the Cartesian product of the sample spaces for all other random variables, characterizing the outcomes Q_c of the coin flips and vectors of variables V_i,c used for determining which children of working mothers are transferred from the treatment to control group in the last step across all cohorts c ∈ {0, …, 4}, used in the randomization protocol parameterized by γ*.³⁷ Using this notation we establish the following proposition for any α ∈ (0, 1):

Proposition 1.

Under the model of randomization protocol in Section 3, the hypothesis test that rejects the sharp null hypothesis whenever p_w ≤ α controls the Type I error rate at level α.

Proof.

Let $p_{{\gamma }^{*}}(D)\equiv {\mathbb{P}}_{{\mathrm{\Lambda }}_{\gamma }*}\left\{T\left({\tilde{D}}_{{\gamma }^{*}}\right)⩾T(D)\right\}$ for all γ* ∈ Ξ, let $p_w(D)\equiv {\sup }_{{\gamma }^{*}\in \mathrm{\Xi }}{p}_{{\gamma }^{*}}(D)$ represent the worst-case p-value, and let ${\psi }_{\alpha }(D)\equiv \mathbb{I}\left\{p_w(D)⩽\alpha \right\}$ be the test for a given D, a realization of the random treatment status vector defined on the probability space Λ_γ, where γ is the true value of the model parameter. Since p_γ(D) ≤ p_w(D) by construction, it follows that ${\mathbb{E}}_{{\mathrm{\Lambda }}_{\gamma }}\left[{\psi }_{\alpha }(D)\right]={\mathbb{E}}_{{\mathrm{\Lambda }}_{\gamma }}\left[\mathbb{I}\left\{p_w(D)⩽\alpha \right\}\right]⩽{\mathbb{E}}_{{\mathrm{\Lambda }}_{\gamma }}\left[\mathbb{I}\left\{p_{\gamma }(D)⩽\alpha \right\}\right]={\mathbb{P}}_{{\mathrm{\Lambda }}_{\gamma }}\left\{p_{\gamma }(D)⩽\alpha \right\}⩽\alpha$ under ${\mathcal{H}}_{\mathcal{F}}$. This is a trivial extension of the simple standard argument used to show the finite-sample validity of randomization tests (see Lehmann and Romano, 2005). The proposition can be equivalently stated in terms of a critical value for the test statistic, as in Heckman et al. (2020).

Although it would be ideal to compute the exact value of p_w, it is computationally not feasible. As is common practice in computing permutation and randomization p-values (see Lehmann and Romano, 2005), we resort to stochastic approximations. Even so, there are two challenges in estimating the worst-case p-value. First, approximating the probability ${\mathbb{P}}_{{\mathrm{\Lambda }}_{\gamma }*}\left\{T\left({\tilde{D}}_{{\gamma }^{*}}\right)⩾T(D)\right\}$ for a given value γ* ∈ Ξ is computationally demanding. Second, estimating p_w based on such tail probability estimates for a finite number of points on Ξ is also challenging. We tackle these two challenges sequentially and discuss how we handle some forms of stochastic approximation errors.

4.3.1. Approximating Tail Probabilities of Randomization Distributions

The first challenge is to approximate ${\mathbb{P}}_{{\mathrm{\Lambda }}_{{\gamma }^{*}}}\left\{T\left({\tilde{D}}_{{\gamma }^{*}}\right)⩾T(D)\right\}$ for a given value γ* in the partially identified set, i.e., for ${\gamma }^{*}=\left({\delta }_0^{*},\dots ,{\delta }_4^{*},{\eta }_{0,1}^{*},{\eta }_2^{*},{\eta }_3^{*},{\eta }_4^{*}\right)\in \mathrm{\Xi }$. Our approach is to break up the sample space of Λ_γ* into two parts, compute the tail probability (measuring how extreme the observed test statistic is in its randomization null distribution) for each of these two parts, and then use the law of total probability and Monte Carlo methods to get the desired final result. To do so, we introduce additional notation. Let ${\delta }_c^{†}$ be the lower bound of the partially identified set for the true value of the satisficing threshold δ_c for c ∈ {0, …, 4}. Then, for any given value ${\delta }_c^{*}⩾{\delta }_c^{†}$, observe that

$${\mathbb{U}}_c\left({\delta }_c^{*}\right)={\mathcal{X}}_c\cup {\mathcal{Y}}_c\left({\delta }_c^{*}\right),$$ (4.9)

where

$${\mathcal{X}}_c=\left\{(\mathcal{A},\mathcal{B})\in {\mathbb{U}}_c(\infty ):{\tau }_c^2(\mathcal{A},\mathcal{B})⩽{\delta }_c^{†}\right\}$$ (4.10)

and

$${\mathcal{Y}}_c\left({\delta }_c^{*}\right)=\left\{(\mathcal{A},\mathcal{B})\in {\mathbb{U}}_c(\infty ):{\delta }_c^{†}<{\tau }_c^2(\mathcal{A},\mathcal{B})⩽{\delta }_c^{*}\right\},$$ (4.11)

for all c ∈ {0, …, 4}. Thus, we can use ${\mathbb{U}}_c(\infty )$, which is the set with an infinite satisficing threshold such that all allowed partitions of ${\mathcal{S}}_c$ are satisfactory, to construct ${\mathcal{X}}_c$, ${\mathcal{Y}}_c\left({\delta }_c^{*}\right)$, and ${\mathbb{U}}_c\left({\delta }_c^{*}\right)$. The set ${\mathcal{X}}_c$ has elements with Hotelling statistics below the lower bound ${\delta }_c^{†}$ of the partially identified set for the satisficing threshold. The other set ${\mathcal{Y}}_c\left({\delta }_c^{*}\right)={\mathbb{U}}_c\left({\delta }_c^{*}\right)\backslash {\mathcal{X}}_c$ has elements with Hotelling statistics between ${\delta }_c^{†}$ and ${\delta }_c^{*}$. Let ${\tilde{\mathrm{\Omega }}}^{\mathcal{X}}={\times }_{c=0}^4{\mathcal{X}}_c$ be the Cartesian product of the sets ${\mathcal{X}}_c$ across cohorts, and let ${\tilde{\mathrm{\Omega }}}_{{\gamma }^{*}}^{\mathcal{Y}}=\mathbb{U}\left({\delta }_0^{*},\dots ,{\delta }_4^{*}\right)\backslash {\tilde{\mathrm{\Omega }}}^{\mathcal{X}}=\left\{X_{c=0}^4{\mathbb{U}}_c\left({\delta }_c^{*}\right)\right\}\backslash {\tilde{\mathrm{\Omega }}}^{\mathcal{X}}$. Both ${\tilde{\mathrm{\Omega }}}^{\mathcal{X}}$ and ${\tilde{\mathrm{\Omega }}}_{{\gamma }^{*}}^{\mathcal{Y}}$ can be constructed using $\mathbb{U}(\infty ,\dots ,\infty )$ by discarding elements in their respective complements. Since the sets ${\mathcal{X}}_c$ do not depend on the values ${\delta }_c^{*}$, the set ${\tilde{\mathrm{\Omega }}}^{\mathcal{X}}$ remains constant. Notice that

$${\mathrm{\Omega }}_{{\gamma }^{*}}=\left({\tilde{\mathrm{\Omega }}}^{\mathcal{X}}\cup {\tilde{\mathrm{\Omega }}}_{{\gamma }^{*}}^{\mathcal{Y}}\right)\times {\mathrm{\Omega }}_{Q,V_{{\gamma }^{*}}}=\left({\tilde{\mathrm{\Omega }}}^{\mathcal{X}}\times {\mathrm{\Omega }}_{Q,V_{\gamma *}}\right)\cup \left({\tilde{\mathrm{\Omega }}}_{{\gamma }^{*}}^{\mathcal{Y}}\times {\mathrm{\Omega }}_{Q,V_{{\gamma }^{*}}}\right).$$ (4.12)

Let ${\mathrm{\Lambda }}_{{\gamma }^{*}}^{\mathcal{X}}$ and ${\mathrm{\Lambda }}_{{\gamma }^{*}}^{\mathcal{Y}}$ be the uniform probability spaces over the sample spaces ${\mathrm{\Omega }}_{{\gamma }^{*}}^{\mathcal{X}}\equiv {\tilde{\mathrm{\Omega }}}^{\mathcal{X}}\times {\mathrm{\Omega }}_{Q,V_{\gamma }*}$ and ${\mathrm{\Omega }}_{{\gamma }^{*}}^{\mathcal{Y}}\equiv {\tilde{\mathrm{\Omega }}}_{{\gamma }^{*}}^{\mathcal{Y}}\times {\mathrm{\Omega }}_{Q,V_{{\gamma }^{*}}}$, respectively. In addition, let

$$x\left({\gamma }^{*}\right)\equiv \frac{\left|{\mathrm{\Omega }}_{{\gamma }^{*}}^{\mathcal{X}}\right|}{\left|{\mathrm{\Omega }}_{{\gamma }^{*}}\right|}=\frac{\left|{\tilde{\mathrm{\Omega }}}^{\mathcal{X}}\times {\mathrm{\Omega }}_{Q,V_{{\gamma }^{*}}}\right|}{\left|{\mathrm{\Omega }}_{{\gamma }^{*}}\right|}=\frac{\left|{\tilde{\mathrm{\Omega }}}^{\mathcal{X}}\right|\cdot \left|{\mathrm{\Omega }}_{Q,V_{{\gamma }^{*}}}\right|}{\left|{\tilde{\mathrm{\Omega }}}^{\mathcal{X}}\cup {\tilde{\mathrm{\Omega }}}_{{\gamma }^{*}}^{\mathcal{Y}}\right|\cdot \left|{\mathrm{\Omega }}_{Q,V_{{\gamma }^{*}}}\right|}=\frac{\left|{\tilde{\mathrm{\Omega }}}^{\mathcal{X}}\right|}{\left|{\tilde{\mathrm{\Omega }}}^{\mathcal{X}}\cup {\tilde{\mathrm{\Omega }}}_{{\gamma }^{*}}^{\mathcal{Y}}\right|},$$ (4.13)

which is the proportion of elements in the sample space Ω_γ* belonging to ${\mathrm{\Omega }}_{{\gamma }^{*}}^{\mathcal{X}}$. Note that the last equality above implies that x(γ*) can be simply computed with the sets ${\tilde{\mathrm{\Omega }}}^{\mathcal{X}}$ and ${\tilde{\mathrm{\Omega }}}_{{\gamma }^{*}}^{\mathcal{Y}}$ constructed using $\mathbb{U}(\infty ,\dots ,\infty )$.³⁸ Then, by the law of total probability, we have that

$${\mathbb{P}}_{{\mathrm{\Lambda }}_{\gamma }*}\left\{T\left({\tilde{D}}_{{\gamma }^{*}}\right)⩾T(D)\right\}=x\left({\gamma }^{*}\right)\cdot {\mathbb{P}}_{{\mathrm{\Lambda }}_{{\gamma }^{*}}^{\chi }}\left\{T\left({\tilde{D}}_{{\gamma }^{*}}^{\mathcal{X}}\right)⩾T(D)\right\}+y\left({\gamma }^{*}\right)\cdot {\mathbb{P}}_{{\mathrm{\Lambda }}_{{\gamma }^{*}}^{\mathcal{Y}}}\left\{T\left({\tilde{D}}_{{\gamma }^{*}}^{\mathcal{Y}}\right)⩾T(D)\right\},$$ (4.14)

where ${\tilde{D}}_{{\gamma }^{*}}^{\mathcal{X}}$ and ${\tilde{D}}_{{\gamma }^{*}}^{\mathcal{Y}}$ represent random treatment status vectors defined on the probability spaces ${\mathrm{\Lambda }}_{{\gamma }^{*}}^{\mathcal{X}}$ and ${\mathrm{\Lambda }}_{{\gamma }^{*}}^{\mathcal{Y}}$, respectively, and y(γ*) = 1 – x(γ*). Since the sample spaces in the model are large, we use Monte Carlo draws from the probability spaces through rejection sampling to stochastically approximate the tail probability ${\mathbb{P}}_{{\mathrm{\Lambda }}_{\gamma }*}\left\{T\left({\tilde{D}}_{{\gamma }^{*}}\right)⩾T(D)\right\}$.^39,40 Our approach provides a feasible way to estimate ${\mathbb{P}}_{{\mathrm{\Lambda }}_{\gamma }*}\left\{T\left({\tilde{D}}_{{\gamma }^{*}}\right)⩾T(D)\right\}$ for points γ* in Ξ efficiently using rejection sampling.

4.3.2. Estimating and Bounding the Worst-Case Tail Probability

The second challenge is to estimate or bound the worst-case tail probability p_w. Taking the supremum of tail probabilities over points in the set Ξ may seem intractable, since Ξ is the Cartesian product of a finite set and a non-compact set.⁴¹ However, we exploit the fact that ${\mathbb{U}}_c(\nu )={\mathbb{U}}_c(\infty )$ for all $\nu ⩾{\mathrm{\Delta }}_c$, where ${\mathrm{\Delta }}_c=\max \left\{{\tau }_c^2(\mathcal{A},\mathcal{B}):(\mathcal{A},\mathcal{B})\in {\mathbb{U}}_c(\infty )\right\}$, since ${\mathbb{U}}_c(\infty )$ is a finite set, for all c ∈ {0, …, 4}. Let Ξ° be the compact subset of Ξ given by

$${\mathrm{\Xi }}^{°}=\left\{\tilde{\gamma }\equiv \left({\tilde{\delta }}_0,\dots ,{\tilde{\delta }}_4,{\tilde{\eta }}_{0,1},{\tilde{\eta }}_2,{\tilde{\eta }}_3,{\tilde{\eta }}_4\right)\in \mathrm{\Xi }:{\delta }_c^{†}⩽{\tilde{\delta }}_c⩽{\mathrm{\Delta }}_c\forall c\right\}.$$ (4.15)

It then follows that

$$p_w\equiv \underset{{\gamma }^{*}\in \mathrm{\Xi }}{\sup }{\mathbb{P}}_{{\mathrm{\Lambda }}_{\gamma }*}\left\{T\left({\tilde{D}}_{{\gamma }^{*}}\right)⩾T(D)\right\}=\underset{{\gamma }^{*}\in {\mathrm{\Xi }}^{°}}{\max }{\mathbb{P}}_{{\mathrm{\Lambda }}_{\gamma }*}\left\{T\left({\tilde{D}}_{{\gamma }^{*}}\right)⩾T(D)\right\}.$$ (4.16)

Thus, it suffices to estimate the worst-case tail probability over the set Ξ°, which is compact.⁴² We use stochastic approximations for this purpose as well. It is computationally infeasible to compute a p-value for each of the points in the set Ξ° and take the maximum of those p-values. To deal with this challenge, we first write ${\mathrm{\Xi }}^{°}={\bigcup }_{l=1}^L{\mathrm{\Xi }}_l^{°}$, where ${\mathrm{\Xi }}_1^{°},\dots ,{\mathrm{\Xi }}_L^{°}$ are disjoint hyper-rectangles that form a partition of the set Ξ°. In our application, L = 20, and each hyper-rectangle represents the partially identified set for (δ₀, …, δ₄) at fixed values of (η_0,1, η₂, η₃, η₄).⁴³ Then, note that

$$p_w=\max \left\{p_w^1,\dots ,p_w^L\right\},$$ (4.17)

where

$$p_w^l=\underset{{\gamma }^{*}\in {\mathrm{\Xi }}_l^o}{\max }{\mathbb{P}}_{{\mathrm{\Lambda }}_{\gamma }*}\left\{T\left({\tilde{D}}_{{\gamma }^{*}}\right)⩾T(D)\right\}$$ (4.18)

for l ∈ {1, …, L}. We approximate $p_w^l$ for each l ∈ {1, …, L} using the p-values $p_{14}^l,\dots ,p_{(S)}^l$ arranged in descending order for S = 900 uniformly sampled random points on the set ${\mathrm{\Xi }}_l^{°}$.⁴⁴

We estimate $p_w^l$ for each l ∈ {1, …, L} using the maximum order statistic ${\tilde{p}}_M^l$:

$${\tilde{p}}_M^l=\underset{1⩽s⩽S}{\max }{p}_{(s)}^l=p_{(1)}^l,$$ (4.19)

which converges almost surely to $p_w^l$ as S → ∞. However, this estimate may have stochastic approximation error. One way to deal with stochastic approximation-related uncertainty in ${\tilde{p}}_M^l$ is by constructing a confidence band for $p_w^l$. To do so, we construct an upper bound based on de Haan’s (1981) 90% asymptotic confidence band for the true maximum using the S randomly sampled p-values. The upper confidence bound ${\tilde{p}}_D^l$ is given by

$${\tilde{p}}_D^l=p_{(1)}^l+\left(p_{(1)}^l-p_{(2)}^l\right)\cdot K_{dH}^l,$$ (4.20)

where $K_{dH}^l$ is a factor provided by de Haan (1981) for the 90% asymptotic confidence bound.⁴⁵ Thus, the 90% confidence interval for $p_w^l$ is given by $\left[{\tilde{p}}_M^l,{\tilde{p}}_D^l\right]$. Finally, the true worst-case p-value p_w can be approximated by the worst-case maximum (max.) p-value ${\tilde{p}}_M$ given by

$${\tilde{p}}_M=\max \left\{{\tilde{p}}_M^1,\dots ,{\tilde{p}}_M^L\right\},$$ (4.21)

and its upper confidence bound is given by the worst-case de Haan p-value ${\tilde{p}}_D$ as follows:

$${\tilde{p}}_D=\max \left\{{\tilde{p}}_D^1,\dots ,{\tilde{p}}_D^L\right\},$$ (4.22)

which provides at least 90% coverage as S → ∞. Of course, these stochastic approximations affect the exact finite-sample validity of the resulting hypothesis tests, but the validity of these approximations can be arbitrarily increased with adequate additional computational power. This is an issue common to most resampling methods in statistics (see Lehmann and Romano, 2005).

In the previous discussion, the test statistic T(·) used to compute the worst-case tail probability is left general. There is reason to suspect that the choice of the test statistic matters, as shown for permutation tests by Chung and Romano (2013, 2016). Wu and Ding (2020) show that using studentized test statistics in certain randomization tests can control type I error asymptotically under certain weak null hypotheses while preserving finite-sample validity under sharp null hypotheses. Their theory ignores covariates and is limited to completely randomized factorial experiments and stratified or clustered experiments. However, they conjecture that “the strategy [of using studentized test statistics to make randomization tests asymptotically robust under weak null hypotheses while retaining their finite-sample validity under sharp null hypotheses may also be] applicable for experiments with general treatment assignment mechanisms” Wu and Ding (2020). While we do not attempt to prove or disprove their conjecture in the Perry experimental setting, we take it seriously given their results for certain randomization tests along with Chung and Romano’s (2013, 2016) results for permutation tests. Thus, we provide worst-case p-values using both the nonstudentized and studentized test statistics.

4.3.3. Multiple Testing

Since ${\mathbb{P}}_{{\mathrm{\Lambda }}_{\gamma }}\left\{p_w⩽\alpha \right\}⩽\alpha$ under ${\mathcal{H}}_{\mathcal{F}}$ for any α ∈ (0, 1) by Proposition 1, Holm (1979) tests of multiple hypotheses based on the worst-case p-values would also have finite-sample validity. Multiplicity-adjusted p-values can be computed as follows. Let ρ₍₁₎, …, ρ_(K) be the associated single worst-case p-values arranged in ascending order. Then, the Holm stepdown p-values adjusted for multiple testing are given by ${\varrho }_{(k)}={\max }_{j⩽k}\min \left(1,(K-j+1){\rho }_{(j)}\right)$ for k ∈ {1, …, K}. However, these adjusted p-values can be even more conservative because they assume least favorable dependence structure between the single worst-case p-values Romano et al. (2010), making this the “worst-case” of the “worst-case.” However, slightly less conservative multiple hypothesis tests are available in the literature (see Romano and Wolf, 2005; Romano and Shaikh, 2010). Since it is unclear how much improvement in terms of power they provide relative to Holm tests in our context, we do not discuss the more computationally involved stepdown procedures in this paper.

4.4. Comparing Methods for Inference with Imperfect Randomization

Our approach complements that of Heckman et al. (2020), who improve on the methodology of Heckman et al. (2010a) by (i) exploiting a symmetry generated by the Perry randomization protocol, (ii) using finite-sample inference that accounts for imperfect randomization, and (iii) making transfers in the fifth step of the randomization protocol depend on a binary variable indicating whether the mother is available for home visits, assuming program infrastructure is available to support it. It is only partially observed in their model. We also exploit the symmetry: Q_c represents the result of a fair coin flip to determine which of the two initially undesignated groups becomes the intended treatment group. However, we model other features of the protocol differently.

Heckman et al. (2020) model the reassignment of children of some working women by introducing a partially observed binary variable U_i that equals 1 if the mother of participant i was unavailable for home visits and 0 otherwise. It is known only for children of non-working mothers, for whom U_i = 0, and for the children of working mothers in the final treatment group, who also have U_i = 0. For children of working mothers in the control group, U_i is not known and could be either 0 or 1. To deal with this difficulty, Heckman et al. (2020) construct two permutation tests. The first test sets U_i to 0 for all children of working mothers in the final control group and conducts a generalized permutation test accordingly. The second test: (i) samples a vector of U_i from the space of possibilities for U_i; (ii) conducts a generalized permutation test given the sampled vector of U_i and obtains the corresponding permutation p-value; and (iii) repeats steps (i) and (ii) until the space of possibilities is exhausted. It then takes the maximum p-value among the computed p-values. Our worst-case inferential methods are similar in spirit. However, there are three key differences between our approach and theirs.

First, Heckman et al. (2020) interpret U_i as a fixed trait of mothers regardless of the (random) circumstances facing program administrators. However, whether or not a working mother and her child are visited at home (through special arrangements, e.g., on a weekend) depends, at least in part, on the availability and capacity constraints of the Perry staff. While U_i = 0 for non-working mothers in both papers, we do not view U_i as a fixed binary trait of working mothers. Consistent with our review of the randomization protocol, we assume that children of working mothers are able to participate in the program if special arrangements, such as weekend home visits, are made for them. In our model, there are capacity constraints for making special arrangements, so only a limited number of slots are available.⁴⁶ In their model, if U_i = 1 for a working mother, her child would always be placed in the control group, because she would not accept any special accommodations even if provided by the Perry staff. Unlike the V_i,c variable that determines post-randomization transfers in our model, the U_i characteristic in their model is allowed to be related to potential outcomes, but this is a consequence of its interpretation as a fixed trait of mothers independent of the capacities of program administrators.

Second, their procedure assumes that “some participants were exchanged between the treatment and control groups in order to ‘balance’ gender and socio-economic status score while keeping Stanford-Binet IQ score roughly constant.”⁴⁷ However, as shown in Appendix B, Perry data from wave 4 reveal that the exchanges were not necessarily between consecutively ranked IQ pairs. Our approach accommodates this feature while also making more explicit the notion of balance.

Third, on a more minor note, we incorporate the five children (out of the original 128) who dropped out of the study due to extraneous reasons, since those five children were also a part of the initial randomization protocol. Our approach can also more readily be applied than that of Heckman et al. (2020) to a variety of compromised experiments, including many discussed by Bruhn and McKenzie (2009). We next demonstrate that there are important differences between inferences obtained from our procedure and theirs.

5. REANALYSIS OF HECKMAN ET AL. (2020)

This section uses the methods developed in this paper to reconsider the conclusions reached by Heckman et al. (2020) on the Perry participants through age 40. In a companion paper, we apply these methods to analyze fresh samples of participants through age 55 and their adult children Heckman (2020). Analyzing the new wave of data in this paper would raise a variety of new issues about sampling procedures better left for another occasion.

We first list our estimators of treatment effects. Using the corresponding test statistics, we then apply our worst-case inferential methods to reanalyze the results in Heckman et al. (2020).

5.1. Estimators and Test Statistics for Hypothesis Testing

A variety of test statistics and estimators can be used in our approach and that of Heckman et al. (2020). Our empirical work focuses conventional ones often used in practice. Let D_i represent the treatment status of participant i, and let Z_i be the vector of baseline variables.⁴⁸ In addition, let Y_i denote the observed outcome of interest of participant i in a relevant subsample $\mathcal{P}$ containing $N_{\mathcal{P}}=|\mathcal{P}|$ participants, and let $Y_i^d$ be the counterfactual outcome of participant i when his or her treatment status D_i is fixed at d ∈ {0, 1}. In switching regression notation Quandt (1958, 1972),

$$Y_i=D_i{Y}_i^1+\left(1-D_i\right)Y_i^0.$$ (5.23)

The average treatment effect $\overline{\tau }$ in the subsample $\mathcal{P}$ given by

$$\overline{\tau }=\frac{1}{N_{\mathcal{P}}}\sum _{i\in \mathcal{P}}\left(Y_i^1-Y_i^0\right)$$ (5.24)

and is conventionally estimated by a difference-in-means (DIM) estimator that takes raw mean differences between non-attrited treated and control observations. However, the randomization procedures used in Perry and other similar experiments only justify conditional independence: $\left(Y_i^1,Y_i^0\right)⫫D_i\mid Z_i$. Exploiting this property and controlling for Z_i in a regression of Y_i on D_i and Z_i using complete case observations, we obtain the ordinary least squares (OLS) estimator.⁴⁹ It would be desirable to relax linearity, but the Perry sample size makes this impractical.

All of these estimators assume that non-response is determined at random or at random conditional on observed covariates. Let R_i be an indicator of whether Y_i is missing. It could depend on the treatment status D_i and the pre-program covariates Z_i. The augmented inverse probability weighting (AIPW) estimator allows for this possibility by using the weaker assumption that $Y_i\perp R_i\mid D_i,Z_i$. The AIPW estimator of the treatment effect is

$${\widehat{\mathrm{\Pi }}}_{\text{AIPW}}=\frac{1}{N_{\mathcal{P}}}\sum _{i\in \mathcal{P}}\left({\widehat{\pi }}_i^1-{\widehat{\pi }}_i^0\right),$$ (5.25)

where

$${\widehat{\pi }}_i^d={\widehat{Y}}_i^d+\frac{\mathbb{I}\left\{R_i=1,D_i=d\right\}}{{\widehat{\lambda }}_i^d{\widehat{\varphi }}_i^d}\left(Y_i^d-{\widehat{Y}}_i^d\right).$$ (5.26)

In this expression, ${\widehat{Y}}_i^d$ is the ordinary least squares (projection) estimator of the conditional expectation $\mathbb{E}\left[Y_i\mid Z_i,D_i=d,R_i=1\right]$ for d ∈ {0, 1}, ${\widehat{\varphi }}_i^d$ is an estimator of Pr(D_i = d|Z_i), the i-th participant’s propensity of being in experimental state d, and ${\widehat{\lambda }}_i^d$ is an estimator of $\mathrm{Pr}\left(R_i^1=1\mid Z_i,D_i=d\right)$, which is the propensity of having a non-missing outcome after fixing the treatment status D_i, for d ∈ {0, 1}. Propensity scores are often estimated using logits. The AIPW estimator adjusts the outcome based on pre-program covariates and corrects for covariate imbalance and various forms of non-response.⁵⁰ It is asymptotically unbiased and has a double robustness property: the estimator is robust to misspecification of either the propensity score models or the models for counterfactual outcomes, but not both.⁵¹ For this reason, the AIPW estimator is sometimes preferred over the DIM and OLS estimators.⁵² We use the studentized version of the AIPW estimate as our main test statistic in our empirical analysis.⁵³

We could use a local average treatment effect (LATE) estimator, and other standard estimation methods dealing with imperfect compliance, if we knew each observation’s initial treatment status. However, in the Perry example, we do not know which members were transferred from the initial treatment group to the control group in the last step of the randomization protocol. Given this problem, we do not present LATE estimates.⁵⁴

5.2. Empirical Analysis

We first conduct a head-to-head comparison of Heckman et al.’s (2020) methods and ours using the same outcomes they analyze. Additionally, to compare the impact of using mean differences versus AIPW test statistics in the conventional inferential approaches and our design-based worst-case inference, we extend the outcomes they study and analyze data on violent crime.

Tables 2 and 3 below report our reanalyses of Heckman et al. (2020), analyzing each outcome one at a time using the doubly robust attrition-adjusted AIPW estimator. Tables 4 and 5 provide stepdown p-values for the outcomes based on multiple testing. Extended versions of these tables are presented in Appendices E through H using alternative test statistics for inference.⁵⁵

Table 2.

Reanalysis of Male Outcomes in Heckman et al. (2020) Using Single Tests

Variable	Age	Untreated mean	Treated mean	AIPW estimate	Asymptotic p-value	Bootstrap p-value	Permutation p-value	Worst-case max. p	Worst-case de Haan p
Stanford–Binet IQ	4	83.077	94.909	8.988	0.0000	0.0000	0.0004	0.0049	0.0056
Stanford–Binet IQ	5	84.793	95.400	9.167	0.0000	0.0002	0.0004	0.0071	0.0071
Stanford–Binet IQ	6	85.821	91.485	3.056	0.0557	0.0512	0.0712	0.0872	0.1229
Stanford–Binet IQ	7	87.711	91.121	1.576	0.2040	0.2143	0.2104	0.2002	0.2198
Stanford–Binet IQ	8	89.054	88.333	−3.829	0.0512	0.0719	0.0556	0.1461	0.2396
Stanford–Binet IQ	9	89.026	88.394	−4.167	0.0398	0.0577	0.0472	0.1289	0.1457
Stanford–Binet IQ	10	86.026	83.697	−4.722	0.0225	0.0412	0.0292	0.0707	0.1022
CAT reading score	14	9.000	13.926	1.815	0.2957	0.3221	0.3112	0.3488	0.3823
CAT arithmetic score	14	8.107	16.000	3.095	0.2410	0.2629	0.2608	0.2909	0.3216
CAT language score	14	6.536	14.333	5.029	0.0815	0.0995	0.1076	0.1771	0.2098
CAT mechanics score	14	6.964	15.556	5.979	0.0538	0.0638	0.0712	0.1333	0.1467
CAT spelling score	14	11.536	18.519	3.171	0.2652	0.2865	0.2600	0.2734	0.3016
High school graduate	19	0.513	0.485	0.015	0.4550	0.4540	0.4868	0.5607	0.6036
Vocational training	40	0.333	0.394	0.071	0.2762	0.2886	0.2932	0.3984	0.4353
Highest grade completed	19	11.282	11.364	0.087	0.3902	0.3901	0.4240	0.4589	0.6118
Grade point average	19	1.794	1.814	−0.035	0.4366	0.4336	0.4328	0.5414	0.6563
Total non-juvenile arrests	40	11.718	7.455	−3.895	0.0461	0.0368	0.0668	0.1019	0.1795
Total crime cost	40	775.901	424.679	−313.263	0.1376	0.1361	0.1764	0.2060	0.2329
Total charges	40	13.385	9.000	−4.132	0.0678	0.0579	0.0920	0.1242	0.1789
Non-victimless charges	40	3.077	1.485	−1.444	0.0274	0.0238	0.0372	0.0755	0.1332
Currently employed	19	0.410	0.545	0.147	0.1263	0.1315	0.1292	0.2989	0.3351
Unemployed last year	19	0.128	0.242	0.102	0.1817	0.1827	0.2148	0.3050	0.4203
Jobless months (past 2 yrs)	19	3.816	5.281	1.367	0.2572	0.2501	0.2928	0.3371	0.4217
Currently employed	27	0.564	0.600	0.089	0.2156	0.2259	0.2452	0.3348	0.3703
Unemployed last year	27	0.308	0.242	−0.081	0.2238	0.2190	0.2388	0.3460	0.3776
Jobless months (past 2 yrs)	27	8.795	5.133	−3.868	0.0438	0.0430	0.0588	0.1193	0.2030
Currently employed	40	0.500	0.700	0.266	0.0089	0.0075	0.0204	0.0444	0.0640
Unemployed last year	40	0.462	0.364	−0.143	0.0843	0.0957	0.0912	0.1629	0.1671
Jobless months (past 2 yrs)	40	10.750	7.233	−4.758	0.0154	0.0200	0.0188	0.0632	0.0722

Note: This table reports p-values for single hypothesis tests of treatment effects on various outcomes of male participants at the given ages. The inferences are based on the studentized AIPW test statistic.

Table 3.

Reanalysis of Female Outcomes in Heckman et al. (2020) Using Single Tests

Variable	Age	Untreated mean	Treated mean	AIPW estimate	Asymptotic p-value	Bootstrap p-value	Permutation p-value	Worst-case max. p	Worst-case de Haan p
Stanford–Binet IQ	4	83.692	96.360	13.425	0.0000	0.0000	0.0004	0.0034	0.0040
Stanford–Binet IQ	5	81.650	94.316	14.157	0.0008	0.0006	0.0064	0.0273	0.0382
Stanford–Binet IQ	6	87.160	90.913	5.271	0.0365	0.0281	0.0636	0.0820	0.0959
Stanford–Binet IQ	7	86.000	92.520	7.347	0.0313	0.0154	0.0564	0.0858	0.1232
Stanford–Binet IQ	8	83.600	87.840	4.669	0.1144	0.0896	0.1704	0.2040	0.2141
Stanford–Binet IQ	9	83.043	86.739	4.809	0.0633	0.0679	0.1128	0.1992	0.2628
Stanford–Binet IQ	10	81.789	86.750	6.480	0.0277	0.0323	0.0596	0.1600	0.1976
CAT reading score	14	8.444	16.500	7.345	0.0130	0.0128	0.0268	0.0573	0.0935
CAT arithmetic score	14	6.889	11.818	6.227	0.0102	0.0138	0.0284	0.0544	0.0710
CAT language score	14	7.833	19.455	11.923	0.0009	0.0013	0.0044	0.0178	0.0232
CAT mechanics score	14	8.833	20.636	12.425	0.0014	0.0015	0.0072	0.0208	0.0269
CAT spelling score	14	10.722	29.500	18.270	0.0017	0.0042	0.0064	0.0180	0.0253
High school graduate	19	0.231	0.840	0.570	0.0000	0.0000	0.0004	0.0051	0.0075
Vocational training	40	0.077	0.240	0.183	0.0286	0.0494	0.0420	0.1165	0.2231
Highest grade completed	19	10.750	11.760	1.202	0.0023	0.0106	0.0120	0.0284	0.0500
Grade point average	19	1.527	2.415	0.958	0.0000	0.0155	0.0004	0.0112	0.0381
Total non-juvenile arrests	40	4.423	2.160	−1.938	0.0657	0.0795	0.0880	0.1566	0.1925
Total crime cost	40	293.497	22.165	−246.242	0.1475	0.1227	0.2436	0.2615	0.3036
Total charges	40	4.923	2.240	−2.309	0.0439	0.0528	0.0580	0.1366	0.1407
Non-victimless charges	40	0.308	0.040	−0.249	0.0365	0.0263	0.0612	0.0853	0.1201
Currently employed	19	0.154	0.440	0.297	0.0054	0.0048	0.0152	0.0585	0.0619
Unemployed last year	19	0.577	0.240	−0.354	0.0029	0.0033	0.0104	0.0313	0.0377
Jobless months (past 2 yrs)	19	10.421	5.217	−4.197	0.0723	0.1386	0.1140	0.2020	0.2780
Currently employed	27	0.545	0.800	0.215	0.0471	0.0604	0.0648	0.0960	0.1237
Unemployed last year	27	0.542	0.250	−0.269	0.0523	0.0457	0.0728	0.1663	0.2242
Jobless months (past 2 yrs)	27	10.455	6.240	−1.298	0.3328	0.3449	0.2916	0.4526	0.7373
Currently employed	40	0.818	0.833	−0.016	0.4536	0.4586	0.4912	0.6550	0.6802
Unemployed last year	40	0.409	0.160	−0.194	0.0807	0.1079	0.1324	0.1934	0.2386
Jobless months (past 2 yrs)	40	5.045	4.000	0.057	0.4910	0.4927	0.4700	0.6114	0.6379

Note: This table reports p-values for single hypothesis tests of treatment effects on various outcomes of female participants at the given ages. The inferences are based on the studentized AIPW test statistic.

Table 4.

Reanalysis of Male Outcomes in Heckman et al. (2020) Using Stepdown Tests

Variable	Age	Untreated mean	Treated mean	AIPW estimate	Asymptotic p-value	Bootstrap p-value	Permutation p-value	Worst-case max. p	Worst-case de Haan p
Stanford–Binet IQ	4	83.077	94.909	8.988	0.0001	0.0002	0.0028	0.0346	0.0394
Stanford–Binet IQ	5	84.793	95.400	9.167	0.0003	0.0012	0.0028	0.0425	0.0425
Stanford–Binet IQ	6	85.821	91.485	3.056	0.1593	0.2058	0.1888	0.3534	0.5112
Stanford–Binet IQ	7	87.711	91.121	1.576	0.2040	0.2143	0.2104	0.3866	0.5112
Stanford–Binet IQ	8	89.054	88.333	−3.829	0.1593	0.2058	0.1888	0.3866	0.5112
Stanford–Binet IQ	9	89.026	88.394	−4.167	0.1593	0.2058	0.1888	0.3866	0.5112
Stanford–Binet IQ	10	86.026	83.697	−4.722	0.1126	0.2058	0.1460	0.3534	0.5112
CAT reading score	14	9.000	13.926	1.815	0.7229	0.7886	0.7800	0.8202	0.9047
CAT arithmetic score	14	8.107	16.000	3.095	0.7229	0.7886	0.7800	0.8202	0.9047
CAT language score	14	6.536	14.333	5.029	0.3260	0.3980	0.4304	0.7084	0.8390
CAT mechanics score	14	6.964	15.556	5.979	0.2690	0.3190	0.3560	0.6667	0.7335
CAT spelling score	14	11.536	18.519	3.171	0.7229	0.7886	0.7800	0.8202	0.9047
High school graduate	19	0.513	0.485	0.015	1.0000	1.0000	1.0000	1.0000	1.0000
Vocational training	40	0.333	0.394	0.071	1.0000	1.0000	1.0000	1.0000	1.0000
Highest grade completed	19	11.282	11.364	0.087	1.0000	1.0000	1.0000	1.0000	1.0000
Grade point average	19	1.794	1.814	−0.035	1.0000	1.0000	1.0000	1.0000	1.0000
Total non-juvenile arrests	40	11.718	7.455	−3.895	0.1384	0.1103	0.2004	0.3057	0.5366
Total crime cost	40	775.901	424.679	−313.263	0.1384	0.1361	0.2004	0.3057	0.5366
Total charges	40	13.385	9.000	−4.132	0.1384	0.1158	0.2004	0.3057	0.5366
Non-victimless charges	40	3.077	1.485	−1.444	0.1096	0.0952	0.1488	0.3020	0.5327
Currently employed	19	0.410	0.545	0.147	0.3790	0.3946	0.3876	0.8968	1.0000
Unemployed last year	19	0.128	0.242	0.102	0.3790	0.3946	0.4296	0.8968	1.0000
Jobless months (past 2 yrs)	19	3.816	5.281	1.367	0.3790	0.3946	0.4296	0.8968	1.0000
Currently employed	27	0.564	0.600	0.089	0.4313	0.4380	0.4776	0.6697	0.7405
Unemployed last year	27	0.308	0.242	−0.081	0.4313	0.4380	0.4776	0.6697	0.7405
Jobless months (past 2 yrs)	27	8.795	5.133	−3.868	0.1313	0.1290	0.1764	0.3580	0.6089
Currently employed	40	0.500	0.700	0.266	0.0268	0.0225	0.0564	0.1333	0.1920
Unemployed last year	40	0.462	0.364	−0.143	0.0843	0.0957	0.0912	0.1629	0.1920
Jobless months (past 2 yrs)	40	10.750	7.233	−4.758	0.0309	0.0399	0.0564	0.1333	0.1920

Note: This table reports Holm stepdown p-values for multiple hypothesis tests of treatment effects on various outcomes of male participants at the given ages. The inferences are based on the studentized AIPW test statistic. The blocks used for multiple testing are indicated above using divider lines.

Table 5.

Reanalysis of Female Outcomes in Heckman et al. (2020) Using Stepdown Tests

Variable	Age	Untreated mean	Treated mean	AIPW estimate	Asymptotic p-value	Bootstrap p-value	Permutation p-value	Worst-case max. p	Worst-case de Haan p
Stanford–Binet IQ	4	83.692	96.360	13.425	0.0000	0.0000	0.0028	0.0239	0.0278
Stanford–Binet IQ	5	81.650	94.316	14.157	0.0046	0.0035	0.0384	0.1637	0.2290
Stanford–Binet IQ	6	87.160	90.913	5.271	0.1387	0.1125	0.2820	0.4099	0.4796
Stanford–Binet IQ	7	86.000	92.520	7.347	0.1387	0.0771	0.2820	0.4099	0.4929
Stanford–Binet IQ	8	83.600	87.840	4.669	0.1387	0.1359	0.2820	0.4801	0.5928
Stanford–Binet IQ	9	83.043	86.739	4.809	0.1387	0.1359	0.2820	0.4801	0.5928
Stanford–Binet IQ	10	81.789	86.750	6.480	0.1387	0.1125	0.2820	0.4801	0.5928
CAT reading score	14	8.444	16.500	7.345	0.0205	0.0255	0.0536	0.1088	0.1421
CAT arithmetic score	14	6.889	11.818	6.227	0.0205	0.0255	0.0536	0.1088	0.1421
CAT language score	14	7.833	19.455	11.923	0.0043	0.0064	0.0220	0.0889	0.1162
CAT mechanics score	14	8.833	20.636	12.425	0.0056	0.0064	0.0256	0.0889	0.1162
CAT spelling score	14	10.722	29.500	18.270	0.0056	0.0127	0.0256	0.0889	0.1162
High school graduate	19	0.231	0.840	0.570	0.0000	0.0000	0.0016	0.0202	0.0299
Vocational training	40	0.077	0.240	0.183	0.0286	0.0494	0.0420	0.1165	0.2231
Highest grade completed	19	10.750	11.760	1.202	0.0046	0.0318	0.0240	0.0567	0.1142
Grade point average	19	1.527	2.415	0.958	0.0000	0.0318	0.0016	0.0336	0.1142
Total non-juvenile arrests	40	4.423	2.160	−1.938	0.1461	0.1589	0.2320	0.4098	0.4803
Total crime cost	40	293.497	22.165	−246.242	0.1475	0.1589	0.2436	0.4098	0.4803
Total charges	40	4.923	2.240	−2.309	0.1461	0.1585	0.2320	0.4098	0.4803
Non-victimless charges	40	0.308	0.040	−0.249	0.1461	0.1051	0.2320	0.3413	0.4803
Currently employed	19	0.154	0.440	0.297	0.0107	0.0099	0.0312	0.1171	0.1237
Unemployed last year	19	0.577	0.240	−0.354	0.0088	0.0099	0.0312	0.0939	0.1132
Jobless months (past 2 yrs)	19	10.421	5.217	−4.197	0.0723	0.1386	0.1140	0.2020	0.2780
Currently employed	27	0.545	0.800	0.215	0.1412	0.1371	0.1944	0.2879	0.3712
Unemployed last year	27	0.542	0.250	−0.269	0.1412	0.1371	0.1944	0.3325	0.4485
Jobless months (past 2 yrs)	27	10.455	6.240	−1.298	0.3328	0.3449	0.2916	0.4526	0.7373
Currently employed	40	0.818	0.833	−0.016	0.9072	0.9173	0.9400	1.0000	1.0000
Unemployed last year	40	0.409	0.160	−0.194	0.2421	0.3237	0.3972	0.5803	0.7157
Jobless months (past 2 yrs)	40	5.045	4.000	0.057	0.9072	0.9173	0.9400	1.0000	1.0000

Note: This table reports Holm stepdown p-values for multiple hypothesis tests of treatment effects on various outcomes of female participants at the given ages. The inferences are based on the studentized AIPW test statistic. The blocks used for multiple testing are indicated above using divider lines.

In Tables 6 and 7, we reproduce Heckman et al.’s (2020) results and provide a side-by-side comparison of their inferences with our own. The most stringent (max-U) single p-values they report for the effects on the California Achievement Test (CAT) reading, arithmetic, language, mechanics, and spelling scores at age 14 in the male sample using the studentized DIM test statistic are 0.036, 0.086, 0.012, 0.023, 0.012, respectively, which are lower than the asymptotic p-values we report in Table 2. After adjusting for multiple testing, their adjusted max-U p-values are no more than 0.086, based on which they conclude that these effects are statistically significant. In contrast, using our approach, the worst-case maximum (single) p-values using studentized DIM test statistic are 0.171, 0.119, 0.075, 0.054, 0.123, respectively. As shown in our Table 2, using the studentized AIPW test statistic, our worst-case maximum p-values are 0.349, 0.291, 0.177, 0.133, 0.273, respectively,⁵⁶ implying that the effects on the CAT scores for males are not statistically significant. Of course, the stepdown p-values for these outcomes shown in Table 4 are also insignificant. Our inference for the female sample is qualitatively similar to theirs. As shown in Table 3, the block of CAT scores for females is statistically significant at the 10% level. However, the multiplicity-adjusted stepdown worst-case de Haan p-values in Table 5 are 0.12 or larger.

Table 6.

Comparing Heckman et al.’s (2020) DIM-Based Inference with Ours for Male Sample

		Heckman et al.’s (2020) p-values				Worst-case p-values using our method
Variable	Age	U = 0 p-value (unadj.)	U = 0 p-value (adjusted)	Max-U p-value (unadj.)	Max-U p-value (adjusted)	Worst-case max. p (unadjusted)	Worst-case max. p (adjusted)	Worst-case de Haan p (unadjusted)	Worst-case de Haan p (adjusted)
Stanford–Binet IQ	4	0.001	0.001	0.008	0.008	0.0048	0.0333	0.0053	0.0368
Stanford–Binet IQ	5	0.022	0.691	0.077	0.800	0.0076	0.0456	0.0095	0.0572
Stanford–Binet IQ	6	0.033	0.034	0.094	0.102	0.0249	0.1247	0.0306	0.1531
Stanford–Binet IQ	7	0.103	0.172	0.247	0.374	0.0782	0.3128	0.1185	0.4741
Stanford–Binet IQ	8	0.599	0.691	0.733	0.800	0.5480	1.0000	0.5743	1.0000
Stanford–Binet IQ	9	0.450	0.548	0.631	0.680	0.5580	1.0000	0.6044	1.0000
Stanford–Binet IQ	10	0.684	0.691	0.790	0.800	0.2568	0.7704	0.3509	1.0000
CAT reading score	14	0.017	0.035	0.036	0.086	0.1711	0.3572	0.1776	0.5135
CAT arithmetic score	14	0.032	0.035	0.086	0.086	0.1191	0.3572	0.1555	0.5135
CAT language score	14	0.001	0.004	0.012	0.027	0.0748	0.2994	0.1284	0.5135
CAT mechanics score	14	0.006	0.007	0.023	0.035	0.0535	0.2673	0.0681	0.3407
CAT spelling score	14	0.003	0.035	0.012	0.086	0.1225	0.3572	0.1503	0.5135
High school graduate	19	0.614	0.674	0.704	0.716	0.6322	1.0000	0.6718	1.0000
Vocational training	40	0.341	0.567	0.547	0.608	0.3674	1.0000	0.4281	1.0000
Highest grade completed	19	0.383	0.622	0.410	0.669	0.3659	1.0000	0.4031	1.0000
Grade point average	19	0.457	0.674	0.567	0.716	0.5133	1.0000	0.5493	1.0000
Total non-juvenile arrests	40	0.036	0.038	0.100	0.115	0.0701	0.2493	0.0896	0.3249
Total crime cost	40	0.037	0.049	0.042	0.143	0.1695	0.2493	0.2133	0.3249
Total charges	40	0.049	0.049	0.143	0.143	0.0946	0.2493	0.1038	0.3249
Non-victimless charges	40	0.025	0.037	0.063	0.091	0.0623	0.2493	0.0812	0.3249
Currently employed	19	0.050	0.164	0.224	0.290	0.2809	0.7459	0.3864	1.0000
Unemployed last year	19	0.901	0.901	0.922	0.922	0.2486	0.7459	0.3673	1.0000
Jobless months (past 2 yrs)	19	0.821	0.849	0.873	0.890	0.3291	0.7459	0.4019	1.0000
Currently employed	27	0.268	0.295	0.485	0.512	0.3629	0.6294	0.4130	0.7278
Unemployed last year	27	0.235	0.295	0.360	0.512	0.3147	0.6294	0.3639	0.7278
Jobless months (past 2 yrs)	27	0.020	0.020	0.036	0.051	0.0965	0.2894	0.1265	0.3795
Currently employed	40	0.103	0.116	0.130	0.146	0.0566	0.1697	0.0853	0.2559
Unemployed last year	40	0.154	0.154	0.216	0.216	0.1684	0.1697	0.2494	0.2559
Jobless months (past 2 yrs)	40	0.064	0.116	0.070	0.146	0.0693	0.1697	0.0957	0.2559

Note: This table compares inferences reported by Heckman et al. (2020) with the inferences obtained using our worst-case tests. The first two columns list the blocks of outcomes analyzed by Heckman et al. (2020). The next four columns reproduce their zero-U (U = 0) p-values and max-U p-values before and after adjusting for multiplicity of hypotheses. Since all of their tests are based on studentized DIM estimate, we report our inferences (using the studentized DIM test statistic) side by side for comparison. The last four columns report our worst-case maximum p-values and worst-case de Haan p-values before and after adjusting for multiplicity of hypotheses. The unadjusted p-values refer to single p-values that are unadjusted for multiplicity of hypotheses. The adjusted p-values refer to stepdown p-values after adjusting for multiple testing.

Table 7.

Comparing Heckman et al.’s (2020) DIM-Based Inference with Ours for Female Sample

		Heckman et al.’s (2020) p-values				Worst-case p-values using our method
Variable	Age	U = 0 p-value (unadj.)	U = 0 p-value (adjusted)	Max-U p-value (unadj.)	Max-U p-value (adjusted)	Worst-case max. p (unadjusted)	Worst-case max. p (adjusted)	Worst-case de Haan p (unadjusted)	Worst-case de Haan p (adjusted)
Stanford–Binet IQ	4	0.008	0.008	0.020	0.020	0.0052	0.0362	0.0066	0.0460
Stanford–Binet IQ	5	0.012	0.203	0.014	0.354	0.0183	0.1095	0.0500	0.2999
Stanford–Binet IQ	6	0.094	0.164	0.160	0.346	0.1397	0.5589	0.1476	0.5904
Stanford–Binet IQ	7	0.133	0.137	0.191	0.222	0.0734	0.3671	0.0978	0.4890
Stanford–Binet IQ	8	0.152	0.164	0.339	0.346	0.1487	0.5589	0.2143	0.5904
Stanford–Binet IQ	9	0.203	0.203	0.354	0.354	0.2134	0.5589	0.2272	0.5904
Stanford–Binet IQ	10	0.203	0.203	0.267	0.354	0.1398	0.5589	0.1748	0.5904
CAT reading score	14	0.078	0.082	0.136	0.167	0.0413	0.0825	0.0771	0.1542
CAT arithmetic score	14	0.035	0.082	0.074	0.167	0.1000	0.1000	0.1357	0.1542
CAT language score	14	0.008	0.070	0.020	0.144	0.0111	0.0514	0.0235	0.0941
CAT mechanics score	14	0.047	0.082	0.097	0.167	0.0116	0.0514	0.0287	0.0941
CAT spelling score	14	0.043	0.082	0.082	0.167	0.0103	0.0514	0.0115	0.0577
High school graduate	19	0.008	0.008	0.020	0.020	0.0050	0.0200	0.0075	0.0299
Vocational training	40	0.078	0.078	0.144	0.144	0.1233	0.1233	0.1489	0.1489
Highest grade completed	19	0.070	0.070	0.113	0.113	0.0249	0.0497	0.0474	0.0948
Grade point average	19	0.039	0.039	0.082	0.082	0.0079	0.0237	0.0094	0.0299
Total non-juvenile arrests	40	0.020	0.133	0.121	0.158	0.1150	0.2641	0.1337	0.3160
Total crime cost	40	0.024	0.133	0.082	0.158	0.0660	0.2641	0.0790	0.3160
Total charges	40	0.020	0.067	0.043	0.090	0.0979	0.2641	0.1382	0.3160
Non-victimless charges	40	0.125	0.133	0.158	0.158	0.0693	0.2641	0.0803	0.3160
Currently employed	19	0.008	0.031	0.035	0.090	0.0518	0.1200	0.0705	0.1742
Unemployed last year	19	0.024	0.031	0.074	0.090	0.0400	0.1200	0.0581	0.1742
Jobless months (past 2 yrs)	19	0.125	0.125	0.206	0.206	0.0807	0.1200	0.1044	0.1742
Currently employed	27	0.110	0.149	0.175	0.198	0.0712	0.2137	0.0867	0.2600
Unemployed last year	27	0.078	0.149	0.128	0.175	0.1026	0.2137	0.1325	0.2650
Jobless months (past 2 yrs)	27	0.110	0.149	0.166	0.198	0.1881	0.2137	0.2449	0.2650
Currently employed	40	0.442	0.442	0.567	0.567	0.4786	0.8773	0.5336	0.9701
Unemployed last year	40	0.047	0.070	0.113	0.160	0.0825	0.2475	0.1250	0.3750
Jobless months (past 2 yrs)	40	0.352	0.367	0.540	0.540	0.4386	0.8773	0.4850	0.9701

Note: This table compares inferences reported by Heckman et al. (2020) with the inferences obtained using our worst-case tests. The first two columns list the blocks of outcomes analyzed by Heckman et al. (2020). The next four columns reproduce their zero-U (U = 0) p-values and max-U p-values before and after adjusting for multiplicity of hypotheses. Since all of their tests are based on studentized DIM estimate, we report our inferences (using the studentized DIM test statistic) side by side for comparison. The last four columns report our worst-case maximum p-values and worst-case de Haan p-values before and after adjusting for multiplicity of hypotheses. The unadjusted p-values refer to single p-values that are unadjusted for multiplicity of hypotheses. The adjusted p-values refer to stepdown p-values after adjusting for multiple testing.

Table 4 reports stepdown p-values for male outcomes. No estimated effect (after age 5) remains statistically significant at the 10% level after adjusting for multiple hypothesis testing using the worst-case maximum or worst-case de Haan p-values. However, in Table 5, which presents stepdown analysis of female outcomes, the treatment effects on post-program outcomes (related to CAT scores and education) are statistically significant at the 10% level using our worst-case maximum p-value. Nevertheless, all of these effects on female outcomes, except for one (high school graduation), disappear when worst-case de Haan p-values are used.

Tables 2 through 5 use the studentized AIPW test statistic for inference. Heckman et al. (2020) use the studentized DIM test statistic instead. Tables 6 and 7 compare their inferences with ours using the same test statistic. The effects for males on post-program outcomes remain statistically insignificant at the 10% level using stepdown worst-case de Haan p-values, whereas treatment effects on CAT scores are statistically significant in Heckman et al.’s (2020) analysis.

The contrast between Tables 5 and 7 reveals the importance of the choice of test statistic. In Table 7, several effects on female outcomes are statistically significant at the 10% level using stepdown worst-case de Haan p-values based on the studentized DIM test statistic. However, in Table 5, using the studentized AIPW test statistic, only one effect (on high school graduation) after age 5 is statistically significant based on the worst-case de Haan p-values.

Heckman et al. (2020) do not analyze the Perry treatment effects on convictions for violent crime, which are substantial and play an important role in cost-benefit analyses of early childhood programs (see Heckman et al., 2010b). Using administrative data on the criminal activity of participants, we illustrate their importance and, at the same time, the importance of long-term follow-up. Tables 8, 9, and 10 provide estimates and measures of statistical significance of treatments effects in the pooled sample (of all participants) on cumulative convictions for violent misdemeanors and felonies at various ages. Appendix D presents expanded versions of these tables reporting inference for various estimators and test statistics for the pooled sample as well as the male and female subsamples. As shown in Table 9, the AIPW estimates of the treatment effect on cumulative violent misdemeanor convictions are below −0.5 at ages 30 and 40. The treatment effects on violent misdemeanor convictions are statistically significant at the 2% and 10% levels before and after multiple hypothesis testing, respectively, regardless of the method used for inference on the pooled sample (see Appendix D).

Table 8.

DIM-Based Single Hypothesis Tests on Cumulative Convictions for Violent Crime

Type	Age	Untreated mean	Treated mean	DIM estimate	Asymptotic p-value	Bootstrap p-value	Permutation p-value	Worst-case max. p	Worst-case de Haan p
Misdemeanor	30	0.5231	0.0517	−0.4714	0.0109	0.0021	0.0036	0.0122	0.0154
Misdemeanor	40	0.6825	0.0877	−0.5948	0.0033	0.0005	0.0004	0.0053	0.0065
Felony	30	0.2846	0.1897	−0.0950	0.2301	0.2263	0.2624	0.4057	0.5318
Felony	40	0.4762	0.1930	−0.2832	0.0333	0.0332	0.0384	0.0708	0.0900

Note: This table reports p-values for single hypothesis tests of treatment effects on cumulative misdemeanor and felony convictions for violent crime at ages 30 and 40, using the pooled sample of participants. The inferences are based on the studentized DIM (differece-in-means) test statistic.

Table 9.

AIPW-Based Single Hypothesis Tests on Cumulative Convictions for Violent Crime

Type	Age	Untreated mean	Treated mean	AIPW estimate	Asymptotic p-value	Bootstrap p-value	Permutation p-value	Worst-case max. p	Worst-case de Haan p
Misdemeanor	30	0.5231	0.0517	−0.5300	0.0064	0.0020	0.0024	0.0099	0.0154
Misdemeanor	40	0.6825	0.0877	−0.6491	0.0021	0.0010	0.0008	0.0133	0.0181
Felony	30	0.2846	0.1897	−0.0561	0.3174	0.3217	0.3488	0.4820	0.5078
Felony	40	0.4762	0.1930	−0.2052	0.0664	0.0778	0.0708	0.1543	0.1754

Note: This table reports p-values for single hypothesis tests of treatment effects on cumulative misdemeanor and felony convictions for violent crime at ages 30 and 40, using the pooled sample of participants. The inferences are based on the studentized AIPW test statistic.

Table 10.

AIPW-Based Multiple Hypothesis Tests on Cumulative Convictions for Violent Crime

Type	Age	Untreated mean	Treated mean	AIPW estimate	Asymptotic p-value	Bootstrap p-value	Permutation p-value	Worst-case max. p	Worst-case de Haan p
Misdemeanor	30	0.5231	0.0517	−0.5300	0.0192	0.0059	0.0072	0.0394	0.0617
Misdemeanor	40	0.6825	0.0877	−0.6491	0.0085	0.0039	0.0032	0.0398	0.0617
Felony	30	0.2846	0.1897	−0.0561	0.3174	0.3217	0.3488	0.4820	0.5078
Felony	40	0.4762	0.1930	−0.2052	0.1327	0.1556	0.1416	0.3086	0.3507

Note: This table reports Holm stepdown p-values for multiple hypothesis tests of treatment effects on cumulative misdemeanor and felony convictions for violent crime at ages 30 and 40, using the pooled sample of participants. The inferences are based on the studentized AIPW test statistic. All the above four variables, which represent cumulative crime outcomes at different ages, are treated as a block for multiple testing.

The choice of inferential method becomes more important in analyzing treatment effects on cumulative convictions for felonies. At age 30, there are no statistically significant treatment effects. At age 40, as shown in Table 9, the magnitude of the treatment effect is higher at about −0.21, which represents more than a four-tenths reduction in the control mean. However, using simple difference-in-means estimates and conventional p-values can be misleading. Using conventional p-values, the effect at age 40 appears to be statistically significant at the 10% level, as shown in Table 8. However, the design-based worst-case p-values, especially those associated with the AIPW estimate, are much higher. The worst-case de Haan p-values for the studentized DIM and AIPW estimates are about 0.090 and 0.175, respectively.

The four variables at ages 30 and 40 considered in Tables 8 and 9 are conceptually related, since they are cumulative crime outcomes measured at different ages. To account for this, we treat these outcomes as a single block of variables and conduct multiple hypothesis testing using the more conservative Holm stepdown procedure, producing results in Table 10. After multiple testing, the effects on cumulative convictions for violent misdemeanors remain statistically significant at the 10% level at both ages 30 and 40, whereas the the effects on violent felonies are insignificant at both ages. These analyses, and those in the appendices, show that use of small-sample inference and the method used to account for compromised randomization matter in analyzing the data. Failure to account for either can give a very positive spin to the Perry program. Accounting for it qualifies such conclusions. We have not, however, established the superiority of our approach. We have established that a very cautious design-based approach produces conservative inference, which by itself is not surprising. Our reanalysis of Heckman et al. (2020) is very conservative. Nonetheless, a few conclusions survive. We test Fisher’s sharp null hypothesis H_F of no treatment effect for each participant. It may in fact be the case that there are treatment effects for many participants and yet we do not reject the sharp null hypothesis because of our worst-case approach.

6. CONCLUSION

In this paper, we develop and apply a design-based finite-sample inferential method for analyzing social experiments with compromised randomization. Compromises come in many forms. They include incompletely documented re-randomization procedures used to improve baseline covariate balance between treatment and control groups. They also include reassignment of treatment status due to administrative constraints.

We build a behavioral model of satisficing experimenters who seek balance in baseline covariates across treatments and controls and who provide readers of their reports qualitative, and sometimes conflicting, summaries of the actual experimental protocols used. We model the randomization protocol as only partially known to the user of experimental data. The empirical researcher recognizes and tries to account for the guiding principles experimenters used in the reassignment of treatment status for balancing baseline covariates while operating under administrative constraints. We show how to partially identify model parameters and construct worst-case (least favorable) randomization tests over a set of possibilities for the actual treatment assignment mechanism.

Our analysis of the Perry program serves as a proof-of-concept of the usefulness of our worst-case finite-sample testing approaches, which are applicable to other compromised experiments, such as those discussed by Bruhn and McKenzie (2009). Our approach is more portable than that of Heckman et al. (2020), which utilizes very specific features of the Perry randomization protocol. Application of our procedures result in conservative finite-sample inferences. It is remarkable that when we apply our worst-case methods to the latest wave of Perry data on the participants at late midlife and their adult children Heckman (2020), we find many statistically significant policy-relevant beneficial treatment effects that survive application of worst-case inferential procedures.