Bias Mechanisms in Intention-to-Treat Analysis With Data Subject to Treatment Noncompliance and Missing Outcomes

Abstract

An analytical approach was employed to compare sensitivity of causal effect estimates with different assumptions on treatment noncompliance and non-response behaviors. The core of this approach is to fully clarify bias mechanisms of considered models and to connect these models based on common parameters. Focusing on intention-to-treat analysis, systematic model comparisons are performed on the basis of explicit bias mechanisms and connectivity between models. The method is applied to the Johns Hopkins school intervention trial, where assessment of the intention-to-treat effect on school children’s mental health is likely to be affected by assumptions about intervention noncompliance and nonresponse at follow-up assessments. The example calls attention to the importance of focusing on each case in investigating relative sensitivity of causal effect estimates with different identifying assumptions, instead of pursuing a general conclusion that applies to every occasion.

1. Introduction

Intention-to-treat (ITT) analysis has been used as a gold standard in estimating treatment effects in randomized trials. In this method, average outcomes are compared across groups categorized by assigned treatments regardless of actual compliance with the treatment. With the protection of random assignment, this standard ITT analysis provides unbiased estimates of treatment assignment effects despite noncompliance of some individuals. If we are not particularly interested in assessing treatment effects given compliance status, standard ITT analysis seems to be the most suitable method of causal effect estimation. However, the robustness of ITT analysis can be challenged in the presence of missing outcomes. Randomized trials often suffer not only from noncompliance but also from missing outcomes due to dropout or nonresponse at follow-up assessments. In this case, the usual practice with ITT analysis is to do an estimation, ignoring any possible association between noncompliance and nonresponse behaviors. The underlying assumption in this type of analysis is missing completely at random (MCAR; Little & Rubin, 2002), in the sense that we allow for association neither between nonresponse and unobserved compliance status, nor between nonresponse and observed compliance status. Frangakis and Rubin (1999) showed that this kind of analysis may be subject to bias in the estimation of treatment assignment effects if compliance behavior is related to response behavior, which is often likely given the similar nature of the two behaviors.

To take into account the association between compliance and response behaviors in causal effect estimation, previous studies used various extensions of the instrumental variable (IV) approach (e.g., Dunn et al., 2003; Frangakis & Rubin, 1999; Mealli, Imbens, Ferro, & Biggeri, 2004; O’Malley & Normand, 2004; Peng, Little, & Raghunathan, 2004; Yau & Little, 2001). These methods commonly adopt the framework of Angrist, Imbens, and Rubin (1996), in the sense that the outcome exclusion restriction and the monotonicity assumptions play key roles in identifying causal effects. Imposing the outcome exclusion restriction means that the effect of treatment assignment on outcomes is allowed for compliers (individuals who do what they are assigned to do) but disallowed for never-takers (individuals who do not receive the treatment, regardless of treatment assignment) and for always-takers (individuals who would receive the treatment, regardless of treatment assignment). Under monotonicity, there are no defiers (individuals who do the opposite of what they are assigned to do, regardless of treatment assignment). These two assumptions are, however, not sufficient to identify causal effects that take into account the association between compliance and response behaviors. Depending on the additional assumption on outcome missing indicators, causal effects can be differently identified. One option is to do an analysis assuming missing at random (MAR; Little & Rubin, 2002), in the sense that we allow for association between nonresponse and observed compliance status but do not allow for association between nonresponse and unobserved compliance status. Another option is to achieve identifiability by imposing the response exclusion restriction (RER). Under RER, the effect of treatment assignment on response is allowed for compliers but is not allowed for never-takers (Frangakis & Rubin, 1999) or for always-takers (Mealli et al., 2004).

Whereas it is straightforward to estimate the additional bias in the ITT estimate by choosing an MCAR model instead of an MAR model, the comparison between MAR and RER models is not so simple because their bias mechanisms involve unidentifiable parameters. In principle, it is possible to do analyses without identifying assumptions, relying on auxiliary information such as from proper priors and covariates. In this case, bias due to deviations from identifying assumptions can be examined by comparing models with and without imposing these assumptions, and then by comparing models with different identifying assumptions. In fact, if the identifying assumptions can be relaxed, there is less need for comparing models with different assumptions. The effect of violating the exclusion restriction imposed on observed outcomes has been previously studied using this approach (Hirano, Imbens, Rubin, & Zhou, 2000; Imbens & Rubin, 1997; Jo, 2002). A drawback of this method is that the causal effect estimates tend to be quite imprecise, even when the exclusion restriction on outcomes alone is relaxed. Another way to compare sensitivity of causal effect estimates with different identifying assumptions is to conduct Monte Carlo simulation studies, considering different levels of deviation from the identifying assumptions in various randomized trial settings. This method has been used in previous studies (Frangakis & Rubin, 1999; Peng et al., 2004) to explore general patterns of the relative sensitivity of MAR and RER models.

This study focuses on comparison and selection of causal effect estimation models given a specific case (i.e., data at hand). The study employs an analytical approach, where models are compared on the basis of explicit bias mechanisms and connectivity between alternative identifying assumptions. First, degrees of deviation from identifying assumptions (or the level of plausibility of identifying assumptions) are put on the same scale based on connectivity between the assumptions. That is, the level of plausibility of one assumption can be translated to the level of plausibility of another assumption based on common parameters related to both assumptions. In this way, plausibility of different identifying assumptions can be compared based on their systematic relationship, instead of relying on intuitive notions of which assumptions are more or less plausible than the others. Second, degrees of deviation from identifying assumptions are translated into resulting bias quantities on the basis of explicit bias mechanisms. Third, sensitivity of causal effect estimates is compared across different models based on bias quantities and the comparability of plausibility established in the first step. Finally, more practical conclusions can be derived within a scientifically plausible range of deviations from identifying assumptions. The proposed approach is demonstrated through analytical comparisons of three ITT modeling options (MCAR, MAR, and RER), which have not been explicitly examined previously. Throughout the article, identification/estimation of causal effects and derivations of biases is based on the IV approach, which is basically a method of moments estimator (MME).

This article is organized as follows. Section 2 describes the Johns Hopkins school intervention trial, which motivated this study. Section 3 describes the randomized trial setting and model assumptions that will be commonly used in the study. Section 4 describes modeling options for the estimation of ITT effect. In Section 5, plausibility of identifying assumptions and connectivity across assumptions is discussed. Section 6 describes bias mechanisms based on underlying model assumptions. In Section 7, the proposed model comparison approach is applied to the Johns Hopkins trial. Section 8 provides a conclusion.

2. Johns Hopkins University Preventive Intervention Research Center School Intervention Study

A school intervention study was conducted by the Johns Hopkins University Preventive Intervention Research Center (JHU PIRC) in 1993–1994 (Ialongo et al., 1999). The study was designed to improve academic achievement and to reduce early behavioral problems of school children. Teachers and first-grade children were randomly assigned (i.e., classroom-level randomization) to the control condition or to the intervention condition. In the Family-School Partnership (FSP) intervention condition, parents were asked to implement 66 take-home activities related to literacy and mathematics, whereas no special instructions were given to control condition children’s parents. Various outcomes related to academic achievement and behavioral problems were measured at the baseline, and approximately 6 months and 18 months from the baseline. One of the main questions in this trial is whether the FSP intervention had any positive effect overall.

The problem of noncompliance arises in this trial because a large number of parents failed to complete a substantial portion of the assigned activities; that is, the intervention might not have had any desirable effect on a child unless the parent had completed a sufficient number of activities. Overreporting of completion level was also expected because parents self-reported their level of activity completion. In this situation, receipt of the intervention may have little meaning unless parents report a quite high level of completion. When the receipt of intervention is defined as completing at least 45 (about two thirds) of the activities, 46% of children in the intervention condition properly received the intervention treatment. The trial also suffered from subsequent missing outcomes. Based on the shy behavior outcome, which will be analyzed in the example, the overall response rate is 0.825 (0.869 in the intervention, 0.781 in the control) at the 6-month follow-up and 0.745 (0.747 in the intervention, 0.744 in the control) at the 18-month follow-up assessment.

A possible association between compliance and response behaviors is also observed in this trial. In the intervention condition, the average response rate was 0.911 for those who completed 45 or more activities and 0.833 for those who completed fewer than 45 activities at the 6-month follow-up, and 0.792 for those who completed 45 or more activities and 0.708 for those who completed fewer than 45 activities at the 18-month follow-up. However, the relationship between compliance and response behaviors cannot be completely observed from the data because intervention receipt status of individuals is unknown in the control condition. Depending on the choice of assumption on this relationship, causal effects of the intervention assignment will be differently identified. A practical question in this situation is how to select a model assumption that will lead to the least biased causal effect estimate within a reasonable range of deviation from the assumption.

3. Common Settings and Notations

A randomized trial is assumed, where individuals are randomly assigned either to the treatment or to the control condition. This setting excludes the possibility of significant contact or relationship among individuals assigned to different conditions, which makes the stable unit treatment value (SUTVA; Rubin, 1978, 1980, 1990) very plausible. It is assumed that treatment receipt status is binary (i.e., received or not) and that treatment receipt status can be observed only among individuals assigned to the treatment condition. The treatment assignment status Z_i = 1 (i = 1, …, n) if person i is assigned to the treatment, and Z_i = 0 if person i is assigned to the control condition, and Z_i is always observed. The observed treatment receipt status D_i = 1 if person i actually received the treatment, and D_i = 0 if person i did not receive the treatment. D_i is always observed.

• Random assignment to two conditions: treatment (Z_i = 1) or control (Z_i = 0).

• Two treatment receipt conditions: receives (D_i = 1) or does not receive (D_i = 0).

• Stable unit treatment value (SUTVA): Potential outcomes for each person are unrelated to the treatment status of other individuals.

Let D_i(1) denote the potential treatment receipt status for individual i when assigned to the treatment, and D_i(0) when assigned to the control condition. The latent compliance status C_i = 1 (complier) if person i would receive the treatment when offered (D_i(1) = 1 and D_i(0) = 0), and C_i = 0 (never-taker) if person i would not receive the treatment regardless of treatment assignment (D_i(1) = 0 and D_i(0) = 0). In this setting, C_i is observed when Z_i = 1. Based on random assignment, it is assumed that E(C_i|Z_i = 1) = E(C_i|Z_i = 0) = E(C_i). Let π_c : = E(C_i). From the observed data, π_c is directly estimable. As in the JHU trial, it is assumed that individuals assigned to the control condition do not have access to the treatment. Therefore, the two possible compliance types are complier and never-taker.

• Two compliance types (C_i):

  1. Complier (C_i = 1)—receives the treatment only if assigned to the treatment condition. π_c = proportion of compliers in the population.

  2. Never-taker (C_i = 0)—does not receive the treatment regardless of the treatment assignment. 1 − π_c = proportion of never-takers in the population.

It is assumed that outcome response status is binary (i.e., responds or does not respond). The response indicator R_i = 1 if outcome Y_i is observed, and R_i = 0 if outcome Y_i is missing, and R_i is always observed. Let ${\pi }_z^R:=E(R_i\mid Z_i=z)$. Based on observed data, ${\pi }_z^R$ is directly estimable. Let ${\pi }_{c,z}^R:=E(R_i\mid C_i=c,Z_i=z)$, where c ∈ {0, 1} and z ∈ {0, 1}. Because C_i is observed when Z_i = 1, ${\pi }_{c,z}^R$ is directly estimable only among individuals with Z_i = 1.

• Two outcome response conditions: responds (outcome is observed, R_i = 1) or does not respond (outcome is unobserved, R_i = 0).

• Three observable average responses when $Z=1:{\pi }_{1,1}^R,{\pi }_{0,1}^R$, and ${\pi }_1^R$.

• Two unobservable average responses when $Z=0:{\pi }_{1,0}^R$ and ${\pi }_{0,0}^R$.

• One observable average response when $Z=0:{\pi }_0^R$.

• Large-sample based approximately unbiased estimates of ${\pi }_{1,1}^R,{\pi }_{0,1}^R,{\pi }_1^R$, and ${\pi }_0^R$ are ${\widehat{\pi }}_{1,1}^R,{\widehat{\pi }}_{0,1}^R,{\widehat{\pi }}_1^R$, and ${\widehat{\pi }}_0^R$.

Under random assignment and SUTVA, the average causal effect of treatment assignment on the outcome Y is defined as

$$\text{ITT}={\mu }_1-{\mu }_0,$$ (1)

where μ₁ : = E(Y_i|Z_i = 1) and μ₀ : = E(Y_i|Z_i = 0).

The outcome Y_i can be observed when R_i = 1. Let ${\mu }_z^{\mathit{obs}}:=E(Y_i\mid R_i=1,Z_i=z)$. In the standard respondent-based ITT analysis, an estimator of Equation 1 is constructed as

$${\text{ITT}}^{\mathit{obs}}={\mu }_1^{\mathit{obs}}-{\mu }_0^{\mathit{obs}}.$$ (2)

As shown in Frangakis and Rubin (1999), ITT^obs is not a consistent estimator of Equation 1 unless R_i is independent of Y_i given Z_i. Based on the following definition, the association between compliance and response behaviors can be taken into account in constructing an estimator of ITT.

Considering compliance, Equation 1 can be rewritten as

$$\begin{array}{l}\text{ITT}={\mu }_1-{\mu }_0\\ =[{\pi }_c{\mu }_{1,1}+(1-{\pi }_c){\mu }_{0,1}]-[{\pi }_c{\mu }_{1,0}+(1-{\pi }_c){\mu }_{0,0}]\\ ={\pi }_c({\mu }_{1,1}-{\mu }_{1,0})+(1-{\pi }_c)({\mu }_{0,1}-{\mu }_{0,0}),\end{array}$$ (3)

where μ_c,z : = E(Y_i|C_i = c, Z_i = z).

Along with SUTVA and random assignment, the assumption of latent ignorability (LI; Frangakis & Rubin, 1999) provides the basis for identification of the ITT effect. Under LI, the probability of outcome being recorded is not associated with the outcome conditional on treatment assignment and latent compliance status. In other words, Y_i ⊥ R_i|Z_i, C_i. Latent ignorability is a special case of missing data mechanisms that assume missing not at random (MNAR; Little & Rubin, 2002), where associations between unobserved variables and response patterns are allowed. Because LI alone does not build identifiability of causal effects, additional identifying assumptions are necessary. Violation of LI affects all ITT estimators discussed in this article including the standard ITT estimator in Equation 2, where LI is usually not explicitly mentioned.

LI implies that E(Y_i|R_i = r, C_i = c, Z_i = z) = E(Y_i|C_i = c, Z_i = z) = : μ_c,z. Because C_i is observed when Z_i = 1 and Y_i is observed when R_i = 1, μ_c,z is directly estimable among individuals with Z_i = 1 and R_i = 1. Among individuals with Z_i = 0 and R_i = 1, additional identifying assumptions are necessary to estimate μ_1,0 and μ_0,0. In both situations, μ_c,z is estimated assuming LI.

• Latent ignorability (LI): The probability of outcome being recorded is not associated with the outcome, conditional on treatment assignment and compliance status.

• Three observable average outcomes when Z = 1 : μ_1,1, μ_0,1, and ${\mu }_1^{\mathit{obs}}$.

• Two unobservable average outcomes when Z = 0 : μ_1,0 and μ_0,0.

• One observable average outcome when $Z=0:{\mu }_0^{\mathit{obs}}$.

• Large-sample based approximately unbiased estimates of μ_1,1, μ_0,1, ${\mu }_1^{\mathit{obs}}$, and ${\mu }_0^{\mathit{obs}}$ are μ̂_1,1, μ̂_0,1, ${\widehat{\mu }}_1^{\mathit{obs}}$, and ${\widehat{\mu }}_0^{\mathit{obs}}$.

In identifying μ_0,0, which is not directly estimable from the data, the outcome exclusion restriction (OER) is commonly assumed. Under OER, the distributions of the potential outcomes are independent of the treatment assignment for never-takers and always-takers (Angrist et al., 1996). In this setting, OER applies to never-takers, and therefore μ_0,0 is simply identified as μ̂_0,1.

• Outcome exclusion restriction (OER): The distributions of the potential outcomes are independent of the treatment assignment for never-takers and always-takers.

• Under OER, μ_0,1 = μ_0,0, and therefore μ_0,0 is estimable from the data.

Simultaneous considerations of R_i, C_i, and Z_i are necessary to understand identification of μ_1,0, which is the last unknown parameter and the only parameter in Equation 3 that is differently identified in the three ITT models considered.

The average response ${\pi }_z^R$ can be written given Z_i and C_i as

$${\pi }_z^R={\pi }_c{\pi }_{1,z}^R+(1-{\pi }_c){\pi }_{0,z}^R,$$ (4)

where ${\pi }_{c,z}^R:=E(R_i\mid C_i=c,Z_i=z)$.

The observed average outcome ${\mu }_z^{\mathit{obs}}$ can be written given R_i, C_i, and Z_i as

$$\begin{array}{l}{\mu }_z^{\mathit{obs}}=E\{E(Y_i\mid R_i=1,Z_i=z,C_i)\mid R_i=1,Z_i=z\}\\ =pr(C_i=1\mid R_i=1,Z_i=z){\mu }_{1,z}+pr(C_i=0\mid R_i=1,Z_i=z){\mu }_{0,z}\\ =\frac{{\pi }_{1,z}^R}{{\pi }_z^R}{\pi }_c{\mu }_{1,z}+\frac{{\pi }_{0,z}^R}{{\pi }_z^R}(1-{\pi }_c){\mu }_{0,z}.\end{array}$$ (5)

The observed average outcome of the control condition is

$${\mu }_0^{\mathit{obs}}=\frac{{\pi }_{1,0}^R}{{\pi }_0^R}{\pi }_c{\mu }_{1,0}+\frac{{\pi }_{0,0}^R}{{\pi }_0^R}(1-{\pi }_c){\mu }_{0,0}.$$ (6)

From Equations 4 and 6, μ_1,0 can be written as

$${\mu }_{1,0}=\frac{{\mu }_0^{\mathit{obs}}{\pi }_0^R-{\mu }_{0,0}{\pi }_{0,0}^R(1-{\pi }_c)}{{\pi }_0^R-{\pi }_{0,0}^R(1-{\pi }_c)},$$ (7)

where ${\pi }_0^R$ and π_c are directly estimable, and μ_0,0 is identified as μ̂_0,1 under OER. However, further restriction is necessary to identify ${\pi }_{0,0}^R$. This is where the three ITT models discussed in the following sections differ.

4. Three Estimators of ITT Effect

4.1. MAR Estimator

In addition to LI and OER, this model assumes MAR (Little & Rubin, 2002) for its identification. Under MAR, the probability of outcome being recorded is not associated with the outcome conditional on treatment assignment and observed treatment receipt status (Y_i ⊥ R_i|Z_i, D_i). It is implied under MAR that pr(R_i|Y_i, Z_i, D_i) = pr(R_i|Z_i, D_i). The assumption of MAR provides a key to the identification of ${\pi }_{0,0}^R$ in Equation 7. In this setting, a sufficient restriction to impose MAR is that ${\pi }_{1,0}^R={\pi }_{0,0}^R={\pi }_0^R$.

• Missing at random (MAR): The probability of outcome being recorded is not associated with the outcome conditional on treatment assignment and observed treatment receipt status.

• A sufficient restriction to impose MAR is that ${\pi }_{0,0}^R={\pi }_{1,0}^R={\pi }_0^R$. Under this restriction, ${\pi }_{0,0}^R$ is directly estimable.

Under LI, E(Y_i|R_i = r, C_i = c, Z_i = z) = E(Y_i|C_i = c, Z_i = z) = : μ_c,z. Under OER, μ_0,0 = μ_0,1. Under LI, OER, and MAR, μ_1,0 can be rewritten from Equation 7 as

$${\mu }_{1,0}=\frac{{\mu }_0^{\mathit{obs}}-{\mu }_{0,1}(1-{\pi }_c)}{{\pi }_c}.$$ (8)

Based on Equations 3 and 8, the ITT^MAR estimator is defined as

$${\text{ITT}}^{\text{MAR}}={\pi }_c{\mu }_{1,1}-{\mu }_0^{\mathit{obs}}+{\mu }_{0,1}(1-{\pi }_c),$$ (9)

where all the involved parameters are directly estimable.

4.2. Respondent-Based MCAR Estimator

This estimator refers to the standard respondent-based ITT estimator (i.e., completer-only analysis). If we focus only on the relationship between compliance and response behaviors, the missing data mechanism assumed in this estimator is MCAR, because association between compliance and response behaviors is disallowed regardless of whether compliance is observed or not. In principle, we can disallow this association but still keep the cases with missing outcomes. However, the MCAR estimator is used in this study to refer to the estimator that uses data from respondents only and disallows association between compliance and response.

In addition to LI and OER, this model assumes MCAR. That is, Y_i ⊥ R_i. Because ITT analysis does not involve parameters that are not conditional on Z, the assumption can be replaced by a weaker version. That is, Y_i ⊥ R_i|Z_i. Given that, MCAR implies that pr(R_i|Y_i, Z_i, D_i) = pr(R_i|Z_i). Under MCAR, response behavior is not associated either with observed treatment receipt status D_i or with latent compliance status C_i. To impose MCAR, it is assumed not only that ${\pi }_{1,0}^R={\pi }_{0,0}^R$ but also that ${\pi }_{1,1}^R={\pi }_{0,1}^R$. As shown above, MAR is a sufficient assumption in identifying ITT. The additional assumption that ${\pi }_{1,1}^R={\pi }_{0,1}^R$ does not contribute to the identification of the MCAR model. Although it operates under a more restricted missing data assumption than necessary, the model is commonly used in practice.

• Missing completely at random (MCAR): The probability of outcome being recorded is not associated with the outcome conditional on treatment assignment.

• To impose MCAR, two restrictions are applied. That is, ${\pi }_{1,0}^R={\pi }_{0,0}^R={\pi }_0^R$ (i.e., MAR) and ${\pi }_{1,1}^R={\pi }_{0,1}^R={\pi }_1^R$.

Under MCAR, ${\pi }_{1,1}^R={\pi }_{0,1}^R={\pi }_1^R$, and therefore, the average compliance after deleting cases with missing outcomes is the same as the average compliance without deleting those cases. That is, it is assumed that ${\pi }_c{\pi }_{1,1}^R/{\pi }_1^R={\pi }_c$. Let a new notation ${\pi }_c^{\mathit{del}}(={\pi }_c{\pi }_{1,1}^R/{\pi }_1^R)$ denote the average compliance after deleting cases with missing outcomes. Under LI, OER, and MCAR, μ_1,0 can be rewritten from Equation 7 as

$${\mu }_{1,0}=\frac{{\mu }_0^{\mathit{obs}}-{\mu }_{0,1}(1-{\pi }_c^{\mathit{del}})}{{\pi }_c^{\mathit{del}}}.$$ (10)

Based on Equations 3 and 10, the ITT estimator is defined as

$${\text{ITT}}^{\text{MCAR}}={\pi }_c^{\mathit{del}}\left\{{\mu }_{1,1}-\left[\frac{{\mu }_0^{\mathit{obs}}-{\mu }_{0,1}(1-{\pi }_c^{\mathit{del}})}{{\pi }_c^{\mathit{del}}}\right]\right\},$$ (11)

where all the involved parameters are directly estimable.

Note that ITT^MCAR ≡ ITT^obs, which is the common definition in the standard respondent-based analysis as shown in Equation 2. Therefore, the simple definition of ITT^obs is sufficient for the estimation of the ITT effect. However, the definition in Equation 11 is useful in defining the explicit bias mechanism.

4.3. RER Estimator

In addition to LI and OER, this model assumes the exclusion restriction on outcome missing indicators (RER) for its identification. Because the model assumes both OER and RER, the combined assumption is called the compound exclusion restriction (CER; Frangakis & Rubin, 1999). Under RER, for never-takers or always-takers, response behavior is not affected by treatment assignment status. In this setting, for never-takers, R_i ⊥ Z_i|C_i = 0. This implies that ${\pi }_{0,0}^R={\pi }_{0,1}^R$.

• Response exclusion restriction (RER): For never-takers or always-takers, the probability of outcome being recorded is not associated with treatment assignment.

• This implies that ${\pi }_{0,0}^R={\pi }_{0,1}^R$, and therefore, ${\pi }_{0,0}^R$ becomes estimable.

Under LI, OER, and RER, μ_1,0 can be rewritten from Equation 7 as

$${\mu }_{1,0}=\frac{{\mu }_0^{\mathit{obs}}{\pi }_0^R-{\mu }_{0,1}{\pi }_{0,1}^R(1-{\pi }_c)}{{\pi }_0^R-{\pi }_{0,1}^R(1-{\pi }_c)}.$$ (12)

Based on Equations 3 and 12, the ITT estimator is defined as

$${\text{ITT}}^{\text{RER}}={\pi }_c\left\{{\mu }_{1,1}-\left[\frac{{\mu }_0^{\mathit{obs}}{\pi }_0^R-{\mu }_{0,1}{\pi }_{0,1}^R(1-{\pi }_c)}{{\pi }_0^R-{\pi }_{0,1}^R(1-{\pi }_c)}\right]\right\},$$ (13)

where all the involved parameters are directly estimable.

5. Plausibility of Response Assumptions

5.1. Deviation From MAR

Let us define the deviation from the MAR assumption as $\delta (={\pi }_{1,0}^R-{\pi }_{0,0}^R)$. Nonzero δ values indicate that response probabilities of compliers and never-takers differ when assigned to the control condition. Positive δ values indicate a higher response probability among compliers, and negative values indicate a higher response probability among never-takers. Because ${\pi }_{1,0}^R$ and ${\pi }_{0,0}^R$ are not directly estimable from the data, δ cannot be estimated. The level of plausibility of MAR is the key to a good estimation of causal effects assuming MAR.

• The level of plausibility of MAR can be expressed by the level of deviation from MAR (i.e., δ).

• $\delta ={\pi }_{1,0}^R-{\pi }_{0,0}^R$, and is not estimable from the data.

In the JHU PIRC trial, some deviation from MAR is expected. Poor compliance is a good indicator of family instability, meaning that these families are more likely to move from place to place (or children are more likely to be sent to live with a relative or placed in foster care) due to financial stress or other reasons related to drug or alcohol problems, and therefore, it is harder to locate these parents and their children at follow-up assessments. In other words, response probability is likely to be higher among potentially well-complying families (i.e., δ > 0).

5.2. Deviation From MCAR

One part of the MCAR assumption (i.e., ${\pi }_{1,1}^R={\pi }_{0,1}^R$) involves parameters that are directly estimable from the data. Let us define the deviation from this assumption as ${\pi }_{1,1}^R-{\pi }_{0,1}^R=\alpha$. Nonzero α values indicate that response probabilities of compliers and never-takers differ when assigned to the treatment condition. Positive α values indicate a higher response probability among compliers, and negative values indicate a higher response probability among never-takers. By comparing sample response rates of compliers and never-takers in the treatment condition, the level of deviation from MCAR (i.e., α) can be estimated. The other part of the MCAR assumption (i.e., ${\pi }_{1,0}^R={\pi }_{0,0}^R$) involves parameters that are not directly estimable from the data and is the same as the MAR assumption. In other words, MCAR is a stronger assumption than MAR.

• The level of plausibility of MCAR can be expressed by the level of deviation from MCAR (i.e., α and δ).

• $\alpha ={\pi }_{1,1}^R-{\pi }_{0,1}^R$, and is estimable from the data.

5.3. Deviation From RER

Let us define the deviation from the RER assumption as $\beta (={\pi }_{0,1}^R-{\pi }_{0,0}^R)$. Nonzero β values indicate that treatment assignment status does not affect response probability of never-takers. Positive β values indicate that never-takers are more likely to respond when assigned to the treatment condition than when assigned to the control condition. Although ${\pi }_{0,1}^R$ is directly estimable from the data, ${\pi }_{0,0}^R$ is not. Therefore, β cannot be estimated. The level of plausibility of RER is the key to a good estimation of causal effects assuming RER.

• The level of plausibility of RER can be expressed by the level of deviation from RER (i.e., β).

• $\beta ={\pi }_{0,1}^R-{\pi }_{0,0}^R$, and is not estimable from the data.

Some deviation from RER is expected in the JHU PIRC trial. Poorly complying families might have felt some benefit from the intervention and might have felt more obliged to respond than families in the control condition, who would have complied poorly if the intervention had been offered, resulting in a higher response probability when assigned to the treatment condition (i.e., β >0). Another possibility is that poorly complying families might have been demoralized by failing to comply with the intervention and might have responded less than families in the control condition, who would have complied poorly if the intervention had been offered, resulting in a lower response probability for those families assigned to the treatment condition (i.e., β < 0).

5.4. Connectivity between MAR and RER

Although one assumption may intuitively seem more plausible than the other, degrees of deviation from the MAR and RER assumptions cannot be compared unless they can be viewed from the same assumption. For example, a small deviation from one assumption might be equivalent to a much larger deviation from the other assumption. Translation between different assumptions can be done by simple calculations, which may reveal a quite surprising relationship between the two assumptions.

The level of plausibility of one assumption can be translated into the level of plausibility of the other assumption based on common parameters related to both assumptions. The two assumptions MAR and RER are connected through the same parameter ${\pi }_{0,0}^R$. Therefore, imposing any restrictions on plausibility of one assumption immediately affects the other assumption. Note that β denotes a certain degree of deviation from RER (i.e., ${\pi }_{0,1}^R-{\pi }_{0,0}^R=\beta$), and δ denotes a certain degree of deviation from MAR (i.e., ${\pi }_{1,0}^R-{\pi }_{0,0}^R=\delta$).

If β is fixed at a certain value, ${\pi }_{0,0}^R$ can be solved for (i.e., ${\widehat{\pi }}_{0,0}^R={\widehat{\pi }}_{0,1}^R-\beta$). Then, ${\pi }_{1,0}^R$ can be identified from the mixture ${\pi }_0^R={\pi }_c{\pi }_{1,0}^R+(1-{\pi }_c){\pi }_{0,0}^R$ as

$${\widehat{\pi }}_{1,0}^R=\frac{{\widehat{\pi }}_0^R-({\widehat{\pi }}_{0,1}^R-\beta )(1-{\widehat{\pi }}_c)}{{\widehat{\pi }}_c},$$ (14)

where ${\pi }_{0,1}^R$ is directly estimable from the observed data. Given that ${\pi }_{1,0}^R$ and ${\pi }_{0,0}^R$ are identified, deviation from MAR also can be identified as

$$\widehat{\delta }={\widehat{\pi }}_{1,0}^R-{\widehat{\pi }}_{0,0}^R=\frac{{\widehat{\pi }}_0^R-{\widehat{\pi }}_{0,1}^R+\beta }{{\widehat{\pi }}_c},$$ (15)

which is the degree of deviation from MAR that can be compared with the degree of deviation from RER (i.e., β).

Similarly, if δ is fixed at a certain value, ${\widehat{\pi }}_{0,0}^R={\widehat{\pi }}_{1,0}^R-\delta$, where ${\pi }_{0,0}^R$ and ${\pi }_{1,0}^R$ are still not identified. If we replace ${\pi }_{0,0}^R$ with ${\pi }_{1,0}^R-\delta$, however, ${\pi }_{1,0}^R$ can be identified from the mixture ${\pi }_0^R={\pi }_c{\pi }_{1,0}^R+(1-{\pi }_c){\pi }_{0,0}^R$ as

$${\widehat{\pi }}_{1,0}^R={\widehat{\pi }}_0^R+\delta (1-{\widehat{\pi }}_c).$$ (16)

Then, ${\pi }_{0,0}^R$ is identified as

$${\widehat{\pi }}_{0,0}^R={\widehat{\pi }}_0^R+\delta (1-{\widehat{\pi }}_c)-\delta ={\widehat{\pi }}_0^R-\delta {\widehat{\pi }}_c,$$ (17)

and deviation from RER can be identified as

$$\widehat{\beta }={\widehat{\pi }}_{0,1}^R-{\widehat{\pi }}_0^R+\delta {\widehat{\pi }}_c,$$ (18)

which is the degree of deviation from RER that can be compared with the degree of deviation from MAR (i.e., δ).

Once the degrees of deviation from the two assumptions are put on the same scale, plausibility of the assumptions can be easily compared. Connectivity between the assumptions also allows us to examine plausibility from two different angles. That is, we can evaluate plausibility of MAR in terms of RER, and plausibility of RER in terms of MAR. For example, in a double-blind trial, the range of deviation from RER is likely to be quite narrow, if there is any deviation. In this case, if the restriction that δ = 0 results in a large β estimate in Equation 18, we may conclude that MAR is unlikely to hold or less plausible than RER. When we are highly confident with plausibility of at least one assumption, as in this example, relative plausibility may well be translated into relative sensitivity. However, in more common situations, where we only have sketchy information on plausibility, evaluation of relative sensitivity should wait until deviation from the assumptions is translated into bias.

Given alternative model assumptions, the ultimate interest is usually in comparing sensitivity of causal effect estimates rather than in comparing plausibility of assumptions. The reason that comparison of plausibility cannot serve as a comparison of sensitivity is that each assumption follows its own bias mechanism. In other words, comparable δ and β values do not necessarily result in the same bias, and the assumption with higher plausibility (less deviation) may not lead to smaller bias because the two assumptions follow different bias mechanisms. Therefore, relative sensitivity of causal effect estimates to different assumptions cannot be evaluated unless possible ranges of deviation from the assumptions are put on the same scale, and bias is quantified based on each assumption’s bias mechanism.

6. Bias Mechanisms

6.1. Deviation From MAR

In extreme cases where we have definite confidence in either MAR or RER, bias in the ITT estimate can be easily identified by subtracting one estimate from the other. That is, given the common assumptions (LI and OER), the differences between the two ITT estimators can be written as

$${\text{ITT}}^{\text{MAR}}-{\text{ITT}}^{\text{RER}}={\text{MAR}}_{\mathit{bias}}-{\text{RER}}_{\mathit{bias}},$$ (19)

which shows that bias due to deviation from one assumption can be identified if the other assumption holds. For example, if RER holds, RER_bias = 0. Therefore, ${\widehat{\text{MAR}}}_{\mathit{bias}}={\widehat{\text{ITT}}}^{\text{MAR}}-{\widehat{\text{ITT}}}^{\text{RER}}$. However, in most cases, we do not have definite confidence in plausibility of the two assumptions, and therefore, bias needs to be quantified based on bias mechanisms. Each assumption follows its own bias mechanism that translates the degree of deviation from the assumption into bias.

Note that δ is the deviation from MAR. That is, $\delta ={\pi }_{1,0}^R-{\pi }_{0,0}^R$. If MAR holds (δ = 0), ${\pi }_{1,0}^R={\pi }_{0,0}^R={\pi }_0^R$. Under LI and OER, the difference between the specification of μ_1,0 in Equation 8 assuming MAR and the specification in Equation 7 without assuming MAR is

$$\frac{\delta (1-{\pi }_c)({\mu }_0^{\mathit{obs}}-{\mu }_{0,1})}{{\pi }_0^R+\delta (1-{\pi }_c)}.$$ (20)

Therefore, under LI and OER, the bias in the estimation of ITT due to deviation from MAR can be written as

$${\text{MAR}}_{\mathit{bias}}=\frac{-\delta (1-{\pi }_c)({\mu }_0^{\mathit{obs}}-{\mu }_{0,1})}{{\pi }_0^R+\delta (1-{\pi }_c)},$$ (21)

where all the parameters involved in the bias mechanism are estimable except δ.

6.2. Deviation From RER

Note that β is the deviation from RER. That is, $\beta ={\pi }_{0,1}^R-{\pi }_{0,0}^R$. Under LI and OER, the difference between the specification of μ_1,0 in Equation 12 assuming RER and the specification in Equation 7 without assuming RER is

$$\frac{\beta {\pi }_0^R(1-{\pi }_c)({\mu }_0^{\mathit{obs}}-{\mu }_{0,1})}{[{\pi }_0^R-({\pi }_{0,1}^R-\beta )(1-{\pi }_c)][{\pi }_0^R-{\pi }_{0,1}^R(1-{\pi }_c)]}.$$ (22)

Therefore, under LI and OER, the bias in the estimation of ITT due to deviation from RER can be written as

$${\text{RER}}_{\mathit{bias}}=\frac{-\beta {\pi }_c{\pi }_0^R(1-{\pi }_c)({\mu }_0^{\mathit{obs}}-{\mu }_{0,1})}{[{\pi }_0^R-({\pi }_{0,1}^R-\beta )(1-{\pi }_c)][{\pi }_0^R-{\pi }_{0,1}^R(1-{\pi }_c)]},$$ (23)

where all the parameters involved in the bias mechanism are estimable except β.

6.3. Deviation From MCAR

Under MCAR, ${\pi }_{1,0}^R={\pi }_{0,0}^R$ and ${\pi }_{1,1}^R={\pi }_{0,1}^R$. The first part of the assumption is the same as MAR. The second part of the assumption is unique to the MCAR assumption and is actually testable based on sample statistics. The deviation from the second part of the assumption has been defined as ${\pi }_{1,1}^R-{\pi }_{0,1}^R=\alpha$.

Given that LI, MAR, and OER are commonly assumed in ITT^MCAR and ITT^MAR, the additional bias by assuming MCAR instead of MAR can be written as

$$\begin{array}{l}{\text{MCAR}}_{\mathit{bias}}={\text{ITI}}^{\text{MCAR}}-{\text{ITT}}^{\text{MAR}}\\ =\frac{\alpha (1-{\pi }_c){\pi }_c({\mu }_{1,1}-{\mu }_{0,1})}{{\pi }_1^R},\end{array}$$ (24)

where all the parameters involved in the bias mechanism are estimable including α. Although bias due to deviation from MCAR can be simply estimated by subtracting ${\widehat{\text{ITT}}}^{\text{MAR}}$ from ${\widehat{\text{ITT}}}^{\text{MCAR}}$, the explicit definition in Equation 24 is useful when one wants to learn whether the resulting bias is substantial or trivial before involving more complex estimators such as ITT^MAR and ITT^RER.

7. Application to the JHU PIRC Study

The FSP intervention condition and the control condition are compared in this example (221 students in the intervention, and 219 in the control condition). Parents who would complete at least 45 activities only when assigned to the intervention condition were categorized as high compliers (D_i(1) = 1 and D_i(0) = 0). Parents who would complete less than 45 activities regardless of the intervention assignment were categorized as low compliers (D_i(1) = 0 and D_i(0) = 0). Because study participants were not allowed to receive a different intervention treatment from the one that they were assigned to, these two are the only possible compliance types based on binary treatment receipt and binary treatment assignment status. To be consistent with the compliance categories used in previous sections, the same notation is used to indicate compliance status in the JHU PIRC data (i.e., C_i = 1 for a high complier, and C_i = 0 for a low complier).

Among various measures of behavioral problems, shy behavior rated by the teacher is the outcome focused on. Shy behavior is a composite variable that includes items such as the following: is friendly to classmates, interacts with classmates, plays with classmates, and initiates interactions with classmates. Change scores for shy behavior are calculated by subtracting the shy behavior score assessed at 6 months after the intervention and at 18 months after the intervention from the baseline score. To illustrate different patterns of missing data and resulting biases, ITT analysis was separately conducted with each change score as a univariate outcome.

Table 1 shows the key sample statistics necessary to estimate causal effects of the intervention considering noncompliance and nonresponse. ${\widehat{\mu }}_0^{\mathit{obs}}$ is the sample mean of the control-condition individuals who responded at the follow-up assessment. μ_1,1 is the sample mean of high compliers, and μ̂_0,1 is the sample mean of low compliers assigned to the FSP intervention condition. ${\widehat{\pi }}_0^R$ is the sample mean response rate of the control-condition individuals. ${\widehat{\pi }}_{1,1}^R$ is the sample mean response rate of high compliers, and ${\widehat{\pi }}_{0,1}^R$ is the sample mean response rate of low compliers assigned to the intervention condition. π̂_c is the sample mean compliance rate among individuals assigned to the intervention condition.

TABLE 1.

Necessary Sample Statistics for Causal Effect Estimation

Follow-Up	${\widehat{\mu }}_0^{\mathit{obs}}$	μ̂1,1	μ̂1,0	${\widehat{\pi }}_0^R$	${\widehat{\pi }}_{1,1}^R$	${\widehat{\pi }}_{0,1}^R$	π̂c
6 months	−.319	−.177	.248	.781	.911	.833	.457
18 months	−.066	−.047	.197	.744	.792	.708	.457

Table 2 shows the ITT analysis results based on different causal effect estimation models. Standard errors were estimated using the delta method. Positive values of ITT estimates can be interpreted as desirable effects of the intervention, meaning that shy behavior increased less among individuals in the intervention condition. At both 6- and 18-month follow-up assessments, different ITT effect estimation models yielded very similar results, implying that the choice of missing data models was not that critical in assessing the ITT effect of the intervention (standard deviation pooled across the intervention and the control condition ignoring compliance status is 1.319 at the 6-month follow-up and 1.370 at the 18-month follow-up). However, for the purpose of method illustration, let us treat these small differences as substantial. At the 6-month follow-up, ITT^MCAR presents the smallest and ITT^RER presents the largest effect of the intervention. At the 18-month follow-up, ITT^RER presents the smallest effect and ITT^MAR presents the largest effect.

TABLE 2.

Intention-to-Treat (ITT) Effects of Family-School Partnership (FSP) Intervention on Shy Behavior

Follow-Up	${\widehat{\text{ITT}}}^{\text{MCAR}}$	${\widehat{\text{ITT}}}^{\text{MAR}}$	${\widehat{\text{ITT}}}^{\text{RER}}$
6 months	.363 (.140)	.373 (.140)	.422 (.160)
18 months	.145 (.152)	.152 (.152)	.137 (.145)

Note: Standard errors are in parentheses.

The unique part of the MCAR assumption implies that response behavior, at least on the average, does not vary across observed compliance types ( $\alpha ={\pi }_{1,1}^R-{\pi }_{0,1}^R=0$), which is actually testable based on sample statistics. That is, $\widehat{\alpha }={\widehat{\pi }}_{1,1}^R-{\widehat{\pi }}_{0,1}^R$. From Table 1, α̂ = 0.911 −0.833 = 0.078 at the 6-month follow-up and α̂ = 0.792 − 0.708 = 0.084 at the 18-month follow-up, indicating that the compliance rate of high compliers was slightly higher than that of low compliers assigned to the intervention condition. The estimate of bias due to deviation from MCAR is 0.010 at the 6-month follow-up and 0.007 at the 18-month follow-up, which can be obtained from Equation 24 or simply by subtracting ${\widehat{\text{ITT}}}^{\text{MAR}}$ from ${\widehat{\text{ITT}}}^{\text{MCAR}}$. These bias quantities show that intervention effects tend to be slightly underestimated by assuming MCAR instead of MAR.

There are a few possible scenarios for the relation between ITT^MAR and ITT^RER estimates: (a) both estimators underestimate, (b) both estimators overestimate, and (c) one underestimates and the other overestimates the ITT effect. In the case of (a), it would be reasonable to choose the largest ITT effect estimate. If (b) or (c) is the case, a conservative choice would be the smallest estimate. Furthermore, based on the choice of the ITT estimate, one may want to know the possible range of bias. To facilitate this model selection/evaluation process, the proposed method simultaneously considers plausibility of model assumptions, bias mechanisms, and interrelationship between model assumptions.

In the JHU PIRC trial, the MAR assumption implies that response probability does not vary across compliance types in the control condition ( $\delta ={\pi }_{1,0}^R-{\pi }_{0,0}^R=0$), and the RER assumption implies that response probability of low compliers does not vary depending on intervention assignment ( $\beta ={\pi }_{0,1}^R-{\pi }_{0,0}^R=0$). In terms of MAR, some deviation from the assumption is expected because poor compliance is a good indicator of family instability, meaning that these families are more likely to move from place to place due to financial stress or other reasons related to drug or alcohol problems, and therefore it is harder to locate these families at follow-up assessments than potentially well-complying families (i.e., δ > 0). Indirect evidence in the observed data is that the response rate of high compliers was higher than that of low compliers in the intervention condition. Some deviation from RER is also expected, but the direction of deviation is not as predictable as that of MAR. Poorly complied families in the intervention condition might have felt somewhat benefited from the intervention and might have felt more obliged to respond than families in the control condition, who would have complied poorly if the intervention had been offered (i.e., β > 0). However, it is also possible that poorly complying families might have been demoralized by failing to comply with the intervention activities and might have responded less at follow-up than their counterparts in the control condition (i.e., β < 0).

Although deviations from MAR and RER are not estimable, the rest of the parameters related to the bias mechanisms are. Therefore, if we know δ and β, we can estimate resulting bias quantities as shown in Equations 21 and 23. Also, as shown in Equations 15 and 18, if either β or δ is fixed at a certain value, the other can be easily estimated.

Table 3 shows some possible combinations of deviations from missing data assumptions and related parameters estimated based on the large sample theory and observed sample statistics at the 6-month follow-up. As discussed earlier, if the value of one of the four parameters ( ${\pi }_{0,0}^R,{\pi }_{1,0}^R$, δ, β) is known, the rest can be estimated, and which variable is fixed affects neither the values reported in Table 3 nor bias estimates reported in Figure 1. The minimum and the maximum deviations from MAR and RER were determined by the natural range of ${\pi }_{1,0}^R$ and ${\pi }_{0,0}^R$, which cannot exceed 1 or fall below 0. For example, if ${\pi }_{0,0}^R=1.0$, from the mixture ${\pi }_0^R={\pi }_c{\pi }_{1,0}^R+(1-{\pi }_c){\pi }_{00}^R$, we can identify ${\pi }_{1,0}^R$ as $({\pi }_0^R-1+{\pi }_c)/{\pi }_c$. If ${\pi }_{1,0}^R=1.0$, we can identify ${\pi }_{0,0}^R$ as $({\pi }_0^R={\pi }_c)/(1-{\pi }_c)$. When β = 0, the δ estimate is −0.115, meaning that the response rate of low compliers is 11.5% higher than that of high compliers in the control condition, which is very unlikely given the observation in the intervention condition and given the circumstances of the trial. When δ = 0, the β estimate is .053, meaning that the response rate of low compliers is slightly higher when assigned to the intervention condition than when assigned to the control condition. In general, in the JHU PIRC trials, it is very plausible that δ ≥ 0. However, the direction of RER deviation is not as predictable as that of MAR deviation.

TABLE 3.

Some Possible Combinations of Deviations From Missing Data Assumptions at the 6-Month Follow-Up

${\pi }_{0,0}^R$	${\pi }_{1,0}^R$	δ	β
1.000	.520	−.480	−.167
.933	.600	−.334	−.100
.833	.718	−.115	.000
.781	.781	.000	.053
.733	.837	.104	.100
.596	1.000	.404	.237

FIGURE 1. Bias and mean squared error (MSE) in the intention-to-treat analysis at the 6-month follow-up.

Note: The horizontal dashed line is set at bias estimate = 0, which indicates the most desirable situation. The vertical dashed line is set at ${\pi }_{0,0}^R=0.781$, which indicates that MAR holds. MAR = missing at random; RER = response exclusion restriction.

Within the natural range of ${\pi }_{0,0}^R$ and ${\pi }_{1,0}^R$, Figure 1 shows biases from the two ITT estimators at the 6-month follow-up. Bias mechanisms in Equations 21 and 23 are used to estimate bias based on sample statistics. Mean squared error (MSE) was calculated based on the variance of an estimator and its bias. With the general restriction in the possible range of deviation from MAR (i.e., δ ≥ 0, or ${\pi }_{0,0}^R\le 0.781$), we can conclude from Figure 1 that both estimators overestimate ITT effect, and the estimator assuming RER overestimates more. The relative quality of ITT estimates within this range is also supported by MSE estimates. Given these results, it seems reasonable to prefer the ITT^MAR estimator to the ITT^RER estimator in assessing the ITT effect of the intervention at the 6-month follow-up.

Table 4 shows some possible combinations of MAR/RER deviations and resulting biases at the 18-month follow-up. The minimum and the maximum deviations from MAR and RER were again determined by the natural range of ${\pi }_{1,0}^R$ and ${\pi }_{0,0}^R$. The RER and OER assumptions seem more realistic at the 18-month follow-up than at the 6-month follow-up, because it is very unlikely that the effect of poorly completed FSP intervention will last for a long period of time. Also, at the 18-month follow-up, if β = 0, the δ estimate is 0.079, meaning that the response rate of compliers is 7.9% higher than that of compliers in the control condition. This is a very realistic situation, further supporting plausibility of RER.

TABLE 4.

Some Possible Combinations of Deviations From Missing Data Assumptions at the 18-Month Follow-Up

${\pi }_{0,0}^R$	${\pi }_{1,0}^R$	δ	β
1.000	.440	−.560	−.292
.808	.668	−.140	−.100
.744	.744	.000	−.036
.708	.787	.079	.000
.608	.906	.298	.100
.529	1.000	.470	.179

Figure 2 shows biases from the two ITT estimators at the 18-month follow-up. If we maintain only the general restriction that δ ≥ 0 (i.e., ${\pi }_{0,0}^R\le 0.744$), we can conclude that both estimators assuming MAR and RER, in general, overestimate the ITT effect and the estimator assuming MAR overestimates more. Within the small range where β < 0 and δ > 0 (i.e., $0.708<{\pi }_{0,0}^R>0.744$), the ITT^MAR estimator slightly overestimates and the ITT^RER estimator underestimates the ITT effect. Therefore, taking a more conservative side, it seems reasonable to prefer the ITT^RER estimator to the ITT^MAR estimator in assessing the ITT effect of the FSP intervention at the 18-month follow-up. The relative quality of ITT estimators within this range is also supported by MSE estimates.

FIGURE 2. Bias and mean squared error (MSE) in the intention-to-treat analysis at the 18-month follow-up.

Note: The horizontal dashed line is set at bias estimate = 0. The vertical dashed line is set at ${\pi }_{0,0}^R=0.744$, which indicates that MAR holds. MAR = missing at random; RER = response exclusion restriction.

To examine possible variation of ITT effect and bias estimates, this study employs an unrestricted analysis method, where analyses are simply conducted separately for subgroups of the whole sample. Table 5 shows the results of separate ITT analyses at the 6-month follow-up on the basis of parents’ racial background, here categorized as African American or not African American, which is the most significant predictor of compliance in the JHU PIRC trial. The results show that the choice of missing data models was not that critical in assessing the ITT effect of the intervention, in particular for the African American sample. For both racial categories, at the 6-month follow-up, ITT^MCAR presents the smallest effect and ITT^RER presents the largest effect of the intervention, which is consistent with the result of the single group analyses. The effect of intervention assignment was much larger for African American families compared with families of other racial backgrounds.

TABLE 5.

Intention-to-Treat (ITT) Effects of Family-School Partnership (FSP) Intervention on Shy Behavior at the 6-Month Follow-Up: Separate Analyses Based on Parents’ Racial Background

Racial Background	${\widehat{\text{ITT}}}^{\text{MCAR}}$	${\widehat{\text{ITT}}}^{\text{MAR}}$	${\widehat{\text{ITT}}}^{\text{RER}}$
African American (N = 310)	.453 (.166)	.460 (.166)	.507 (.187)
Not African American (N = 130)	.118 (.256)	.138 (.257)	.181 (.297)

Note: Standard errors are in parentheses.

Figures 3 and 4 show bias estimates from the two ITT estimators at the 6-month follow-up for different racial groups. As in the estimation of ITT effect, bias estimation was conducted separately without imposing any restrictions (e.g., equality across groups on some parameters) on the relation between the two groups. In comparing bias from the two samples, it should be noted that response and treatment receipt probabilities differ across groups, and therefore, their admissible ranges of ${\pi }_{0,0}^R$ and ${\pi }_{1,0}^R$ are also different. Figures 3 and 4 show that the quality of ITT effect estimation, in terms of bias and MSE, is quite different for different racial categories (i.e., smaller bias and MSE for the African American sample). However, the general conclusion on relative performance of different ITT estimators is consistent across the two groups. That is, with the general restriction in the possible range of deviation from MAR (i.e., δ ≥ 0), we can conclude that both estimators overestimate ITT effect, and the estimator assuming RER overestimates more. Therefore, for both racial groups, the ITT^MAR estimator is preferred to the ITT^RER estimator in assessing the ITT effect of the intervention at the 6-month follow-up.

FIGURE 3. African Americans: Bias and mean squared error (MSE) in the intention-to-treat analysis at the 6-month follow-up.

Note: The horizontal dashed line is set at bias estimate = 0. The vertical dashed line is set at ${\pi }_{0,0}^R=0.826$, which indicates that MAR holds. MAR = missing at random; RER = response exclusion restriction.

FIGURE 4. Not African Americans: Bias and mean squared error (MSE) in the intention-to-treat analysis at the 6-month follow-up.

Note: The horizontal dashed line is set at bias estimate = 0. The vertical dashed line is set at ${\pi }_{0,0}^R=0.672$, which indicates that MAR holds. MAR = missing at random; RER = response exclusion restriction.

8. Concluding Remarks

This study took an analytical approach in comparing sensitivity of causal effect estimates with different assumptions on treatment noncompliance and nonresponse behaviors. It was demonstrated that model comparisons can be performed in a more explicit way via decomposition of identifying assumptions and clarification of bias mechanisms. Interrelationship among identifying assumptions, which was not emphasized in previous research, turned out to be critical in judging relative performance of different causal effect estimation models. The JHU PIRC example showed that different model assumptions may be preferred even with the same outcome in the same trial, depending on the measured time points, which sheds light on the importance of investigating relative sensitivity by focusing on each case, instead of pursuing a general conclusion that applies to every occasion (i.e., one assumption always works better than the other).

It was demonstrated that it is quite straightforward to make a model selection based on potential biases due to violation of model-specific missing data assumptions. It is a common practice to focus on assumptions that distinguish one statistical model from the other when comparing biases from different models. However, biases due to violation of common assumptions also can affect the size and the direction of the total bias. An ideal model comparison should be based on such a complete picture of bias mechanisms that considers accumulation or cancellation of biases being combined. In this setting, all three models commonly assume LI and therefore are subject to bias due to violation of LI. As shown in the appendix, very detailed information is needed to predict bias due to deviation from LI, which is not a realistic option.

Further research is needed for the new method to be readily applicable to diverse settings of randomized trials. Because assumptions related to covariates further complicate the investigation of bias mechanisms, covariate-related parameters were not explicitly modeled in this study for simplicity. However, potential advantages of having covariates in causal effect estimation models make it worth investigating the role of covariates. For example, it is not well known how different assumptions (and deviations from them) on covariate-related parameters affect sensitivity of causal effect estimates with different missing-data assumptions. How other auxiliary information, rather than from covariates, affects bias mechanisms is also an important matter to be studied. This study used the instrumental variable approach based on the method of moments estimator. More efficient estimators based on maximum likelihood and other data augmentation methods may substantially adjust the expected biases.

Biography

BOOIL JO is an assistant professor in the Department of Psychiatry & Behavioral Sciences, Stanford University, 401 Quarry Road, Stanford, CA 94305-5795; booil@stanford.edu. Her areas of interest include latent variable modeling, causal inference, missing data analysis, and longitudinal data analysis.

Appendix

Deviations From Latent Ignorability (LI) and Corresponding Bias Mechanisms

Although LI can be violated due to various unobserved (or latent) variables that are not included in the causal effect estimation models, let us assume that there is only one omitted covariate associated with outcome missingness. For simplicity, it is also assumed that the covariate is binary. However, in practice, unobserved covariates are likely to have more complex forms. The focus here is given to the demonstration of possible complexities in the bias mechanism that involves violation of LI.

In this missing not at random (MNAR) setting, Y_i ⊥ R_i|Z_i, C_i, X_i. This implies that E(Y_i|R_i = r, C_i = c, Z_i = z, X_i = x) = E(Y_i|C_i = c, Z_i = z, X_i = x) = : μ_{c,z,X = x}. A covariate X is binary (1/0) and its information is completely missing. Let X̄_c,z : = E(X_i|C_i = c, Z_i = z).

The average response ${\pi }_{c,z}^R$ can be written given X as

$${\pi }_{c,z}^R={\overline{X}}_{c,z}{\pi }_{c,z,X=1}^R+(1-{\overline{X}}_{c,z}){\pi }_{c,z,X=0}^R,$$ (A1)

where ${\pi }_{c,z}^R:=E(R_i\mid C_i=c,Z_i=z)$ and ${\pi }_{c,z,X=x}^R:=E(R_i\mid C_i=c,Z_i=z,X_i=x)$.

Assuming that the proposed MNAR setting is correct,

$$\begin{array}{l}{\mu }_{c,z}=E(Y_i\mid C_i=c,Z_i=z)\\ =E[E(Y_i\mid C_i=c,Z_i=z,X_i)\mid C_i=c,Z_i=z]\\ =Pr(X_i=1\mid C_i=c,Z_i=z){\mu }_{c,z,X=1}+Pr(X_i=0\mid C_i=c,Z_i=z){\mu }_{c,z,X=0}\\ ={\overline{X}}_{c,z}{\mu }_{c,z,X=1}+(1-{\overline{X}}_{c,z}){\mu }_{c,z,X=0}.\end{array}$$ (A2)

Therefore, correct specifications of average outcomes given C, Z, R, and X are

$${\mu }_{1,1}={\overline{X}}_{1,1}{\mu }_{1,1,X=1}+(1-{\overline{X}}_{1,1}){\mu }_{1,1,X=0},$$ (A3)

$${\mu }_{1,0}={\overline{X}}_{1,0}{\mu }_{1,0,X=1}+(1-{\overline{X}}_{1,0}){\mu }_{1,0,X=0},$$ (A4)

$${\mu }_{0,1}={\overline{X}}_{0,1}{\mu }_{0,1,X=1}+(1-{\overline{X}}_{0,1}){\mu }_{0,1,X=0},$$ (A5)

$${\mu }_{0,0}={\overline{X}}_{0,0}{\mu }_{0,0,X=1}+(1-{\overline{X}}_{0,0}){\mu }_{0,0,X=0}.$$ (A6)

Under LI, it is assumed that $E(Y_i\mid R_i=r,C_i=c,Z_i=z,X_i=x)=E(Y_i\mid C_i=c,Z_i=z)=:{\mu }_{c,z}^{\text{LI}}$. The average outcome assuming LI can be written given R_i, C_i, Z_i, and X_i as

$$\begin{array}{l}{\mu }_{c,z}^{\text{LI}}=E\{E(Y_i\mid R_i=1,Z_i=z,C_i=c,X_i)\mid R_i=1,Z_i=z,C_i=c\}\\ =pr(X_i=1\mid R_i=1,Z_i=z,C_i=c){\mu }_{c,z,X=1}+pr(X_i=0\mid R_i=1,Z_i=z,C_i=c){\mu }_{c,z,X=0}\\ ={\overline{X}}_{c,z}\frac{{\pi }_{c,z,X=1}^R}{{\pi }_{c,z}^R}{\mu }_{c,z,X=1}+(1-{\overline{X}}_{c,z})\frac{{\pi }_{c,z,X=0}^R}{{\pi }_{c,z}^R}{\mu }_{c,z,X=0}.\end{array}$$ (A7)

That is,

$${\mu }_{1,1}^{\text{LI}}={\overline{X}}_{1,1}\frac{{\pi }_{1,1,X=1}^R}{{\pi }_{1,1}^R}{\mu }_{1,1,X=1}+(1-{\overline{X}}_{1,1})\frac{{\pi }_{1,1,X=0}^R}{{\pi }_{1,1}^R}{\mu }_{1,1,X=0},$$ (A8)

$${\mu }_{1,0}^{\text{LI}}={\overline{X}}_{1,0}\frac{{\pi }_{1,0,X=1}^R}{{\pi }_{1,0}^R}{\mu }_{1,0,X=1}+(1-{\overline{X}}_{1,0})\frac{{\pi }_{1,0,X=0}^R}{{\pi }_{1,0}^R}{\mu }_{1,0,X=0},$$ (A9)

$${\mu }_{0,1}^{\text{LI}}={\overline{X}}_{0,1}\frac{{\pi }_{0,1,X=1}^R}{{\pi }_{0,1}^R}{\mu }_{0,1,X=1}+(1-{\overline{X}}_{0,1})\frac{{\pi }_{0,1,X=0}^R}{{\pi }_{0,1}^R}{\mu }_{0,1,X=0},$$ (A10)

$${\mu }_{0,0}^{\text{LI}}={\overline{X}}_{0,0}\frac{{\pi }_{0,0,X=1}^R}{{\pi }_{0,0}^R}{\mu }_{0,0,X=1}+(1-{\overline{X}}_{0,0})\frac{{\pi }_{0,0,X=0}^R}{{\pi }_{0,0}^R}{\mu }_{0,0,X=0}.$$ (A11)

LI ignoring X implies that ${\pi }_{c,z,X=1}^R={\pi }_{c,z,X=0}^R={\pi }_{c,x}^R$. The difference between the specification of μ_c,z in Equations A8 through A11 and the specification in Equations A3 through A6 without assuming LI can be written as

$${\text{LI}}_{\mathit{bias}11}={\overline{X}}_{1,1}\left(\frac{{\pi }_{1,1,X=1}^R}{{\pi }_{1,1}^R}-1\right){\mu }_{1,1,X=1}+(1-{\overline{X}}_{1,1})\left(\frac{{\pi }_{1,1,X=0}^R}{{\pi }_{1,1}^R}-1\right){\mu }_{1,1,X=0},$$ (A12)

$${\text{LI}}_{\mathit{bias}10}={\overline{X}}_{1,0}\left(\frac{{\pi }_{1,0,X=1}^R}{{\pi }_{1,0}^R}-1\right){\mu }_{1,0,X=1}+(1-{\overline{X}}_{1,0})\left(\frac{{\pi }_{1,0,X=0}^R}{{\pi }_{1,0}^R}-1\right){\mu }_{1,0,X=0},$$ (A13)

$${\text{LI}}_{\mathit{bias}01}={\overline{X}}_{0,1}\left(\frac{{\pi }_{0,1,X=1}^R}{{\pi }_{0,1}^R}-1\right){\mu }_{0,1,X=1}+(1-{\overline{X}}_{0,1})\left(\frac{{\pi }_{0,1,X=0}^R}{{\pi }_{0,1}^R}-1\right){\mu }_{0,1,X=0},$$ (A14)

$${\text{LI}}_{\mathit{bias}00}={\overline{X}}_{0,0}\left(\frac{{\pi }_{0,0,X=1}^R}{{\pi }_{0,0}^R}-1\right){\mu }_{0,0,X=1}+(1-{\overline{X}}_{0,0})\left(\frac{{\pi }_{0,0,X=0}^R}{{\pi }_{0,0}^R}-1\right){\mu }_{0,0,X=0}.$$ (A15)

Then, from Equation 3, the total bias in the ITT estimator due to deviation from LI can be written as

$${\text{LI}}_{\mathit{bias}}={\pi }_c({\text{LI}}_{\mathit{bias}11}-{\text{LI}}_{\mathit{bias}10})+(1-{\pi }_c)({\text{LI}}_{\mathit{bias}01}-{\text{LI}}_{\mathit{bias}00}).$$ (A16)