Diagnosing Covariate Balance Across Levels of Right-Censoring Before and After Application of Inverse-Probability-of-Censoring Weights

Abstract

Covariate balance is a central concept in the potential outcomes literature. With selected populations or missing data, balance across treatment groups can be insufficient for estimating marginal treatment effects. Recently, a framework for using covariate balance to describe measured confounding and selection bias for time-varying and other multivariate exposures in the presence of right-censoring has been proposed. Here, we revisit this framework to consider balance across levels of right-censoring over time in more depth. Specifically, we develop measures of covariate balance that can describe what is known as “dependent censoring” in the literature, along with its associated selection bias, under multiple mechanisms for right censoring. Such measures are interesting because they substantively describe the evolution of dependent censoring mechanisms. Furthermore, we provide weighted versions that can depict how well such dependent censoring has been eliminated when inverse-probability-of-censoring weights are applied. These results provide a conceptually grounded way to inspect covariate balance across levels of right-censoring as a validity check. As a motivating example, we applied these measures to a study of hypothetical “static” and “dynamic” treatment protocols in a sequential multiple-assignment randomized trial of antipsychotics with high dropout rates.

Covariate balance is a central concept in the potential outcomes literature (1). In expectation, randomization balances all covariates in experimental studies, and propensity scores balance measured covariates in observational studies. When treatment groups have similar distributions of prognostic covariates that jointly affect or associate with outcomes and treatments, marginal contrasts between groups identify causal effects of treatment. Many experimental and observational studies of time-fixed treatments that employ propensity scores inspect covariate balance as a validity check (2, 3). See Jackson (4) for extensions to time-varying treatments. Although balance checks are limited under unmeasured confounding and misspecified causal models, they can be substantively meaningful and helpful in evaluating potential residual bias after adjustments for confounding (e.g., matching, subclassification, weighting). For example, imbalance can reflect positivity violations and effects of weight truncation (4, 5).

With missing data, balance across treatment groups can be insufficient for estimating marginal treatment effects (6–10). Those with complete versus incomplete data might have different distributions of covariates that correlate with the outcome, which can bias estimates of absolute and relative causal contrasts. Here, we revisit a previous framework of covariate balance for time-varying and other multivariate exposures in the presence of right censoring (4). Our contribution is to develop extensible measures of covariate balance that can substantively describe what is known as “dependent censoring” (7) under multiple mechanisms for right-censoring that evolve over time. Furthermore, we provide weighted versions that can depict how well such dependent censoring has been eliminated when inverse-probability-of-censoring weights (IPCW) (11) are applied to adjust for selection bias, as a conceptually grounded validity check.

As a motivating example, we applied these measures to a study of symptom trends under alternate antipsychotic treatment protocols in a randomized trial. This study had 2 censoring mechanisms, one of which is study attrition. The other is induced when the investigators estimate a per-protocol effect by artificially censoring those that deviate from protocol. We show how to describe both censoring mechanisms in the context of intention-to-treat, static, and dynamic treatment protocols. Our balance measures for (in)dependent censoring extend beyond intention-to-treat and per-protocol analyses of a trials (12) and their emulations (13) to other cohort designs (14).

We begin with notation and formal definitions of (in)dependent censoring and their role in causal identification. We apply these definitions to justify extensible measures of covariate balance across levels of censoring in unweighted and weighted data. We then demonstrate their use in our motivating example and discuss generalizations. The justifications in Web Appendixes 1–4 (available at https://academic.oup.com/aje) are given to highlight implications for balance, but are subsumed by the seminal work of Robins and colleagues (15–18).

NOTATION

Let $V(x)$ represent the value that the random variable $V$ would take had the random variable $X$ been set to value $x$. Let $t$ equal discrete time (i.e., study visit number) with baseline indexed at $t=0$ and the end of follow-up indexed at $t=T$. Let an overbar represent the entire history vector of $\mathit{X}$ from baseline through the indexed time (i.e., ${\overline{\mathbf{X}}}_{\mathbf{t}}=\{X_0,\dots ,X_t\}$). Let an underbar represent the vector of realizations from the indexed time through the end of follow-up (i.e., ${\underset{\_}{\mathbf{X}}}_{\mathbf{t}+1}=\{X_{t+1},\dots ,X_T\}$.) Let the expression $V{\coprod }^{}X|W$ denote conditional independence between $V$ and $X$ given $W$. Throughout, outcomes measured at time $t+1$ will be denoted as continuous $Y_{t+1}$, treatment actually taken at time $t$ will be denoted as binary $A_t$, and prognostic covariates, which might include earlier measures of the outcome, will be denoted as continuous or binary $L_t$. $U$ represents unmeasured prognostic factors not used to guide treatment decisions. Censoring mechanisms for study dropout $S_t$ by time $t$ and artificial censoring by the investigator $S_t^{†}$ by time $t$ are indicator variables (1 = censored, 0 = not censored). We assume the time-specific order $S_t,L_t,A_t,S_t^{†}$ such that if $S_t=1$ the rest are missing. Last, $G$ is describes a treatment protocol of interest to the investigator, a collective set of time-specific treatment rules $g_t$ (i.e., ${\overline{\mathbf{g}}}_{\mathbf{T}}=[g_0(\cdot ),\dots ,g_T(\cdot )]$), that a person’s treatment history ${\overline{\mathbf{A}}}_{\mathbf{t}}$ might be observed to be consistent with through time $t$ (19). Static treatment protocols do not depend on $L_t$ whereas dynamic ones do (20). See Web Appendix 1 for further details.

DEFINITION OF (IN)DEPENDENT CENSORING

We define a censoring mechanism to be a particular way in which a person’s data ceases to contribute to the analysis. One mechanism might be exit from a study for administrative reasons or loss to follow-up. Another mechanism might be due to the analytical strategy. For example, investigators might censor someone once their data are no longer consistent with a protocol of interest (17, 18, 21). The reasons that persons leave a study or switch treatments might be associated with outcomes. In what follows, we formally express this dependence between censoring and outcomes. Our arguments align with those used by Robins and other investigators in the context of survival analysis (6, 7), as well as missing data mechanisms (22) advanced by Rubin (23).

We define independent censoring as when, for all follow-up times, no censoring mechanism is associated with the history of prognostic covariates, measured or unmeasured, that are associated with subsequent outcomes. We define dependent censoring as when, for any follow-up time, any censoring mechanism is associated with the history of any prognostic covariate, measured or unmeasured. Under these definitions, censoring that is dependent will be associated with the history of prognostic covariates as well as future outcomes. Conversely, censoring that is independent will be associated with neither the history of prognostic covariates nor future outcomes. There is no way to empirically verify that independent censoring holds, even in randomized trials.

Implicitly, these definitions condition on those who have remained uncensored by any mechanism up to the considered time, because only those previously uncensored can become censored. Furthermore, for causal identification, these definitions condition on the history of treatment. Here, this simply means that definitions of (in)dependent censoring are specific to the treatment protocol arm, capturing the plausible possibility that the nature of dependent censoring varies by protocol arm. Participants in a placebo arm might exit for lack of efficacy, but participants in a protocol arm exit for intolerability. Last, these definitions condition on time because the way in which censoring mechanisms relate to the history of prognostic covariates might evolve over time. For example, attrition might be driven by adverse events and treatment efficacy early in a trial but towards the end be driven by social support or quality of life.

We can equivalently define (in)dependent censoring via the (in)dependence between censoring and either 1) the history of prognostic covariates or 2) potential outcomes, with factual outcomes realized under consistency with observed treatment history. With 2 mechanisms, attrition and departure from treatment protocol, we can define independent censoring as independence from the history of measured prognostic factors ${\overline{\mathbf{L}}}_{\mathbf{t}}$ and unmeasured prognostic factors ${\overline{\mathbf{U}}}_{\mathbf{t}}$:

$$\{{\overline{\mathbf{L}}}_{\mathbf{t}},{\overline{\mathbf{U}}}_{\mathbf{t}}\}{\coprod }^{}{S}_{t+1}|{\overline{\mathbf{S}}}_{\mathbf{t}}^{†}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},G=g$$ (1a)

$$\{{\overline{\mathbf{L}}}_{\mathbf{t}},{\overline{\mathbf{U}}}_{\mathbf{t}}\}{\coprod }^{}{S}_t^{†}|{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}-1}^{†}=\overline{\mathbf{0}},G=g.$$ (1b)

Equivalently, we can define independent censoring as independence from potential outcomes:

$${\underset{\_}{\mathbf{Y}}}_{\mathbf{t}+1}\left(g,{\overline{\mathbf{s}}}_{\mathbf{T}}=\overline{\mathbf{0}},{\overline{\mathbf{s}}}_{\mathbf{T}-1}^{†}=\overline{\mathbf{0}}\right){\coprod }^{}{S}_{t+1}|{\overline{\mathbf{S}}}_{\mathbf{t}}^{†}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},G=g$$ (2a)

$${\underset{\_}{\mathbf{Y}}}_{\mathbf{t}+1}\left(g,{\overline{\mathbf{s}}}_{\mathbf{T}}=\overline{\mathbf{0}},{\overline{\mathbf{s}}}_{\mathbf{T}-1}^{†}=\overline{\mathbf{0}}\right){\coprod }^{}{S}_t^{†}|{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}-1}^{†}=\overline{\mathbf{0}},G=g.$$ (2b)

Dependent censoring is defined as when either of the above independencies do not hold. When we have measured all the prognostic factors responsible for the violation of independent censoring, we can define conditional forms of independent censoring:

$${\overline{\mathbf{U}}}_{\mathbf{t}}{\coprod }^{}{S}_{t+1}|{\overline{\mathbf{L}}}_{\mathbf{t}},{\overline{\mathbf{S}}}_{\mathbf{t}}^{†}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},G=g$$ (3a)

$${\overline{\mathbf{U}}}_{\mathbf{t}}{\coprod }^{}{S}_t^{†}|{\overline{\mathbf{L}}}_{\mathbf{t}},{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}-1}^{†}=\overline{\mathbf{0}},G=g.$$ (3b)

Equivalently, we can also define conditional independent censoring as:

$${\underset{\_}{\mathbf{Y}}}_{\mathbf{t}+1}\left(g,{\overline{\mathbf{s}}}_{\mathbf{T}}=\overline{\mathbf{0}},{\overline{\mathbf{s}}}_{\mathbf{T}-1}^{†}=\overline{\mathbf{0}}\right){\coprod }^{}{S}_{t+1}|{\overline{\mathbf{L}}}_{\mathbf{t}},{\overline{\mathbf{S}}}_{\mathbf{t}}^{†}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},G=g$$ (4a)

$${\underset{\_}{\mathbf{Y}}}_{t+1}\left(g,{\overline{\mathbf{s}}}_{\mathbf{T}}=\overline{\mathbf{0}},{\overline{\mathbf{s}}}_{\mathbf{T}-1}^{†}=\overline{\mathbf{0}}\right){\coprod }^{}{S}_t^{†}|{\overline{\mathbf{L}}}_{\mathbf{t}},{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}-1}^{†}=\overline{\mathbf{0}},G=g.$$ (4b)

In 4a and 4b we include independence with the potential outcomes of the response ${\underset{\_}{\mathbf{Y}}}_{\mathbf{t}+1}\left(g,{\overline{\mathbf{s}}}_{\mathbf{T}}=\overline{\mathbf{0}},{\overline{\mathbf{s}}}_{\mathbf{T}-1}^{†}=\overline{\mathbf{0}}\right)$ alone, which is sufficient for causal identification of per-protocol effects under static protocols (19). We can alternatively define conditional independent censoring, also equivalent to 3a and 3b, that includes independence with potential outcomes of prognostic factors ${\underset{\_}{\mathbf{L}}}_{\mathbf{t}+1}\left(g,{\overline{\mathbf{s}}}_{\mathbf{T}}=\overline{\mathbf{0}},{\overline{\mathbf{s}}}_{\mathbf{T}-1}^{†}=\overline{\mathbf{0}}\right)$ for identification of effects under dynamic protocols: (19)

$$\begin{array}{c}\{{\underset{\_}{\mathbf{Y}}}_{\mathbf{t}+1}\left(g,{\overline{\mathbf{s}}}_{\mathbf{T}}=\overline{\mathbf{0}},{\overline{\mathbf{s}}}_{\mathbf{T}-1}^{†}=\overline{\mathbf{0}}\right),{\underset{\_}{\mathbf{L}}}_{t+1}\left(g,{\overline{\mathbf{s}}}_{\mathbf{T}}=\overline{\mathbf{0}},{\overline{\mathbf{s}}}_{\mathbf{T}-1}^{†}=\overline{\mathbf{0}}\right)\}\\ {\coprod }^{}{S}_{t+1}|{\overline{\mathbf{L}}}_{\mathbf{t}},{\overline{\mathbf{S}}}_{\mathbf{t}}^{†}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},G=g\end{array}$$ (4c)

$$\begin{array}{c}\{{\underset{\_}{\mathbf{Y}}}_{\mathbf{t}+1}\left(g,{\overline{\mathbf{s}}}_{\mathbf{T}}=\overline{\mathbf{0}},{\overline{\mathbf{s}}}_{\mathbf{T}-1}^{†}=\overline{\mathbf{0}}\right),{\underset{\_}{\mathbf{L}}}_{t+1}\left(g,{\overline{\mathbf{s}}}_{\mathbf{T}}=\overline{\mathbf{0}},{\overline{\mathbf{s}}}_{\mathbf{T}-1}^{†}=\overline{\mathbf{0}}\right)\}\\ {\coprod }^{}{S}_{\mathit{t}}^{†}|{\overline{\mathbf{L}}}_{\mathbf{t}},{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}-1}^{†}=\overline{\mathbf{0}},G=g\end{array}$$ (4d)

(IN)DEPENDENT CENSORING AND CAUSAL IDENTIFICATION

Intention-to-treat estimand

These definitions are not just useful for substantively describing the censoring mechanisms at play. They are critical assumptions that underlie any causal analysis of right-censored data. For example, suppose we are interested in estimating the outcome trend under an intention-to-treat protocol in a baseline randomized trial with study attrition, as represented by the causal graph in Figure 1A and Web Figure 1. There is only one censoring mechanism at play here, that of study dropout. We want to estimate the outcome trend had, counter to fact, everyone been observed for the duration of the study and followed the intention-to-treat protocol. That is, we want to estimate ${\overline{\mathbf{Y}}}_{\mathbf{T}}\left(g,{\overline{\mathbf{s}}}_{\mathbf{T}}=\overline{\mathbf{0}}\right)$. Those who initiate different protocols are exchangeable by virtue of the randomization. If time-varying prognostic factors $L_t$ and $A_t$ (as well as $U_t$) are independent of dropout $S_{t+1}$ given the protocol arm, the uncensored are exchangeable with the censored. This sequential exchangeability would be reflected by a lack of backdoor paths between $G$ and ${\overline{\mathbf{Y}}}_{\mathbf{T}}$, and also between $S_{t+1}$ and ${\underset{\_}{\mathbf{Y}}}_{\mathbf{t}+1}$ given ${\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}}$ and $G=g$. Under dependent censoring where $U_t$ is still independent of $S_{t+1}$, such backdoor paths would be present. But the paths between $S_{t+1}$ and ${\underset{\_}{\mathbf{Y}}}_{\mathbf{t}+1}$ would be blocked upon further conditioning on $L_t$ and $A_t$, rendering the uncensored and censored conditionally exchangeable. There are several ways to leverage such conditional exchangeability to identify our causal estimand. Interestingly, IPCW for study dropout accomplishes this by producing a pseudopopulation where there is independent censoring with respect to the measured covariates (15). More directly, if conditional independent censoring holds in the crude study data along with positivity, independent censoring with respect to measured and unmeasured covariates holds in the weighted population. It does this by reweighting the data in such a way that, for each level of $L_t$ and $A_t$ among the previously uncensored, the proportion of those who depart versus remain in the study does not depend on $L_t$ or $A_t$. In our context, the stabilized weights (11) $W_t$ are:

$$W_t={\prod }_{j=0}^t\frac{\mathrm{Pr}[{\mathit{S}}_{\mathit{j}+1}=s|{\overline{\mathbf{S}}}_{\mathbf{j}}=\overline{\mathbf{0}},G=g]}{\mathrm{Pr}[S_{j+1}=s|L_j,A_j,{\overline{\mathbf{S}}}_{\mathbf{j}}=\overline{\mathbf{0}},G=g]}$$ (5)

Figure 1.

Causal graph relating protocol arm at baseline $G$, actual treatment $A_t$ taken, prognostic covariates $L_t$, unmeasured prognostic factors $U$, subsequent outcomes $Y_{t+1}$, and censoring due to study dropout $S_t$ and/or artificial censoring $S_t^{†}$ for an intention-to-treat analysis (A), static per-protocol analysis (B), dynamic per-protocol analysis (C).

Static per-protocol estimand

Suppose that we were now interested in estimating the outcome trend under adherence to a static treatment protocol (always take a given treatment) in a baseline randomized trial with study attrition, as represented by the causal graph in Figure 1B and Web Figure 2, where there is no unmeasured confounding of time-varying treatment $A_t$. We want to estimate the outcome trend had, counter to fact, everyone been observed for the duration of the study and followed the protocol. That is, we want to estimate ${\overline{\mathbf{Y}}}_{\mathbf{T}}\left(g,{\overline{\mathbf{s}}}_{\mathbf{T}}=\overline{\mathbf{0}},{\overline{\mathbf{s}}}_{\mathbf{T}-1}^{†}=\overline{\mathbf{0}}\right)$ and do so by artificially censoring persons when they their actual treatment conflicts with the protocol. With treatments randomized at baseline, those who initiate the protocol versus those who do not are exchangeable at baseline. However, there are 2 censoring mechanisms at play here, that of study dropout and also that of protocol nonadherence. If time-varying factors $L_t$ (and also $U_t$) are independent of dropout $S_{t+1}$ and nonadherence $S_t^{†}$ in each arm, as in the case of independent censoring, then for both censoring mechanisms the censored are exchangeable with the uncensored. Under dependent censoring where $U_t$ is still independent of $S_{t+1}$ and also independent of $S_t^{†}$ through its lack of association with $A_t$, we have $L_t$ associated with $S_{t+1}$ or with $S_t^{†}$ through its association with $A_t$. This leads to backdoor paths between one or both censoring mechanisms ($S_{t+1}$ or $S_t^{†}$) and the outcome ${\overline{\mathbf{Y}}}_{\mathbf{t}+1}$. However, conditional on $L_t$, these pathways would be blocked, such that the censored and uncensored are conditionally exchangeable. Interestingly, joint IPCW for study dropout and static protocol nonadherence identifies our counterfactual trend by producing a pseudopopulation where there is independent censoring with respect to measured covariates. It does this by reweighting the data in such a way that, for each level of $L_t$ among those previously uncensored by any mechanism, the proportion of those who drop out versus remain in the study, and adhere versus do not adhere to protocol, does not depend on $L_t$. In our context, the stabilized weights (17, 18) are:

$$\begin{array}{cc}{W}_t& =\prod _{j=0}^t\frac{\mathrm{P}\mathrm{r}[S_{j+1}=s|{\overline{\mathbf{S}}}_{\mathbf{j}}^{†}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{j}}=\overline{\mathbf{0}},G=g]}{\mathrm{P}\mathrm{r}[S_{j+1}=s|L_j,{\overline{\mathbf{S}}}_{\mathbf{j}}^{†}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{j}}=\overline{\mathbf{0}},G=g]}\\ & \times \prod _{j=0}^t\frac{\mathrm{P}\mathrm{r}[S_j^{†}=s^{†}|{\overline{\mathbf{S}}}_{\mathbf{j}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{j}-1}^{†}=\overline{\mathbf{0}},G=g]}{\mathrm{P}\mathrm{r}[S_j^{†}=s^{†}|L_j,{\overline{\mathbf{S}}}_{\mathbf{j}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{j}-1}^{†}=\overline{\mathbf{0}},G=g]}\end{array}$$ (6)

Dynamic per-protocol estimand

These concepts also extend to the case for estimating the outcome trend under adherence to a dynamic treatment protocol that depends on prognostic factors (e.g., initiate a given treatment, switch to another when a prognostic factor drops below a certain threshold), as in Figure 1C and Web Figure 3, where there is no unmeasured confounding of time-varying treatment $A_t$. Suppose we are interested in estimating this counterfactual trend by censoring persons when their treatment conflicts with the protocol. Even though treatments are randomized, the protocol of interest depends on prognostic factors $L_t$, so those who initiate protocol versus those who do not are only conditionally exchangeable at baseline given $L_0$. As with the static case, there are 2 censoring mechanisms at play here, that of study dropout and that of protocol nonadherence, which depends on $L_t$. Thus, censoring will always be dependent in the crude data, even when the treatments $A_t$ are randomized. Provided that $U_t$ is independent of $S_{t+1}$ and $A_t$, conditioning on $L_t$ blocks backdoor paths between each censoring mechanism ($S_{t+1}$ and $S_t^{†}$) and the outcome ${\underset{\_}{\mathbf{Y}}}_{\mathbf{t}+1}$, making the uncensored exchangeable with the censored. The joint IPCW for study dropout and dynamic protocol nonadherence identify our counterfactual trend by producing a pseudopopulation with independent censoring with respect to measured covariates. In our context, the stabilized weights are of the same form as equation 6.

MEASURES OF COVARIATE BALANCE FOR (IN)DEPENDENT CENSORING

The definitions outlined above have direct implications for using covariate balance to describe (in)dependent censoring and the nature of its associated selection bias. The general approach is, for each censoring mechanism, to compare at each time the distribution of covariate history among those who become censored versus those who do not, among those following the same treatment protocol who have not yet been censored by any mechanism. This definition implies an ordering of mechanisms within each time interval that is known a priori, could be ascertained, or could be arbitrarily chosen for mechanisms whose timing within time intervals coincides. They also highlight why it is important to distinguish mechanisms: The prognostic factors involved in dependent censoring plausibly vary by mechanism.

Using the standardized mean difference, we can define the following balance measures. The first measure assesses balance across levels of censoring due to study dropout, among those in arm $G=g$ who have not dropped out of the study or deviated from protocol up to that point:

$$\frac{\begin{array}{c}E[L_t\times W_{t,s_{t+1}=1,s_t^{†}=0}\times I(S_{t+1}=1,S_t^{†}=0)|{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}-1}^{†}=\overline{\mathbf{0}},G=g]\\ -E[L_t\times W_{t,s_{t+1}=0,s_t^{†}=0}\times I(S_{t+1}=0,S_t^{†}=0)|{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}-1}^{†}=\overline{\mathbf{0}},G=g]\end{array}}{{\sigma }_{S_{t+1}=0|{\overline{\mathbf{S}}}_{\mathbf{t}}^{†}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},G=g}}$$ (7a)

where $W_{t,s_{t+1}=s^{\text{'}},s_t^{†}=0}=1$ for assessment of balance in the unweighted data. If we used the weights specified in equation 6 evaluated at the person’s observed values for censoring due to study dropout and protocol nonadherence, we would have:

$$\begin{array}{ll}{W}_{t,s_{t+1}=s^{\text{'}},s_t^{†}=0}& =\frac{\mathrm{Pr}\left[S_{t+1}=s^{\text{'}}|{\overline{\mathbf{S}}}_{\mathbf{t}}^{†}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},G=g\right]}{\mathrm{Pr}\left[S_{t+1}=s^{\text{'}}|L_t,{\overline{\mathbf{S}}}_{\mathbf{t}}^{†}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_t=\overline{\mathbf{0}},G=g\right]}\\ & \times \prod _{j=0}^{t-1}\frac{\mathrm{Pr}\left[S_{j+1}=0|{\overline{\mathbf{S}}}_{\mathbf{j}}^{†}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{j}}=\overline{\mathbf{0}},G=g\right]}{\mathrm{Pr}\left[S_{j+1}=0|L_j,{\overline{\mathbf{S}}}_{\mathbf{j}}^{†}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{j}}=\overline{\mathbf{0}},G=g\right]}\\ & \times \prod _{j=\mathrm{0}}^t\frac{\mathrm{P}\mathrm{r}[S_j^{†}=\mathrm{0}|{\overline{\mathbf{S}}}_{\mathbf{j}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{j}-1}^{†}=\overline{\mathbf{0}},G=g]}{\mathrm{P}\mathrm{r}[S_j^{†}=0|L_j,{\overline{\mathbf{S}}}_{\mathbf{j}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{j}-1}^{†}=\overline{\mathbf{0}},G=g]}.\end{array}$$

The second measure assesses balance across levels of censoring due to protocol nonadherence, among those in arm $G=g$ who have not dropped out the study or deviated from protocol up to that point:

$$\frac{\begin{array}{c}E[L_t\times W_{t,s_{t+1}=s,s_t^{†}=1}\times I(S_{t+1}=s,S_t^{†}=1)|{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}-1}^{†}=\overline{\mathbf{0}},G=g]\\ -E[L_t\times W_{t,s_{t+1}=s,s_t^{†}=0}\times I(S_{t+1}=s,S_t^{†}=0)|{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}-1}^{†}=\overline{\mathbf{0}},G=g]\end{array}}{{\mathrm{\sigma }}_{{\mathit{S}}_{\mathbf{t}}^{†}=0|{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}-1}^{†}=\overline{\mathbf{0}},G=g}}.$$ (7b)

where $W_{t,s_{t+1}=s,s_t^{†}=s^{\text{'}}}=1$ for assessment of balance in the unweighted data. If we used the weights specified in equation 6 evaluated at the person’s observed values for censoring due to study dropout and protocol nonadherence, we would have:

$$\begin{array}{cc}{W}_{t,s_{t+1}=s,s_t^{†}=s\prime }& =\frac{\mathrm{Pr}\left[S_{t+1}=s|{\overline{\mathbf{S}}}_{\mathbf{t}}^{†}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},G=g\right]}{\mathrm{Pr}\left[S_{t+1}=s|L_t,{\overline{\mathbf{S}}}_{\mathbf{t}}^{†}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},G=g\right]}\\ & \times \prod _{j=0}^{t-1}\frac{\mathrm{Pr}\left[S_{j+1}=0|{\overline{\mathbf{S}}}_{\mathbf{j}}^{†}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{j}}=\overline{\mathbf{0}},G=g\right]}{\mathrm{Pr}\left[S_{j+1}=0|L_j,{\overline{\mathbf{S}}}_{\mathbf{j}}^{†}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{j}}=\overline{\mathbf{0}},G=g\right]}\\ & \times \frac{\mathrm{P}\mathrm{r}[S_t^{†}=s^{\text{'}}|{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}-1}^{†}=\overline{\mathbf{0}},G=g]}{\mathrm{P}\mathrm{r}[S_t^{†}=s^{\text{'}}|L_t,{\overline{\mathbf{S}}}_{\mathbf{t}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{t}-1}^{†}=\overline{\mathbf{0}},G=g]}\\ & \times \prod _{j=0}^{t-1}\frac{\mathrm{P}\mathrm{r}[S_j^{†}=0|{\overline{\mathbf{S}}}_{\mathbf{j}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{j}-1}^{†}=\overline{\mathbf{0}},G=g]}{\mathrm{P}\mathrm{r}[S_j^{†}=0|L_j,{\overline{\mathbf{S}}}_{\mathbf{j}}=\overline{\mathbf{0}},{\overline{\mathbf{S}}}_{\mathbf{j}-1}^{†}=\overline{\mathbf{0}},G=g]}.\end{array}$$

Note that for both measures, for efficiency we estimate the standard deviation $\mathrm{\sigma }$ of $L_t$ among the currently uncensored in $G=g$, but other choices are possible, such as pooling across censoring levels. Note also that the weights in 7b are depicted as they are to make clear that assessing balance across levels of protocol nonadherence is not restricted to those present at the next study visit. They also make clear that the measures capture the effects of the joint weights on balance, not just their individual components. As for interpretation, following Ho et al. (24), independent censoring would call for minimizing imbalance without limit.

We need measures 7a and 7b when both mechanisms for censoring are present. In the case of censoring due only to study dropout, we would need only the relevant balance measure: 7a for study dropout or 7b for nonadherence. If there were more than 2 censoring mechanisms, we could expand our approach by evaluating time-specific balance for each form of censoring, conditional on the protocol arm and not having been previously censored by any mechanism. We argue in Web Appendix 5 that, although possible (4), it is not necessary to evaluate balance across finer gradations of censoring levels (e.g., Web Figure 4), such as documented reasons for study dropout or reasons for treatment switching.

APPLICATION TO MOTIVATING EXAMPLE

Clinical management of schizophrenia requires balancing treatment response and adverse effects, such as weight gain and onset of movement disorders. Because treatment-specific response unpredictably varies by patient, clinicians usually adapt a treatment strategy until a suitable course is found. The Clinical Antipsychotic Trial of Intervention Effectiveness schizophrenia study (CATIE; NCT00014001) was a landmark comparative effectiveness trial (n = 1,462) in the United States that evaluated effects of first- and second-generation antipsychotics over 18 months of follow-up of persons with schizophrenia (25, 26). Treatment choices were randomized at baseline and again upon treatment failure or intolerability at the patient and treating physician’s discretion. Despite this robust design, high attrition rates across treatment protocol arms (44%–57%) might bias assessments of symptom trajectories, such as the Positive and Negative Syndrome Scale (PANSS) (27). High attrition rates are common in trials of schizophrenia patients (28, 29).

A previous study (30) used the same study’s data to study the effects of several hypothetical treatment protocols on 18-month PANSS scores among patients without tardive dyskinesia, a movement disorder that occurs more frequently with first-generation antipsychotics (31). We consider 3 of these protocols that trade between PANSS score reduction and risk of tardive dyskinesia. The first protocol is static and always treats with the first-generation antipsychotic perphenazine. The second protocol, also static, always treats with a second-generation antipsychotic. The third protocol depends on a threshold PANSS score, $\mathrm{\theta }$. If baseline PANSS score is above threshold, treatment begins with perphenazine. Once the PANSS score drops below threshold, treatment continues with a second-generation antipsychotic from that point forward. Conversely, if baseline PANSS score is below threshold, the third protocol always treats with a second-generation antipsychotic. Protocols 2 and 3 allow patients to switch between second-generation antipsychotics as needed. For estimation, we used the “clone-and-censor” strategy with IPCW for attrition and protocol deviation (18) to examine trends in PANSS score under these protocols with the threshold $\mathrm{\theta }$ set to 50 in protocol 3 (see Li et al. (32) for an overview). Following the previous study (30), we excluded those with tardive dyskinesia at baseline, but we differed by choosing to not censor patients who were randomized to clozapine or ziprasidone or who entered the open-label phase of the study. Our use of IPCW rather than imputation to address attrition also differs from the previous study (30). The purpose of our application is demonstrative. Web Appendix 6 and Web Table 1 provide further analytical details.

Table 1 shows that, with the exception of baseline PANSS scores, baseline covariates had similar distributions across protocol arms. Figure 2A and 2B show, respectively, the absolute distributions of study dropout and protocol nonadherence over time in each arm for selected times. Study dropout is right-skewed over time. The nonadherence patterns show that for those who initiated perphenazine (arm 1) or second-generation antipsychotics (arm 2), very few departed from these strategies until month 3. The distribution of the weights (Web Figure 5) show somewhat well-behaved weights for arm 2 but not the others. Figures 3 and 4 show, respectively, the arm-specific covariate balance across study dropout and protocol nonadherence, before and after weighting. Due to space limitations, these display data for months 1, 6, and 12 (see Web Figures 6 and 7 for all months). Each plot reports standardized mean differences (x-axis) comparing the means of those who drop out/depart from protocol with those who do not at a given month. Panels arrange the data by month in increasing order. The standardized means for each covariate (y-axis) are further distinguished by protocol arm (shapes). These distinctions allow us to examine whether the covariate associations with censoring vary over time and protocol arm.

Table 1.

Baseline Characteristics and Censoring Frequency of Those Initiating Different Hypothetical Treatment Protocols, Using Data From the Clinical Antipsychotic Treatment Intervention Effectiveness Study^a, United States, 2000–2004

Characteristic	Protocol 1b			Protocol 2c			Protocol 3d
	No.	%	Mean (SD)	No.	%	Mean (SD)	No.	%	Mean (SD)
Sample size	151	100		1,055	100		205	100
Age, years			38.8 (11.0)			39.3 (10.9)			39.9 (10.5)
Male sex	103	68		781	74		145	70
Female sex	48	32		274	26		61	30
Race/ethnicity
White	90	60		633	60		120	58
Black	52	34		372	35		75	36
AIAN/Asian/NHPI/Mixed race	9	6		50	5		11	5
High-school graduate	113	75		787	75		159	78
Unemployed	124	82		882	84		161	78
Recent hospitalization	46	30		295	28		56	27
Phase 1A stratume	0	0		0	0		0	0
Ziprasidone stratumf	151	100		591	56		179	87
PANSSg			75.1 (18.5)			75.6 (17.5)			66.5 (21.6)
CGI severityg			3.9 (0.9)			4 (1.0)			3.5 (1.0)
Calgary depression			1.9 (0.8)			1.9 (0.9)			1.7 (0.8)
Quality of life			2.7 (1.1)			2.8 (1.1)			3 (1.2)
Body weight, lb			192.6 (46.3)			195.8 (47.7)			194.9 (46.5)
Simpson-Agnes EPS			0.2 (0.3)			0.2 (0.3)			0.2 (0.3)
Drug use scale			1.4 (0.7)			1.4 (0.7)			1.3 (0.6)
Censoring reason
Protocol nonadherence	57	38		136	13		69	33
Study discontinuation	74	49		464	44		93	45
Either reason	120	79		586	56		152	74

Abbreviations: AIAN, American Indian and Alaskan Native; CGI, Clinical Global Impressions; EPS, extrapyramidal symptoms; NHPI, Native Hawaiian and Other Pacific Islander; PANSS, Positive and Negative Syndrome Scale; SD, standard deviation.

^a A multisite pragmatic trial (ClinicalTrials.gov: NCT00014001).

^b Protocol 1: Always take perphenazine, a first-generation antipsychotic.

^c Protocol 2: Always take a second-generation antipsychotic.

^d Protocol 3: Initiate perphenazine if PANSS ≥50, switch to second-generation antipsychotic when PANSS <50; always second-generation antipsychotic if PANSS <50.

^e Patients with tardive dyskinesia.

^f Patients randomized after approval of (and thus eligibility to be assigned) ziprasidone.

^g Positive and Negative Syndrome Scale (27) and Clinical Global Impressions (45).

Figure 2.

Histogram of censoring events over time, using data from Clinical Antipsychotic Trials of Intervention Effectiveness, United States, 2000–2004. A) Study dropout; B) protocol departure. White: protocol arm 1, always perphenazine; gray: protocol arm 2, always second-generation antipsychotic; black: protocol arm 3, dynamic.

Figure 3.

Covariate balance across levels of censoring at months 1, 6, and 12, using data from Clinical Antipsychotic Trials of Intervention Effectiveness, United States, 2000–2004. A) Due to study dropout in the crude data; B) after applying inverse-probability-of-censoring weights. Reference lines for the standardized mean difference are located at −0.25, 0, and 0.25. Some standardized mean differences are beyond the limits of (−1,1); see Web Figure 6 for all times and for all months. Others could not be calculated due to lack of covariate variation among the uncensored (see Web Table 2 for these unstandardized differences). Circles: protocol arm 1, always perphenazine; triangles: protocol arm 2, always second-generation antipsychotic; squares: protocol arm 3, dynamic. CGI, Clinical Global Impressions (45); EPS, extrapyramidal symptoms; PANSS, Positive and Negative Syndrome Scale (27).

Figure 4.

Covariate balance across levels of censoring at months 1, 6, and 12, using data from Clinical Antipsychotic Trials of Intervention Effectiveness, United States, 2000–2004. A) Due to protocol nonadherence in the crude data; B) after applying inverse-probability-of-censoring weights. Reference lines for the standardized mean difference are located at −0.25, 0, and 0.25. Some standardized mean differences are beyond the limits of (−1,1); see Web Figure 7 for all times and for all months. Others could not be calculated due to lack of covariate variation among the uncensored (see Web Table 2 for these unstandardized differences). Circles: protocol arm 1, always perphenazine; triangles: protocol arm 2, always second-generation antipsychotic; squares: protocol arm 3, dynamic. CGI, Clinical Global Impressions (45); EPS, extrapyramidal symptoms; PANSS, Positive and Negative Syndrome Scale (27).

In Figure 3A, we see that those who dropped out had higher levels of illness severity, higher use of illicit substances, lower declines in PANSS scores, and less weight gain. Speculatively, these clinical parameters are consistent with poorer treatment effectiveness, which can occur with concomitant illicit substance use. These relationships vary across arm and also over time, with larger differences between dropouts versus nondropouts towards the end of the study. In Figure 3B we see that after applying IPCW, these differences were ameliorated in many cases. In other cases, they persisted or were exacerbated but there is no clear pattern. In Figure 4A we see that, initially, those who deviated from the dynamic protocol had lower depression symptoms, illness severity, and weight gain, suggesting that those who adhere to protocol early on are sicker. From month 6 on, those who depart from protocol had, at times, higher levels of illness severity, higher drug use, and lower quality of life. Those who departed from the static protocol to always treat with a second-generation antipsychotic did not often differ greatly from those who adhered, although there were some exceptions across covariates at certain times. Those who departed from the static protocol to always treat with perphenazine also showed fewer differences earlier on in the trial. Later, those who departed had higher levels of depression, illicit drug use, and weight gain. In Figure 4B, we can see that after applying IPCW these differences are sometimes ameliorated. Particularly for those following protocol 1, the smallest group, differences persist, or are exacerbated or qualitatively change direction.

We did not observe significant differences in PANSS score trends across protocol arms at the $\alpha =0.05$ significance level, either before or after weighting (see Web Figure 8 for estimated trends). However, both sets of results likely suffer from some degree of selection bias and should be interpreted with caution. The inability of IPCW to fully remove covariate differences between the censored and uncensored is not altogether unsurprising, given the small size of the groups initiating protocols 1 and 3. It can be difficult to model heterogeneous selection mechanisms over time well with small samples, and positivity violations might also be present. Our findings support the decision of the previous studies of these regimes (30, 33) to pursue imputation methods for dealing with the high rates of study attrition in the Clinical Antipsychotic Trial of Intervention Effectiveness schizophrenia study.

DISCUSSION

Assumptions about the censoring processes are essential for identification, and the (in)dependent nature of censoring will be reflected in balance of covariates across levels of censoring. We can never be sure that independent censoring holds in the raw data. However, we can reject it when measured prognostic factors associate with censoring, which can substantively enrich our understanding of it. Further, we can use substantive knowledge to consider the causal architecture of censoring and measure sufficient prognostic factors to achieve conditional independent censoring. Finally, if we use IPCW to correct for dependent censoring, we can verify whether our estimated weights produce independent censoring with respect to measured covariates in the pseudopopulation. We have theoretically justified covariate balance measures across levels of multiple censoring mechanisms for each of these purposes and demonstrated their utility.

The present work could be expanded in several ways. Our measures of covariate balance assess only the balance of marginal covariate distributions at the mean, which is appropriate only when the underlying structural model lacks nonlinearities and interactions in the covariates (34). Given that the measures need only condition on the protocol arm and being previously uncensored, it would be straightforward to adapt more detailed assessments of covariate balance that have been considered in the propensity-score literature (35, 36). These would include balance of product terms, cumulative distributions, the Kolmogorov distance, and kernel-based estimators, among others. Second, the plots themselves might become unwieldy when there are several points of follow-up and rich covariate history. Previous work for plotting and summarizing covariate balance with high-dimensional measurements could easily be adapted here (4). The mean values and distribution of time-specific stabilized weights (5) are compact, easy to compute, and informative about aggregate balance (4). However, they do not substantively describe how the censored differ from the uncensored. Another important avenue for future work is to explore potential for bias, given that there might be little bias when the proportion of dropout is very low. There is an emerging literature exploring the presence, direction, and magnitude of bias under various causal structures for selection bias, including the absence of scale-dependent treatment effect heterogeneity (on censoring and/or outcomes) (9, 10, 37, 38). This literature could be used to define bias metrics for dependent censoring that incorporate a priori substantive knowledge about the censoring mechanisms. Concerning our restriction to a time-fixed exposure, our balance measures could be expanded to condition on a function of time-varying exposure history (4), as would be relevant for epidemiologic cohort designs that do not emulate trials (14). Last, our results could also be incorporated into efforts that use covariate balance to directly optimize estimation of causal effects (39–43).

Covariate balance can be useful for describing multiple censoring mechanisms over time, not only for describing potential selection bias but also residual selection bias in the application of IPCW. See the example code in the Web Appendix for implementation in R (R Foundation for Statistical Computing, Vienna, Austria) (44).

Abbreviations

IPCW

inverse-probability-of-censoring weights

PANSS

Positive and Negative Syndrome Scale