Covariate Adjustment in Randomized Clinical Trials with Missing Covariate and Outcome Data

Abstract

When analyzing data from randomized clinical trials, covariate adjustment can be used to account for chance imbalance in baseline covariates and to increase precision of the treatment effect estimate. A practical barrier to covariate adjustment is the presence of missing data. In this paper, in the light of recent theoretical advancement, we first review several covariate adjustment methods with incomplete covariate data. We investigate the implications of the missing data mechanism on estimating the average treatment effect in randomized clinical trials with continuous or binary outcomes. In parallel, we consider settings where the outcome data are fully observed or are missing at random; in the latter setting, we propose a full weighting approach that combines inverse probability weighting for adjusting missing outcomes and overlap weighting for covariate adjustment. We highlight the importance of including the interaction terms between the missingness indicators and covariates as predictors in the models. We conduct comprehensive simulation studies to examine the finite-sample performance of the proposed methods and compare with a range of common alternatives. We find that conducting the proposed adjustment methods generally improves the precision of treatment effect estimates regardless of the imputation methods when the adjusted covariate is associated with the outcome. We apply the methods to the Childhood Adenotonsillectomy Trial to assess the effect of adenotonsillectomy on neurocognitive functioning scores.

1. Introduction

Randomized controlled trials (RCT) are the gold standard for evaluating the efficacy and safety of new treatments. Randomization ensures both measured and unmeasured covariates are balanced in large samples. However, chance imbalance of patient baseline characteristics may occur due to the random nature in allocating the treatment (Senn, 1989; Ciolino et al., 2015) especially when the sample size is small (Thompson et al., 2015). Adjusting for imbalanced covariates, particularly the ones that are predictive of the outcome, that is, prognostic risk factors, in analysis can increase precision in estimating the treatment effects as well as improve face validity (Ciolino et al., 2015; Pocock et al., 2002; Lin, 2013; Benkeser et al., 2021). One common method for covariate adjustment in RCT is regression adjustment (Yang and Tsiatis, 2001; Kahan et al., 2016; Leon et al., 2003; Tsiatis et al., 2008; Zhang et al., 2008), where the outcome is regressed on the treatment and covariates (Tsiatis et al., 2008; Lin, 2013), and the treatment effect is estimated by the coefficient of the treatment variable. This is also known as the analysis of covariance (ANCOVA) method. Recently, propensity score weighting has been proposed as an alternative covariate adjustment method with several conceptual and practical advantages (Williamson et al., 2014; Shen et al., 2014; Colantuoni and Rosenblum, 2015; Zeng et al., 2021). The two methods are shown to be asymptotically equivalent (Williamson et al., 2014; Zeng et al., 2021) and empirical analyses also suggest similar finite sample performances in many situations.

A practical barrier to covariate adjustment is the presence of missing data. When covariates are partially observed, the analysts have several choices of estimating the treatment effect: (i) the unadjusted analysis, where one obtains a simple difference-in-mean estimator without using any covariate information; (ii) the complete-case analysis, where only units with fully observed covariates are used in the adjustment; (iii) the complete-covariate analysis, where only fully observed covariates are used in the adjustment; (iv) the missing-indicator method (Groenwold et al., 2012), where one includes a dummy variable for missingness for the partially observed covariate in regression adjustment, and all missing values are set to the same value; (v) the imputation method, where one first imputes the missing covariate values and then conducts adjustment on the completed data. A consensus in the literature is that the complete-case analysis should be avoided (Schemper and Smith, 1990; White and Thompson, 2005; Groenwold et al., 2012; Sullivan et al., 2018; Kayembe et al., 2020, 2022). Moreover, White and Thompson (2005) showed that the missing-indicator method generally performs well regardless of whether missingness of baseline covariates is predictive of the outcome, whereas mean imputation is adequate when missingness is completely random (MCAR). Groenwold et al. (2012) clarified that the missing-indicator method is valid for randomized experiments and obeys the intention-to-treat principle, but it can lead to bias in observational studies, which was also pointed out earlier (Greenland and Finkle, 1995). Sullivan et al. (2018) and Kayembe et al. (2020) investigated the use of the popular multiple imputation (MI) (Rubin, 1987) method in regression adjustment; both concluded that MI does not improve over simpler imputation methods in most cases.

An important recent theoretical development regarding regression adjustment with missing covariates is due to Zhao and Ding (2022). They proposed a modified missing-indicator method, where one imputes the missing covariates with zeros and then applies Lin’s regression estimator (Lin, 2013), which postulates an outcome model including all the imputed covariates, the missing indicators, and the covariate-treatment interactions. They showed that this method is asymptotically more efficient than the unadjusted estimator, regardless of the missing data mechanism. They also showed theoretically that the choice of imputation method and the validity of the imputation model matters little, that is, simple imputation by a constant leads to the same precision gain as imputations based on correctly specified models.

Despite the above advancements, covariate adjustment with missing data still has not been widely adopted in practice. And a few important questions remain. First, propensity score weighting has been advocated as an outcome-model-free and asymptotically equivalent alternative to regression adjustment, but it is unclear how to accommodate partially missing covariates in this approach. Second, more importantly, most clinical trials have missing data in both covariates and outcomes. Despite the large literature on the analysis of missing outcome data in randomized trials, there has been little investigation on the case of missingness in both covariates and outcomes. When there is missing data in the outcomes, complete case analysis that ignores units with missing outcomes can lead to biased treatment effect estimates as well as efficiency loss. Under the missing at random (MAR) assumption (Rubin, 1976), two commonly used approaches for handling missing outcomes are: (i) the imputation method (Rubin, 1996), where one first imputes the missing outcomes based on a model and then estimate the treatment effects; (ii) the inverse probability weighting (IPW) method (Horvitz and Thompson, 1952; Li et al., 2013), where one estimates the probability of missingness of each unit given the observed variables and weighs the outcome by the inverse of that probability in estimating the treatment effect (Seaman and White, 2013). Because one cannot verify the correctness of the MAR assumption using the observed data, sensitivity analyses such as those based on pattern-mixture and selection models are recommended to evaluate the robustness of conclusions under alternative assumptions about the missingness process (Little et al., 2012). It is important to note that consideration of missing data is closely tied with the target estimand. The focus of this paper is on handling missing covariates in estimating treatment effects with covariate adjustments. Recognizing that outcome missingness is common in practice, we consider both the settings when the outcome data are complete or are partially observed, in the latter case, we will assume an MAR mechanism and adopt an inverse probability weighted approach.

The rest of the paper is organized as follows. We first briefly review covariate adjustment with complete data (Section 2). We then discuss covariate adjustment with (i) missingness only in covariates (Section 3.1) and (ii) missingness in both covariates and outcome (Section 3.2). We specify two versions of missing at random assumptions of the outcome missingness mechanisms and provide corresponding models to estimate the outcome missingness probability (i.e., the propensity). We further propose a full weighting approach that combines inverse probability weighting for adjusting missing outcomes and overlap weighting for covariate adjustment (Zeng et al., 2021). We then conduct extensive simulation studies to compare the proposed methods with a range of alternatives (Section 4). In Section 5, we apply the methods to the Childhood Adenotonsillectomy Trial (CHAT) (Marcus et al., 2013) and illustrate one way to assess how the magnitude of potential efficiency gain vary according to the proportion of missingness and the covariate’s association with the outcome.

2. Covariate adjustment with complete data

Consider a randomized trial with two arms and $N$ patients, where $N_1$ and $N_0$ patients are randomized into the treatment and control arm, respectively. Let $Z_i=z$ be the binary treatment indicator, with $z=1$ indicates treatment and $z=0$ control. Under the potential outcome framework (Neyman 1923) and the stable unit treatment value assumption (SUTVA), each unit has two potential outcomes $\left\{Y_i\left(1\right),Y_i\left(0\right)\right\}$, corresponding to the treatment and the control conditions, respectively. For each unit, only the potential outcome under the actual assigned condition is observed, denoted by $Y_i=Z_i{Y}_i\left(1\right)+\left(1-Z_i\right)Y_i\left(0\right)$. A collection of $p$ baseline variables are recorded for each unit, denoted by $X_i={\left(X_{i1},\dots ,X_{ip}\right)}^T$. A common causal estimand is the average treatment effect (ATE):

$$\tau =E\left\{Y_i\left(1\right)-Y_i\left(0\right)\right\}$$ (1)

In randomized trials, the treatment $Z$ is randomly assigned with a fixed probability $r$, that is $\mathrm{P}\mathrm{r}\left(Z_i=\right.\left.1\mid X_i,Y_i\left(1\right),Y_i\left(0\right)\right)=\mathrm{P}\mathrm{r}\left(Z_i=1\right)=r$. A common study design uses balanced assignment with $r=1/2$. Under randomization, we have $\tau =E\left(Y_i\mid Z_i=1\right)-E\left(Y_i\mid Z_i=0\right)$, and thus an unbiased estimator of the ATE is the unadjusted difference-in-means estimator: ${\hat{\tau }}^{\mathrm{U}\mathrm{N}\mathrm{A}\mathrm{D}\mathrm{J}}=\sum _{i=1}^N{Z}_i{Y}_i/\sum _{i=1}^n{Z}_i-\sum _{i=1}^N\left(1-Z_i\right)Y_i/\sum _{i=1}^n\left(1-Z_i\right)$. Previous research has shown that adjusting for chance imbalance in covariates that are predictive of the outcome, namely prognostic covariates, can substantially improve statistical efficiency over the unadjusted estimator.

There are two classes of covariate adjustment methods in randomized trials. The first class is outcome regression (Lin, 2013; Yang and Tsiatis, 2001; Kahan et al., 2016; Leon et al., 2003; Tsiatis et al., 2008; Zhang et al., 2008)—also known ordinary least square (OLS) regression—based on the analysis of covariance (ANCOVA) models, where the outcome is regressed on the treatment and covariates and the OLS estimator, ${\tau }^{ols}$, is estimated by the coefficient of the treatment variable (Yang and Tsiatis, 2001; Leon et al., 2003; Tsiatis et al., 2008). Lin (2013) showed that for efficient outcome regression adjustment it is critical to include the full set of covariate-treatment interaction terms in the OLS model:

$$Y_i\sim Z_i+X_i+Z_i{X}_i.$$ (2)

When the randomization probability is 1/2, OLS returns consistent point and interval estimates even if the outcome model is misspecified (Yang and Tsiatis, 2001; Lin, 2013; Wang et al., 2019). However, misspecification of the outcome model can decrease precision in unbalanced experiments with treatment effect heterogeneity (Freedman, 2008). Despite the asymptotic argument, including a full set of covariate-treatment interaction terms sometimes leads to higher variance in small samples. OLS models can also lead to unstable estimates when applied to rare binary outcomes (Zeng et al., 2021).

The second class of covariate adjustment methods is propensity score weighting (Williamson et al., 2014; Shen et al., 2014; Colantuoni and Rosenblum, 2015; Zeng et al., 2021). The general form of a weighted estimator for the ATE is

$${\hat{\tau }}^w=\frac{\sum _{i=1}^n{w}_1\left(x_i\right)Z_i{Y}_i}{\sum _{i=1}^n{w}_1\left(x_i\right)Z_i}-\frac{\sum _{i=1}^n{w}_0\left(x_i\right)\left(1-Z_i\right)Y_i}{\sum _{i=1}^n{w}_0\left(x_i\right)\left(1-Z_i\right)},$$ (3)

where $w_1\left(x\right)=h\left(x\right)/e\left(x\right)$, $w_0\left(x\right)=h\left(x\right)/\{1-e(x\left)\right\}$ are some type of balancing weights (Li et al., 2018), $e\left(x\right)=\mathrm{P}\mathrm{r}\left(Z_i=1\mid X_i=x\right)$ is the treatment propensity score, i.e., the conditional probability of treatment given the covariates (Rosenbaum and Rubin, 1983), and $h\left(x\right)$ is a pre-specified function of the covariates, known as the tilting function (Li and Li, 2019). In RCT, the true propensity score is known and usually fixed at $e_i\left(x\right)=\mathrm{P}\mathrm{r}\left(Z_i=1\right)=r$ for all units. In that case, one can show that the weighting estimator ${\hat{\tau }}^w$ is unbiased for the ATE $\tau$ regardless of the specific choice of $h\left(x\right)$ as long as $h\left(x\right)$ is a function of the propensity score $e\left(x\right)$. Two examples of the balancing weights are (i) inverse probability weights $\left(\mathrm{I}\mathrm{P}\mathrm{W}\right):\left(w_1,w_0\right)=(1/e(x),1/\{1-e\left(x\right)\left\}\right)$, obtained by $h\left(x\right)=1$ (Williamson et al., 2014; Shen et al., 2014; Colantuoni and Rosenblum, 2015); and (ii) overlap weights (OW): $\left(w_1,w_0\right)=(1-e(x),e(x\left)\right)$, obtained by $h\left(x\right)=e\left(x\right)(1-e(x\left)\right)$ (Li et al., 2018, 2019; Zeng et al., 2021). In the context of covariate adjustment, the propensity score $e_i$ in the weighting estimator (3) is replaced by a working propensity score ${\hat{e}}_i=e(X_i;\hat{\theta })$, estimated from a “working” logistic model

$$e_i=e\left(X_i;\theta \right)=\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left({\theta }_0+X_i^T{\theta }_1\right)}{1+\mathrm{e}\mathrm{x}\mathrm{p}\left({\theta }_0+X_i^T{\theta }_1\right)},$$ (4)

with parameters $\theta ={\left({\theta }_0,{\theta }_1^T\right)}^T$ and $\hat{\theta }$ being the maximum likelihood estimate of $\theta$. For example, the OW estimator for ATE is

$${\hat{\tau }}^{\mathrm{O}\mathrm{W}}=\frac{\sum _{i=1}^n\left(1-{\hat{e}}_i\right)Z_i{Y}_i}{\sum _{i=1}^n\left(1-{\hat{e}}_i\right)Z_i}-\frac{\sum _{i=1}^n{\hat{e}}_i\left(1-Z_i\right)Y_i}{\sum _{i=1}^n{\hat{e}}_i\left(1-Z_i\right)}.$$ (5)

OW has a unique exact balance property: when the treatment propensity scores are estimated from a logistic model, the weighted average of any predictor in the model is exactly the same between the treatment and control arms (Li et al., 2018).

Zeng et al. (2021) proved that the weighting estimator ${\hat{\tau }}^w$ with any balancing weights is asymptotically equivalent to the efficient OLS estimator for continuous outcomes. They further demonstrated via analytical derivations and simulations that the exact balance property of OW improves finite-sample efficiency in estimating ATE compared to IPW. Therefore, we will focus on OW in the next section. Weighting methods have two practical advantages compared to outcome regression adjustment. First, weighting obviates the need to specify an outcome model and has the same form for all types of outcome. This is particularly desirable in handling rare binary or categorical outcomes, where outcome regression can fail to converge. Second, specification of the treatment propensity score model is usually simpler than that of the outcome model because the former does not include covariate-treatment interactions; this can be beneficial for finite-sample performance when the sample size is small. Third, implementation of weighting methods is more straightforward than outcome regression, which requires centering of all covariates. Moreover, because weighting only involves modeling the baseline data, conceptually it might be more appealing to analysts who would avoid directly modeling the outcome data.

3. Covariate adjustment with incomplete data

3.1. Adjustment with missingness in covariates

We introduce additional notation in the presence of incomplete data. We first consider the case of missingness only in covariates. For simplicity, we describe the case with one partially observed covariate $X$, centered based on the observed values, and rest of the covariates are fully observed and centered, denoted as $V$. For each unit $i(i=1,\dots n)$, let $R_i^x$ be the missingness indicator for $X_i$, with $R_i^x=1$ if the value is observed and 0 if missing. Let $X_i^{\text{imp}}$ denote the imputed value based on some imputation method if the value $X_i$ is missing. For notational simplicity, below we omitted the fully observed covariate $V$ but stress that one should consider adjustment of imbalanced prognostic variables and stratification factors used in the stratified randomization.

When there is missingness only in $X$, Zhao and Ding (2022) showed theoretically that it matters little how to impute the missing $X$ as long as the missingness indicator is included as in the following ANCOVA model for covariate adjustment. Specifically, they showed the minimum specification of regression adjustment to ensure asymptotic efficiency gain is as follows:

$$Y_i\sim Z_i+X_i{R}_i^x+R_i^x+Z_i\left(X_i{R}_i^x+R_i^x\right).$$ (6)

This result is proved from a design-based perspective without any assumption on the missing data mechanism of $X$, and the only requirement is that $R^x$ is independent of the treatment assignment. A technical insight is that the interaction term $X_i{R}_i^x$ is numerically equivalent to (i) $X_i$ when the covariate is observed and (ii) zero when the covariate is missing regardless how the missing value is imputed. Therefore, the term $X_i{R}_i^x$ effectively imputes the missing values as zero, and we can generically denote $X_i^{\text{imp}}=X_i{R}_i^x$. In programming, we suggest to use median instead zero as the imputed values because for categorical variables (both lead to identical model fit), the imputed zeros are indistinguishable from the actual zeros, which may cause confusion.

The theoretical explanation of Zhao and Ding’s result lies in Lin’s result (Lin, 2013) that adding more covariates into the OLS adjustment model (2) would always increase the asymptotic efficiency and the fact that the missingness indicators $R^x$ are essentially additional covariates (comparing models (2) and (6)). We now also provide some empirical intuition for Zhao and Ding’s result. Consider a case where there is a covariate that is predictive of a unit’s outcome and also his/her propensity to missing data, and that covariate is not fully observed. As a hypothetical example, suppose that older patients are more likely to have a higher outcome value, and also tend to fail to report some covariates, but age is missing for some patients. In this case, the covariate missingness indicator is a strong predictor of the outcome, and thus ensuring its balance would improve the efficiency of the treatment effect estimate.

Recognizing the importance of the covariate missingness indicators $R^x$, for the propensity score weighting method for covariate adjustment, we suggest to add $R^x$’s into the working propensity score model, as follows:

$$\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\left\{e\left(X_i,R_i^x\right)\right\}={\theta }_0+{\theta }_1{X}_i{R}_i^x+{\theta }_2{R}_i^x,$$ (7)

and then plug in the estimated propensity score ${\hat{e}}_i$ to formula (5) to estimate the treatment effect.

Below we generalize the exact balancing property of overlap weighting to the case with missing covariates. For simplicity in notation, we present the proposition with only one covariate, which is straightforward to be extended to multiple covariates.

Proposition 1

Assume the working treatment propensity scores are estimated from the logistic regression model (7): ${\hat{e}}_i={\{1+\mathrm{e}\mathrm{x}\mathrm{p}[-({\hat{\theta }}_0+{\hat{\theta }}_1{X}_i{R}_i^x+{\hat{\theta }}_2{R}_i^x)]\}}^{-1}$, where $\hat{\theta }$ is the maximum likelihood estimate of the model coefficients. Then the resulting overlap weights lead to exact balance in the means of any predictor, denoted by $W$, in the variable set $\left(X{R}^x,R^x\right)$, between the treatment and control groups. That is,

$$\frac{\sum _i{W}_i{Z}_i\left(1-{\hat{e}}_i\right)}{\sum _i{Z}_i\left(1-{\hat{e}}_i\right)}=\frac{\sum _i{W}_i\left(1-Z_i\right){\hat{e}}_i}{\sum _i\left(1-Z_i\right){\hat{e}}_i}.$$ (8)

The proof follows the same as that in Li et al. (2018) and thus is omitted here. Similar to the case with complete covariates, this exact balance property is the key to improve the finite-sample efficiency over other weighting methods (Zeng et al., 2021).

Variance of the covariate-adjusted estimators ${\widehat{\tau }}^{ols}$ and ${\widehat{\tau }}^w$ can be estimated via bootstrap, where one resamples the original data with missing values and repeats the analysis detailed above. The bootstrap takes into account the sample variability as well as the uncertainty of imputing the missing data, but it can be computationally intensive. Alternatively, one can use the analytical Eicker-Huber-White robust standard error for the OLS estimator (Lin, 2013; Zhao and Ding, 2022). This estimator is asymptotically conservative but serves as a convenient approximation to the true standard error. For the weighting estimator, one can use the sandwich variance estimator (Lunceford and Davidian, 2004; Li and Li, 2019), both of which are implemented in the $\mathrm{R}$ package PSweight (Zhou et al., 2020).

3.2. Adjustment with missingness in covariates and outcomes

We now consider the case of missingness in both covariates and outcome. Let $R_i^y$ denote the outcome missingness indicator for unit $i$, with $R_i^y=1$ if $Y_i$ is observed and 0 if missing. A common method in the presence of missing $Y$ is the complete-outcome analysis, which removes all units with missing outcome. Such method implicitly assumes the outcome is missing completely at random (MCAR) (Rubin, 1976), that is, $\mathrm{P}\mathrm{r}\left(R_i^y=1\mid Y_i,X_i,R_i^x,V_i,Z_i\right)=\mathrm{P}\mathrm{r}\left(R_i^y=1\right)$. This assumption is usually questionable, and can lead to large bias as well as loss of sample size (White and Thompson, 2005). It should be avoided unless there is a strong reason to believe MCAR holds. A more principled approach is to inversely weigh each observed outcome by its probability of being observed (Seaman and White, 2013). Below we discuss how to proceed with the inverse weighting method in the presence of missing $Y$ under the assumption of outcome missing at random (MAR), that is, the outcome missingness can depend on the fully observed covariates and the missing covariates indicators.

Assumption 1

The missing data mechanism of $Y$:

$\mathrm{P}\mathrm{r}(R_i^y=1\mid Y_i,X_i,R_i^x,V_i,Z_i)=\mathrm{P}\mathrm{r}(R_i^y=1\mid R_i^x,V_i,Z_i)$.

To proceed with inverse probability weighting, one needs to estimate the probability of $Y_i$ being observed, $p_i=\mathrm{P}\mathrm{r}\left(R_i^y=1\mid R_i^x,Z_i,V_i\right)$. Similar to the estimation strategy for the treatment propensity score, given Assumption 1, we can postulate a model to estimate the outcome missingness propensity. For example, a simple choice is the following logistic model:

$$\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\left\{\mathrm{P}\mathrm{r}\left(R_i^y=1\mid R_i^x,V_i,Z_i\right)\right\}={\gamma }_0+{\gamma }_1{R}_i^x+{\gamma }_2{V}_i+{\gamma }_3{Z}_i.$$ (9)

We now discuss an alternative MAR outcome missingness mechanism that is weaker than the assumption 1. We assume: (i) if the covariate value for unit $i$ is observed, i.e. $R_i^x=1$, then the missingness of $Y_i$ can be dependent on $X_i,Z_i,V_i:\mathrm{P}\mathrm{r}\left(R_i^y=1\mid Y_i,X_i,R_i^x,V_i,Z_i\right)=\mathrm{P}\mathrm{r}\left(R_i^y=1\mid X_i,V_i,Z_i\right)$, and (ii) if the covariate value is missing, i.e. $R_i^x=0$, then the missingness of $Y_i$ can be dependent on $Z_i,V_i:\mathrm{P}\mathrm{r}\left(R_i^y=1\mid Y_i,X_i,R_i^x,V_i,Z_i\right)=\mathrm{P}\mathrm{r}\left(R_i^y=1\mid V_i,Z_i\right)$. That is,

Assumption 2

The missing data mechanism of $Y$:

$\mathrm{P}\mathrm{r}\left(R_i^y=1\mid Y_i,X_i,R_i^x,V_i,Z_i\right)=\mathrm{P}\mathrm{r}\left(R_i^y=1\mid \right.\left.R_i^x,X_i{R}_i^x,V_i,Z_i\right)$.

Assumption 2 is weaker than Assumption 1 in the sense that it allows the missingness of $Y$ to also depend on the observed values of the partially observed covariates. This assumption was first made in Rosenbaum and Rubin (1984). Given Assumption 2, we can postulate the following model to estimate the outcome missingness probability (or propensity score):

$$\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\left\{\mathrm{P}\mathrm{r}\left(R_i^y=1\mid X_i,R_i^x,V_i,Z_i\right)\right\}={\gamma }_0+{\gamma }_1{X}_i{R}_i^x+{\gamma }_2{R}_i^x+{\gamma }_3{V}_i+{\gamma }_4{Z}_i.$$ (10)

As discussed before, the interaction term $X_i{R}_i^x$ effectively imputes all the missing values as zero regardless of the actual imputed values.

The analysts can choose either Assumption 1 or 2 depending on the specific study, and estimate the outcome missing propensity accordingly. It is straightforward to extend Model (9) and (10) to include multiple partially observed covariates: for each partially observed covariate, one needs to include the corresponding missing indicator $R_i^x$ or/and the interaction term $X_i{R}_i^x$ as predictors. Note that the role of the outcome missingness propensity $\left(p_i\right)$ is different from that of the “working” treatment propensity $\left(e_i\right)$. Specifically, the treatment propensity score is used to adjust for chance imbalance in baseline characteristics to improve efficiency and it is never misspecified because of the randomization. On the other hand, the outcome propensity score is used to adjust for potential bias due to informative missingness in outcomes and its validity depends on the correct specification of the outcome missingness process.

Once we estimate the outcome missingness and the treatment propensity (denoted by ${\hat{p}}_i$ and ${\hat{e}}_i$, respectively), there are two approaches for covariate adjustment. The first approach is to conduct a weighted least square (WLS) by weighting the ANCOVA (6) on the observed outcomes with the weights ${\widehat{p}}_i$, and the estimated coefficient of $Z$ is the treatment effect ${\widehat{\tau }}^{wls}$. Below we instead focus on the weighting approach following the discussion in Section 2, partly due to the coherency of a full weighting approach. Specifically, we propose a full weighting approach that combines overlap weighting for covariate adjustment and inverse probability weighting for adjusting for the missing outcomes, as follows.

S1. Impute the missing covariate values by zeros or median of the observed values of that covariate.

S2. Estimate the outcome missingness propensity for each unit ${\widehat{p}}_i$, e.g. via Model (9) or (10).

S3. Estimate the working treatment propensity score for each unit ${\widehat{e}}_i$, via the logistic model (7).

S4. Estimate the covariate-adjusted treatment effect by an inverse weighted Hajék estimator based on units with observed outcomes

$${\widehat{\tau }}^{ow-ipw}=\frac{\sum _{i=1}^n{w}_1\left(X_i\right)R_i^y{Z}_i{Y}_i}{\sum _{i=1}^n{w}_1\left(X_i\right)R_i^y{Z}_i}-\frac{\sum _{i=1}^n{w}_0\left(X_i\right)R_i^y\left(1-Z_i\right)Y_i}{\sum _{i=1}^n{w}_0\left(X_i\right)R_i^y\left(1-Z_i\right)},$$ (11)

where the weight $w$ is the product of the overlap weight based on ${\widehat{e}}_i$ and the inverse of the missingness propensities ${\widehat{p}}_i:w_1\left(X_i\right)=\left(1-{\widehat{e}}_i\right)/{\widehat{p}}_i$ and $w_0\left(X_i\right)={\widehat{e}}_i/{\widehat{p}}_i$.

The variance of the covariate-adjusted estimators ${\widehat{\tau }}^{ow-ipw}$ can be estimated via bootstrap, where one resamples the original data with missing values and repeats the analysis detailed above. The bootstrap takes into account the sample variability, the uncertainty of imputing the missing data, as well as the uncertainty in estimating both the treatment and outcome missingness propensity scores.

4. Simulations

We conduct extensive simulations to investigate the finite-sample performance of the proposed estimator with a variety of alternatives. In particular, we examine these questions: (i) Is the choice of imputation methods important for efficiency? Does misspecified imputation models decrease efficiency? (ii) Does the asymptotic equivalence between outcome regression and propensity score weighting extend to the case with missing covariates and translate into finite-sample similarities? (iii) Is it important to account for the missingness in the outcome data at the same time? We first conduct simulation studies with missing covariates only. We then extend the simulation to incorporate both missing covariates and outcomes.

4.1. Simulations with missing covariates and fully observed outcomes

4.1.1. Continuous outcome

This section focuses on the setting with fully observed continuous outcomes. We consider a setting with $p=3$ baseline covariates, ${\mathbf{X}}_i={\left(X_{i1},X_{i2},X_{i3}\right)}^T$, among which $X_1$ is highly predictive of the outcome but subject to missingness, while $X_2,X_3$ are less predictive and completely observed. In practice, this could correspond to the scenario where $X_1$ is the baseline measurement of the outcome, and $X_2$ and $X_3$ are some other less important covariates. Covariates $X_{i1},X_{i2}$ are generated from a bivariate normal distribution with mean ${(0,0)}^T$, variance ${(1,1)}^T$, and correlation 0.3; $X_{i3}$ is simulated from a centered Bernoulli distribution, i.e. Bern(0.5) - 0.5. Let $R_i^{x_1}$ denote the missing indicator of $X_{i1}$ and $R_i^{x_1}=1$ if $X_{i1}$ is observed and 0 otherwise. We focus on simple randomization of sample size $N=100$ or 500 with a balanced design $(r=1/2)$. The potential outcomes are generated from the following linear model: for $z=0,1$

$$Y_i\left(z\right)\sim 𝒩\left({\beta }_0+\alpha z+{\mathbf{X}}_i^T{\mathit{\beta }}_1+z{\mathbf{X}}_i^T{\mathit{\beta }}_2,{\sigma }_y^2\right)$$ (12)

The observed outcome is set to be $Y_i=Y_i\left(Z_i\right)=Z_i{Y}_i\left(1\right)+\left(1-Z_i\right)Y_i\left(0\right)$. Since all covariates have mean 0, the coefficient $\alpha$ corresponds to the true ATE (i.e., $\tau$) on the additive scale, and we fix $\alpha =\tau =0$. We set the intercept ${\beta }_0=0.8$ and the covariate main effects ${\mathit{\beta }}_1=(3,0.3,0.42)$ and the treatment-covariate interaction effects ${\mathit{\beta }}_2=(0.75,0.53,0.38)$. The residual variance ${\sigma }_y^2$ is set to 1.

We generate data based on the following three types of missing data mechanisms:

1. Missing completely at random (MCAR): $R_i^{x_1}\stackrel{iid}{\sim }\mathrm{B}\mathrm{e}\mathrm{r}\mathrm{n}(q)$

2. Missing at random (MAR): $\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\left\{\mathrm{P}\mathrm{r}\left(R_i^{x_1}=1\right)\right\}={\gamma }_0+{\gamma }_1{X}_{i2}+{\gamma }_2{X}_{i3}$

3. Missing not at random (MNAR): $\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\left\{\mathrm{P}\mathrm{r}\left(R_i^{x_1}=1\right)\right\}={\zeta }_0+{\zeta }_1{X}_{i1}$

We fix ${\gamma }_1={\gamma }_2={\zeta }_1=-1$, and for each missing mechanism, we simulate data with approximately 10% or 30% of missingness in $X_{i1}$ through varying the values of $q,{\gamma }_0,{\zeta }_0$ respectively.

To summarize, our simulation scenarios consist of 2 × 2 × 3 = 12 combinations of the total sample size (2 levels), the covariates missing proportion (2 levels) and the missing data mechanism (3 levels), and for each scenario, we conduct covariate adjustment using both OW and ANCOVA. We compare three types of imputation methods: (1) mean imputation; (2) correct model imputation where we impute with the predicted values from the linear model $X_1\sim X_2$ fitted to those with $X_1$ observed; (3) wrong model imputation where we impute with the predicted values from the linear model $X_1\sim X_2^2+X_3$. For each type of imputation methods, covariate adjustment is performed both with and without the missing indicators. That is, for ANCOVA, we compare the model $Y_i\sim Z_i+X_{i1}^{imp}+X_{i2}+X_{i3}+Z_i(X_{i1}^{imp}+X_{i2}+X_{i3})$ with $Y_i\sim Z_i+X_{i1}^{imp}+X_{i2}+X_{i3}+R_i^{x_1}+Z_i(X_{i1}^{imp}+X_{i2}+X_{i3}+R_i^{x_1})$. And for OW, the treatment propensity score model without missing indicator is $\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\{Pr(Z=1)\}\sim X_{i1}^{imp}+X_{i2}+X_{i3}$, and with missing indicator it becomes $\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\left\{Pr\left(Z_i=1\right)\right\}\sim X_{i1}^{imp}+X_{i2}+X_{i3}+R_i^{x_1}$. Additionally, we include two types of complete data analysis (complete-unit and complete-covariate), as well as an unattainable ideal case where ANCOVA/OW is applied to the full data with no missingness. The resulting estimators of $\tau$ are evaluated in terms of their empirical bias and efficiency. The relative efficiency is defined as the ratio between the Monte Carlo variance of the unadjusted estimator to that of the adjusted estimator. The higher the relative efficiency, the more efficient that estimator is compared to the unadjusted estimator.

Table 1 summarizes the simulation results for sample size $N=100$. All approaches produce nearly unbiased estimates of the ATE except for the complete-unit approach under the MAR and MNAR settings, where the magnitude of bias increases as the proportion of missingness increases from 10% to 30%. Overall, performing covariate adjustment using either ANCOVA or OW improves efficiency over the unadjusted estimators, and adjustment with imputed $X_1$ results in further efficiency gain over adjustment with complete covariates only. On the other hand, comparisons among different imputation methods render minimal differences, which echoes the theoretical results by (Zhao and Ding, 2022). Inclusion of the missing indicator in the ANCOVA or the propensity score model does not appear to impact efficiency much under MCAR and MAR, as seen in (Kayembe et al., 2022), but can substantially improve efficiency under the MNAR setting. Similar trends are observed for sample size $N=500$ and the results are provided in Table 2.

Table 1:

Simulation results for continuous outcomes with sample size 100. RE stands for relative efficiency compared to the unadjusted estimator; MSI stands for missing indicators. The results are based on 5000 iterations.

	Covariates Missingness Mechanism
	MCAR				MAR				MNAR
	Bias		RE		Bias		RE		Bias		RE
Method	ANCOVA	OW	ANCOVA	OW	ANCOVA	OW	ANCOVA	OW	ANCOVA	OW	ANCOVA	OW
N = 100, 30% missing
Full data	.00	.00	11.22	11.04	.00	.00	11.22	11.04	.00	.00	11.22	11.04
Complete covariate	.02	.02	1.20	1.20	.02	.02	1.20	1.20	.02	.02	1.20	1.20
Complete unit	.00	.00	7.78	7.66	.20	.20	7.77	7.69	.23	.23	7.72	7.59
Mean imputation w/o MSI	.01	.01	2.92	2.92	.01	.01	2.98	2.97	.00	.00	2.44	2.45
Mean imputation w/ MSI	.01	.01	2.91	2.92	.01	.01	2.99	3.00	.00	.00	3.42	3.45
Correct model imputation w/o MSI	.01	.01	3.05	3.04	.01	.01	3.08	3.08	.00	.01	2.60	2.61
Correct model imputation w/ MSI	.01	.01	3.05	3.05	.01	.00	3.07	3.08	.00	.00	3.58	3.60
Wrong model imputation w/o MSI	.01	.01	2.89	2.86	.01	.01	2.88	2.85	.00	.01	2.86	2.42
Wrong model imputation w/ MSI	.01	.01	2.85	2.86	.01	.01	2.85	2.87	.00	.00	3.37	3.40
N = 100, 10% missing
Full data	.00	.00	11.22	11.04	.00	.00	11.22	11.04	.00	.00	11.22	11.04
Complete covariate	.02	.02	1.20	1.20	.02	.02	1.20	1.20	.02	.02	1.20	1.20
Complete unit	.00	.00	10.12	9.93	.07	.07	10.17	9.95	.08	.08	10.32	10.13
Mean imputation w/o MSI	.00	.00	5.70	5.64	.00	.00	5.61	5.54	.00	.00	4.42	4.40
Mean imputation w/ MSI	.00	.00	5.44	5.83	.00	.00	5.52	5.86	.00	.00	5.90	6.31
Correct model imputation w/o MSI	.00	.00	5.93	5.87	.00	.00	5.94	5.86	.00	.00	4.72	4.69
Correct model imputation w/ MSI	.00	.00	5.68	6.04	.00	.00	5.73	6.04	.00	.00	6.12	6.52
Wrong model imputation w/o MSI	.00	.00	5.68	5.58	.00	.01	5.47	5.34	.00	.00	4.66	4.36
Wrong model imputation w/ MSI	.00	.00	5.38	5.76	.00	.00	5.32	5.68	.00	.00	5.83	6.24

Table 2:

Simulation results for continuous outcomes with sample size 500. RE stands for relative efficiency compared to the unadjusted estimator; MSI stands for missing indicators. The results are based on 5000 iterations.

	MCAR				MAR				MNAR
	Bias		RE		Bias		RE		Bias		RE
Method	ANCOVA	OW	ANCOVA	OW	ANCOVA	OW	ANCOVA	OW	ANCOVA	OW	ANCOVA	OW
N = 500, 30% missing
Full data	.00	.00	11.00	10.95	.00	.00	11.00	10.95	.00	.00	11.00	10.95
Complete covariate	.01	.01	1.20	1.20	.01	.01	1.20	1.20	.01	.01	1.20	1.20
Complete unit	.00	.00	7.88	7.85	.21	.21	7.97	7.92	.23	.23	7.77	7.74
Mean imputation w/o MSI	.00	.00	2.92	2.92	.00	.00	2.96	2.95	.01	.01	2.50	2.49
Mean imputation w/ MSI	.00	.00	2.91	2.92	.00	.00	3.00	3.00	.00	.00	3.41	3.41
Correct model imputation w/o MSI	.00	.00	3.06	3.06	.00	.00	3.12	3.12	.01	.01	2.60	2.60
Correct model imputation w/ MSI	.00	.00	3.06	3.06	.00	.00	3.13	3.13	.00	.00	3.49	3.49
Wrong model imputation w/o MSI	.00	.00	2.90	2.90 0	.00	.00	2.89	2.86	.00	.01	2.90	2.48
Wrong model imputation w/ MSI	.00	.00	2.89	2.90	.00	.00	2.90	2.90	.00	.00	3.39	3.39
N = 500, 10% missing
Full data	.00	.00	11.00	10.95	.00	.00	11.00	10.95	.00	.00	11.00	10.95
Complete covariate	.01	.01	1.20	1.20	.01	.01	1.20	1.20	.01	.01	1.20	1.20
Complete unit	.00	.00	9.86	9.81	.07	.07	10.01	9.96	.08	.08	10.05	10.02
Mean imputation w/o MSI	.00	.00	5.73	5.71	.00	.00	5.65	5.63	.00	.00	4.50	4.49
Mean imputation w/ MSI	.00	.00	5.66	5.74	.00	.00	5.76	5.82	.00	.00	6.21	6.28
Correct model imputation w/o MSI	.00	.00	5.98	5.96	.00	.00	6.00	5.98	.00	.00	4.82	4.81
Correct model imputation w/ MSI	.00	.00	5.91	5.98	.00	.00	5.98	6.04	.00	.00	6.44	6.51
Wrong model imputation w/o MSI	.00	.00	5.71	5.70	.00	.00	5.58	5.53	.00	.00	4.73	4.47
Wrong model imputation w/ MSI	.00	.00	5.64	5.72	.00	.00	5.67	5.73	.00	.00	6.18	6.26

4.1.2. Binary outcome

We perform another set of simulations under the setting with fully observed binary outcomes. We consider a balanced randomization design ($r=0.5$) and use the same covariate generating processes and missingness processes as in Section 4.1.1. The potential outcomes are generated from the following logistic model: for $z=0,1$,

$$\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\left\{\mathrm{P}\mathrm{r}\left(Y_i\left(z\right)=1\right)\right\}={\beta }_0+\alpha z+{\mathbf{X}}_i^T{\mathit{\beta }}_1+z{\mathbf{X}}_i^T{\mathit{\beta }}_2$$ (13)

We fixed ${\beta }_0=\alpha =0$ and ${\mathit{\beta }}_1={(4,1,1)}^T$, ${\mathit{\beta }}_2={(-3.5,0.3,0.3)}^T$. Same as before, our target estimand $\tau$ is the ATE on the additive scale, namely the marginal risk difference between the treatment and control arms. Under the current parameter setup, we have $\tau =0$. The simulations for binary outcomes follow the same 2×2×3 factorial design as for the continous outcomes.

Non-collapsibility of odds ratio is a well-known issue for binary outcomes (Greenland et al., 1999). If one applies ANCOVA-type logistic regression model to estimate the marginal risk difference, one needs to first estimate the group means by standardization using the coefficients obtained from the fitted model. To avoid such complication, we conduct ANCOVA-type covariate adjustment through the standard linear regression, which allows us to directly interpret the coefficient of the treatment indicator as the marginal risk difference. In contrast, implementation of OW is more straightforward and the ATE estimator takes the form in eq (4) for both continuous and binary outcomes. Similar to Section 4.1.1, we compare the same types of imputation methods (i.e., mean imputation, correctly specified, and incorrectly specified model-based imputation), two types of complete data analysis (i.e., complete-unit and complete-covariate), and the unattainable ideal case (i.e., application of ANCOVA/OW to the full data with no missingness). We again consider both bias and the relative efficiency for each estimator to the unadjusted estimator.

The results for binary outcomes are presented in Table 3 for $N=100$ and in Table 4 for $n=500$. In general, results from binary outcomes are similar to those from the continuous outcome settings. The ANCOVA and OW estimators perform similarly for all settings with negligible differences. The complete-unit analyses can be biased when covariates are MAR or MNAR. The complete-covariate analyses are seen with reduced efficiency in these settings due to the omission of the partially-observed covariate ${\mathit{X}}_1$ that is predictive of the outcome. Incorporating ${\mathit{X}}_1$ in estimating the ATE in general improves efficiency, even when the imputation model is misspecified. When covariates are MNAR, including missingness indicator in the adjustment model can lead to further efficiency improvement.

Table 3:

Simulation results for binary outcomes with sample size 100. RE stands for relative efficiency compared to the unadjusted estimator; MSI stands for missing indicators. The results are based on 5000 iterations.

	Covariates Missingness Mechanism
	MCAR				MAR				MNAR
	Bias		RE		Bias		RE		Bias		RE
Method	ANCOVA	ow	ANCOVA	OW	ANCOVA	ow	ANCOVA	OW	ANCOVA	ow	ANCOVA	OW
N = 100, 30% missing
Full data	.00	.00	1.53	1.52	.00	.00	1.53	1.52	.00	.00	1.53	1.52
Complete covariate	.00	.00	1.22	1.22	.00	.00	1.22	1.22	.00	.00	1.22	1.22
Complete unit	.00	.00	1.02	1.01	.02	.03	1.02	1.01	.06	.06	1.03	1.03
Mean imputation w/o MSI	.00	.00	1.40	1.39	.00	.00	1.41	1.40	.00	.00	1.38	1.37
Mean imputation w/ MSI	.00	.00	1.39	1.38	.00	.00	1.40	1.40	.00	.00	1.43	1.42
Correct model imputation w/o MSI	.00	.00	1.41	1.40	.00	.00	1.42	1.41	.00	.00	1.39	1.38
Correct model imputation w/ MSI	.00	.00	1.40	1.39	.00	.00	1.40	1.40	.00	.00	1.43	1.42
Wrong model imputation w/o MSI	.00	.00	1.39	1.38	.00	.00	1.41	1.40	.00	.00	1.38	1.37
Wrong model imputation w MSI	.00	.00	1.38	1.38	.00	.00	1.39	1.39	.00	.00	1.42	1.41
N = 100, 10% missing
Full data	.00	.00	1.53	1.52	.00	.00	1.53	1.52	.00	.00	1.53	1.52
Complete covariate	.00	.00	1.22	1.22	.00	.00	1.22	1.22	.00	.00	1.22	1.22
Complete unit	.00	.00	1.36	1.35	.01	.01	1.35	1.34	.02	.02	1.35	1.34
Mean imputation w/o MSI	.00	.00	1.49	1.48	.00	.00	1.49	1.48	.00	.00	1.46	1.45
Mean imputation w/ MSI	.00	.00	1.47	1.47	.00	.00	1.48	1.48	.00	.00	1.48	1.48
Correct model imputation w/o MSI	.00	.00	1.49	1.48	.00	.00	1.50	1.48	.00	.00	1.47	1.46
Correct model imputation w/ MSI	.00	.00	1.47	1.47	.00	.00	1.48	1.48	.00	.00	1.48	1.48
Wrong model imputation w/o MSI	.00	.00	1.49	1.47	.00	.00	1.49	1.48	.00	.00	1.46	1.45
Wrong model imputation w/ MSI	.00	.00	1.46	1.46	.00	.00	1.47	1.47	.00	.00	1.48	1.48

Table 4:

Simulation results for binary outcomes with sample size 500. RE stands for relative efficiency compared to the unadjusted estimator; MSI stands for missing indicators. The results are based on 5000 iterations.

	MCAR				MAR				MNAR
	Bias		RE		Bias		RE		Bias		RE
Method	ANCOVA	OW	ANCOVA	OW	ANCOVA	OW	ANCOVA	OW	ANCOVA	OW	ANCOVA	OW
N = 500, 30% missing
Full data	.00	.00	1.51	1.51	.00	.00	1.51	1.51	.00	.00	1.51	1.51
Complete covariate	.00	.00	1.23	1.23	.00	.00	1.23	1.23	.00	.00	1.23	1.23
Complete unit	.00	.00	1.06	1.05	.02	.02	1.04	1.04	.06	.06	1.05	1.05
Mean imputation w/o MSI	.00	.00	1.42	1.42	.00	.00	1.42	1.42	.00	.00	1.41	1.41
Mean imputation w/ MSI	.00	.00	1.42	1.42	.00	.00	1.42	1.42	.00	.00	1.45	1.45
Correct model imputation w/o MSI	.00	.00	1.42	1.42	.00	.00	1.42	1.42	.00	.00	1.41	1.41
Correct model imputation w/ MSI	.00	.00	1.42	1.42	.00	.00	1.42	1.42	.00	.00	1.45	1.45
Wrong model imputation w/o MSI	.00	.00	1.42	1.42	.00	.00	1.42	1.42	.00	.00	1.41	1.41
Wrong model imputation w/MSI	.00	.00	1.42	1.42	.00	.00	1.42	1.42	.00	.00	1.45	1.45
N] = 500, 10% missing
Full data	.00	.00	1.51	1.51	.00	.00	1.51	1.51	.00	.00	1.51	1.51
Complete covariate	.00	.00	1.23	1.23	.00	.00	1.23	1.23	.00	.00	1.23	1.23
Complete unit	.00	.00	1.36	1.36	.01	.01	1.36	1.35	.02	.02	1.36	1.35
Mean imputation w/o MSI	.00	.00	1.47	1.47	.00	.00	1.49	1.48	.00	.00	1.48	1.48
Mean imputation w/ MSI	.00	.00	1.47	1.47	.00	.00	1.49	1.48	.00	.00	1.50	1.50
Correct model imputation w/o MSI	.00	.00	1.47	1.47	.00	.00	1.48	1.48	.00	.00	1.49	1.49
Correct model imputation w/ MSI	.00	.00	1.47	1.47	.00	.00	1.48	1.48	.00	.00	1.50	1.50
Wrong model imputation w/o MSI	.00	.00	1.47	1.47	.00	.00	1.49	1.48	.00	.00	1.48	1.48
Wrong model imputation w/ MSI	.00	.00	1.47	1.47	.00	.00	1.49	1.48	.00	.00	1.50	1.50

4.2. Simulations with missing covariates and missing outcomes

In this section, we expand on the scenarios presented in 4.1.1 by introducing missingness in outcomes. We consider the same balanced randomized design ($r=0.5$), the same covariate and continuous outcome generating processes, and the same covariate missingness processes (i.e., MCAR, MAR, and MNAR) as before. For the outcome missingness process, we consider the following logistic model:

$$logit\left(\mathrm{P}\mathrm{r}\left(R_i^y=1\mid R_i^{x_1},Z_i,X_{i2},X_{i3}\right)\right)={\theta }_0+{\theta }_Z{Z}_i+{\theta }_{Z{R}^{x_1^{\mathrm{*}}}}{Z}_i{R}_i^{x_1^{\mathrm{*}}}+\sum _{j=2}^3{\theta }_{Z{X}_j}{Z}_i{X}_{ij},$$ (14)

where $R^{x_1^{\mathrm{*}}}=R^{x_1}-\overline{R^{x_1}}$ is the centered covariate missingness indicator. We fixed $\mathit{\theta }=\left({\theta }_0,{\theta }_Z,{\theta }_{Z{R}^{x_1^{\mathrm{*}}}},{\theta }_{Z{X}_2},{\theta }_{Z{X}_3}\right)=(0.8,1,1,1,1$), yielding around 25% of missing outcomes. To demonstrate the importance of correcting potential bias due to informative missing outcomes, we first ignore covariate adjustment and compare the complete-outcome analysis (i.e., difference-in-mean estimator based on units with complete outcomes) with the standard IPW method using the correctly specified propensity score model for outcome missingness. We further investigate different strategies in dealing with missing covariates based on our proposed weighting estimator (11). In particular, we correctly specify the propensity score model for outcome missingness and consider treatment propensity score models under the complete-covariate analysis (i.e., ignoring $X_1$), the unattainable ideal case (i.e., using full covariates data), and the same covariates imputation methods from Section 4.1. The performance of each estimator is then evaluated by its empirical bias and efficiency compared to the IPW estimator without covariate adjustment.

Table 5 shows the simulation results for $N=100$. As anticipated, the complete-outcome analysis without the IPW adjustment ignores informative missing outcomes, which results in bias across all settings. However, the bias disappears with the IPW adjustment, irrespective of the methods used to adjust for covariates. When the treatment propensity score model fails to account for $X_1$, the efficiency gain is reduced compared to estimators that include imputed $X_1$ in the model, even when $X_1$ is imputed under the misspecified model. When covariates are MNAR, including covariate missingness indicators in the adjustment model always leads to further efficiency gain, consistent with the simulation results in Section 4.1.1 and 4.1.2. Similar trends are observed for sample size $N=500$, and the results are presented in Table 6.

Table 5:

Simulation results with missing covariates and missing continuous outcomes (N=100, 5000 iterations). RE stands for relative efficiency compared to the IPW estimator without covariate adjustment; MSI stands for missing indicators.

	Covariates Missingness Mechanism
	MCAR		MAR		MNAR
	Bias	RE	Bias	RE	Bias	RE
N = 100, 30% missing X
Complete outcome w/o cov. adj.	.34	1.08	.29	1.07	.23	1.06
IPW w/o cov. adj.	.03	1.00	.02	1.00	.01	1.00
IPW & Full X	.01	2.62	.01	2.81	.00	2.74
IPW & Complete covariate	.02	1.10	.02	1.11	.01	1.12
IPW & Mean imputation w/o MSI	.02	1.77	.02	1.91	.01	1.68
IPW & Mean imputation w/ MSI	.03	1.77	.02	1.91	.00	1.92
IPW & Correct model imputation w/o MSI	.02	1.81	.02	1.93	.01	1.72
IPW & Correct model imputation w/ MSI	.02	1.81	.02	1.93	.00	1.94
IPW & Wrong model imputation w/o MSI	.02	1.77	.02	1.86	.00	1.67
IPW & Wrong model imputation w/ MSI	.03	1.77	.02	1.87	.00	1.91
N = 100, 10% missing X
Complete outcome w/o cov. adj.	.34	1.10	.33	1.07	.30	1.07
IPW w/o cov. adj.	.03	1.00	.03	1.00	0.02	1.00
IPW & Full X	.02	2.69	.01	2.76	.00	2.68
IPW & Complete covariate	.03	1.11	.03	1.12	.02	1.12
IPW & Mean imputation w/o MSI	.02	2.33	.01	2.35	.01	2.16
IPW & Mean imputation w/ MSI	.02	2.36	.01	2.37	.00	2.41
IPW & Correct model imputation w/o MSI	.02	2.36	.01	2.39	.01	2.20
IPW & Correct model imputation w/ MSI	.02	2.39	.01	2.39	.00	2.42
IPW & Wrong model imputation w/o MSI	.02	2.32	.01	2.34	.01	2.15
IPW & Wrong model imputation w/ MSI	.02	2.35	.01	2.36	.00	2.40

Table 6:

Simulation results with missing covariates and missing continuous outcomes (N=500, 5000 iterations). RE stands for relative efficiency compared to the IPW estimator without covariate adjustment; MSI stands for missing indicators.

	Covariates Missingness Mechanism
	MCAR		MAR		MNAR
	Bias	RE	Bias	RE	Bias	RE
N = 500, 30% missing X
Complete outcome w/o cov. adj.	.35	1.08	.29	1.03	.23	1.03
IPW w/o cov. adj.	.00	1.00	.00	1.00	.00	1.00
IPW & Full X	.00	2.84	.00	3.20	.00	3.10
IPW & Complete covariate	.00	1.13	.00	1.15	.00	1.14
IPW & Mean imputation w/o MSI	.00	1.95	.00	2.01	.00	1.82
IPW & Mean imputation w/ MSI	.00	1.96	.00	2.02	.00	2.14
IPW & Correct model imputation w/o MSI	.00	2.00	.00	2.06	.00	1.87
IPW & Correct model imputation w/ MSI	.00	2.00	.00	2.06	.00	2.17
IPW & Wrong model imputation w/o MSI	.00	1.95	.00	1.97	.00	1.82
IPW & Wrong model imputation w/ MSI	.00	1.95	.00	1.99	.00	2.13
N = 500, 10% missing X
Complete outcome w/o cov. adj.	.34	1.06	.32	1.04	.29	1.04
IPW w/o cov. adj.	.01	1.00	0.00	1.00	.00	1.00
IPW & Full X	.01	3.01	.00	3.16	.00	3.09
IPW & Complete covariate	.01	1.14	.00	1.14	.00	1.14
IPW & Mean imputation w/o MSI	.00	2.54	.00	2.64	.00	2.32
IPW & Mean imputation w/ MSI	.00	2.55	.00	2.67	.00	2.63
IPW & Correct model imputation w/o MSI	.00	2.57	.00	2.70	.00	2.39
IPW & Correct model imputation w/ MSI	.00	2.58	.00	2.70	.00	2.66
IPW & Wrong model imputation w/o MSI	.00	2.54	.00	2.62	.00	2.32
IPW & Wrong model imputation w/ MSI	.00	2.54	.00	2.65	.00	2.62

5. Application

The Childhood Adenotonsillectomy Trial (CHAT) (Marcus et al., 2013; Zhang et al., 2018) is a multi-center, single-blind, randomized, controlled trial designed to test whether children, ages 5 to 9.9 years, with mild to moderate obstructive sleep apnea randomized to early adenotonsillectomy (eAT, treatment) will show better neurocognitive functioning than children randomized to watchful waiting with supportive care (WWSC, control). Children were randomized with a 1:1 ratio to both arms. Evaluations of participants were conducted at baseline as well as at 7 month post-baseline. This paper focuses on the change of the score from the Behavior Rating Inventory of Executive Function (BRIEF). Lower BRIEF scores indicate better executive functioning. An unadjusted analysis based on data from 392 children who had valid BRIEF measurements (195 and 197 in the eAT and WWSC arm, respectively) reveals that the average decrease in BRIEF score of the eAT arm exceeds that of the WWSC arm by 3.71 points and the difference is statistically significant (p < .001). In this section, we first investigate whether adjusting for baseline covariates can improve the efficiency of the treatment effect estimation based on patients with a complete BRIEF score. We further apply our proposed weighting estimator (11) to not only adjust for baseline covariates but also account for 61 patients without a valid BRIEF score.

Table 7 displays a summary of the baseline demographic and clinical characteristics based on participants with a valid BRIEF score (392 patients) and all participants (453 patients). For each variable, we present the unadjusted absolute standardized difference (ASD) as a measure of baseline imbalance:

$$\mathrm{A}\mathrm{S}\mathrm{D}=\left|{\sum }_{i=1}^N{X}_{ij}{Z}_i/{\sum }_{i=1}^N{Z}_i-{\sum }_{i=1}^N{X}_{ij}\left(1-Z_i\right)/{\sum }_{i=1}^N\left(1-Z_i\right)\right|/\sqrt{[\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}\left(X_{ij}\mid Z_i=1\right)+\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}\left(X_{ij}\mid Z_i=0\right)]/2},$$

where $\widehat{\mathrm{V}\mathrm{a}\mathrm{r}}(\cdot )$ denote the sample variance $s^2$ for a continuous variable, and $\hat{p}(1-\hat{p})$ for a binary variable with empirical prevalence $\hat{p}$. Two binary covariates, gender and tonsil size, have ASD larger than 10%, which has been regarded as a common threshold for imbalance (Austin and Stuart, 2015). In addition, univariate analyses reveal that the baseline BMI z-score, total score from the Pediatric Sleep Questionnaire (PSQ score) and total score from the Obstructive Sleep Apnea Quality of Life Survey (OSA score) are significantly correlated with the outcome (p < 0.05). The two covariates with baseline imbalance (gender and tonsil size), together with the three that are significantly correlated with the outcome (BMI z-score, PSA score and OSA score), as well as baseline BRIEF score, are subsequently included in the covariate adjustment analyses. These covariates exhibit mild missingness ranging from 0.4% to 3.3%, except for gender, which is observed for all participants.

Table 7:

Baseline characteristics of the CHAT study by treatment groups. The unadjusted ASD and correlation with outcome are computed ignoring missing values.

Participants with valid BRIEF score	Overall N = 392	eAT group N1 = 195	WWSC group N0 = 197	Unadjusted ASD	Missing (%)	Corr. with outcome (p value)
Baseline categorical covariates (count)
Gender (male)	193	88	105	.16	0	−.08 (.11)
Race (black)	207	102	105	.02	0	.05 (.33)
Tonsil size (0–50%)	98	44	54	.11	.8	.05 (.33)
Family income (≤ 40000)	154	73	81	.04	14	−.03 (.61)
Second-hand smoking (yes)	123	61	62	<.01	0	.06 (.22)
History of asthma (yes)	126	61	65	.03	2.3	−.06 (.26)
History of allergy (yes)	165	81	84	.03	.3	.01 (.89)
History of eczema (yes)	105	53	52	.02	.8	−.02 (.70)
History of prematurity (yes)	60	27	33	.09	1.3	.05 (.29)
Baseline continuous covariates (mean (sd))
Age (years)	6.6 (1.4)	6.6 (1.4)	6.6 (1.4)	<.01	0	−.10 (.06)
BMI z-score	.9 (1.3)	.9(1.4)	.9 (1.3)	.03	3.3	−.13 (.01)
Birth weight (oz)	109 (24.5)	110 (23.5)	108 (25.4)	.10	6.9	−.03 (.55)
PSQ score	.5 (.2)	.5 (.2)	0.5 (.2)	.02	.8	−.19 (<.01)
OSA score	53.7 (18.6)	53.4 (17.9)	54.0 (19.4)	.03	.5	−.23 (<.01)
BRIEF Score
Baseline	50.1 (11.4)	50.1 (11.2)	50.1 (11.5)
Endline	48.7 (11.9	46.8 (11.6)	50.5 (11.9)
Change	−1.46 (8.8)	−3.33 (8.5)	0.39 (8.8)
All Participants	Overall N = 453	eAT group N1 = 227	WWSC group N0 = 226	Unadjusted ASD	Missing (%)	Corr. with outcome (p value)
Baseline categorical covariates (count)
Gender (male)	219	101	118	.15	0	−.08 (.11)
Race (black)	249	126	123	.03	0	.05 (.33)
Tonsil size (0–50%)	113	50	63	.13	.9	.05 (.33)
Family income (≤ 40000)	182	90	92	.02	14.1	−.02 (.72)
Second-hand smoking (yes)	142	72	70	.02	0	.06 (.22)
History of asthma (yes)	143	69	74	.05	2.2	−.06 (.26)
History of allergy (yes)	186	90	96	.05	.2	.00 (.89)
History of eczema (yes)	123	62	61	.02	1.1	−.02 (.08)
History of prematurity (yes)	68	32	36	.05	1.1	.05 (.29)
Baseline continuous covariates (mean (sd))
Age (years)	6.6 (1.4)	6.6 (1.4)	6.5 (1.4)	.05	0	−.10 (.06)
BMI z-score	.9 (1.3)	.9(1.4)	.9 (1.2)	.05	2.9	−.13 (.01)
Birth weight (oz)	108 (24)	109 (22.9)	107 (25.1)	.10	6.6	−.03 (.05)
PSQ score	.5 (.2)	.5 (.2)	.5 (.2)	.06	1.3	−.19 (<.01)
OSA score	53.6 (18.6)	53.1 (18.3)	54.1 (18.8)	.05	.4	−.22 (<.01)
BRIEF Score
Baseline	49.9 (11.3)	49.9 (11.1)	50 (11.5)		.7
Endline	48.7 (11.9)	46.8 (11.6)	50.6 (11.8)		12.8
Change	−1.46 (8.8)	−3.33 (8.5)	0.39 (8.8)		13.5

We perform both mean imputation and model-based imputation for the missing covariates. For model-based imputation, we fit linear regression models with all the fully-observed covariates as predictors. We then conduct covariate adjustment using ANCOVA and OW with the 6 selected covariates based on participants with valid BRIEF measurements. The ANCOVA models include main effects of treatment, centered covariates, and centered missing indicators of those covariates with missingness, together with treatment-covariate interactions and treatment-missing-indicator interactions. Correspondingly, the treatment propensity score models contain the covariates and their missing indicators. To account for potential informative missing outcomes, we apply our proposed weighting estimator (11) using the same treatment propensity score model. We use LASSO for covariate selection of the propensity score model for outcome missingness, where race, treatment-race interaction, and missing indicators for history of eczema, BMI z-score, PSQ score, and baseline BRIEF scores, are selected for the final model.

We contrast the bootstrap variance estimates resulted from the covariate adjustment analyses to that from the unadjusted analysis. The results are presented in Table 8. The relative efficiency is computed as the ratio of the bootstrap variance of the unadjusted estimate to those of the adjusted ones. Overall, different combinations of imputation methods and adjustment methods produce similar treatment effect estimates, and on average adjusting for these three covariates leads to a 6%–7% of efficiency improvement over the unadjusted analysis. Inclusion of missing indicators in the treatment propensity score models does not appear to impact efficiency gain, implying that the covariate missingness is likely to fall under the MCAR or MAR setting. Table 7 demonstrates that the covariate distributions for participants with a valid BRIEF score and all participants are very similar. This might explain that outcome missingness is not informative, yielding similar treatment effect estimates regardless of adjustment for the missing outcomes.

Table 8:

Modeling results

Results based on patients with complete outcomes
Covariates	Method		$\widehat{\tau }$	${\widehat{V}}_{boot}$	Rel.Eff.boot
None	Unadjusted		−3.71	0.86	1.00
Gender, Tonsil size, baseline BRIEF, BMI z-score, PSQ score, OSA score w/o missing indicators	Mean imp.	ANCOVA	−3.69	0.82	1.06
		OW	−3.69	0.81	1.07
	Model-based imp.	ANCOVA	−3.69	0.82	1.06
		OW	−3.69	0.81	1.07
Gender, Tonsil size, baseline BRIEF, BMI z-score, PSQ score, OSA score w/ missing indicators	Mean imp.	ANCOVA	−3.70	0.82	1.06
		OW	−3.71	0.81	1.06
	Model-based imp.	ANCOVA	−3.70	0.82	1.06
		OW	−3.71	0.81	1.07
Results based on all patients
Covariates	Method		$\widehat{\tau }$	${\widehat{V}}_{boot}$	Rel.Eff.boot
None	IPW w/o cov. adj.		−3.64	0.88	1.00
Gender, Tonsil size, baseline BRIEF, BMI z-score, PSQ score, OSA score, w/o missing indicators	Mean imp.	IPW + OW	−3.67	0.82	1.07
	Model-based imp.		−3.67	0.82	1.07
Gender, Tonsil size, baseline BRIEF, BMI z-score, PSQ score, OSA score, w/ missing indicators	Mean imp.	IPW + OW	−3.70	0.82	1.06
	Model-based imp.		−3.70	0.82	1.06

Through the use of a pseudo variable, we further investigate how a covariate’s correlation with the outcome and its level of missingness relate to the amount of possible efficiency gain from adjusting for that covariate. Based on participants with a valid BRIEF score, we generate $X_i=\rho {\hat{s}}_i+\sqrt{\left(1-{\rho }^2\right){\hat{v}}_s}{\epsilon }_i$, where ${\epsilon }_i\stackrel{iid}{\sim }N(0,1)$, ${\hat{s}}_i$’s are the residuals from fitting the unadjusted model $Y\sim Z$ and ${\hat{v}}_s$ is the sample variance of ${\hat{s}}_i$. The correlation between the $X_i$’s generated in this way and $Y_i$ is approximately $\rho$. By removing the effect of $Z_i,X_i$ is uncorrelated with the randomization indicator $Z_i$. We then consider missingness of the variable $X$ following either a MCAR mechanism or a MNAR one where $\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\left\{Pr\left(R_i^x=1\right)\right\}={\delta }_0+X_i$. The value of ${\delta }_0$ is varied to achieve specific levels of missingness. The empirical efficiency of the treatment effect estimates resulting from ANCOVA with missing indicators after mean imputation of missing values in $X$ over repeated experiments is then compared to that of the unadjusted estimators. Fig. 1 shows the contour plot of relative efficiency for varying levels of $\rho$ and proportions of missingness under MCAR and MNAR. The relative efficiencies are computed as the ratios of the average bootstrap variances of the estimators from covariate adjustment analyses to that from the unadjusted analysis. From the plot for MCAR, we observe that in order to gain a 30% increase in efficiency, the covariate needs to have a correlation of approximately 0.5 or higher with the outcome. The efficiency gain diminishes with increased percentages of missingness, which is expected. The plot for the MNAR case is largely similar.

Figure 1:

Relative efficiency of ANCOVA with missing indicators after mean imputation to the unadjusted analyses under MCAR and MNAR mechanisms with varying values of $\rho$ and missing proportions

6. Discussion

We investigate covariate adjustment in randomized controlled trials with missing data in both baseline covariates and outcomes. When there are missing data in covariates only, performing covariate adjustment on the imputed data generally reduces the variance of treatment effect estimate. To gain additional efficiency, it is important to add the missingness indicator in the covariate adjustment model, but how the missing data is imputed is not important. When both covariates and outcomes are partially observed, we specify MAR assumptions for the missing outcomes, where the outcome missingness can depend on the fully-observed pre-treatment covariates and treatment indicators, the missingness indicators of the partially observed covariates, and the values of the partially observed covariates when observed. Under these assumptions, as long as the propensity model for the probability of $Y$ being observed includes the missingness indicator of $X$ and its interaction with $X$ (equivalent to imputing the missing values by zero) as the predictors, it does not matter how the missing covariate values are imputed. Based on this result, we propose two methods, one via outcome regression and one via weighting, with specific models for covariate adjustment.

Our simulations confirm the analytical results. Regardless of the missingness mechanism for the partially observed covariate, we observe improvement in the precision of the treatment effect estimate when performing covariate adjustment based on the imputed data. When covariates are missing not at random, including the covariate missingness indicators in the adjustment model could lead to further efficiency gain. The magnitude of improvement depends on the proportion of missingness and the strength of the association between the adjusted covariate and the outcome. Simulation results also suggest that the outcome regression and overlap weighting method lead to similar results in many settings. Therefore, we conjecture that the asymptotic equivalence between the outcome regression and propensity score weighting adjustment methods with complete covariates extends to the case of incomplete data. Nonetheless, the overlap weighting method is arguably more stable and easier to implement than outcome regression for non-continuous outcomes. As a general recommendation, in the presence of incomplete data in randomized trials, we recommend analysts to impute all the missing covariate values with mean (or median for categorical variables), include the covariate missingness indicators and their interaction with $X$ in either the treatment propensity score or outcome models, and perform a specific covariate adjustment method (either outcome regression or propensity score weighting) on the imputed data.

The use of missing indicator method in the context of observational studies has been investigated extensively (Jones, 1996; Greenland and Finkle, 1995; Song et al., 2021). However, randomized experiments are distinct from observational studies. Heuristically, randomization balances all measured and unmeasured covariates, including the missingness patterns in the baseline covariates, between treatment and control units in large samples. Thus adding the missingness indicators in the outcome regression leads to consistent estimator of the ATE and could improve efficiency when the missingness patterns are predictive of the outcome, regardless of the specific imputed values. This no longer stands in observational studies and we expect that the imputation data quality affects final estimates of the treatment effects.

In randomized clinical trials, stratified randomization is commonly used to avoid potential imbalance on important baseline covariates during the randomization stage. When stratified randomization is used, because the randomization is based on study participants’ stratification factors, these variables are typically fully observed. After implementing stratified randomization, the recommended analysis approach is to adjust the baseline covariates used in the stratified randomization process (Kahan and Morris, 2012; Wang et al., 2021). To further improve precision of treatment effect estimation, one can additionally adjust for baseline prognostic variables not used for stratification (Zhang and Zhang, 2021). The analysis methods for covariate adjustment in the presence of missing data we investigate in this paper can be applied to studies that utilize simple randomization or stratified randomization; in the latter case, the stratification factors and additional prognostic variables that are fully observed can be included in the set of covariates $V$. The partially-observed prognostic baseline variables can be included in the set of covariates $X$.

We compare covariate adjustment methods for partially-observed covariates ($X$) based on outcome regression and propensity score weighing in terms of efficiency gain in randomized studies. Various doubly robust (DR) methods that combine a propensity score model and an outcome model have been proposed (Robins et al., 1994, 1995; Lipsitz et al., 1999; Lunceford and Davidian, 2004; Neugebauer and van der Laan, 2005; Bang and Robins, 2005). In causal inference with observational data, a DR estimator can be constructed by combining a propensity score model for the treatment assignment mechanism and a model for the distribution of the counterfactual data. When the outcome $Y$ is missing at random, a DR estimator can be constructed combining a propensity score model for the missingness mechanism and a model for the distribution of the complete data. In both cases, the resulting doubly robust estimators remain consistent when either model is correctly specified. Augmented inverse probability weighted estimators that account for both confounding and missing data have also been investigated (Williamson et al., 2012). In randomized studies with fully observed outcome data, because the treatment propensity score model is always correct (the true propensity is a constant for all units), the additional outcome model becomes unnecessary (Zeng et al., 2021) When the outcome $Y$ is missing at random, it would be useful to investigate how to construct DR estimators in the presence of missing covariates in future research.

In this paper, we focus on continuous and binary outcomes, but time-to-event outcomes with right censoring is common in randomized trials. For the setting of missing covariate values in the Cox models for a survival outcome, White and Royston compared the use of several imputation models and recommended including the event indicator and the Nelson-Aalen estimator of the cumulative baseline hazard in the imputation model (White and Royston, 2009). Yi et al. considered Cox regression with survival-time-dependent missing covariate values, proposed an inverse propensity weighting method with the propensity estimated by nonparametric kernel regression, and pointed out the need for further investigation of survival-time-dependent time-varying missingness propensity estimation (Yi et al., 2020). However, there lacks investigation on covariate adjustment with missing data both in covariates and survival outcomes, an important direction for future research. Another direction is to investigate the implication of missing data when performing covariate adjustment in cluster randomized trials, where the unit of randomization is a cluster (such as hospitals) and outcomes and covariates are measured at individual level (Turner et al., 2017). Systematic reviews suggested that the median number of clusters in a CRT is around 20 (Murray et al., 2008). Such small number of clusters renders covariate imbalance prevalent in CRT. When data are fully observed, an augmented generalized estimating equations approach has been proposed to improve efficiency and validity of estimation in cluster randomized trials leveraging cluster-level and individual-level covariates (Stephens et al., 2012). Doubly robust estimators for the marginal treatment effect have also been proposed in the presence of informative missing outcome data (Prague et al., 2016). It is critical to differentiate imbalance caused by random chance and by post-randomization selection: the former can be addressed by covariate adjustment but the latter can not and would require additional data and assumptions (Li et al., 2022).

Data Availability Statement

The Childhood Adenotonsillectomy Trial data are available upon reasonable request at https://sleepdata.org.