Robust Estimation of Area Under ROC Curve Using Auxiliary Variables In the Presence of Missing Biomarker Values

Summary:

In medical research, the receiver operating characteristic (ROC) curves can be used to evaluate the performance of biomarkers for diagnosing diseases or predicting the risk of developing a disease in the future. The area under the ROC curve (AUC), as a summary measure of ROC curves, is widely utilized, especially when comparing multiple ROC curves. In observational studies, the estimation of the AUC is often complicated by the presence of missing biomarker values, which means that the existing estimators of the AUC are potentially biased. In this article, we develop robust statistical methods for estimating the ROC AUC and the proposed methods use information from auxiliary variables that are potentially predictive of the missingness of the biomarkers or the missing biomarker values. We are particularly interested in auxiliary variables that are predictive of the missing biomarker values. In the case of missing at random (MAR), i.e., missingness of biomarker values only depends on the observed data, our estimators have the attractive feature of being consistent if one correctly specifies, conditional on auxiliary variables and disease status, either the model for the probabilities of being missing or the model for the biomarker values. In the case of missing not at random (MNAR), i.e., missingness may depend on the unobserved biomarker values, we propose a sensitivity analysis to assess the impact of MNAR on the estimation of the ROC AUC. The asymptotic properties of the proposed estimators are studied and their finite sample behaviors are evaluated in simulation studies. The methods are further illustrated using data from a study of maternal depression during pregnancy.

1. Introduction

The receiver operating characteristic (ROC) curve plots the fraction of true positives (sensitivity) against the fraction of false positives (1–specificity) as the discrimination threshold (e.g., of a biomarker for a disease) is varied, and it is often used to evaluate the performance of biomarkers for diagnosing diseases or predicting the risk of developing diseases in the future. It was originally developed for the analysis of signal detection (Green and Swets, 1966) and was first used in medicine for the assessment of imaging devices (Zweig and Campbell, 1993). In medical studies, summary measures of ROC curves are often used and they are particularly powerful when comparing several ROC curves. The most widely used summary measure is the area under the ROC curve (ROC AUC) (Bamber, 1975). The ROC AUC is bounded between 0.5 and 1, and has the interpretation of the probability of a randomly selected observation from the diseased (non-diseased) population having a higher biomarker value than that from the non-diseased (diseased) population. Therefore, a large AUC value represents good separation in the biomarker values between the diseased and non-diseased populations. In particular, a perfect test would achieve an AUC of 1.0, whereas an uninformative test would have an AUC of 0.5. A wealth of literature has been developed for this type of research (Pepe (2003) and references therein).

In practice, the biomarker value may be missing for some subjects, especially in observational studies. Take for example a self-rated mental illness score collected from pregnant women in a psychiatric study, where the disease of interest is the presence (or absence) of a major depressive episode throughout pregnancy (see Section 4 for more details). Since the biomarker score is self-rated, it is possible that some subjects did not complete the self-evaluation and hence the score is missing. In such studies, additional variables including demographic and baseline variables are often available, which are referred to as auxiliary variables. While these variables are not of primary interest themselves, they are potentially predictive of the missingness of the biomarker value or the value itself, and can be incorporated in a data analysis to improve its robustness and/or efficiency. If an auxiliary variable is predictive of missingness but independent of the missing values, then using it in an analysis will not affect the results. Thus, we are interested in auxiliary variables that are predictive of the missing values, especially if they are also predictive of the missingness.

As with the general setting discussed in Little and Rubin (2002) and references therein, a naive analysis that only uses complete observations may lead to bias and loss of efficiency in the estimation of the ROC AUC. First, when the biomarker is missing completely at random (MCAR), i.e., the missingness does not depend on either observed or unobserved data, the naive analysis is valid but is not efficient. Second, when the biomarker is missing at random (MAR), i.e., the missingness is conditionally independent of the missing data given the observed data, the naive analysis is biased and other methods, e.g., inverse-weighted (IW) methods, can be extended for consistent estimation. IW methods weight each complete case by the inverse of the probability of observing the biomarker value. Despite its conceptual simplicity, IW methods have limitations. Most notably, IW methods are not efficient and are subject to bias if one misspecifies the model for the missingness. Alternatively, one can extend the methods that are doubly robust and more efficient (Robins et al., 1994; Scharfstein et al., 1999) for estimating the ROC AUC. In the case of missing not at random (MNAR), i.e., missingness depends on unobserved biomarker values even after conditioning on the observed data, it is common practice to conduct sensitivity analysis (Zhou, 1994; Rotnitzky and Robins, 1997; Scharfstein et al., 1999; Kosinski and Barnhart, 2003). In all cases, auxiliary variables can be used to potentially reduce bias and improve efficiency when associated with the probability of missing and the value of biomarkers, or simply improve efficiency when only associated with the value of biomarkers.

We confine the scope of this paper to the case where the disease status is always confirmed and a set of auxiliary variables are fully observed but the biomarker values are missing for some subjects, and we are interested in estimating the ROC AUC. Our setting is to be distinguished from the existing research on verification bias (Zhou, 1993, 1998; Rotnitzky et al., 2006; Fluss et al., 2009). In the presence of verification bias, the biomarker values are always observed whereas the true disease status is only verified for a non-random sample of the population of interest, e.g., the selection for testing may depend on the disease status or other variables. In particular, Rotnitzky et al. (2006) extended the doubly robust method developed in Rotnitzky and Robins (1997) to the estimation of the ROC AUC in the presence of verification biases. As a result of different problem setups (i.e. biomarker values missing vs. disease status unconfirmed for a subset of subjects), there are important differences between our work and theirs. In our setting, a working model on biomarker values, which can be continuous or categorical, is utilized, whereas a working model on the presence (or absence) of the disease, a binary variable, was utilized in Rotnitzky et al. (2006); consequently, our methods require modeling of the conditional distribution of biomarker values. Furthermore, we study and compare parametric and nonparametric approaches for estimating this conditional distribution and discuss two types of MAR assumptions, which have different implications on the estimation of AUC.

The remainder of the article is organized as follows. In Section 2, we describe the proposed estimators and their theoretical properties under MAR and propose a sensitivity analysis under MNAR. In Section 3, we evaluate the finite sample performance of the proposed estimators through simulations. In Section 4, we apply the proposed methods to a psychiatric study of maternal depression during pregnancy. We conclude with a discussion in Section 5.

2. Methodology

Suppose that a random sample of n subjects is selected from a population of interest to evaluate the performance of a diagnostic or predictive test using a biomarker. Each subject i, i = 1, … , n, is classified into one of two groups, diseased (D_i = 1) or non-diseased (D_i = 0), based on a gold standard. For each subject i, denote the biomarker value by X_i, which is used to diagnose or predict the disease status (D_i). X_i is not observed in a subset of the subjects, and let δ_i denote the missing indicator for X_i, i.e., δ_i = 1 when X_i is observed and δ_i = 0 if X_i is missing. In addition, p auxiliary variables that may be associated with the value of X_i and/or its missingness (δ_i) are also collected and denoted by $Z_i={\left(Z_i^{(1)},\dots ,Z_i^{(p)}\right)}^T$. Then for subject i, the complete data are (D_i, Z_i, δ_i, X_i). When δ_i = 1, the observed data are O_i = (D_i, Z_i, δ_i, X_i) and subject i is called a complete case; when δ_i = 0, the observed data are O_i = (D_i, Z_i, δ_i) and subject i is called an incomplete case. We denote by O the collection of observed data for all subjects. When δ_i is independent of X_i conditional on D_i and Z_i, it is a case of MAR; when δ_i is dependent on X_i conditional on D_i and Z_i, it is a case of MNAR.

We are interested in estimating the ROC AUC, which is equivalent to a U-statistic (Bamber, 1975), θ = Pr(X_i > X_j | D_i = 1, D_j = 0), assuming that the diseased tend to have higher biomarker values. When all data are completely observed, an unbiased estimator of θ is

$$\widehat{\theta }=\frac{1}{{\sum }_{i\ne j}{D}_i\left(1-D_j\right)}\sum _{i\ne j}{D}_i\left(1-D_j\right)I_{ij},$$

where I_ij = I(X_i > X_j) + 0.5I(X_i = X_j) with I(A) equals to 1 if A is true and 0 if A is false. When X is missing for some subjects, a naive extension of the above estimator using only the complete observations (i.e., δ_i = 1) is

$${\widehat{\theta }}_0=\frac{1}{{\sum }_{i\ne j}{D}_i\left(1-D_j\right){\delta }_i{\delta }_j}\sum _{i\ne j}{D}_i\left(1-D_j\right){\delta }_i{\delta }_j{I}_{ij}.$$ (1)

It is straightforward to verify the following proposition:

Proposition 1: (i) When δ is independent of X given D, ${\widehat{\theta }}_0$ is an unbiased estimator of θ; (ii) when δ is dependent on X given D, then ${\widehat{\theta }}_0$ is subject to potential bias.

We note that (i) includes the case of MCAR and a special case of MAR where δ may depend on D and Z and is independent of X given D and (ii) includes the case of MNAR and a special case of MAR where δ is dependent on X conditional on D but is independent of X conditional on D and Z. We refer to ${\widehat{\theta }}_0$ as the naive estimator throughout this article.

2.1. Inverse-Weighted Estimator

In the case of MAR, we first study an inverse-weighted estimator,

$${\widehat{\theta }}_{IW}=\frac{1}{{\sum }_{i\ne j}\frac{{\delta }_i{\delta }_j}{{\widehat{\pi }}_i{\widehat{\pi }}_j}{D}_i\left(1-D_j\right)}\sum _{i\ne j}\frac{{\delta }_i{\delta }_j}{{\widehat{\pi }}_i{\widehat{\pi }}_j}{D}_i\left(1-D_j\right)I_{ij},$$ (2)

where ${\widehat{\pi }}_i$ is an estimate of the probability of observing X_i, namely, π_i = Pr(δ_i = 1), conditional on Z_i and D_i under MAR. We denote by $\left(\mathcal{M}1\right)$ the working model for π_i given Z_i and D_i with a set of unknown parameters, α, and denote by $\mathcal{A}(\alpha ;O)={\sum }_i{\mathcal{A}}_i(\alpha ;O)$ the estimating equations for computing the estimate of α, namely, $\widehat{\alpha }$, based on the observed data. For instance, one can use a logistic regression model for $\left(\mathcal{M}1\right)$, i.e. logit(π_i) = W(Z_i, D_i; α) where W(Z_i, D_i; α) is a function of Z_i and D_i and is parameterized by α; $\mathcal{A}(\alpha ;O)$ can be taken as the likelihood equations for the logistic regression model. $\left(\mathcal{M}1\right)$ is also known as the propensity score model (Rosenbaum and Rubin, 1983). It can be readily shown that if the working model $\left(\mathcal{M}1\right)$ is correctly specified, ${\widehat{\theta }}_I$ is a consistent estimator of θ under MAR.

2.2. Doubly Robust Estimators

In the case of MAR, we propose a second estimator

$${\widehat{\theta }}_{DR}=\frac{1}{{\sum }_{i\ne j}\frac{{\delta }_i{\delta }_j}{{\widehat{\pi }}_i{\widehat{\pi }}_j}{D}_i\left(1-D_j\right)}\sum _{i\ne j}{D}_i\left(1-D_j\right)\left\{\frac{{\delta }_i{\delta }_j}{{\widehat{\pi }}_i{\widehat{\pi }}_j}{I}_{ij}-\frac{{\delta }_i{\delta }_j-{\widehat{\pi }}_i{\widehat{\pi }}_j}{{\widehat{\pi }}_i{\widehat{\pi }}_j}E\left(I_{ij}\mid Z_i,Z_j,D_i=1,D_j=0\right)\right\}$$ (3)

where ${\widehat{\pi }}_i$ is the same as previously defined and E(I_ij | Z_i, Z_j, D_i = 1, D_j = 0) can be estimated based on the joint conditional distribution of X_i and X_j given the observed data. Specifically, we denote by $\left(\mathcal{M}2\right)$ the working model for characterizing the conditional distribution of X given Z and D with a set of unknown parameters, β, and denote by $\mathcal{B}(\beta ;O)={\sum }_i{\mathcal{B}}_i(\beta ;O)$ the estimating equations for computing the estimate of β, namely, $\widehat{\beta }$, based on the observed data. We note that the conditional mean of X given Z and D is only part of $\left(\mathcal{M}2\right)$. It can be readily shown that if either $\left(\mathcal{M}1\right)$ or $\left(\mathcal{M}2\right)$ is correctly specified, ${\widehat{\theta }}_{DR}$ is a consistent estimator of θ under MAR.

We consider two options for the working model $\left(\mathcal{M}2\right)$. In the first option, X given Z and D is assumed to follow a known parametric distribution with unknown parameters β. One special case is the Gaussian distribution, i.e., $\left[X_i\mid Z_i,D_i\right]~N\left(V\left(Z_i,D_i;{\beta }_1\right),{\sigma }_1^2{D}_i+{\sigma }_0^2\left(1-D_i\right)\right)$, where V(Z_i, D_i; β₁) is a function of Z_i and D_i parameterized by β₁. Let $\beta ={\left({\beta }_1^T,{\sigma }_1^2,{\sigma }_0^2\right)}^T$ denote all parameters of interest, and it follows that

$$X_i-X_j\mid Z_i,Z_j,D_i=1,D_j=0~N\left(V\left(Z_i,D_i=1;{\beta }_1\right)-V\left(Z_j,D_j=0;{\beta }_1\right),{\sigma }_1^2+{\sigma }_0^2\right),$$

and hence

$$E\left\{I_{ij}\mid Z_i,Z_j,D_i=1,D_j=0\right\}=\Phi \left(\frac{V\left(Z_i,D_i=1;{\beta }_1\right)-V\left(Z_j,D_j=0;{\beta }_1\right)}{\sqrt{{\sigma }_1^2+{\sigma }_0^2}}\right),$$

where Φ(·) is the cumulative distribution function (c.d.f.) of a standard normal random variable. ${\widehat{\theta }}_{DR}$ can be rewritten as

$${\widehat{\theta }}_{DR}=\frac{1}{{\sum }_{i\ne j}\frac{{\delta }_i{\delta }_j}{{\widehat{\pi }}_i{\pi }_j}{D}_i\left(1-D_j\right)}\sum _{i\ne j}{D}_i\left(1-D_j\right)\left[\frac{{\delta }_i{\delta }_j}{{\widehat{\pi }}_i{\widehat{\pi }}_j}{I}_{ij}-\frac{{\delta }_i{\delta }_j-{\widehat{\pi }}_i{\widehat{\pi }}_j}{{\widehat{\pi }}_i{\widehat{\pi }}_j}\Phi \left\{b_{ij}(\widehat{\beta })\right\}\right],$$ (4)

where $b_{ij}(\widehat{\beta })=\frac{V\left(Z_i,D_i=1;{\widehat{\beta }}_1\right)-V\left(Z_j,D_j=0;{\widehat{\beta }}_1\right)}{\sqrt{{\widehat{\sigma }}_1^2+{\widehat{\sigma }}_0^2}}$ and $\widehat{\beta }$ can be obtained through, say, linear regression for $\left(\mathcal{M}2\right)$ using the observed data. From here on, let ${\widehat{\theta }}_{DR}$ denote the doubly robust estimator in Equation (4), which assumes that the conditional distribution of X is Gaussian.

In the second option, suppose X_i = V(Z_i, D_i; β₁)+ε_1iD_i+ε_0i(1−D_i), where {ε_1i, i = 1, …, n₁} and {ε_0i, i = 1, …, n₀} are independent and identically distributed (i.i.d.) random errors in the diseased and non-diseased, respectively, and their respective distributions are unknown. In this case, the conditional distribution of X_i given Z_i and D_i can be estimated nonparametrically. We denote the set of observed residuals by $\left\{{\widehat{\epsilon }}_{1k}=X_k-V\left(Z_k,D_k=1;{\widehat{\beta }}_1\right),k=1,\dots ,n_1^o\right\}$ and $\left\{{\widehat{\epsilon }}_{0l}=X_l-V\left(Z_l,D_l=0;{\widehat{\beta }}_1\right),l=1,\dots ,n_0^o\right\}$ for the diseased and non-diseased, respectively, where $n_1^o$ and $n_0^o$ are the number of subjects with observed X in the diseased and non-diseased, respectively. An empirical sample of the estimated conditional distribution of X_i given Z_i and D_i can be constructed as $\left\{{\tilde{X}}_{ik}^1=V\left(Z_i,D_i=1;{\widehat{\beta }}_1\right)+{\widehat{\epsilon }}_{1k},k=1,\dots ,n_1^o\right\}$ in the diseased and $\left\{{\tilde{X}}_{il}^0=V\left(Z_i,D_i=0;{\widehat{\beta }}_1\right)+{\widehat{\epsilon }}_{0l},l=1,\dots ,n_0^o\right\}$ in the non-diseased. E(I_ij | Z_i, Z_j, D_i = 1, D_j = 0) in Equation (3) can then be estimated using $\frac{1}{n_1^0{n}_0^{\sigma }}{\sum }_{k=1}^{n_1^0}{\sum }_{l=1}^{n_0^o}\left\{\text{I}\left({\tilde{X}}_{ik}^1>{\tilde{X}}_{jl}^0\right)+0.5\text{I}\left({\tilde{X}}_{ik}^1={\tilde{X}}_{jl}^0\right)\right\}$, where i and j go through all subjects including those with missing X, and we denote the resulting nonparametric estimator of θ by ${\widehat{\theta }}_{DR-N}$. When random errors are not i.i.d., e.g., the variance changes as the mean of X changes, the above procedure needs to be modified accordingly, e.g., performed within strata of the mean of X.

When computing ${\widehat{\theta }}_{IW}$, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$, the weights $\left(\frac{1}{{\widehat{\pi }}_i}\right)$ may be large and unstable, and lead to extra noise in estimation, in particular, when computing the bootstrap SE of ${\widehat{\theta }}_{DR-N}$. Thus, we consider a simple modification to stabilize the weights, namely, replacing $\frac{1}{{\widehat{\pi }}_i}$ with $\frac{1}{{\widehat{\pi }}_i}\frac{n}{{\sum }_i{\delta }_i/{\widehat{\pi }}_i}$. When $\left(\mathcal{M}1\right)$ is correctly specified, it can be readily shown that $\frac{1}{n}{\sum }_i{\delta }_i/{\widehat{\pi }}_i$ converges to 1 in probability, hence $\frac{1}{{\widehat{\pi }}_i}\frac{n}{{\sum }_i{\delta }_i/{\widehat{\pi }}_i}$ is equivalent to $\frac{1}{{\widehat{\pi }}_i}$ asymptotically.

2.3. Theoretical Properties

Following our previous notation, we further define the following,

$${\mathcal{U}}_{i,j}(\theta ,\alpha )\equiv \theta \frac{{\delta }_i{\delta }_j}{{\pi }_i{\pi }_j}{D}_i\left(1-D_j\right)-\frac{{\delta }_i{\delta }_j}{{\pi }_i{\pi }_j}{D}_i\left(1-D_j\right)I_{ij},$$

$${\mathcal{V}}_{i,j}(\theta ,\alpha ,\beta )\equiv \theta \frac{{\delta }_i{\delta }_j}{{\pi }_i{\pi }_j}{D}_i\left(1-D_j\right)-\frac{{\delta }_i{\delta }_j}{{\pi }_i{\pi }_i}{D}_i\left(1-D_j\right)I_{ij}+\frac{{\delta }_i{\delta }_j-{\pi }_i{\pi }_j}{{\pi }_i{\pi }_j}{D}_i\left(1-D_j\right)E\left(I_{ij}\mid Z_i,Z_j,D_i,D_j\right),$$

where π_i depends on α and E(I_ij | Z_i, Z_j, D_i, D_j) depends on β. It follows that $\mathcal{U}={\sum }_{i\ne j}{\mathcal{U}}_{i,j}(\theta ,\widehat{\alpha })$ and $\mathcal{V}={\sum }_{i\ne j}{\mathcal{V}}_{i,j}(\theta ,\widehat{\alpha },\widehat{\beta })$ are the set of estimating equations for ${\widehat{\theta }}_{IW}$ and ${\widehat{\theta }}_{DR}$, respectively. Let α₀ and β₀ be the probability limits of $\widehat{\alpha }$ and $\widehat{\beta }$, respectively, which usually exist.

Theorem 1: Under the regularity conditions (A1)–(A3) given in Web Appendix A, if either or both of $\left(\mathcal{M}1\right)$ and $\left(\mathcal{M}2\right)$ are correctly specified, then $\sqrt{n}\left({\widehat{\theta }}_{DR}-\theta \right)\to N(0,\Omega )$ in distribution, where $\Omega =\mathit{\text{Var}}\left[{\left\{E\frac{{\delta }_i{\delta }_j}{{\pi }_i{\pi }_j}{D}_i\left(1-D_j\right)\right\}}^{-1}{Q}_i\left(\theta ,{\alpha }_0,{\beta }_0\right)\right]$, and

$$Q_i=-E\left\{{\mathcal{V}}_{i,j}(\theta ,\alpha ,\beta )+{\mathcal{V}}_{j,i}(\theta ,\alpha ,\beta )\mid O_i\right\}+\left[\frac{\partial }{\partial \alpha }E\left\{{\mathcal{V}}_{i,j}(\theta ,\alpha ,\beta )\right\}\right]\times {\left[\frac{\partial }{\partial \alpha }E\left\{{\mathcal{A}}_i(\alpha )\right\}\right]}^{-1}{\mathcal{A}}_i(\alpha )+\left[\frac{\partial }{\partial \beta }E\left\{{\mathcal{V}}_{i,j}(\theta ,\alpha ,\beta )\right\}\right]\times {\left[\frac{\partial }{\partial \beta }E\left\{{\mathcal{B}}_i(\beta )\right\}\right]}^{-1}{\mathcal{B}}_i(\beta ).$$

Ω can be consistently estimated by $\widehat{\Omega }=\frac{1}{{\gamma }^{2}n}{\sum }_{i=1}^n{\widehat{Q}}_i^2$ with $\gamma =\frac{1}{n^2}{\sum }_{i,j}\frac{{\delta }_i{\delta }_j}{{\widehat{\pi }}_i{\widehat{\pi }}_j}{D}_i\left(1-D_j\right)$ and

$${\widehat{Q}}_i=-\frac{1}{n}\left[{\sum }_j\left\{{\mathcal{V}}_{i,j}\left({\widehat{\theta }}_{DR},\widehat{\alpha },\widehat{\beta }\right)+{\mathcal{V}}_{j,i}\left({\widehat{\theta }}_{DR},\widehat{\alpha },\widehat{\beta }\right)\right\}\right]+\frac{1}{n}\left[{{\sum }_{i\ne j}\frac{\partial {\mathcal{V}}_{i,j}\left({\widehat{\theta }}_{DR},\alpha ,\widehat{\beta }\right)}{\partial \alpha }|}_{\alpha =\widehat{\alpha }}\right]{\left[{{\sum }_i\frac{\partial {\mathcal{A}}_i(\alpha )}{\partial \alpha }|}_{\alpha =\widehat{\alpha }}\right]}^{-1}{\mathcal{A}}_i(\widehat{\alpha })+\frac{1}{n}\left[{{\sum }_{i\ne j}\frac{\partial {\mathcal{V}}_{i,j}\left({\widehat{\theta }}_{DR},\widehat{\alpha },\beta \right)}{\partial \beta }|}_{\beta =\widehat{\beta }}\right]{\left[{{\sum }_i\frac{\partial {\mathcal{B}}_i(\beta )}{\partial \beta }|}_{\beta =\widehat{\beta }}\right]}^{-1}{\mathcal{B}}_i(\widehat{\beta }).$$

Theorem 2: Under the regularity conditions similar to (A1)–(A3) given in Web Appendix A, if $\left(\mathcal{M}1\right)$ is correctly specified, then $\sqrt{n}\left({\widehat{\theta }}_{IW}-\theta \right)\to N(0,\Omega )$ in distribution, where $\Omega =\mathit{\text{Var}}\left[{\left\{E\frac{{\delta }_i{\delta }_j}{{\pi }_i{\pi }_j}{D}_i\left(1-D_j\right)\right\}}^{-1}{R}_i\left(\theta ,{\alpha }_0\right)\right]$, and

$$R_i=-E\left\{u_{i,j}(\theta ,\alpha )+u_{j,i}(\theta ,\alpha )\mid O_i\right\}+\left[\frac{\partial }{\partial \alpha }E\left\{{\mathcal{U}}_{i,j}(\theta ,\alpha )\right\}\right]\times {\left[\frac{\partial }{\partial \alpha }E\left\{{\mathcal{A}}_i(\alpha )\right\}\right]}^{-1}{\mathcal{A}}_i(\alpha ).$$

Ω can be consistently estimated by $\widehat{\Omega }=\frac{1}{{\gamma }^{2}n}{\sum }_{i=1}^n{\widehat{R}}_i^2$ with $\gamma =\frac{1}{n^2}{\sum }_{i,j}\frac{{\delta }_i{\delta }_j}{{\widehat{\pi }}_i{\widehat{\pi }}_j}{D}_i\left(1-D_j\right)$ and

$${\widehat{R}}_i=-\frac{1}{n}\left[{\sum }_j\left\{u_{i,j}\left({\widehat{\theta }}_{IW},\widehat{\alpha }\right)+u_{j,i}\left({\widehat{\theta }}_{IW},\widehat{\alpha }\right)\right\}\right]+\frac{1}{n}\left[{{\sum }_{i\ne j}\frac{\partial {\mathcal{U}}_{i,j}\left({\widehat{\theta }}_{IW},\alpha \right)}{\partial \alpha }|}_{\alpha =\widehat{\alpha }}\right]{\left[{{\sum }_i\frac{\partial {\mathcal{A}}_i(\alpha )}{\partial \alpha }|}_{\alpha =\widehat{\alpha }}\right]}^{-1}{\mathcal{A}}_i(\widehat{\alpha }).$$

A sketch of proof for Theorems 1 and 2 is provided in Web Appendix A, which is along the similar lines of Rotnitzky et al. (2006). The underlying idea is to derive the influence functions for ${\widehat{\theta }}_{IW}$ or ${\widehat{\theta }}_{DR}$ by plugging in the influence functions for $\widehat{\alpha }$ and $\widehat{\beta }$. The consistency of ${\widehat{\theta }}_{DR-N}$ is straightforward to show when either $\left(\mathcal{M}1\right)$ or $\left(\mathcal{M}2\right)$ holds and its SE can be computed using a bootstrap procedure, which resamples the data with replacement within disease strata.

A few remarks are in order. First, as stated in Proposition 1, ${\widehat{\theta }}_0$ is unbiased when δ is independent of X given D; but if δ and X are associated with Z, ${\widehat{\theta }}_{IW}$, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$ are potentially more efficient when the working models are correctly specified. Second, when δ is dependent on X given D but independent of X given D and Z, ${\widehat{\theta }}_{IW}$, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$ are still consistent under suitable conditions, while ${\widehat{\theta }}_0$ is subject to potential bias. Thirdly, ${\widehat{\theta }}_{DR}$ assumes that the residuals are Gaussian in $\left(\mathcal{M}2\right)$ and is subject to model misspecification even if the mean model is correctly specified; ${\widehat{\theta }}_{DR-N}$ does not impose this restriction.

2.4. MNAR: Sensitivity Analysis

We now consider a case of MNAR, where δ is dependent on X conditional on Z and D; thus, a working model $\left(\mathcal{M}1\right)$ that only includes Z and D is misspecified. We investigate a sensitivity analysis to assess the impact on ${\widehat{\theta }}_{IW}$, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$, as the effect of X on δ is varied. To fix the idea, suppose that logit(π_i) = S(Z_i, D_i; α_S) + U(X_i, α_X), where α_S and α_X are two sets of unknown parameters associated with known functions S and U, respectively. α_X represents the effect of biomarker values on the probability of being missing. Since α_S and α_X can not be jointly estimated using the observed data, we fix α_X at a set of pre-determined values and estimate α_S using the following set of estimating equations,

$$\sum _{i=1}^n\left(\frac{{\delta }_i}{{\pi }_i}-1\right)W\left(Z_i,D_i\right),$$ (5)

where W(Z_i, D_i) is an arbitrary known vector function with the same dimension as α_S. For instance, if S(Z_i, D_i; α_S) = α_SW(Z_i, D_i), then W(Z_i, D_i) is the covariate vector for i, which may include interaction terms. Compared to the likelihood equations for the logistic regression, one advantage of the estimation equations (5) is that π_i is not needed when X_i is missing. For every pre-determined value of α_X, we can compute ${\widehat{\alpha }}_S$ using (5) and ${\widehat{\pi }}_i$ for subjects with observed X_i; subsequently we can compute ${\widehat{\theta }}_{IW}$, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$, all of which do not need ${\widehat{\pi }}_i$ for subjects with missing X_i. This procedure is repeated for a grid of α_X values, and the resulting estimators are compared to assess the impact of α_X and hence the impact of MNAR. U(X_i, α_X) = 0 corresponds to the case of MAR, and U(X_i, α_X) ≠ 0 corresponds to the case of MNAR. In this sensitivity analysis, we do not assume that the estimation of the parameters of $\left(\mathcal{M}2\right)$ is not affected by MNAR. To simplify the sensitivity analysis and, in particular, avoid performing sensitivity analysis for two working models, we exploit the doubly robust property, i.e., if $\left(\mathcal{M}1\right)$ is correctly specified then the proposed estimators are consistent, and focus on $\left(\mathcal{M}1\right)$ only.

3. Simulation studies

We conducted simulations to evaluate the finite sample performance of the proposed estimators, first in the case of MAR where δ is independent of X given D and Z, then in the case of MNAR where δ is dependent on X given D and Z. In our simulations, ${\widehat{\theta }}_0$, ${\widehat{\theta }}_{IW}$, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$ were compared. In addition, we considered another estimator, namely, ${\widehat{\theta }}_{IMP}=\frac{1}{{\sum }_{i\ne j}{D}_i\left(1-D_j\right)}{\sum }_{i\ne j}{D}_i\left(1-D_j\right)\left[{\delta }_i{\delta }_j{I}_{ij}-\left({\delta }_i{\delta }_j-1\right)\Phi \left\{b_{ij}(\widehat{\beta })\right\}\right]$, which only relies on $\left(\mathcal{M}2\right)$ and is not doubly robust. While it is not of primary interest in this article, ${\widehat{\theta }}_{IMP}$ under the correctly specified $\left(\mathcal{M}2\right)$ can be used as an optimal benchmark for efficiency as suggested by a referee. To benchmark bias and loss of efficiency due to missing data, a so-called gold standard (GS) estimator was also computed, i.e., ${\widehat{\theta }}_{GS}=\frac{1}{{\sum }_{i\ne j}{D}_i\left(1-D_j\right)}{\sum }_{i=1}^n{\sum }_{j=1}^n{D}_i\left(1-D_j\right)I_{ij}$, which uses the underlying “true” biomarker values for all subjects and is not applicable in real data analysis. In Tables 1–3, modified weights as described in Section 2.2 were used for ${\widehat{\theta }}_{IW}$, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$; to compute the standard error for ${\widehat{\theta }}_{DR-N}$, we used 200 bootstrap datasets randomly sampled with replacement from the data while stratified on the disease status. In each simulation, we generated a random sample of n = 200 independent subjects with an equal number of diseased and non-diseased subjects. For each simulation setting, 500 Monte Carlo datasets were generated and the results were summarized using the following measures: the mean relative bias (RB), mean of the standard error estimates (SE), Monte Carlo standard deviation of parameter estimates (SD), square root of mean squared errors (SMSE) and coverage rate (CR) of 95% Wald’s confidence interval using a logistic transform of θ as suggested in Pepe (2003) (Ch. 5).

Table 1.

Results of simulation study under MAR: comparison of ${\widehat{\theta }}_0$, ${\widehat{\theta }}_{IW}$, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$ using modified weights, when Z₁ and Z₂ are identical. ε is Gaussian (i.e., ε ~ N(0, 1)) or non-Gaussian (i.e., ε = 20{η – E(η)} with η ~ Beta(5, 1)). True θ is 0.722 for Gaussian ε and 0.675 for non-Gaussian ε. The details of true models and misspecified working models are provided in Section 3.1.

	Gaussian ε					non-Gaussian ε
	RB (%)	SE	SD	SMSE	CR (%)	RB (%)	SE	SD	SMSE	CR (%)
${\widehat{\theta }}_{GS}$	−0.2	0.036	0.037	0.037	94.0	0.0	0.038	0.038	0.038	95.8
${\widehat{\theta }}_0$	11.6	0.050	0.054	0.099	70.0	10.8	0.057	0.056	0.092	80.4
	Both mean models correctly specified
${\widehat{\theta }}_{IMP}$	−0.1	0.040	0.042	0.042	95.0	−0.8	0.054	0.054	0.054	94.8
${\widehat{\theta }}_{IW}$	0.5	0.048	0.052	0.052	93.0	1.0	0.056	0.058	0.059	95.0
${\widehat{\theta }}_{DR}$	0.0	0.045	0.043	0.043	96.4	0.5	0.057	0.055	0.055	96.4
${\widehat{\theta }}_{DR-N}$	0.0	0.042	0.043	0.043	94.4	0.5	0.056	0.056	0.056	96.0
	Mean model for $\left(\mathcal{M}1\right)$ misspecified
${\widehat{\theta }}_{IW}$	8.4	0.050	0.054	0.081	78.6	8.0	0.056	0.058	0.079	84.8
${\widehat{\theta }}_{DR}$	0.0	0.040	0.043	0.043	94.0	0.6	0.052	0.055	0.055	94.4
${\widehat{\theta }}_{DR-N}$	0.0	0.041	0.043	0.043	95.4	0.4	0.056	0.055	0.055	95.8
	Mean model for $\left(\mathcal{M}2\right)$ misspecified
${\widehat{\theta }}_{DR}$	0.5	0.054	0.050	0.050	96.2	0.9	0.061	0.058	0.058	96.2
${\widehat{\theta }}_{DR-N}$	0.5	0.050	0.050	0.050	94.0	0.9	0.059	0.058	0.058	95.6
	Both mean models misspecified
${\widehat{\theta }}_{DR}$	8.4	0.050	0.053	0.081	78.6	7.9	0.057	0.058	0.079	86.0
${\widehat{\theta }}_{DR-N}$	8.4	0.051	0.053	0.081	79.0	7.9	0.058	0.058	0.079	86.2

Table 3.

Results of simulation study under MNAR: comparison of ${\widehat{\theta }}_0$, ${\widehat{\theta }}_{IW}$, ${\widehat{\theta }}_{DR}$, ${\widehat{\theta }}_{DR-N}$, ${\widehat{\theta }}_{IW-S}$, ${\widehat{\theta }}_{DR-S}$, and ${\widehat{\theta }}_{DR-N-S}$ using modified weights, when ε ~ N(0, 1) and Z₁ and Z₂ are identical. True θ is 0.722. The details of true models and misspecified working models are provided in Section 3.2.

	RB (%)	SE	SD	SMSE	CR (%)
${\widehat{\theta }}_{GS}$	−0.2	0.036	0.037	0.037	94.0
${\widehat{\theta }}_0$	−8.1	0.053	0.055	0.080	78.2
Correct subset of Z1 and D included in $\left(\mathcal{M}1\right)$
${\widehat{\theta }}_{IW}$	−5.8	0.052	0.055	0.069	85.4
${\widehat{\theta }}_{IW-S}$	−1.0	0.049	0.052	0.053	93.0
Correct subset of Z and D included in both models
${\widehat{\theta }}_{DR}$	−0.5	0.039	0.040	0.041	95.0
${\widehat{\theta }}_{DR-N}$	−0.5	0.039	0.040	0.040	94.4
${\widehat{\theta }}_{DR-S}$	−0.2	0.038	0.039	0.039	93.8
${\widehat{\theta }}_{DR-N-S}$	−0.2	0.038	0.039	0.039	93.6
Incorrect subset of Z1 and D included in $\left(\mathcal{M}1\right)$
${\widehat{\theta }}_{DR}$	−0.6	0.039	0.040	0.041	94.6
${\widehat{\theta }}_{DR-N}$	−0.5	0.039	0.040	0.040	93.2
${\widehat{\theta }}_{DR-S}$	−0.2	0.038	0.039	0.039	94.0
${\widehat{\theta }}_{DR-N-S}$	−0.2	0.038	0.039	0.039	93.4
Incorrect subset of Z2 and D included in $\left(\mathcal{M}2\right)$
${\widehat{\theta }}_{DR}$	−5.3	0.049	0.053	0.065	85.0
${\widehat{\theta }}_{DR-N}$	−5.3	0.050	0.053	0.065	86.2
${\widehat{\theta }}_{DR-S}$	−0.8	0.047	0.049	0.049	92.8
${\widehat{\theta }}_{DR-N-S}$	−0.8	0.048	0.049	0.049	93.8

3.1. MAR: δ independent of X given D and Z

Under MAR, we considered two settings, namely, δ dependent on X given D and δ independent of X given D. Corresponding to each setting, we generated the auxiliary variables, $Z_1=\left(Z_1^{(1)},Z_1^{(2)},Z_1^{(3)}\right)$, which are associated with δ, and $Z_2=\left(Z_2^{(1)},Z_2^{(2)},Z_2^{(3)}\right)$, which are associated with X. In the first setting, Z₁ = Z₂ and they were generated from a multivariate Gaussian distribution with a mean μ_Z = (3, −2, −1) and a variance matrix Σ_Z = diag(0.25, 0.25, 0.25), which implies that δ is dependent on X given D and hence ${\widehat{\theta }}_0$ is subject to potential bias. In the second setting, Z₁ and Z₂ were generated from two independent multivariate Gaussian distributions with the same mean and variance as in the first setting, which implies that δ is independent of X given D and hence ${\widehat{\theta }}_0$ is unbiased. Next, we generated X as follows, X = β₀ + β₁D + β₂Z₂ + β₃DZ₂ + ε with β₀ = 1, β₁ = 2.5, β₂ = (3, 3, 3), and β₃ = (.5, .5, .5), which is the true underlying model for $\left(\mathcal{M}2\right)$. Two different residual distributions were considered so that we could compare the performance of ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$; specifically, ε ~ N(0, σ²) or ε = 20{η − E(η)} with η ~ Beta(5, 1). The resulting true θ is 0.722 for Gaussian ε and 0.675 for non-Gaussian ε. Subsequently, we generated the missing indicator δ from a Bernoulli distribution with mean π which satisfies logit(π) = α₀ + α₁D + α₂Z₁ + α₃DZ₁ with α₀ = 0.3, α₁ = 0.3, α₂ = (0.4,0.5,0.3), and α₃ = (−0.7, −0.7, −0.9); this is the underlying true model for $\left(\mathcal{M}1\right)$. The resulting average probability of missing X is 66.4% in the diseased group and 55.8% in the non-diseased group.

When computing ${\widehat{\theta }}_{IW}$, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$, we fitted the two working models for δ and X, namely, $\left(\mathcal{M}1\right)$ and $\left(\mathcal{M}2\right)$, under the following four scenarios: 1) the mean structure is correctly specified for both working models, i.e., Z₁ and D are included in $\left(\mathcal{M}1\right)$, and Z₂ and D are included in $\left(\mathcal{M}2\right)$; 2) the mean structure is misspecified for $\left(\mathcal{M}1\right)$, i.e., only $Z_1^{(1)}$ and D are used in $\left(\mathcal{M}1\right)$; 3) the mean structure is misspecified for $\left(\mathcal{M}2\right)$, i.e., only $Z_2^{(1)}$ and D are used in $\left(\mathcal{M}2\right)$; and 4) the mean structure is misspecified for both working models, i.e., only $Z_1^{(1)}$ and D are included in $\left(\mathcal{M}1\right)$ and only $Z_2^{(1)}$ and D are included in $\left(\mathcal{M}2\right)$. We note that ${\widehat{\theta }}_{DR}$ assumes that X follows Gaussian distributions. Consequently, if the residuals for X follow a Gaussian distribution, e.g. ε ~ N(0, σ²), then the correct specification of the mean structure in $\left(\mathcal{M}2\right)$ also indicates the correct specification of the conditional distribution for X when computing ${\widehat{\theta }}_{DR}$. However, if the residual distribution is not Gaussian, e.g., ε = 20{η−E(η)} with η ~ Beta(5, 1), the conditional distribution for X is misspecified when computing ${\widehat{\theta }}_{DR}$, even if the mean structure is correctly specified in $\left(\mathcal{M}2\right)$. Since ${\widehat{\theta }}_{DR-N}$ is robust to the mis-specification of distributions of the residuals for X, it should remain consistent in both cases.

3.1.1. The case of δ dependent on X given D.

In this setting, we let Z₁ and Z₂ be identical, hence δ is dependent on X given D. Table 1 presents the results for two different residual distributions for X. We first discuss the case of Gaussian ε. ${\widehat{\theta }}_0$ shows a large RB of 11.6% with a low coverage rate of 70.0%. ${\widehat{\theta }}_{IW}$ exhibits negligible bias and a CR close to the nominal level when $\left(\mathcal{M}1\right)$ is correctly specified; however, its bias becomes substantial and CR degrades considerably to 78.6% when $\left(\mathcal{M}1\right)$ is misspecified. When at least one working model is correctly specified, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$ show negligible bias that is comparable to ${\widehat{\theta }}_{GS}$ and good coverage properties. In particular, as long as $\left(\mathcal{M}2\right)$ is correctly specified, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$ are more efficient than ${\widehat{\theta }}_{IW}$, and is almost as efficient as ${\widehat{\theta }}_{IMP}$; in this case, negligible loss of efficiency is observed even if $\left(\mathcal{M}1\right)$ is misspecified. By contrast, when $\left(\mathcal{M}2\right)$ is misspecified and $\left(\mathcal{M}1\right)$ is correctly specified, the loss of efficiency is considerable for ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$. These observations are consistent with what have been reported in the literature, i.e., the correct specification of $\left(\mathcal{M}2\right)$ for X is more important in terms of improving efficiency of ${\widehat{\theta }}_{DR}$. When both working models are misspecified, the bias and MSE of ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$ are still similar to or less than those of ${\widehat{\theta }}_{IW}$ or ${\widehat{\theta }}_0$.

When the residuals are not Gaussian, $\left(\mathcal{M}2\right)$ is always misspecified for ${\widehat{\theta }}_{DR}$. Our results in Table 1 show that ${\widehat{\theta }}_{DR}$ is fairly robust to the misspecified distribution of ε as long as the conditional mean of X in $\left(\mathcal{M}2\right)$ is correctly specified. In addition, most observations for Gaussian ε are still true for non-Gaussian ε. In this case, ${\widehat{\theta }}_{IMP}$ serves as an approximate benchmark for efficiency, since ${\widehat{\theta }}_{IMP}$ is also fairly robust to a mis-specified distribution for ε and it is generally difficult to obtain an exact “imputation” estimator when ε is non-Gaussian. Similar results were observed in our additional simulations with other non-Gaussian distributions for ε, say, χ² distribution.

3.1.2. The case of δ independent of X given D.

In this setting, Z₁ and Z₂ are two separate sets of auxiliary variables, hence δ is independent of X given D. Table 2 presents the results for both Gaussian and non-Gaussian residuals. In all cases, all estimators exhibit negligible bias and satisfactory coverage properties, which is consistent with our discussion in Section 2. Again, as long as $\left(\mathcal{M}2\right)$ is (approximately) correctly specified, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$ are almost as efficient as ${\widehat{\theta }}_{IMP}$; they perform no worse than ${\widehat{\theta }}_{IW}$ and ${\widehat{\theta }}_0$ in other settings. As with the case of δ dependent on X given D in Section 3.1.1, the results are very similar for two different types of residual distributions for X.

Table 2.

Results of simulation study under MAR: comparison of ${\widehat{\theta }}_0$, ${\widehat{\theta }}_{IW}$, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$ using modified weights, when Z₁ and Z₂ are independent. ε is Gaussian (i.e., ε ~ N(0, 1)) or non-Gaussian (i.e., ε = 20{η ~ E(η)} with η ~ Beta(5, 1)). True θ is 0.722 for Gaussian ε and 0.675 for non-Gaussian ε. The details of true models and misspecified working models are provided in Section 3.1.

	Gaussian ε					non-Gaussian ε
	RB (%)	SE	SD	SMSE	CR (%)	RB (%)	SE	SD	SMSE	CR (%)
${\widehat{\theta }}_{GS}$	0.2	0.036	0.035	0.035	96.0	0.2	0.038	0.036	0.036	96.0
${\widehat{\theta }}_0$	−0.1	0.059	0.057	0.057	95.8	0.4	0.063	0.061	0.061	96.4
	Both mean models correctly specified
${\widehat{\theta }}_{IMP}$	0.1	0.039	0.040	0.040	94.2	−0.6	0.051	0.050	0.050	95.8
${\widehat{\theta }}_{IW}$	−0.1	0.059	0.060	0.059	94.8	0.3	0.062	0.065	0.065	94.6
${\widehat{\theta }}_{DR}$	0.2	0.044	0.040	0.040	96.4	0.1	0.056	0.053	0.053	95.8
${\widehat{\theta }}_{DR-N}$	0.2	0.041	0.041	0.041	95.4	0.1	0.056	0.053	0.053	95.2
	Mean model for $\left(\mathcal{M}1\right)$ misspecified
${\widehat{\theta }}_{IW}$	−0.2	0.058	0.059	0.059	95.0	0.3	0.062	0.062	0.062	95.2
${\widehat{\theta }}_{DR}$	0.2	0.041	0.040	0.040	95.0	0.2	0.053	0.051	0.051	95.8
${\widehat{\theta }}_{DR-N}$	0.2	0.041	0.040	0.040	94.8	0.1	0.054	0.051	0.051	96.2
	Mean model for $\left(\mathcal{M}2\right)$ misspecified
${\widehat{\theta }}_{DR}$	−0.2	0.059	0.055	0.055	96.0	0.4	0.063	0.062	0.062	95.4
${\widehat{\theta }}_{DR-N}$	−0.2	0.057	0.055	0.055	96.4	0.4	0.063	0.062	0.062	95.2
	th Mean Models misspecified
${\widehat{\theta }}_{DR}$.	−0.2	0.054	0.054	0.054	95.0	0.4	0.059	0.059	0.059	95.2
${\widehat{\theta }}_{DR-N}$	−0.2	0.055	0.054	0.054	95.8	0.4	0.061	0.059	0.059	95.8

We repeated the simulations in Tables 1 and 2 using the original weights (Web Appendix B), and the results are almost the same except that the performance of the bootstrap SE for ${\widehat{\theta }}_{DR-N}$ deteriorates somewhat.

3.2. MNAR: δ dependent on X given D and Z

We now consider the case of MNAR where δ is dependent on X conditional on D and Z, i.e., the true model for δ is $\text{logit}(\pi )={\alpha }_0+{\alpha }_Z{Z}_1^{(3)}+{\alpha }_{D}D+{\alpha }_{X}X$ with (α₀, α_Z, α_D, α_X) = (−1, 0.2, 0.5, 0.3). The rest of the simulation setup is identical to that in Section 3.1. The resulting average probability of missing X is 57.4% in the diseased group and 31.5% in the non-diseased group. We focused on the case where Z₁ and Z₂ are identical and ε is Gaussian; in this case, the true θ remains 0.722. Our primary goal is to compare ${\widehat{\theta }}_{IW}$, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$ with their corresponding sensitivity estimators as described in Section 2.4, namely, ${\widehat{\theta }}_{IW-S}$, ${\widehat{\theta }}_{DR-S}$ and ${\widehat{\theta }}_{DR-N-S}$, for which the estimating equations (5) were used to estimate α_S = (α₀, α_Z, α_D) with α_X fixed at its true value. The rest of estimating procedures remain the same for all estimators. As with the case of MAR in Section 3.1, we investigated the impact of the mis-specified $\left(\mathcal{M}1\right)$ and/or $\left(\mathcal{M}2\right)$; specifically, we considered a misspecified $\left(\mathcal{M}1\right)$ that includes $Z_1^{(1)}$ and D and a misspecified $\left(\mathcal{M}2\right)$ that includes only $Z_2^{(3)}$ and D. We also note that X is included as a covariate in $\left(\mathcal{M}1\right)$ for ${\widehat{\theta }}_{IW-S}$, ${\widehat{\theta }}_{DR-S}$ and ${\widehat{\theta }}_{DR-N-S}$, but not for ${\widehat{\theta }}_{IW}$, ${\widehat{\theta }}_{DR}$ or ${\widehat{\theta }}_{DR-N}$. Thus, when D and the correct subset of Z₁ (i.e., $Z_1^{(3)}$) are included in $\left(\mathcal{M}1\right)$, $\left(\mathcal{M}1\right)$ is correctly specified for ${\widehat{\theta }}_{IW-S}$, ${\widehat{\theta }}_{DR-S}$ and ${\widehat{\theta }}_{DR-N-S}$, but is misspecified for ${\widehat{\theta }}_{IW}$, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$.

Table 3 presents the simulation results. First, ${\widehat{\theta }}_0$ again exhibits substantial bias under MNAR. We now compare ${\widehat{\theta }}_{IW}$ and ${\widehat{\theta }}_{IW-S}$. When $\left(\mathcal{M}1\right)$ does not account for the effect of X, ${\widehat{\theta }}_{IW}$ shows considerable bias even if $\left(\mathcal{M}1\right)$ includes D and the correct subset of Z₁. On the other hand, ${\widehat{\theta }}_{IW-S}$, which accounts for the effect of X, shows negligible bias. Next, we compare ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$ with ${\widehat{\theta }}_{DR-S}$ and ${\widehat{\theta }}_{DR-N-S}$. When correct subsets of Z and D are included in both working models, $\left(\mathcal{M}1\right)$ is still misspecified for ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$. However, since $\left(\mathcal{M}2\right)$ is correctly specified, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$ exhibit negligible bias and good coverage properties as a result of their double robustness, and their efficiency is comparable to that of ${\widehat{\theta }}_{DR-S}$ and ${\widehat{\theta }}_{DR-N-S}$. These results still hold when $\left(\mathcal{M}1\right)$ includes the incorrect subset of auxiliary variables and $\left(\mathcal{M}2\right)$ is correctly specified. When an incorrect subset of Z₂ is included in $\left(\mathcal{M}2\right)$, both working models are misspecified for ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$; consequently, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$ exhibit considerable bias. In all three settings, ${\widehat{\theta }}_{DR-S}$ and ${\widehat{\theta }}_{DR-N-S}$ show negligible bias, but their SDs increase when $\left(\mathcal{M}2\right)$ is misspecified, which is consistent with the earlier findings that $\left(\mathcal{M}2\right)$ is more important in terms of improving efficiency.

4. Data Analysis

We illustrate our methods using an observational psychiatric study, which was concerned with the impact of maternal depression during pregnancy on infant outcomes. In this study, participants were enrolled no later than week 28 of gestation and evaluated at each trimester across pregnancy. As part of the study, the presence (or absence) of a major depressive episode (disease status, D) was determined at each visit by the Mood Module of the Structured Clinical Interview for DSMIV Axis I Disorders (SCID) (First et al., 2002), which needs to be administered by a trained research professional and is considerably more time-consuming and difficult to obtain in practice. At the same time, some subjects also completed the self-rated Edinburgh Postnatal Depression Scale (EPDS) (Cox and Holden, 1987) at each visit.

In female mental health research, several rating scales have been developed for identifying postpartum depression (Fergerson et al., 2002; Perfetti et al., 2004), and in particular the self-rated EPDS has emerged as a widely-used instrument for postpartum depression screening and detection (Austin et al., 2005; Felice et al., 2006), which can be obtained fairly easily in practice. In contrast, there are no validated tools to assess depression during pregnancy. In practice, the EPDS, developed for postpartum use, has been increasingly used to identify depression during pregnancy and to screen for those at risk for developing depression during pregnancy. While not designed for such purpose, data collected from this study have been recently used to evaluate EPDS as a biomarker for the diagnosis of maternal depression throughout pregnancy. For the purpose of illustration, we focus on the data collected from the second trimester; a subset of the study population who had data in the second trimester was used and the sample size is n = 517 in the analysis. The outcome of interest is the presence of a major depressive episode (D) and is confirmed for all subjects, whereas EPDS is the biomarker of interest and is missing in 79% of the subjects. Additional auxiliary variables were also measured in this study including the mother’s age, race, marital status and eduction level, whether or not it was the first pregnancy. In addition, a research interviewer masked to treatment status administered the Structured Interview Guide for the Hamilton Rating Scale for Depression to obtain 17-item (HRSD17), which is known to be highly correlated with EPDS. These variables are treated as auxiliary variables (Z) and are used to build $\left(\mathcal{M}1\right)$ and $\left(\mathcal{M}2\right)$.

We conducted a sensitivity analysis for ${\widehat{\theta }}_0$, ${\widehat{\theta }}_{IW}$, ${\widehat{\theta }}_{DR}$ and ${\widehat{\theta }}_{DR-N}$ as described in Section 2.4. Specifically, we considered a $\left(\mathcal{M}1\right)$ that is similar to what is discussed in Section 2.4, i.e., $\text{logit}(\pi )={\alpha }_S^{T}W(Z,D)+{\alpha }_{X}X$, where W(Z, D) include the intercept and interaction terms between auxiliary variables Z and D. In fitting this $\left(\mathcal{M}1\right)$, estimating equations (5) were used with α_X fixed at −1, 0 and 1, where α_X = 0 corresponds to the case of MAR, and α_X = −1 or 1 correspond to the case of MNAR. In our analysis, all continuous variables including X were standardized to have mean 0 and unit standard deviation. Consequently, α_X captures the effect of a one-SD change in X. Table 4 presents the results using modified weights. The impact of different α_X values is moderate on ${\widehat{\theta }}_{IW}$, ${\widehat{\theta }}_{DR}$, and ${\widehat{\theta }}_{DR-N}$, and the estimates using different methods including ${\widehat{\theta }}_0$ are comparable. It indicates that the missingness of X (δ) is likely close to be independent of X given D. Nevertheless, ${\widehat{\theta }}_{DR}$, which incorporates information from auxiliary variables, is more efficient than the other estimators. Since the proportion of missing data is very high in the data, the bootstrap SE of ${\widehat{\theta }}_{DR-N}$ is greater than the SE of ${\widehat{\theta }}_{DR}$, but it is still smaller than the SE of ${\widehat{\theta }}_{IW}$. We repeated this analysis using the original weights in Web Appendix C; while the main results remain similar, a larger bootstrap SE for ${\widehat{\theta }}_{DR-N}$ is observed as a result of large and unstable weights.

Table 4.

Sensitivity analysis using the modified weights for estimating the ROC AUC (θ) in the psychiatric study

			Estimate	SE
		${\widehat{\theta }}_0$	0.861	0.038
	αX = −1		αX = 0		αX = 1
	Estimate	SE	Estimate	SE	Estimate	SE
${\widehat{\theta }}_{IW}$	0.864	0.037	0.851	0.040	0.849	0.042
${\widehat{\theta }}_{DR}$	0.873	0.028	0.852	0.030	0.841	0.032
${\widehat{\theta }}_{DR-N}$	0.873	0.035	0.852	0.038	0.841	0.038

With ${\widehat{\theta }}_{DR}$ ranging from 0.841 to 0.873, our results suggest that EPDS has very good discriminative power during the second trimester. However, in this study, only a subset of the study population had depression status confirmed during each perinatal window. As a result, in addition to missing values in the biomarker, the verification bias is potentially in play as well. Furthermore, both the rating scale and the presence of a major depressive episode were repeatedly measured through the pregnancy. Therefore, it is of substantial interest in the future studies to investigate methods that can account for both missing biomarker values as well as verification bias and accommodate repeatedly measured biomarker values and disease status when estimating the ROC AUC.

5. Discussion

We have proposed and contrasted several estimators of the ROC AUC when the biomarker value is missing for some subjects. Our numerical studies show that the doubly robust estimators perform as well as or better than other estimators in all cases even when both working models are misspecified. ${\widehat{\theta }}_{DR}$ is also fairly robust to the misspecified residual distribution for the biomarker variable (X). Since only ranks of X are used in estimating θ, the correct specified conditional mean is more important and the impact of a misspecified residual distribution may be limited given the correctly specified conditional mean. The bootstrap procedure for obtaining SE of ${\widehat{\theta }}_{DR-N}$ is computationally more expensive and also makes it more susceptible to large and unstable weights. Thus, in practice, we recommend the use of ${\widehat{\theta }}_{DR}$ and stabilized weights such as ours, and emphasize the importance of identifying (approximately) correct $\left(\mathcal{M}2\right)$. We also note that ${\widehat{\theta }}_{DR}$ can readily accommodate categorical biomarker values, e.g., a baseline logit model (Agresti, 2002) can be used to model the conditional distribution of a categorical biomarker variable.

More recently, Cao et al. (2009) investigated alternative doubly robust estimators for estimating a population mean; their methods achieve minimum variance under incorrectly specified $\left(\mathcal{M}2\right)$ and correctly specified $\left(\mathcal{M}1\right)$, and they do not suffer from large and unstable weights. While their enhanced model for $\left(\mathcal{M}1\right)$ can be readily adopted in our methods as an alternative to alleviate the problem of large and unstable weights, it is more involved to extend their approach of minimizing variance under misspecified $\left(\mathcal{M}2\right)$ and correctly specified $\left(\mathcal{M}1\right)$ to the estimation of the ROC AUC as complications arise from the use of U-statistic in our methods. Potential future research may also include extending sensitivity analysis to $\left(\mathcal{M}2\right)$ and investigating more complicated missing patterns, e.g., auxiliary variables are also missing and missingness is not monotone, for which an imputation approach may be more practical.