Statistical Methods without Estimating the Missingness Mechanism: A Discussion of “Statistical Inference for Nonignorable Missing Data Problems: A Selective Review” by Niansheng Tang and Yuanyuan Ju

First of all, I wholeheartedly congratulate Tang and Ju (referred to as TJ hereafter) on a well-written comprehensive review paper that surveys cutting-edge statistical theory and methodology relevant to estimation, in uence analysis and model selection in regression models with missing data.

TJ begins their presentation from the missing data mechanism, a fundamental concept in the missing data literature (Little and Rubin, 2002; Tsiatis, 2006; Kim and Shao, 2013; Molenberghs et al., 2014). In their Section 2, TJ presents a detailed explanation of this definition and underlines its importance to developing downstream statistical methodology. To facilitate this discussion, I adopt the same notation as follows. Consider a regression model where Y is a response variable and X is a p-dimensional explanatory variable, and $\left\{\left({\text{x}}_i,y_i\right):i=1,\dots ,n\right\}$ are n independent and identically distributed realizations of (X, Y ). Assume X is always fully observed but Y is subject to missingness. Let δ be the missing data indicator for Y, that is, δ = 0 if Y is missing, and δ = 1 otherwise. Then the missing data mechanism is the conditional distribution of δ given X and Y, i.e.,

$$\text{π}\left(\mathbf{\text{x}},y\right)=\text{\hspace{0.17em}pr}\left(\delta =1|\mathbf{\text{x}},y\right).$$ (1)

One intrinsic complication of the missing data mechanism is that, only except for a few scenarios (Little, 1988; d’Haultfoeuille, 2010), its underlying truth is difficult to verify. The reason dues to its plausible dependence on Y, an incompletely observed variable. This issue pronounces more clearly when one moves forward to real application, where the investigators would be more satisfied if a statistical method could make the assumption of the mechanism less stringently so that it is able to be flexibly applied to various scenarios.

My discussion, motivated by the need of developing versatile statistical procedures that would provide robust protection to certain mechanism misspecification, showcases the up-to-date statistical treatments where the mechanism model assumption is only imposed at a minimum level. The discussion concentrates on brief introduction of two types of these assumptions and spans diverse statistical topics including model identification, point estimation, hypothesis testing, and high dimensional variable selection.

One distinct feature of the methods in this discussion is that the mechanism model would be treated as a nuisance, hence all the methods could be carried out without the need of estimating the mechanism.

1. Mechanism based on conditional independence

The instrumental variable is a well studied method in econometrics, epidemiology and related disciplines. The key step of applying this method is certain requirement about the conditional independence among variables. Zhao and Shao (2015) proposed to take advantage of the nonresponse instrument Z, a component of X, to analyze missing data, especially nonignorable missing data. The concept of nonresponse instrument shares the similar spirit to the instrumental variable. To be more specific, Zhao and Shao (2015) assumed that

$$\text{pr}\left(\delta =1|\mathbf{\text{x}},\text{\hspace{0.17em}y}\right)=\text{\hspace{0.17em}pr}\left(\delta =1|\mathbf{\text{u}},y\right),$$ (2)

where $\mathbf{\text{x}}={\left({\mathbf{\text{u}}}^{\text{T}},{\mathbf{\text{z}}}^{\text{T}}\right)}^{\text{T}}$. Some further requirement, e.g., $p\left(y|\mathbf{\text{x}}\right)\ne p\left(y|\mathbf{\text{u}}\right)$. is also needed for model identification purpose.

When X by itself serves as the nonresponse instrument, Tang et al. (2003) studied this special situation and proposed to estimate the unknown parameter ϴ in $p\left(y|\mathbf{\text{x}}\right)=p\left(y|\mathbf{\text{x}};\mathit{\theta }\right)$ through the conditional likelihood of $p\left(\mathbf{\text{x}}|y,\delta =1\right)$:

$${\prod _{i=1}^n\left\{\frac{p\left(y_i|{\mathbf{\text{x}}}_i;\mathit{\theta }\right)}{\int p\left(y_i|\mathbf{\text{x}};\mathit{\theta }\right)g\left(\mathbf{\text{x}}\right)d\mathbf{\text{x}}}\right\}}^{{\delta }_i=1},$$

where g(x) represents the unspecified probability density function of X. Then the objective becomes to a semiparametric function:

$$l\left(\mathit{\theta },g\right)=\sum _{i=1}^n{r}_i\left\{\log p\left(y_i|{\mathbf{\text{x}}}_i;\mathit{\theta }\right)-\log \int p\left(y_i|\mathbf{\text{x}};\mathit{\theta }\right)g\left(\mathbf{\text{x}}\right)d\mathbf{\text{x}}\right\}.$$

To solve for $\mathit{\theta }$, an estimator of g(x) is needed. Three straightforward g(x) estimators could be considered: the true g(x); a parametric g(x) = g(x; α) with α estimated as $\widehat{\alpha }$ through full data likelihood method; a nonparametric g(x) with its cumulative distribution function estimated by its empirical version. These three alternatives lead to three different pseudolikelihood estimators of $\mathit{\theta }$: $\widehat{\mathit{\theta }}$_PL0, $\widehat{\mathit{\theta }}$_PL1 and $\widehat{\mathit{\theta }}$_PL2. At rst sight, one would believe that $\widehat{\mathit{\theta }}$_PL0 is superior to the other two in terms of estimation efficiency. However, Tang et al. (2003) showed that $\widehat{\mathit{\theta }}$_PL0 is always less efficient than b_PL1. In a recent paper Zhao and Ma (2018), the authors further proved that $\widehat{\mathit{\theta }}$_PL1 is always less efficient than $\widehat{\mathit{\theta }}$_PL2 and there is no other method which could lead to a more efficient estimator than $\widehat{\mathit{\theta }}$_PL2, hence $\widehat{\mathit{\theta }}$_PL2 is optimal.

Other work along this line includes Miao and Tchetgen Tchetgen (2016) exploring different types of doubly robust estimators and Fang et al. (2018) extending the idea to missing covariate and proposing some imputation approach based on estimating equations.

2. Mechanism based on statistical chromatography

The other unspecified missing data mechanism investigated in the literature is to assume a decomposable model

$$\text{pr}\left(\delta =1|\mathbf{\text{x}},y\right)=s\left(\mathbf{\text{x}}\right)t\left(y\right),$$ (3)

where s(·) and t(·) are two unspecified functions. It is clear that, MCAR (s = t = constant) and MAR (t = constant) are special cases of this assumption. When s = constant, it becomes the case discussed in Section 1 where X on its own serves as the nonresponse instrument.

A pivotal observation following (3) is that, $p\left(y|\mathbf{\text{x}}\right)$ and $p\left(y|\mathbf{\text{x}},\delta =1\right)$ could be bridged as

$$p\left(y|\mathbf{\text{x}},\delta =1\right)=\frac{\text{pr}\left(\delta =1|\mathbf{\text{x}},y\right)}{\text{pr}\left(\delta =1|\mathbf{\text{x}}\right)}p\left(y|\mathbf{\text{x}}\right).$$

Note that $\text{pr}\left(\delta =1|\mathbf{\text{x}},y\right)/\text{pr}\left(\delta =1|\mathbf{\text{x}}\right)$ preserves to be a function of x-only multiples a function of y-only. Using the idea of the conditional likelihood (Kalb eisch, 1978), decomposing the observed y_i’s as its rank statistic and order statistic, considering the likelihood conditional on the order statistic, Liang and Qin (2000) proposed the following objective function to estimating $\mathit{\theta }$:

$$\prod _{1\le i\le j\le m}\frac{p\left(y_i|{\mathbf{\text{x}}}_i;\mathit{\theta }\right)p\left(y_j|{\mathbf{\text{x}}}_j;\mathit{\theta }\right)}{p\left(y_i|{\mathbf{\text{x}}}_i;\mathit{\theta }\right)p\left(y_j|{\mathbf{\text{x}}}_j;\mathit{\theta }\right)+p\left(y_i|{\mathbf{\text{x}}}_j;\mathit{\theta }\right)p\left(y_j|{\mathbf{\text{x}}}_i;\mathit{\theta }\right)},$$ (4)

where the first m subjects are fully observed without loss of generality.

The key here is that we model the data at a more refined granularity of rank and order statistics, so that sophisticated conditioning arguments could be applied to separate the parameter of interest $\mathit{\theta }$ and other nuisance components. Hence we call this procedure statistical chromatography.

We elaborate under the generalized linear model framework where

$$p\left(y|\mathbf{\text{x}};\mathit{\theta }\right)=\exp \left[{\varphi }^{-1}\left\{y\eta -b\left(\eta \right)\right\}+c\left(y;\varphi \right)\right]$$

with link function structure $g\left(\mu \left(\eta \right)\right)=\alpha +{\mathit{\beta }}^{\text{T}}\mathbf{\text{x}}$. With canonical link, to maximize (4) is equivalent to minimizing

$$\sum _{1\le i\le j\le m}\log \left[1+\exp \left\{-\left(y_i-y_j\right){\left({\mathbf{\text{x}}}_i-{\mathbf{\text{x}}}_j\right)}^{\text{T}}\text{γ}\right\}\right],$$

where $\mathit{\gamma }={\varphi }^{-1}{\mathit{\beta }}^{\text{T}},\mathit{\theta }={\left(\alpha ,{\mathit{\beta }}^{\text{T}},\varphi \right)}^{\text{T}}$. Hence to compensate for missing data, we could only estimate γ as opposed to the whole unknown parameter ϴ. Although only γ is estimable, the hypothesis testing β = 0 versus β ≠ 0 could still be carried out since the null hypothesis β = 0 is equivalent to γ = 0. The detailed Wald type test statistic needs the asymptotic distribution of the estimator γ of under this scheme (Zhao and Shao, 2017). With noncanonical link, Zhao and Shao, (2017) showed that, interestingly, the whole unknown parameter $\mathit{\theta }$ is estimable under some situations.

Finally I would like to point out a regularization approach for high dimensional variable selection with missing data using this approach. The essential idea is to identify “important” variables through whether the corresponding estimator ${\widehat{\gamma }}_j$ equals zero or not. The penalized likelihood function is

$$\sum _{1\le i\le j\le m}\log \left[1+\exp \left\{-\left(y_i-y_j\right){\left({\mathbf{\text{x}}}_i-{\mathbf{\text{x}}}_j\right)}^{\text{T}}\mathit{\gamma }\right\}\right]+\sum _{j=1}^p{p}_{\lambda }\left(\left|{\text{γ}}_j\right|\right),$$

where p_λ(·) could be any penalty function, and λ ≥ 0 is the tuning parameter. Zhao et al. (2018) proved that the validity of the selection consistency allows p to grow at a rate exponentially fast with n as log $\log p=o\left(n^{1-4k}/{\left(\log n\right)}^2\right)$ with 0 < k < 1/4. In penalized likelihood approach for variable selection, the determination of the tuning parameter is also critical. Zhao and Yang (2017) further studied some stability enhanced tuning parameter selection methods following this approach.