A note on the relationships between multiple imputation, maximum likelihood and fully Bayesian methods for missing responses in linear regression models

Abstract

Multiple Imputation, Maximum Likelihood and Fully Bayesian methods are the three most commonly used model-based approaches in missing data problems. Although it is easy to show that when the responses are missing at random (MAR), the complete case analysis is unbiased and efficient, the aforementioned methods are still commonly used in practice for this setting. To examine the performance of and relationships between these three methods in this setting, we derive and investigate small sample and asymptotic expressions of the estimates and standard errors, and fully examine how these estimates are related for the three approaches in the linear regression model when the responses are MAR. We show that when the responses are MAR in the linear model, the estimates of the regression coefficients using these three methods are asymptotically equivalent to the complete case estimates under general conditions. One simulation and a real data set from a liver cancer clinical trial are given to compare the properties of these methods when the responses are MAR.

1. INTRODUCTION

Missing data is very common in various experimental settings, including clinical trials, sample surveys and environmental studies. There are essentially three major likelihood-based approaches for handling missing data in a regression problem. These are i) Maximum Likelihood (ML), ii) Multiple Imputation (MI), and iii) Fully Bayesian (FB). The EM algorithm is a technique often used to obtain ML estimates and is useful when the likelihood function of the observed data has no closed form. The recent developments of missing data approaches also include empirical likelihood method [18], parametric fractional imputation [10], among others. In this paper, we investigate theoretical connections between MI, ML (especially within the EM framework), and FB approaches in the linear regression model when the response variable is missing at random (MAR).

It is well known that when the response variable is MAR and the covariates are fully observed, the likelihood function of the observed data is the same as the complete case likelihood (i.e. the likelihood obtained by omitting all cases with missing values), and therefore the ML estimates are identical to the complete case (CC) estimates. However, this result is not obvious under the MI and FB approaches. Although the CC estimates are unbiased and efficient under MAR responses, the MI and FB methods are still used in practice since many researchers are unaware of this special property of the CC estimates. To study the relationships between the three methods in this context, we consider MAR responses in the linear model and investigate the small and large sample properties of the estimates, and derive analytic and asymptotic expressions of the estimates and standard errors for the MI, ML and FB approaches. Under noninformative priors for the MI and FB methods, we show that the estimates and their standard errors under these three approaches are asymptotically equivalent to the CC estimates.

There is much literature on MI, ML and FB, respectively. [21] provide asymptotic results for MI with MAR responses in linear models. [17], [15], and [20] discuss theoretical properties of proper and improper MI. [22] and [19] propose a consistent variance estimator for MI. For ML, one of the earliest references is [12]. [6], [11], and [8] proposes the “EM by the method of weights” and the Monte Carlo EM algorithm (MCEM) for the ML framework in generalized linear models (GLM). [16] examine the problem of using EM to obtain the asymptotic covariance matrix of the parameter estimates. [7] discuss FB methods for MAR covariates in GLM’s. There are two major differences of our work in this paper from previous literature. First, for MAR responses, we derive both the small and large sample properties of the estimates, while the previous work mainly focuses on large sample properties. Secondly, the main purpose of this paper is to investigate the theoretical relationships between MI, ML and FB, as this was only investigated only through simulations before.

The rest of this paper is organized as follows. In Section 2.1, we derive the small sample and asymptotic expressions of the estimates and standard errors for proper MI. In Section 2.2, we derive the expressions for ML via the EM algorithm. The expressions for FB are derived in Section 2.3. In Section 3, we conduct a simulation to demonstrate our results. A real data analysis of a liver cancer clinical trial is given in Section 4, and we conclude the paper with a brief discussion in Section 5.

2. MAR RESPONSES IN THE LINEAR MODEL

Consider the linear model

$$\mathbf{y}=\mathbf{X}\beta +\mathbf{e},$$ (1)

where β is a p × 1 vector of unknown parameters, X is an n×p full rank matrix of explanatory variables including an intercept, and e is an n × 1 vector of random errors with e ~ N_n(0, σ²I), where σ² is assumed unknown throughout. We assume throughout that X is fully observed and the components of y are MAR. For simplicity, we rearrange the data so that y₁ = (y₁, …, y_n1)′ are fully observed and y₂ = (y_n1+1, …, y_n)′ are MAR, and assume that the corresponding n₁ × p and n₂ × p matrices of fixed covariates X₁ and X₂ for y₁ and y₂ are full-rank, n₁+n₂ = n and p < n₁. Therefore, we write $\mathbf{y}={({\mathbf{y}}_1^{\prime },{\mathbf{y}}_2^{\prime })}^{\prime }$ and $\mathbf{X}={({\mathbf{X}}_1^{\prime },{\mathbf{X}}_2^{\prime })}^{\prime }$.

As shown in [13], the maximum likelihood estimates of β and σ² are the same as the CC estimates in the linear regression model, in which cases with any missing values are simply discarded. In fact, this is true for any regression model with MAR responses satisfying conditional independence between y₁ and y₂ given X and γ, where γ = (β, σ²), since the likelihood function of the observed data D_obs = (y₁, X) is given by

$$\begin{array}{l}L(\gamma \mid {\mathbf{D}}_{\mathit{obs}})=\int p({\mathbf{y}}_1,{\mathbf{y}}_2\mid \mathbf{X},\gamma )d{\mathbf{y}}_2\\ =p({\mathbf{y}}_1\mid \mathbf{X},\gamma )\int p({\mathbf{y}}_2\mid \mathbf{X},\gamma )d{\mathbf{y}}_2\\ =p({\mathbf{y}}_1\mid \mathbf{X},\gamma ),\end{array}$$

which is the CC likelihood.

The standard results for the linear regression model with MAR responses are

$${\widehat{\beta }}_{ML}={\widehat{\beta }}_{CC}={({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_1^{\prime }{\mathbf{y}}_1,$$ (2)

which is independent of

$${\widehat{\sigma }}_{ML}^2={\widehat{\sigma }}_{CC}^2={\mathbf{y}}_1^{\prime }(\mathbf{I}-{\mathbf{X}}_1{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_1^{\prime }){\mathbf{y}}_1/n_1$$ (3)

with E(β̂) = β and $E({\widehat{\sigma }}_{ML}^2)=(n_1-p){\sigma }^2/n_1$. The variances of the estimates are

$$\text{Var}({\widehat{\beta }}_{ML})={\sigma }^2{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}$$ (4)

and

$$\text{Var}({\widehat{\sigma }}_{ML}^2)=2(n_1-p){\sigma }^4/n_1^2.$$ (5)

Clearly, we can adjust the estimate of σ² to get an unbiased estimate by letting ${\stackrel{\sim }{\sigma }}_{ML}^2=n_1{\widehat{\sigma }}_{ML}^2/(n_1-p)$. It is worth noting that if we apply the EM algorithm in this setting and use Louis’s method [14] to get the variance estimates, we get same variance estimates of β̂_ML as in Eq. (4), and the variance estimate of ${\widehat{\sigma }}_{ML}^2$ is equal to $2{\widehat{\sigma }}_{ML}^4/n_1$, which is larger than Eq. (5) in small samples but asymptotically equivalent when n₁ → ∞. In the next three subsections, we explore the small and large sample properties of the estimators under MI, ML via the EM algorithm, and FB methods for MAR responses under model (1).

2.1 Multiple imputation (MI)

Multiple imputation has emerged as a very popular technique for inference in missing data problems. In this section, we consider the precision parameter instead of the variance parameter for the development of MI. Therefore, we assume γ^* = (β, τ), where τ = 1/σ². Proper MI is based on creating imputed datasets in which the missing values are sampled from the posterior predictive distribution of the missing data given the observed data, given by

$$p({\mathbf{D}}_{\mathit{mis}}\mid {\mathbf{D}}_{\mathit{obs}})=\int p({\mathbf{D}}_{\mathit{mis}}\mid {\mathbf{D}}_{\mathit{obs}},{\gamma }^{\ast })\pi ({\gamma }^{\ast }\mid {\mathbf{D}}_{\mathit{obs}})d{\gamma }^{\ast },$$ (6)

where D_obs = (y₁, X₁, X₂), and D_mis = y₂ for the current setting. π(γ^*|D_obs) is the posterior distribution of γ^* based on the observed data, given by π(γ^*|D_obs) ∝ {∫ L(γ^*|D_mis, D_obs)dD_mis}π(γ^*), where L(γ^*|D_mis, D_obs) is the likelihood function of the complete-data, ∫ L(γ^*|D_mis, D_obs)dD_mis is the likelihood based on the observed data, and π(γ^*) is the prior distribution of γ^*. Assume ${\mathbf{D}}_{\mathit{mis}}^{(l)}$, l = 1, …, m, are draws of D_mis from the posterior predictive distribution p(D_mis|D_obs) given in Eq. (6). Let ${\widehat{\gamma }}_l^{\ast }$ and V_l denote the posterior mean and covariance matrix of γ^* based on π(γ^*|D_obs) calculated for the lth imputed data set (y₁, ${\mathbf{y}}_2^{(l)}$). Then, the MI estimate of γ^* is ${\widehat{\gamma }}_{MI}^{\ast }=m^{-1}{\sum }_{l=1}^m{\widehat{\gamma }}_l^{\ast }$, and the estimate of the variance of ${\widehat{\gamma }}_{MI}^{\ast }$ is

$$\widehat{\text{Var}}({\widehat{\gamma }}_{MI}^{\ast })=\widehat{\mathbf{V}}+\left(1+\frac{1}{m}\right)\widehat{\mathbf{B}},$$ (7)

where $\widehat{\mathbf{V}}=m^{-1}{\sum }_{l=1}^m{\widehat{\mathbf{V}}}_l$ and $\widehat{\mathbf{B}}={\sum }_{l=1}^m({\widehat{\gamma }}_l^{\ast }-{\widehat{\gamma }}_{MI}^{\ast }){({\widehat{\gamma }}_l^{\ast }-{\widehat{\gamma }}_{MI}^{\ast })}^{\prime }/(m-1)$ is the between-imputation variance. There are several imputation methods that have been proposed for the MI method. In this paper, we concentrate on proper MI using the improper prior,

$$\pi ({\gamma }^{\ast })\propto {\tau }^{-1}.$$ (8)

We note that in MI, the imputation model can be different from the analysis model, but in this paper we only consider the case in which the two models are the same.

Theorem 1 gives the small sample behavior of the estimates of β and σ² for proper MI assuming the improper prior π(γ^*) ∝ τ⁻¹. Large sample properties of the estimates are also given under some general conditions. To derive these properties, we need the following lemma.

Lemma 2.1

If the n×1 random vector z has a multivariate t distribution, denoted S_n(v, μ, V), with density proportional to ${[1+\frac{1}{v}{(\mathbf{z}-\mu )}^{\prime }{\mathbf{V}}^{-1}(\mathbf{z}-\mu )]}^{-\frac{(\mathbf{v}+\mathbf{n})}{\mathbf{2}}}$, and A and B are matrices of constants, then

1. $E({\mathbf{z}}^{\prime }\mathbf{Az})=\frac{v}{v-2}tr(\mathbf{AV})+{\mu }^{\prime }\mathbf{A}\mu$, when v > 2

2. $E({\mathbf{zz}}^{\prime }\mathbf{Az})=\frac{v}{v-2}[{\mathbf{VA}}^{\prime }\mu +\mathbf{VA}\mu +\mu tr(\mathbf{AV})]+\mu {\mu }^{\prime }\mathbf{A}\mu$, when v > 2

3. $E[({\mathbf{z}}^{\prime }\mathbf{Az})({\mathbf{z}}^{\prime }\mathbf{Bz})]=\frac{v^2}{(v-2)(v-4)}[tr(\mathbf{AV})tr(\mathbf{BV})+tr(\mathbf{AVBV})+tr({\mathbf{AVB}}^{\prime }\mathbf{V})]+\frac{v}{v-2}[tr(\mathbf{AV}){\mu }^{\prime }\mathbf{B}\mu +{\mu }^{\prime }\mathbf{A}\mu tr(\mathbf{BV})+{\mu }^{\prime }(\mathbf{AVB}+{\mathbf{A}}^{\prime }\mathbf{VB}+\mathbf{AV}{\mathbf{B}}^{\prime }+{\mathbf{A}}^{\prime }{\mathbf{VB}}^{\prime })\mu ]+{\mu }^{\prime }\mathbf{A}\mu {\mu }^{\prime }\mathbf{B}\mu$, when v > 4.

The proof of Lemma 2.1 is given in the Appendix. For the linear regression model (1) with prior as Eq. (8), the posterior distribution of γ^* based on the observed data is

$$\begin{array}{l}p({\gamma }^{\ast }\mid {\mathbf{y}}_1)\propto p({\mathbf{y}}_i\mid {\gamma }^{\ast })\pi ({\gamma }^{\ast })\\ \propto {\tau }^{n_1/2-1}exp\left\{-\frac{\tau }{2}{({\mathbf{y}}_1-{\mathbf{X}}_1\beta )}^{\prime }({\mathbf{y}}_1-{\mathbf{X}}_1\beta )\right\}\end{array}$$

and the posterior predictive distribution is

$$\begin{array}{l}p({\mathbf{y}}_2\mid {\mathbf{y}}_1)=\int \int p({\mathbf{y}}_2\mid {\mathbf{y}}_1,{\gamma }^{\ast })p({\gamma }^{\ast }\mid {\mathbf{y}}_1)d\beta d\tau \\ \propto {\left[1+\frac{{({\mathbf{y}}_2-{\widehat{\mathbf{y}}}_2)}^{\prime }\mathbf{H}({\mathbf{y}}_2-{\widehat{\mathbf{y}}}_2)}{(n_1-p)s^2}\right]}^{-\frac{n_1+n_2-p}{2}}\end{array}$$

where ${\widehat{\mathbf{y}}}_2={\mathbf{X}}_2{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_1^{\prime }{\mathbf{y}}_1,s^2={\mathbf{y}}_1^{\prime }(\mathbf{I}-{\mathbf{X}}_1{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_1^{\prime }){\mathbf{y}}_1/(n_1-p)$ and $\mathbf{H}=\mathbf{I}-{\mathbf{X}}_2{({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}{\mathbf{X}}_2^{\prime }$. Since ${\mathbf{X}}^{\prime }\mathbf{X}={\mathbf{X}}_1^{\prime }{\mathbf{X}}_1+{\mathbf{X}}_2^{\prime }{\mathbf{X}}_2$, and ${\mathbf{X}}_1^{\prime }{\mathbf{X}}_1$ are of full-rank, it can be shown that H is positive definite with inverse ${\mathbf{H}}^{-1}=\mathbf{I}+{\mathbf{X}}_2{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_2^{\prime }$. Hence, the posterior predictive distribution of [y₂|y₁] is a multivariate t distribution given by

$${\mathbf{y}}_2\mid {\mathbf{y}}_1~S_{n-n_1}(n_1-p,{\mathbf{X}}_2{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_1^{\prime }{\mathbf{y}}_1,s^2{\mathbf{H}}^{-1}).$$ (9)

Theorem 2.1 establishes the small and large sample properties of the estimates based on the MI method.

Theorem 2.1

For the linear regression model (1) with prior (8), let ${\mathbf{y}}_2^{(l)}$, l = 1, …, m, be the samples of y₂ from [y₂|y₁] in Eq. (9). Then

1. the multiple imputation (MI) estimate of β is

   $${\widehat{\beta }}_{MI}=\frac{1}{m}\sum _{l=1}^m{({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}({\mathbf{X}}_1^{\prime }{\mathbf{y}}_1+{\mathbf{X}}_2^{\prime }{\mathbf{y}}_2^{(l)}),$$ (10)
   with mean E(β̂_MI) = β and variance

   $$\mathit{Var}({\widehat{\beta }}_{MI})={\sigma }^2{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}+\frac{{\sigma }^2(n_1-p)}{m(n_1-p-2)}[{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}-{({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}].$$ (11)
2. The MI estimate of σ² is

   $${\widehat{\sigma }}_{MI}^2=\frac{1}{m}\sum _{l=1}^m{y}^{{(l)}^{\prime }}(\mathbf{I}-\mathbf{X}{({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}{\mathbf{X}}^{\prime }){\mathbf{y}}^{(l)}/(n-2),$$ (12)
   with mean

   $$E({\widehat{\sigma }}_{MI}^2)=\frac{(n-p-2)(n_1-p)}{(n-2)(n_1-p-2)}{\sigma }^2$$
   and variance

   $$\mathit{Var}({\widehat{\sigma }}_{MI}^2)=\frac{2(n_1-p){(n-p-2)}^2}{{(n-2)}^2{(n_1-p-2)}^2}{\sigma }^4+a_1{\sigma }^4/m,$$ (13)

   where $a_1=\frac{2(n-n_1)(n-p-2)(n_1-p)(n_1-p+2)}{{(n-2)}^2{(n_1-p-2)}^2(n_1-p-4)}$.

From Theorem 2.1, it can be shown that the MI estimate of β and σ² as well as their variances are asymptotically equivalent to the CC estimates. Furthermore, after some algebra, we can show that when n > n₁ > p+ 2

$$E({\widehat{\sigma }}_{MI}^2\mid {\mathbf{y}}_1)=\frac{n_1(n-p-2)}{(n-2)(n_1-p-2)}{\widehat{\sigma }}_{CC}^2>{\widehat{\sigma }}_{CC}^2,$$

and

$$\text{Var}({\widehat{\sigma }}_{MI}^2)=\frac{n_1^2{(n-p-2)}^2}{{(n_1-p-2)}^2{(n-2)}^2}\text{Var}({\widehat{\sigma }}_{CC}^2)+\frac{2(n-p+2)(n-n_1)}{m(n_1-p+2)(n+2)}{\sigma }^4>\text{Var}({\widehat{\sigma }}_{CC}^2),$$

where ${\widehat{\sigma }}_{CC}^2$ is given in Eq. (3). We note here that throughout this paper, we do not consider the situation in which the number of regression coefficients p increases as n increases, so p is either fixed or increases at a slower rate than n.

Remark 2.1

$E({\widehat{\sigma }}_{MI}^2)$ is independent from m while $\text{Var}({\widehat{\sigma }}_{MI}^2)$ is a function of m, therefore, increasing the number of imputations, m, does not reduce the bias of ${\widehat{\sigma }}_{MI}^2$, but it reduces the variance of ${\widehat{\sigma }}_{MI}^2$.

Remark 2.2

(β̂_MI|y₁)/β̂_CC → 1, $({\stackrel{\sim }{\sigma }}_{MI}^2\mid {\mathbf{y}}_1)/{\stackrel{\sim }{\sigma }}_{CC}^2\to 1$ as m → ∞, where ${\stackrel{\sim }{\sigma }}_{MI}^2=\frac{(n-2)(n_1-p-2)}{(n-p-2)(n_1-p)}{\widehat{\sigma }}_{MI}^2$ and ${\stackrel{\sim }{\sigma }}_{CC}^2=\frac{n_1}{n_1-p}{\widehat{\sigma }}_{CC}^2$ are unbiased estimates of σ². However, this is not true for a fixed m.

We also note that Var(β̂_MI) > Var(β̂_CC) and $\text{Var}({\stackrel{\sim }{\sigma }}_{MI}^2)>\text{Var}({\stackrel{\sim }{\sigma }}_{CC}^2)$, which imply that the MI estimates are less efficient than the CC estimates. This is because the ML estimates for MAR responses in the linear model are the same as the CC estimates, and the ML estimates are most efficient if the model is correct. The extra variability of the MI estimate is induced by the sampling involved in finding the estimator. Even though we are able to improve the MI estimates under the setting of MAR responses in linear regression with small samples, this is not the main aim of this paper. The goal of this paper is to investigate the relationships between MI, ML, and FB approaches. The small sample properties of MI have been studied under more general settings in [1] and [9].

2.2 Maximum likelihood (ML)

As shown in equations (2) and (3), there are closed form estimates of β and σ² using the ML method when the response variable is MAR in the linear model, and those estimates are precisely the CC estimates. However, the ML method is more generally carried out using the EM algorithm, which can be either directly solved when the E-step has a closed form, or it may be obtained using Monte Carlo methods when it does not have a closed form. This latter version of the EM algorithm is referred to as the Monte Carlo EM (MCEM) algorithm and is a more general method of carrying out ML since for most regression models with missing data, the E-step does not have a closed form. We will study ML via MCEM in this subsection in order to study the connections between ML, MI, and FB, and to shed light on examining the properties of the MCEM method when closed form estimates under ML do not exist. ML via MCEM will be the basis of our development in this subsection. In particular, we will derive expressions for the estimates, and their associated variances for both the small sample and large sample situations using MCEM. Following [6] and [8], the Monte Carlo E-Step at the (t + 1)^st EM iteration can be written as

$$\begin{array}{l}Q(\gamma \mid {\gamma }^{(t)})=\int l(\gamma \mid {\mathbf{D}}_{\mathit{obs}},{\mathbf{D}}_{\mathit{mis}},{\gamma }^{(t)})p({\mathbf{D}}_{\mathit{mis}}\mid {\mathbf{D}}_{\mathit{obs}},{\gamma }^{(t)})d{\mathbf{D}}_{\mathit{mis}}\\ \approx \frac{1}{m}\sum _{j=1}^{m}l(\gamma \mid {\mathbf{D}}_{\mathit{obs}},{\mathbf{D}}_{\mathit{mis}}^{(j)},{\gamma }^{(t)}),\end{array}$$

where l(γ | D_obs, D_mis, γ^(t)) is the log-likelihood function based on the complete data given the parameter estimates at the t^th iteration, D_obs = (y₁, X₁, X₂) is the observed data, D_mis = y₂ and the ${\mathbf{D}}_{\mathit{mis}}^{(j)}$’s are the missing values replaced by their j^th sampled values from the full conditional distribution p(D_mis | D_obs, γ^(t)). The M-Step at the (t+1)^st EM iteration maximizes Q(γ | γ^(t)). Standard errors can be calculated by using Louis’s method and the estimated observed information matrix of γ based on Louis’s method is given by

$$I(\widehat{\gamma })=-\ddot{Q}(\widehat{\gamma }\mid \widehat{\gamma })-\frac{1}{m}\sum _{j=1}^{m}S(\widehat{\gamma };{\mathbf{D}}_{\mathit{obs}},{\mathbf{D}}_{\mathit{mis}}^{(j)})S{(\widehat{\gamma };{\mathbf{D}}_{\mathit{obs}},{\mathbf{D}}_{\mathit{mis}}^{(j)})}^{\prime }$$

where γ̂ is the ML estimate at MCEM convergence and Q̈(γ̂|γ̂) is the second derivative matrix of the Q function. The estimate of the asymptotic covariance matrix of γ̂ is therefore [ (γ̂)]⁻¹.

Note that unlike the MI method, which creates m pseudo complete datasets by replacing the missing values with each of the m sets of imputed values, ML via MCEM calculates the estimates from a single dataset and assigns a weight of 1 for complete observations and a weight of 1/m for each sampled value. In order to explore the connections between MI and ML, we consider the imputation distribution [y₂|y₁, β̂] of MCEM, given by

$${\mathbf{y}}_2\mid {\mathbf{y}}_1,\widehat{\beta }~N_{n-n_1}\left({\mathbf{X}}_2{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_1^{\prime }{\mathbf{y}}_1,\left(\frac{n_1-p}{n_1}\right)s^2\mathbf{I}\right),$$ (14)

where $s^2={\mathbf{y}}_1^{\prime }(\mathbf{I}-{\mathbf{X}}_1{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_1^{\prime }){\mathbf{y}}_1/(n_1-p)$. Theorem 2.2 gives the estimates of β and σ² along with their small and large sample properties.

Theorem 2.2

For the linear regression model (1), let ${\mathbf{y}}_2^{(l)}$, l = 1, …, m, be the Gibbs samples of y₂ from [y₂|y₁, β̂] in Eq. (14). Then

1. the maximum likelihood estimate of β using MCEM is

   $${\widehat{\beta }}_{ML\mathit{2}}=\frac{1}{m}\sum _{l=1}^m{({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}({\mathbf{X}}_1^{\prime }{\mathbf{y}}_1+{\mathbf{X}}_2^{\prime }{\mathbf{y}}_2^{(l)}),$$ (15)
   with variance

   $$\mathit{Var}({\widehat{\beta }}_{ML\mathit{2}})={\sigma }^2{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}+\frac{{\sigma }^2(n_1-p)}{{mn}_1}{({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}({\mathbf{X}}_2^{\prime }{\mathbf{X}}_2){({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}.$$ (16)

2. β̂_ML2 is an unbiased estimator of β.

3. The ML estimate of σ² is

   $${\widehat{\sigma }}_{ML\mathit{2}}^2=\frac{1}{m}\sum _{l=1}^m{({\mathbf{y}}^{(l)}-\mathbf{X}{\widehat{\beta }}_{ML\mathit{2}})}^{\prime }({\mathbf{y}}^{(l)}-\mathbf{X}{\widehat{\beta }}_{ML\mathit{2}})/n$$ (17)
   with mean

   $$E({\widehat{\sigma }}_{ML\mathit{2}}^2)=\left(\frac{n_1-p}{n_1}-\frac{(n_1-p)tr(M)}{{\mathit{mnn}}_1}\right){\sigma }^2$$
   and variance

   $$\mathit{Var}({\widehat{\sigma }}_{ML\mathit{2}}^2)=\frac{2(n_1-p)}{n_1^2}{\sigma }^4+\frac{2(n_1-p)}{n^2{n}_1^2}{a}_2{\sigma }^4,$$ (18)

   where $a_2=\frac{{tr}^2(M)+(n_1-p+2)tr(M^2)-2(n_1-p+2)tr(M)}{m^2}+\frac{(n-n_1)(n_1-p+2)-2ntr(M)}{m}$ and $M=(X_2^{\prime }{X}_2){(X^{\prime }X)}^{-1}$.

4. ${\widehat{\sigma }}_{ML\mathit{2}}^2\stackrel{\mathcal{P}}{\to }{\sigma }^2$, as n₁ → ∞ and m → ∞.

Again from Theorem 2.2, it can be easily shown that the estimate of β and its variance based on MCEM are asymptotically equivalent to the CC estimates. In particular, after some algebra, it can be shown that

$$\frac{E({\widehat{\sigma }}_{ML\mathit{2}}^2\mid {\mathbf{y}}_1)}{{\widehat{\sigma }}_{CC}^2}=1-\frac{\text{tr}(M)}{mn}\to 1,$$

as n₁ or m → ∞. The condition that tr(M) → K, 0 ≤ K < ∞, as n → ∞, implies that the information contained in the covariates corresponding to the missing responses is finite compared to the total information in the covariates. The variance of ${\widehat{\sigma }}_{ML\mathit{2}}^2$ in Eq. (18) can also be written as

$$\text{Var}({\widehat{\sigma }}_{ML\mathit{2}}^2)=\text{Var}({\widehat{\sigma }}_{CC}^2)+O\left(\frac{1}{{mn}_1}\right){\sigma }^4,$$

and hence $\text{Var}({\widehat{\sigma }}_{ML\mathit{2}}^2)/\text{Var}({\widehat{\sigma }}_{CC}^2)\to 1$ as n₁ or m go to infinity.

Note that the variance of β̂_ML2 in Eq. (16) is smaller than the variance of β̂_MI in Eq. (11), however, the derivation of Theorem 2.2 is based on the assumption that the imputation distribution of the missing responses yields the ML estimates, which may not be true in practice. Again, note that although we write the estimates of (β, σ²) and their variance as if there were m data sets in order to compare the MI and ML methods, in practice, ML via MCEM calculates the estimates from only one dataset with different weights assigned to the observed and sampled values. In this sense, MCEM augments the data “vertically” and MI augments the data “horizontally”.

Remark 2.3

Both $E({\widehat{\sigma }}_{ML\mathit{2}}^2)$ and $\text{Var}({\widehat{\sigma }}_{ML\mathit{2}}^2)$ are functions of m, the number of Gibbs samples, and therefore, increasing m reduces the bias and variance of ${\widehat{\sigma }}_{ML\mathit{2}}^2$.

2.3 Fully Bayesian (FB)

FB methods for the missing data problem are based on specifying priors for all of the parameters and then the missing data are sampled from their full conditional distributions within the Gibbs sampler. Clearly, ML and MI have Bayesian connections, since ML can be viewed as a large sample Bayesian method, and in many cases, the implementation of Bayesian methods using uniform improper priors on all parameters leads to ML estimates. In this subsection, we consider the FB method under conjugate priors, which yield closed form expressions for the posterior mean and variance of the parameters.

Note that observed data likelihood for MAR responses is the CC likelihood and thus the posterior distribution of γ^* based on the observed data is p(γ^*|y₁, X) ∝ p(y₁|X; γ^*)π(γ^*). Theorem 2.3 provides the properties of the fully Bayesian estimates of β and τ.

Theorem 2.3

For the linear regression model (1), assume that the prior for γ^* = (β, τ) is π(γ^*) = π(β|τ)π(τ), where π(β|τ) = N(μ₀, τ⁻¹Σ₀) and π(τ) = Gamma(δ₀/2, λ₀/2). Then

1. the fully Bayesian estimate of β is

   $${\widehat{\beta }}_{FB}=\frac{1}{m}\sum _{l=1}^m{\beta }^{(m)},$$
   where β_m is the m sample from the posterior distribution

   $$p(\beta \mid D_{\mathit{obs}})~S_p(n_1+{\delta }_0,\stackrel{\sim }{\beta },{\stackrel{\sim }{s}}^2({({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1+{\mathrm{\sum }}_0)}^{-1}))$$

   with β̃ = Λμ₀ + (I − Λ)β̂, $\mathrm{\Lambda }={({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1+{\mathrm{\sum }}_0^{-1})}^{-1}{\mathrm{\sum }}_0^{-1},\widehat{\beta }={({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_1^{\prime }{\mathbf{y}}_1$, and ${\stackrel{\sim }{s}}^2={(n_1+{\delta }_0)}^{-1}[{\mathbf{y}}_1^{\prime }(\mathbf{I}-{\mathbf{X}}_1{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_1^{\prime }){\mathbf{y}}_1+{(\widehat{\beta }-{\mu }_0)}^{\prime }({\mathbf{\Lambda }}^{\prime }{\mathbf{X}}_{\mathbf{1}}^{\prime }{\mathbf{X}}_{\mathbf{1}})(\widehat{\beta }-{\mu }_0)+{\lambda }_0]$.

2. The posterior mean and variance of β based on the observed data are

   $$E(\beta \mid {\mathbf{D}}_{\mathit{obs}})=\stackrel{\sim }{\beta }$$
   and

   $$\mathit{Var}(\beta \mid {\mathbf{D}}_{\mathit{obs}})=(n_1+{\delta }_0){\stackrel{\sim }{s}}^2{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1+{\mathrm{\sum }}_0)}^{-1}/(n_1+{\delta }_0-2).$$
3. The fully Bayesian estimate of τ = 1/σ² is

   $${\widehat{\tau }}_{FB}=\frac{1}{m}\sum _{l=1}^m{\tau }^{(m)}$$

   where p(τ |D_obs) ~ Gamma((n₁ + δ₀)/2, (n₁ + δ₀)s̃²/2) with s̃² defined in (i).

4. The posterior mean and variance of τ are

   $$E(\tau \mid {\mathbf{D}}_{\mathit{obs}})=1/s^2\mathit{and}\mathit{Var}(\tau \mid {\mathbf{D}}_{\mathit{obs}})=2{(n_1+{\delta }_0)}^{-1}{\stackrel{\sim }{s}}^{-4}.$$

The proof of Theorem 2.3 is straightforward and can be found in most Bayesian textbooks. We state it as a Theorem here only to be consistent with other sections.

Remark 2.4

When the prior for γ^* is an improper prior, π(γ^*) ∝ τ⁻¹, s̃² reduces to ${\mathbf{y}}_1^{\prime }(\mathbf{I}-{\mathbf{X}}_1{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_1^{\prime }){\mathbf{y}}_1/n_1$ and the posterior mean and variance of (β, τ) are then equal to the CC estimates given in equations (2) and (3).

Therefore, the CC analysis is recommended over the MI, MCEM, and FB methods for MAR responses in the linear regression model, unless additional information is available to specify informative priors for the MI and FB methods, or the imputation model of MI includes covariates not specified in the analysis model. On the other hand, the loss of efficiency of MI, MCEM, and FB methods can be significantly reduced by increasing the number of imputations for MI or the number of Gibbs samples for MCEM and FB.

3. SIMULATION STUDY

In this section, we will compare inferences about β using the four methods, MI, CC, MCEM and FB using the formulas we developed in Section 2 for a small sample size n and various values of m for MI and MCEM.

We generate N = 1,000 replicates with each simulation consisting of n = 250 independent response variables y_i from the linear regression model as

$$y_i={\beta }_0+{\beta }_1{x}_{i1}+{\beta }_2{x}_{i2}+e_i,$$

where the e_i’s are independent and identically distributed (i.i.d.) as N(0, σ²). The values chosen for the parameters are (β₀, β₁, β₂) = (1.0, 1.5, −1.0) and σ² = 1.0. The covariates (x_i1, x_i2) are i.i.d. and simulated as

$$x_{i1}~N(1.0,1.0)\text{and}{x}_{i2}\mid x_{i1}~N({\alpha }_0+{\alpha }_1{x}_{i1},{\sigma }_x^2)$$

where (α₀, α₁) = (1.0, 1.0) and ${\sigma }_x^2=1.0$.

We assume that y_i is MAR for some i’s and x_i1 and x_i2 are completely observed throughout. In this setting, the model for the missing data mechanism of y_i is given by

$$p(r_{i1}=1\mid x_{i1},x_{i2},\varphi )=\frac{exp({\varphi }_0+{\varphi }_1{x}_{i1}+{\varphi }_2{x}_{i2})}{1+exp({\varphi }_0+{\varphi }_1{x}_{i1}+{\varphi }_2{x}_{i2})},$$

where (ϕ₀₁, ϕ₁₁, ϕ₂₁) = (−5.5, 1.0, 1.0) and r_i1 = 1 if y_i is missing, 0 otherwise.

Table 1 gives the results using the four methods, MI, CC, MCEM and FB, and also gives the estimates based on the full data (i.e., no missing values), as these estimates serve as a benchmark for comparison. From the N = 1,000 simulations, the average number of observations with y_i missing is 19%. We chose the number of samples m equal to 30 and 3 in both the MI and MCEM methods in order to compare the results. [13] note that for proper MI, m, the number of imputed datasets, can be as small as m = 5. However, m in MCEM, the number of Gibbs samples, is usually large, say m = 100 or more, in order to accurately represent the sampling distribution in the E-step, especially in complex models with large missing data fractions. This is consistent with the simulation results. When m = 3 in the MI method, the two forms of the variance estimates give similar coverage rates because both of them adjust well for small m, and when m = 3 in MCEM, equations (16) and (18) give much better coverage rates than the Louis method. On the other hand, for the MI method, considering that the estimates with m = 30 always have smaller variances than the estimates with m = 3 with better coverage probabilities, larger values of m may need to be considered if the computational burden is not heavy. The simulation results confirm the theorems in Section 2, and show that the three methods (MI, ML via MCEM, FB) produce consistent estimates with valid inferences and all are asymptotically equivalent to the CC estimates when the response variable is MAR.

Table 1.

Simulation with MAR responses in the linear regression model. The 95% CR is the coverage rate of a 95% confidence interval. γ̂_F (full data), ${\widehat{\gamma }}_{MI}^{\ast }$ (Multiple Imputation with covariance matrix as Eq. (7), ${\widehat{\gamma }}_{MI}^{†}$ (Multiple Imputation with covariance matrix as Eq. (11) and Eq. (13), γ̂_CC (CC estimates), ${\widehat{\gamma }}_{ML2}^{\ast }$ (MCEM with covariance matrix using Louis’s method, ${\widehat{\gamma }}_{ML2}^{†}$ (MCEM with covariance matrix as Eq. (16) and Eq. (18), and γ̂_FB (fully Bayesian). The number m is defined in the section discussing the corresponding method

Method	m	Estimate (var(×10−3))[95% CR]
		β0 = 1.00	β1 = 1.50	β2 = −1.00	σ2 = 1.0
γ̂F	-	0.994(12)[94]	1.499(8)[96]	−0.998(4)[96]	0.989(8)[94]
${\widehat{\gamma }}_{MI}^{\ast }$	30	0.995(14)[95]	1.496(11)[95]	−0.998(5)[95]	1.000(10)[95]
${\widehat{\gamma }}_{MI}^{†}$	30	0.995(14)[95]	1.496(10)[95]	−0.998(5)[95]	1.000(10)[95]
${\widehat{\gamma }}_{MI}^{\ast }$	3	0.993(14)[95]	1.496(11)[95]	−0.998(6)[94]	1.002(11)[94]
${\widehat{\gamma }}_{MI}^{†}$	3	0.993(14)[95]	1.496(11)[96]	−0.998(6)[95]	1.002(10)[94]
γ̂CC	-	0.995(14)[95]	1.497(10)[95]	−0.9984(5)[95]	0.987(10)[94]
${\widehat{\gamma }}_{ML2}^{\ast }$	30	0.995(14)[95]	1.497(10)[95]	−0.998(5)[95]	0.987(10)[94]
${\widehat{\gamma }}_{ML2}^{†}$	30	0.995(14)[95]	1.497(10)[95]	−0.998(5)[95]	0.987(10)[94]
${\widehat{\gamma }}_{ML2}^{\ast }$	3	0.993(14)[92]	1.497(10)[90]	−0.998(5)[89]	0.987(10)[90]
${\widehat{\gamma }}_{ML2}^{†}$	3	0.993(14)[95]	1.497(10)[95]	−0.998(5)[93]	0.987(10)[93]
γ̂FB	-	0.995(14)[95]	1.497(10)[95]	−0.998(5)[95]	0.992(10)[95]

4. LIVER CANCER DATA

To further illustrate the CC, MI, ML and FB methods, we consider a real dataset on n = 174 patients from two Eastern Cooperative Oncology Group clinical trials, EST 2282 [3] and EST 1286 [4]. We are interested in how the number of cancerous liver nodes (CNT) when entering the trials is predicted by six other baseline characteristics: body mass index (BMI, defined as weight in kilograms divided by the square of height in meters); age (in years); associated jaundice (yes, no); and time since diagnosis of the disease (TSD, in weeks). Thirty four out of 174 (19.5%) patients have a missing response variable (CNT). Throughout, we assume that the response variable CNT is MAR. The square root transformation on CNT and TSD was used in the analyses.

We use linear regression to model the response variable, $\sqrt{\mathit{CNT}}$, as $\sqrt{{\mathit{CNT}}_i}={\beta }_0+{\beta }_1{\text{BMI}}_i+{\beta }_2{\text{Age}}_i+{\beta }_3{\text{Jaundice}}_i+{\beta }_4\sqrt{{\mathit{TSD}}_i}+e_i$, where the e_i’s are i.i.d. normally distributed as e_i~ N(0, σ²).

Table 2 show the results for the CC analysis, MI with m = 3 and m = 30 using equations (11) and (13), MCEM with m = 30 and m = 300 using equations (16) and (18), and the FB method discussed in Section 3. Moreover, the variance estimates are 0.775, 0.715, 0.715, 0.724, 0.739, and 0.712 for CC, MI with m = 3 and m = 30, ML via MCEM with m = 30 and m = 300, and FB, respectively. As shown in the table, the MI, MCEM and FB methods yield very similar estimates with very little differences from the CC estimates. In particular, the p-values of all the covariates except Age are smaller with larger m. The results show that the age of the patients is significantly associated with the number of cancerous liver nodes controlling for body mass index, associated jaundice, and time since diagnosis.

Table 2.

Estimates for liver cancer data

Effect	Method	β̂	Std	P-value
Intercept	CC	2.826	0.445	< .001
	MI m = 3	2.801	0.446	< .001
	MI m = 30	2.763	0.408	< .001
	ML2 m = 30	2.782	0.440	< .001
	ML2 m = 300	2.766	0.395	< .001
	FB	2.770	0.408	< .001
BMI	CC	−0.008	0.015	0.595
	MI m = 3	−0.007	0.015	0.632
	MI m = 30	−0.005	0.013	0.726
	ML2 m = 30	−0.005	0.014	0.740
	ML2 m = 300	−0.005	0.013	0.715
	FB	−0.005	0.013	0.720
Age	CC	−0.012	0.005	0.018
	MI m = 3	−0.012	0.005	0.016
	MI m = 30	−0.012	0.005	0.007
	ML2 m = 30	−0.012	0.005	0.012
	ML2 m = 300	−0.012	0.004	0.006
	FB	−0.012	0.005	0.008
Jaundice	CC	0.190	0.152	0.212
	MI m = 3	0.231	0.146	0.115
	MI m = 30	0.217	0.134	0.107
	ML2 m = 30	0.210	0.144	0.147
	ML2 m = 300	0.204	0.129	0.116
	FB	0.204	0.134	0.129
$\sqrt{\mathit{TSD}}$	CC	0.002	0.034	0.964
	MI m = 3	0.000	0.034	0.998
	MI m = 30	−0.002	0.032	0.957
	ML2 m = 30	−0.003	0.034	0.933
	ML2 m = 300	−0.003	0.030	0.926
	FB	−0.002	0.032	0.943

5. DISCUSSION

It is known in the missing data literature that when the responses are MAR, the CC analysis is unbiased and efficient. However, MI, ML via MCEM, and FB are still commonly used in practice in this setting. This may be due to the fact that (a) the unbiasedness and efficiency properties of the CC method in this setting is not known to general researchers; (b) MI, ML, and FB, as well as some other methods including parametric fractional imputation [10] and empirical likelihood [18] outperform CC in a general setting. To overcome these barriers, it is important to inform researchers and practitioners about these important results. Moreover, we also showed in this paper that the loss of efficiency of MI, ML via MCEM, and FB can be significantly reduced by increasing the number of imputations for MI and the number of Gibbs samples for MCEM and FB. It would be of interest to extend our theoretical results to MAR responses for models other than linear regression. This is a topic of current investigation. It would also be interesting to accommodate missingness in the predictors. Unfortunately, even for linear regression models with normally distributed MAR covariates, no closed form expressions are available for the estimates of the three methods, which makes the comparisons between the methods very hard. A special scenario of it, assuming unit variances for response variable and missing covariates, was investigated in [2] for the maximum likelihood approach.

APPENDIX

Proof of Lemma 2.1

If the n × 1 random vector z has a multivariate t distribution as S_n(v, μ, V), then we can write $\mathbf{z}=\mathbf{x}/\sqrt{y/v}+\mu$, where x is an n × 1 random vector that has a multivariate normal distribution N(0, V), y is a random variable which has a ${\chi }_v^2$ distribution, x and y are independent. Therefore, (i) is straightforward and, to get (ii), we have

$$\begin{array}{l}E({\mathbf{zz}}^{\prime }\mathbf{Az})=E\left[E\left(\frac{{\mathbf{xx}}^{\prime }\mathbf{Ax}}{{(y/v)}^{3/2}}+\frac{\mathbf{x}{\mu }^{\prime }\mathbf{Ax}+{\mathbf{xx}}^{\prime }\mathbf{A}\mu +\mu {\mathbf{x}}^{\prime }\mathbf{Ax}}{(y/v)}+\frac{\mu {\mu }^{\prime }\mathbf{Ax}+\mu {\mathbf{x}}^{\prime }\mathbf{A}\mu }{{(y/v)}^{1/2}}+\mu {\mu }^{\prime }\mathbf{A}\mu \mid \mathbf{y}\right)\right]\\ =\mathbf{0}+E[{vy}^{-1}({\mathbf{VA}}^{\prime }\mu +\mathbf{VA}\mu +\mu \text{tr}(\mathbf{AV}))]+\mathbf{0}+\mu {\mu }^{\prime }\mathbf{A}\mu \\ =\frac{v}{v-2}[{\mathbf{VA}}^{\prime }\mu +\mathbf{VA}\mu +\mu \text{tr}(\mathbf{AV})]+\mu {\mu }^{\prime }\mathbf{A}\mu .\end{array}$$

We substitute $\mathbf{z}=\mathbf{x}/\sqrt{y/v}+\mu$ and calculate the double expectation in the first equality. Because of independence between x and y, we can substitute in expressions for multivariate normal moments in the second equality, and therefore

$$\begin{array}{l}E[({\mathbf{z}}^{\prime }\mathbf{Az})({\mathbf{z}}^{\prime }\mathbf{Bz})]=E[E({(\mathbf{x}/\sqrt{y/v}+\mu )}^{\prime }\mathbf{A}(\mathbf{x}/\sqrt{\mathbf{y}/\mathbf{v}}+\mu ){(\mathbf{x}/\sqrt{\mathbf{y}/\mathbf{v}}+\mu )}^{\prime }\times \mathbf{B}(\mathbf{x}/\sqrt{y/v}+\mu )\mid \mathbf{y})]\\ =E[{(v/y)}^{2}E({\mathbf{x}}^{\prime }{\mathbf{Axx}}^{\prime }\mathbf{Bx})]+E[{(v/y)}^{3/2}\times \mathbf{0}]\\ +E[(v/y)E({\mathbf{x}}^{\prime }\mathbf{Ax}{\mu }^{\prime }\mathbf{B}\mu +{\mu }^{\prime }\mathbf{Ax}{\mu }^{\prime }\mathbf{Bx}+{\mu }^{\prime }\mathbf{Ax}{\mathbf{x}}^{\prime }\mathbf{B}\mu \\ +{\mu }^{\prime }\mathbf{Ax}{\mathbf{x}}^{\prime }\mathbf{B}\mu +{\mathbf{x}}^{\prime }\mathbf{A}\mu {\mu }^{\prime }\mathbf{Bx}+{\mathbf{x}}^{\prime }\mathbf{A}\mu {\mathbf{x}}^{\prime }\mathbf{B}\mu \\ +{\mu }^{\prime }\mathbf{A}\mu {\mathbf{x}}^{\prime }\mathbf{Bx})]+\mathbf{E}[{(\mathbf{v}/\mathbf{y})}^{\mathbf{1}/\mathbf{2}}\times \mathbf{0}]+{\mu }^{\prime }\mathbf{A}\mu {\mu }^{\prime }\mathbf{B}\mu \\ =\frac{v^2}{(v-2)(v-4)}[\text{tr}(\mathbf{AV})tr(\mathbf{BV})+\text{tr}(\mathbf{AVBV})\\ +\text{tr}({\mathbf{AVB}}^{\prime }\mathbf{V})]+\frac{v}{(v-2}[\text{tr}(\mathbf{AV}){\mu }^{\prime }\mathbf{B}\mu +{\mu }^{\prime }\mathbf{A}\mu \text{tr}(\mathbf{BV})\\ +{\mu }^{\prime }(\mathbf{AVB}+{\mathbf{A}}^{\prime }\mathbf{VB}+\mathbf{AV}{\mathbf{B}}^{\prime }+{\mathbf{A}}^{\prime }{\mathbf{VB}}^{\prime })\mu ]\\ +{\mu }^{\prime }\mathbf{A}\mu {\mu }^{\prime }\mathbf{B}\mu .\end{array}$$

In the second equality, the two zero components correspond to the first and third moments of x. In the last equality, we use the second and fourth moments of x, which are available in [5] with a modification for nonsymmetric matrices A and B.

Proof of Theorem 2.1

(i) and (ii): It is straightforward to get the estimate of β as in Eq. (10). It is also straightforward to use double expectations to get E(β̂_MI) = β. To find the variance of β̂_MI, we have

$$\begin{array}{l}\text{Var}({\widehat{\beta }}_{MI})=E(\text{Var}(\widehat{\beta }\mid {\mathbf{y}}_1))+\text{Var}(E(\widehat{\beta }\mid {\mathbf{y}}_1))\\ =\frac{n_1-p}{m(n_1-p-2)}\times E({({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}{\mathbf{X}}_2^{\prime }{s}^2(\mathbf{I}+{\mathbf{X}}_2{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_2^{\prime }){\mathbf{X}}_2{({\mathbf{X}}^{\prime }\mathbf{X})}^{-1})+\text{Var}({({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}{\mathbf{X}}_1^{\prime }{\mathbf{y}}_1+{({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}({\mathbf{X}}_2^{\prime }{\mathbf{X}}_2){({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_1^{\prime }{\mathbf{y}}_1)\\ =\frac{{\sigma }^2(n_1-p)}{m(n_1-p-2)}{({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}({\mathbf{X}}_2^{\prime }{\mathbf{X}}_2){({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}+\text{Var}({({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_1^{\prime }{\mathbf{y}}_1)\\ =\frac{{\sigma }^2(n_1-p)}{m(n_1-p-2)}{({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}({\mathbf{X}}_2^{\prime }{\mathbf{X}}_2){({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}+{\sigma }^2{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}.\end{array}$$

(iii) and (iv): It is straightforward to get the estimate of σ² as in Eq. (12). In order to find $E({\widehat{\sigma }}_{MI}^2)$, we write $(n-2){\widehat{\sigma }}_{MI}^2={\mathbf{y}}_1^{\prime }{\mathbf{Ay}}_1-2{\mathbf{y}}_1^{\prime }\mathbf{B}{\overline{\mathbf{y}}}_2+\frac{1}{m}{\sum }_{l=1}^m{\mathbf{y}}_2^{{(l)}^{\prime }}{\mathbf{Cy}}_2^{(l)}$, where ${\overline{\mathbf{y}}}_2={\sum }_{l=1}^m{\mathbf{y}}_2^{(l)}/m,\mathbf{A}=\mathbf{I}-{\mathbf{X}}_1{({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}{\mathbf{X}}_1^{\prime },\mathbf{B}={\mathbf{X}}_1{(\mathbf{XX})}^{-1}{\mathbf{X}}_2^{\prime }$, and $\mathbf{C}=\mathbf{I}-{\mathbf{X}}_2{({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}{\mathbf{X}}_2^{\prime }$. Let ${\mathbf{P}}_{{\mathbf{X}}_1}=\mathbf{I}-{\mathbf{X}}_1{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_1^{\prime }$ and $\mathbf{D}={\mathbf{X}}_1({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)({\mathbf{X}}_2^{\prime }{\mathbf{X}}_2){({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}{\mathbf{X}}_1^{\prime }$. Then after some algebra, we have

$$\begin{array}{l}E({\widehat{\sigma }}_{MI}^2)=E[E({\sigma }_{MI}^2\mid {\mathbf{y}}_1)]\\ =E[{\mathbf{y}}_1^{\prime }(\mathbf{A}-\mathbf{D}+\frac{(n-n_1)}{(n_1-p-2)}{\mathbf{P}}_{{\mathbf{X}}_1}){\mathbf{y}}_1]/(n-2)\\ =\frac{(n-p-2)}{(n_1-p-2)(n-2)}E({\mathbf{y}}_1^{\prime }{\mathbf{P}}_{{\mathbf{X}}_1}{\mathbf{y}}_1)\\ =\frac{(n-p-2)(n_1-p)}{(n-2)(n_1-p-2)}{\sigma }^2\\ \to {\sigma }^2,\end{array}$$

by noting that y₁ ~ N(X₁β, σ²I), A − D = P_X1, ${\mathbf{P}}_{{\mathbf{X}}_1}^2={\mathbf{P}}_{{\mathbf{X}}_1}$ and ${\mathbf{X}}_1^{\prime }{\mathbf{P}}_{{\mathbf{X}}_1}{\mathbf{X}}_1=\mathbf{0}$.

To find $\text{Var}({\widehat{\sigma }}_{MI}^2)$, we write $\text{Var}((n-2){\widehat{\sigma }}_{MI}^2)=\text{Var}[E((n-2){\widehat{\sigma }}_{MI}^2\mid {\mathbf{y}}_1)]+E[\text{Var}((n-2){\widehat{\sigma }}_{MI}^2\mid {\mathbf{y}}_1)]$. First we have

$$\begin{array}{l}\text{Var}[E((n-2){\widehat{\sigma }}_{MI}^2\mid {\mathbf{y}}_1)]=\frac{{(n-p-2)}^2}{{(n_1-p-2)}^2}\text{Var}((n_1-p)s^2)\\ =\frac{2(n_1-p){(n-p-2)}^2}{{(n_1-p-2)}^2}{\sigma }^4.\end{array}$$

Then we obtain

$$\begin{array}{l}\text{Var((}n-2){\widehat{\sigma }}_{MI}^2\mid {\mathbf{y}}_1)=\text{Var}(-2{\mathbf{y}}_1^{\prime }\mathbf{B}{\overline{\mathbf{y}}}_2+\frac{1}{m}\sum _{l=1}^m{\mathbf{y}}_2^{{(l)}^{\prime }}{\mathbf{Cy}}_2^{(l)}\mid {\mathbf{y}}_1)\\ =\frac{4{\mathbf{y}}_1^{\prime }\mathbf{B}\text{Var}({\mathbf{y}}_2\mid {\mathbf{y}}_1){\mathbf{B}}^{\prime }{\mathbf{y}}_1}{m}+\frac{\text{Var}({\mathbf{y}}_2^{\prime }{\mathbf{Cy}}_2\mid {\mathbf{y}}_1)}{m}-\sum _{l=1}^m\sum _{k=1}^m\frac{4\text{Cov}({\mathbf{y}}_1^{\prime }{\mathbf{By}}_2^{(k)},{\mathbf{y}}_2^{(l)}{\mathbf{Cy}}_2^{(l)}\mid {\mathbf{y}}_1)}{m^2}\\ =\frac{4(n_1-p)}{m(n_1-p-2)}{\mathbf{y}}_1^{\prime }\mathbf{B}\mathrm{\sum }{\mathbf{B}}^{\prime }{\mathbf{y}}_1+\frac{1}{m}\text{Var}({\mathbf{y}}_2^{\prime }{\mathbf{Cy}}_2\mid {\mathbf{y}}_1)-\frac{4}{m}\text{Cov}({\mathbf{y}}_1^{\prime }{\mathbf{By}}_2,{\mathbf{y}}_2{\mathbf{Cy}}_2\mid {\mathbf{y}}_1),\end{array}$$

where $\mathrm{\sum }=s^2(\mathbf{I}+{\mathbf{X}}_2{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_2^{\prime })$ and s² is defined as in Theorem 2.1. The last equality holds because ${\mathbf{y}}_2^{(j)}$ is independent of ${\mathbf{y}}_2^{(k)}$ given y₁, when j ≠ k. Then we use Lemma 2.1 and get

$$\begin{array}{l}E[\text{Var}((n-2){\widehat{\sigma }}_{MI}^2\mid {\mathbf{y}}_1)]=\frac{2(n_1-p)(n-n_1)(n-p-2)}{{(n_1-p-2)}^2(n_1-p-4)}E\left[\frac{{\mathbf{y}}_1^{\prime }{\mathbf{P}}_{{\mathbf{X}}_1}{\mathbf{y}}_1{\mathbf{y}}_1^{\prime }{\mathbf{P}}_{{\mathbf{X}}_1}{\mathbf{y}}_1}{m}\right]=\frac{2(n_1-p)(n-n_1)(n-p-2)(n_1-p+2)}{m{(n_1-p-2)}^2(n_1-p-4)}{\sigma }^4,\end{array}$$

and therefore, we can get $\text{Var}({\widehat{\sigma }}_{MI}^2)$ as in (13). Since $\text{Var}({\widehat{\sigma }}_{MI}^2)\to 0,{\widehat{\sigma }}_{MI}^2\stackrel{\mathcal{P}}{\to }{\sigma }^2$.

Proof of Theorem 2.2

(i) and (ii): It is straightforward to get the estimate of β as in Eq. (15). It is straightforward to use double expectations to get E(β̂_MI) = β. To find the variance of β̂_ML2, we have

$$\begin{array}{l}\text{Var}({\widehat{\beta }}_{ML\mathit{2}})=E(\text{Var}(\widehat{\beta }\mid {\mathbf{y}}_1))+\text{Var}(E(\widehat{\beta }\mid {\mathbf{y}}_1))\\ =\frac{n_1-p}{{mn}_1}E({({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}{\mathbf{X}}_2^{\prime }{s}^2{\mathbf{IX}}_2{({\mathbf{X}}^{\prime }\mathbf{X})}^{-1})+\text{Var}({({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}{\mathbf{X}}_1^{\prime }{\mathbf{y}}_1+{({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}({\mathbf{X}}_2^{\prime }{\mathbf{X}}_2){({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_1^{\prime }{\mathbf{y}}_1)\\ =\frac{{\sigma }^2(n_1-p)}{{mn}_1}{({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}({\mathbf{X}}_2^{\prime }{\mathbf{X}}_2){({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}+\text{Var}({({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}{\mathbf{X}}_1^{\prime }{\mathbf{y}}_1)\\ =\frac{{\sigma }^2(n_1-p)}{{mn}_1}{({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}({\mathbf{X}}_2^{\prime }{\mathbf{X}}_2){({\mathbf{X}}^{\prime }\mathbf{X})}^{-1}+{\sigma }^2{({\mathbf{X}}_1^{\prime }{\mathbf{X}}_1)}^{-1}.\end{array}$$

(iii) and (iv): It is straightforward to get the estimate of σ² as in Eq. (17). In order to find $E({\widehat{\sigma }}_{MI}^2)$, we write $n{\widehat{\sigma }}_{ML\mathit{2}}^2={\mathbf{y}}_1^{\prime }{\mathbf{Ay}}_1+{\sum }_{l=1}^m{\mathbf{y}}_2^{{(l)}^{\prime }}{\mathbf{y}}_2^{(l)}/m-2{\mathbf{y}}_1^{\prime }\mathbf{B}{\overline{\mathbf{y}}}_2-{\overline{\mathbf{y}}}_2^{{(l)}^{\prime }}(\mathbf{I}-\mathbf{C}){\overline{\mathbf{y}}}_2^{(l)}$, where the symbols are same as the proof of Theorem 2.2. Then we have

$$\begin{array}{l}E({\widehat{\sigma }}_{ML\mathit{2}}^2)=E[E({\widehat{\sigma }}_{MI}^2\mid {\mathbf{y}}_1)]\\ =E\left[{\mathbf{y}}_1^{\prime }(\mathbf{A}-\mathbf{D}+\frac{n-n_1}{n_1}{\mathbf{P}}_{{\mathbf{X}}_1}-\frac{\text{tr}(\mathbf{M})}{{mn}_1}{\mathbf{P}}_{{\mathbf{X}}_1}){\mathbf{y}}_1\right]/n\\ =\left(\frac{n_1-p}{n_1}-\frac{(n_1-p)\text{tr}(M)}{{\mathit{mnn}}_1}\right){\sigma }^2\\ \to {\sigma }^2,\end{array}$$

where $M=(X_2^{\prime }{X}_2){(X^{\prime }X)}^{-1}$. To find $\text{Var}({\widehat{\sigma }}_{ML\mathit{2}}^2)$, we have

$$\text{Var}((n+2){\widehat{\sigma }}_{MI}^2)=\text{Var}[E((n+2){\widehat{\sigma }}_{MI}^2\mid {\mathbf{y}}_1)]+E[\text{Var}((n+2){\widehat{\sigma }}_{MI}^2\mid {\mathbf{y}}_1)].$$

Then we have

$$\begin{array}{l}\text{Var}[E(n{\widehat{\sigma }}_{ML\mathit{2}}^2\mid {\mathbf{y}}_1)]=\text{Var}\left(\left(\frac{n}{n_1}-\frac{\text{tr}(M)}{mn}\right){\mathbf{y}}_1^{\prime }{\mathbf{P}}_{{\mathbf{X}}_1}{\mathbf{y}}_1\right)\\ =2{\left(\frac{n}{n_1}-\frac{\text{tr}(M)}{mn}\right)}^2(n_1-p){\sigma }^4.\end{array}$$

Using Chapter 10.9 of [5] and after some algebra, we get

$$\text{Var}(n{\widehat{\sigma }}_{ML\mathit{2}}^2\mid {\mathbf{y}}_1)=2\left(\frac{n-n_1}{{mn}_1^2}-\frac{\text{tr}({\mathbf{M}}^2)}{m^2{n}_1^2}-\frac{2\text{tr}(\mathbf{M})}{m^2{n}_1^2}\right){({\mathbf{y}}_1^{\prime }{\mathbf{P}}_{{\mathbf{X}}_1}{\mathbf{y}}_1)}^2.$$

Therefore, after some algebra, we can get $\text{Var}({\widehat{\sigma }}_{ML\mathit{2}}^2)$ as in Eq. (18), and therefore ${\widehat{\sigma }}_{ML\mathit{2}}^2\stackrel{\mathcal{P}}{\to }{\sigma }^2$.