Bayesian Frequentist hybrid Model wth Application to the Analysis of Gene Copy Number Changes

Abstract

Gene copy number (GCN) changes are common characteristics of many genetic diseases. Comparative genomic hybridization (CGH) is a new technology widely used today to screen the GCN changes in mutant cells with high resolution genome-wide. Statistical methods for analyzing such CGH data have been evolving. Existing methods are either frequentist’s, or full Bayesian. The former often has computational advantage, while the latter can incorporate prior information into the model, but could be misleading when one does not have sound prior information. In an attempt to take full advantages of both approaches, we develop a Bayesian-frequentist hybrid approach, in which a subset of the model parameters is inferred by the Bayesian method, while the rest parameters by the frequentist’s. This new hybrid approach provides advantages over those of the Bayesian or frequentist’s method used alone. This is especially the case when sound prior information is available on part of the parameters, and the sample size is relatively small. Spatial dependence and false discovery rate are also discussed, and the parameter estimation is efficient. As an illustration, we used the proposed hybrid approach to analyze a real CGH data.

1. Introduction

Recent evidences show that gene copy number (GCN) amplifications and deletions are common characteristics of many diseases caused by genetic mutations. For example, GCN can be elevated in cancer cells as demonstrated in the epidermal growth factor receptor (EGFR) gene in patients with non-small cell lung cancer (Cappuzzo et al. 2005) and also higher copy number of CCL3L1 has been associated with susceptibility to human HIV infection (Gonzalez et al. 2005). Thus, identifying these genetic gains and losses provides useful information about the genesis of specific diseases. GCN analysis among normal people within the human genome is also of interest. However, these genetic characteristics are usually not directly observable. Recent technology development on comparative genomic hybridization (CGH) array allows to screen putative mutant cells with high resolution for genome-wide analysis of copy number changes (Solinas et al., 1997; Pinkel et al., 1998; Snijders et al., 2001; Hupe et al., 2004; Myers et al., 2004; Olshen et al., 2004; Huang et al., 2005; Lingjaerde et al., 2005; Wang et al., 2005; Guha et al., 2008). The development of CGH has proved to be an efficient tool to study the genetic mutations responsible for disease development. The assay was first developed by Kallioniemi et al., (1992). The basic strategy of the technique is to use genomic DNA from cancer cells labeled with one fluorochrome and genomic DNA from normal reference cells labeled with a second fluorochrome and co-hybridize the labeled samples to a metaphase spread from a normal reference cell. The ratio, of matched pair on age, gender etc, of the two fluorochrome intensities is then calculated and regions where the cancer DNA are amplified or deleted are readily detected on the metaphase spread. After hybridization, emission from each of the two uorescent dyes is measured, and the signal intensity ratios are indicative of the relative copy number of the two samples. This technique not only gives us information about copy number gains and losses in the disease genomic DNA but also allows us to identify the specific chromosomes and the regions of the chromosomes where these changes have occurred.

However, the CGH data does not provide direct measurements of the GCN changes. Hence, various statistical approaches for analyzing and describing results from these experiments have been developed.

The existing methods are either frequentist’s (Hodgson et al., 2001; Pollack et al., 2002; Cheng et al., 2003; Wang et al., 2004; Yuan et al., 2008), or full Bayesian (Barash and Friedman, 2002; Daruwala et al., 2004; Broet and Richardson, 2005). The former often has computational advantage, while the latter can incorporate useful prior information into the model, but can also be misleading when such prior information is not reliable. For finite dimensional parametric models, it is known that under very general conditions the frequentist’s MLE and Bayesian estimates are asymptotically equivalent, in that the convergence to the truth of one implies that of the other, and both of them have the inverse Fisher information as asymptotic variance(matrix), this is the Bernstainvon Mises theorem. But in practice, sample size is always finite, and the two methods differ in many aspects. For Bayesian inference, the choice of prior is a source of debating, and the computation is much more demanding, that’s the main reason why the real practice is dominated by the frequentist’s methods. However, if the prior represents a good summary of the experience about the underlying parameters, a Bayesian analysis do have small sample advantages over the frequentist’s. For example, if X ~ Binomial(m, θ), π(·) = U(0, 1), then given data x, the posterior is π( θ|x) = Beta(x + 1, m − x + 1). The frequentist estimate of θ is the MLE θ̂ = x. Under the squared error loss, the Bayes estimate of θ is the posterior mean θ̌ = (x + 1)/(m + 2). If x = 0 or 1, the MLE of θ is 0 or 1, while the Bayes estimate is only close to 0 or 1, which is more reasonable. But if we do not have good prior experience, or the prior density does not characterize the parameter distribution reasonably, then the result may be mis-leading, as almost all Bayesian estimates are biased (Blackwell and Grischick, 1954; Blackwell and Bickel, 1967), and computationally more demanding than the frequentist method, so in this case the MLE is more preferred. In this case one may put a non-informative prior on the part of parameters which we do not have confident prior information, to formulate a full Bayesian procedure. But the hybrid model has more flexibility, for example, as we can see in Section 2, that parameter estimates under the hybrid model can be formulated as a EM algorithm, which significantly simplify the computation for the mixture model, in contrast to that of a full Bayesian model.

Often in multi-dimensional parametric analysis, the parameters of interest can be partitioned into two subsets, and in practice we may have good knowledge in one subset of the parameters, and not enough experience on the other subset. In this case, a Bayesian analysis on one subset will benefit from the prior information, but for the other subet a frequentist method is preferred. Thus a hybrid Bayesian-frequentist method for the analysis of full parameters will be advantageous and is proposed in Yuan (2009). In CGH data analysis, each data sequence has three unobserved status to be inferred: copy number deletion, amplification, or normal (no deletion nor amplification). Often, for specified disease and chromosome, the experimenter can accumulate knowledge about each status, but relatively lack of experience on the other unknown nature which constitute part of the model, thus the above method fits the case well. In section 2 we describe the method in detail, then illustrate its application to a real CGH data in section 3. Finally a brief discussion is given in section 4.

2. The Method

2.1 The Method in General Setting

We first give a general description of the hybrid method. The parameters is partitioned as θ = (α, β), on part of the parameter α, the Bayesian method is preferred, and on the other part β, the MLE is more favored. Let x = (x₁, …, x_n) be the sample i.i.d. with density function f(x|α, β), the likelihood is $f({\mathbf{x}}_n\mid \mathit{\alpha },\mathit{\beta })={\prod }_{i=1}^{n}f(x_i\mid \mathit{\alpha },\mathit{\beta })$. Let π(α) be the prior density for α, and π(α|x_n, β) = f(x_n|α, β)π(α)/m(x_n|β) the posterior density of α for given β, where m(x_n|β) = ∫f(x_n|α, β) × π(α)dα is the mixture over α at β. Let be the decision space, d(x_n) ∈ be a decision rule for inferring α, W(d(x_n), α) be the loss function for α, R(d, α, β) = E_(α,β)W (d(x_n), α) be the risk, R(d|β) = ∫R(d, α, β)π(α)dα be the Bayes risk, and R(d|x_n, β) = ∫ W(d(x_n), α)π(α|x_n, β)dα the posterior risk for α at β. Then R(d|β) = ∫R(d|x_n, β)m(x_n|β)dx_n. The Bayes decision for α given β is d^*(·) = d^*(·|β) = arg inf_d∈ R(d|β), and it is known that, d^*(x_n) = arg inf_d∈ R(d|x_n, β) (a.s.), the generalized Bayesian estimator α_n given β.

In this hybrid inference, we infer α by the above generalized Bayesian rule for each fixed β, and at the same time infer β by the frequentist MLE, i.e. we are to find θ_n = (α̌_n, β̂_n) = (α̌(x_n), β̂(x_n)) such that

$$({\check{\mathit{\alpha }}}_n,{\widehat{\mathit{\beta }}}_n)=arg\underset{(\mathbf{d},\mathit{\beta })}{infsup}\int W(\mathbf{d}({\mathbf{x}}_n),\mathit{\alpha })f({\mathbf{x}}_n\mid \mathit{\alpha },\mathit{\beta })\pi (\mathit{\alpha })d\mathit{\alpha }$$ (1)

is the joint optimization over (d, β). Note that by imposing a 0-1 loss and constant prior on β, (1) can be formulated as a double minimization over (d, β) in a full Bayesian sense. Thus (α̌_n, β̂_n) always exists and is generally locally unique. Here in (α̌_n, β̂_n), α̌_n is a generalized Bayes estimator of α and β̂_n the MLE of β. This hybrid estimator is consistent and efficient while keeping the small sample advantage for the components of parameters with good prior knowledge. Some basic asymptotic properties are studied in Yuan (2009), here we discuss some computational aspect of it and address its application. Here, in a way, the use of MLE for parameter estimates are like using non-informative prior distributions for those parameters in the Bayesian sense, but there is a difference: under the commonly used squared loss, the estimate of β is the posterior mean (with respect to the non-informative prior), while in the hybrid method, estimate of β is a MLE. However, if we use the 0-1 loss for β, and a constant prior, a non-informative prior (but not Jeffreys’ non-informative prior), then the hybrid estimate is full Bayesian procedure. This is also true also for the full MLE, which is a special Bayesian estimate under 0-1 loss and constant prior. In this sense, many frequentist’s methods can be regarded as special Bayesian (Wald’s complete class theory states that any admissible procedure can be formulated as a Bayesian or limit of Bayesian procedures). But in the literature they are not called (special) Bayesian for historical and technical reasons, as they differ from standard Bayesian procedures. Similarly we do not call the hybrid method a Bayesian, although it can be formulated as a full Bayesian under special loss and prior. Also, we haven’t seen in the literature a Bayesian model as our hybrid one by choosing such special loss and prior on part of the parameters. They may use a non-informative prior on part of the parameters, but typically a squared error or absolute error loss will be used for all parameters.

Although in some cases, (α̌_n, β̂_n) has closed form expression, but the general solution of (1) may not easy. Denote G_n(d, β) = ∫W(d(x_n), α)f(x_n|α, β)π(α)dα. Let sup_βinf_d denote first taking inf w.r.t. d then taking sup w.r.t. β; inf_{(d, supβ)} the other way round, and $\underset{(\mathbf{d},\mathit{\beta })}{infsup}$ for taking inf w.r.t. d and the sup w.r.t. β simultaneously. Since ${sup}_{\mathit{\beta }}{inf}_{\mathbf{d}}{G}_n(\mathbf{d},\mathit{\beta })\le \underset{(\mathbf{d},\mathit{\beta })}{infsup}{G}_n(\mathbf{d},\mathit{\beta })\le {inf}_{\mathbf{d}}{sup}_{\mathit{\beta }}{G}_n(\mathbf{d},\mathit{\beta })$, and the “=” signs are not generally asserted, so generally arg sup_β inf_d G_n(d, β) ≠ (α̌_n, β̂_n) = arg inf_{(d, supβ)} G_n(d, β) ≠ arg inf_d sup_βG_n(d, β). However, if arg inf_d G_n(d, β) does not depend on β, then (α̌_n, β̂_n) = arg sup_β inf_d G_n(d, β). Similarly, if arg sup_β G_n(d, β) does not depend on d, then (α̌_n, β̂_n) = inf_d sup_β G_n(d, β).

When (α̌_n, β̂_n) is not directly computable, iterative procedures can be used. Below we give two such procedures although they are not directly used in this analysis. The first is a version of the multivariate Newton-Raphson method. Let l(α, β|x_n) be the log-likelihood. We rewrite θ̌ = (α̌, β̂) as the solution of the equations

$$\{\begin{array}{l}\frac{\partial }{\partial \mathit{\alpha }}R(\mathit{\alpha }\mid {\mathbf{x}}_n,\mathit{\beta })=\mathbf{0}\hfill \\ \frac{\partial }{\partial \mathit{\beta }}l(\mathit{\alpha },\mathit{\beta }\mid {\mathbf{x}}_n)=\mathbf{0}\hfill \end{array}.$$

For a vector v denote v′ for its transpose. Let $R^{(1)}(\mathit{\theta })=((\frac{\partial }{\partial \mathit{\alpha }}R{(\mathit{\alpha },\mathit{\beta }))}^{\prime },(\frac{\partial }{\partial \mathit{\beta }}l{{(\mathit{\alpha },\mathit{\beta }\mid {\mathbf{x}}_n))}^{\prime })}^{\prime },R^{(2)}=\frac{\partial }{\partial \mathit{\theta }}{R}^{(1)}(\mathit{\theta })$. Choosing an arbitrary starting point θ⁽⁰⁾, this iterative procedure gives

$${\mathit{\theta }}^{(k+1)}={\mathit{\theta }}^{(k)}+{(R^{(2)})}^{-1}({\mathit{\theta }}^{(k)})(R^{(1)}({\mathit{\theta }}^{(k+1)})-R^{(1)}({\mathit{\theta }}^{(k)})),k=0,1,2,\dots$$ (2)

It iterates between the two equations of $\frac{\partial }{\partial \mathit{\alpha }}R=0$ and $\frac{\partial }{\partial \mathit{\beta }}l=0$ in the above to get the sequence {(α^(k), β^(k))}. By the property of this well known procedure, without proof, we have

Proposition 1

Assume |R⁽²⁾(θ)| ≠ 0 in a neighborhood of θ̌_n, then as k → ∞, {(α^(k), β^(k))} converges to a stationary point such that (1) holds. If further, for fixed n, there is a unique point (α̌_n, β̂_n) such that (1) holds, then as k → ∞,

$$({\mathit{\alpha }}^{(k)},{\mathit{\beta }}^{(k)})\to ({\check{\mathit{\alpha }}}_n,{\widehat{\mathit{\beta }}}_n).$$

More generally, given β, a Bayesian procedure has the form α̌_n = G(π(·|x_n, β)) for some known functional G(·). For squared error loss, G(·) is the posterior mean; for absolute error loss, G(·) is the posterior median; for 0-1 loss, G(·) is the posterior mode. The following algorithm is more intuitive and sometimes convenient. To be specific, start from (α⁽⁰⁾, β⁽⁰⁾), …, given the k-th iteration estimates (α^(k), β^(k)), the next (k + 1)-th update is given by

$${\mathit{\alpha }}^{(k+1)}=G(\pi (\mathit{\alpha }\mid {\mathbf{x}}_n,{\mathit{\beta }}^{(k)})),{\mathit{\beta }}^{(k+1)}=arg\underset{\mathit{\beta }}{sup}l({\mathit{\alpha }}^{(k+1)},\mathit{\beta }\mid {\mathbf{x}}_n).$$ (3)

Let the Fisher information matrix I(θ) = I(α, β) be partitioned into sub-blocks I_ij(θ) (i, j = 1, 2), and I(α|β) = −E(∂²/∂α∂α′ log f(X|α, β)) be the conditional Fisher information given β. We have (Appendix)

Proposition 2

Assume G(π(α|x_n, β)) − G(N (α̂_n, I⁻¹(α|β)/n)) → 0 as n → ∞, that sup_θ∈A ||∂/∂β [G(N(α̂_n, I⁻¹(α|β)/n)]I₁₂(θ)I₂₂(θ)⁻¹|| < 1 for some A ⊃ (α̌_n, β̂_n), and that there is a unique stationary point (α̌_n, β̂_n) satisfying (1), then for large n, the algorithm given by (3) converges as k → ∞:

$$({\mathit{\alpha }}^{(k)},{\mathit{\beta }}^{(k)})\to ({\check{\mathit{\alpha }}}_n,{\widehat{\mathit{\beta }}}_n).$$

Under the squared error or absolute error losses, G(·) is the posterior mean or median, the condition G(π(α|x_n, β)) − G(N (α̂_n, I⁻¹(α|β)/n)) → 0 in the L₁ sense is implied by Theorem 2.2 in Bickel and Yahav (1969).

Especially, under the squared error loss, (3) becomes

$${\mathit{\alpha }}^{(k+1)}=\int \mathit{\alpha }\pi (\mathit{\alpha }\mid {\mathbf{x}}_n,{\mathit{\beta }}^k)d\mathit{\alpha },{\mathit{\beta }}^{(k+1)}=arg\underset{\mathit{\beta }}{sup}l({\mathit{\alpha }}^{(k+1)},\mathit{\beta }\mid {\mathbf{x}}_n).$$

Under the 0-1 loss, it becomes a double maximization procedure,

$${\check{\mathit{\alpha }}}_n=arg\underset{\mathit{\alpha }}{sup}\frac{f({\mathbf{x}}_n\mid \mathit{\alpha },{\widehat{\mathit{\beta }}}_n)}{m({\mathbf{x}}_n\mid {\widehat{\mathit{\beta }}}_n)}=arg\underset{\mathit{\alpha }}{sup}(l(\mathit{\alpha },{\widehat{\mathit{\beta }}}_n\mid {\mathbf{x}}_n)+log\pi (\mathit{\alpha })),{\widehat{\mathit{\beta }}}_n=arg\underset{\mathit{\beta }}{sup}l({\check{\mathit{\alpha }}}_n,\mathit{\beta }\mid {\mathbf{x}}_n)$$

$$({\check{\mathit{\alpha }}}_n,{\widehat{\mathit{\beta }}}_n)=arg\underset{\mathit{\alpha },\mathit{\beta }}{sup}(l(\mathit{\alpha },\mathit{\beta }\mid {\mathbf{x}}_n)+log\pi (\mathit{\alpha })).$$

2.2 The Model for the CGH Data

Now we detail this model for the CGH data. Let x_n = (x₁, …, x_n) be the case-control log-ratio measurement of a genomic sequence (GS), with x_i i = 1, …, n be the observation at the i-th locus. The x_i’s are assumed to be arranged according to their chromosomal location. Here we focus on just one GS, in the case of multiple independent GS’s the corresponding likelihood is a product of those for each GS, and the method is analogues. In practice, the true copy number changes cannot be observed directly, but rather, with a CGH data, in which the uorescence ratios between two samples, case and control, are measured across a genomic region. For loci with copy number deletion/amplification, the corresponding log-ratio measurement tend to be lower/higher. We are to answer the question of what is the probability that a given gene or region has increased or decreased copy number changes. In CGH data analysis, often a three-state mixture model is used: deletion state, normal state and amplification state, and we arbitrarily label them as state 1, 2 and 3. Although in some cases the number k of states can be arbitrary but fixed, the analysis is similar in nature. Some method (Rueda and Díaz-Uriarte, 2007) treat k as a parameter to be estimated using the reversible jump Markov model. Here we focus on the common situation for this problem of k being fixed at 3. Genes with copy number deletion tend to have smaller log-ratio measurements, those with normal status tend to have median measurements, and those with amplification tend to have larger measurements. The goal is to classify each the x_is to one of the states. Without knowing these memberships, the density for the x_is can be specified as the mixture

$$f({\mathbf{x}}_n\mid \mathit{\theta })=\prod _{i=1}^n\sum _{j=1}^3{\gamma }_j\phi (x_i\mid {\alpha }_j,{\sigma }_j^2),$$ (4)

where 0 ≤ γ_j ≤ 1, ${\sum }_{j=1}^3{\gamma }_j=1$ are the mixing proportions of the three states in the observed data, and φ(·|α, σ²) is the density of the normal distribution N(α, σ²). Commonly for this problem, we have or can find good prior knowledge on α = (α₁, …, α₃) from existing such studies, as summarized by the prior density ${\pi }_j({\alpha }_j)~N({\alpha }_{j0},{\sigma }_{j0}^2)$, with (α₁₀, …, α₃₀) known, and $\pi (\mathit{\alpha })={\prod }_{j=1}^3\pi ({\alpha }_j)$. But we don’t have not enough experience for the parameter β = (β₁, …, β₃)′, ${\mathit{\beta }}_j=({\gamma }_j,{\sigma }_j^2)(j=1,2,3)$, as often β is difficult to infer from a prior initial study. One may also put a non-informative prior on β to formulate a full Bayesian model. But we will see below that by using the hybrid model, the parameter estimation can be formulated as a EM algorithm, which typically simplifies the computation considerably for mixture model as compared to a full Bayesian model. Thus we use a hybrid model with Bayesian estimate for α and the MLE for β.

Given β and the data x_n, the posterior density of α is

$$\pi (\mathit{\alpha }\mid {\mathbf{x}}_n,\mathit{\beta })\propto \pi (\mathit{\alpha })\prod _{i=1}^n\sum _{j=1}^3{\gamma }_j\phi (x_i\mid {\alpha }_j,{\sigma }_j^2).$$

Note here π(α|x_n, β) is different from the classical posterior π(α, β|x_n) of the full Bayesian model, the former does not contain any prior information about β. Using the 0-1 loss on α, its Bayesian solution is the posterior mode, the solution (α̌_n, β̂_n) in (1) is the maximizer of π(α|x_n, β) over (α, β)

$$({\check{\mathit{\alpha }}}_n,{\widehat{\mathit{\beta }}}_n)=arg\underset{(\mathit{\alpha },\mathit{\beta })}{sup}\pi (\mathit{\alpha })\prod _{i=1}^n\sum _{j=1}^3{\gamma }_j\phi (x_i\mid {\alpha }_j,{\sigma }_j^2).$$

Here π(α|x_n, β) is the object function to be maximized.

The direct maximization in the above mixture model may not easy. Also, to classify each gene x_i, we need a status indicator for each gene. From computation aspect, more commonly and conveniently, the maximization problem in a mixture model is solved by an EM algorithm on the corresponding “complete data” model, such as in Meng and Rubin (1993). For this, let I_ij (j = 1, 2, 3) be the unobserved status indicator for each x_i, that is if x_i is in state j, then I_ij = 1 and I_il = 0 (l ≠ j). Let I_i = (I_i1, I_i2, I_i3), y_i = (x_i, I_i). Since the I_is are unobserved, we treat them as missing data and y_n = (y₁, …, y_n) as the ‘complete’ data. We model the within chromosome dependence only through the I_is. Then given the I_is the x_is are independent. Given y_n and β, the posterior on α is

$$\pi (\mathit{\alpha }\mid {\mathbf{y}}_n,\mathit{\beta })\propto \pi (\mathit{\alpha })\prod _{i=1}^n\prod _{j=1}^3{({\gamma }_j\phi (x_i\mid {\alpha }_j,{\sigma }_j^2))}^{I_{ij}}$$

and omitting the proportionality constant, the corresponding logarithm is

$$l(\mathit{\alpha },\mathit{\beta }\mid {\mathbf{y}}_n)=\sum _{i=1}^n\sum _{j=1}^3{I}_{ij}(log{\gamma }_j+log\phi (x_i\mid {\alpha }_j,{\sigma }_j^2))+log\pi (\mathit{\alpha }).$$

Here l(α, β|y_n) is different from the traditional log-likelihood, as the prior π(α) is incorporated. Instead of maximizing the ‘incomplete data’ posterior π(α|x_n, β), maximizing the ‘complete data’ log-likelihood is computationally much easier and this leads to an EM algorithm, but not the common EM algorithm as here the object function π₀(α, β|x_n) does not have the full likelihood formulation. We will show it has the same property as the standard EM algorithm.

The estimator (α̌_{n, j}, ${\widehat{\sigma }}_{n,j}^2$) is the solution of two equations with two unknowns of order two, which can be easily obtained by the Newton-Raphson method. To get a closed form solution, we set ${\sigma }_{j0}^2={\sigma }_j^2$ (j = 1, 2, 3) (recall the α_j0’s are known, but here the ${\sigma }_j^2$ are unknown and to be estimated). For given I_ij’s, we get (Appendix), with $n_j={\sum }_{i=1}^n{I}_{ij}$ for j = 1, 2, 3,

$${\stackrel{⌣}{\alpha }}_{n,j}=\frac{{\sum }_{i=1}^n{I}_{ij}{x}_i+{\alpha }_{j0}}{n_j+1},{\widehat{\sigma }}_{n,j}^2=\frac{{\sum }_{i=1}^n{I}_{ij}{(x_i-{\stackrel{⌣}{\alpha }}_{n,j})}^2+{({\stackrel{⌣}{\alpha }}_{n,j}-{\alpha }_{j0})}^2}{n_j+1},(j=1,2,3).$$ (5)

In comparison, if we used a full MLE for this problem, then we would have

$${\widehat{\alpha }}_{n,j}=\sum _{i=1}^n{I}_{ij}{x}_i/n_j,{\widehat{\sigma }}_{n,j}^2=\sum _{i=1}^n{I}_{ij}{(x_i-{\stackrel{⌣}{\alpha }}_{n,j})}^2/n_j,(j=1,2,3).$$

We see that, if the α_j0’s represent useful information about α, α̌_n is more accurate than the MLE α̂_n, especially when the sample size is relatively small, or if the ||α_j||’s have relatively large values.

On the other hand, if a full Bayes estimate is used, although we have the small sample advantage on α̌_n, but since we don’t have accurate information on ${\sigma }^2=({\sigma }_1^2,{\sigma }_2^2,{\sigma }_3^2)$, a prior on it may be miss leading, and the computation will be much more involved, even if we place a non-informative prior. It is known that if the prior is too “thin” around true parameter, the Bayes estimate can be inconsistent (LeCam and Yang, 2000).

Since the I_ij’s are missing, we use the following EM algorithm for the computation. Specifically, start from given (α⁽⁰⁾, β⁽⁰⁾), say,

$${\gamma }_j^{(0)}=1/3,{\alpha }_j^{(0)}={\alpha }_{j0},{\sigma }_j^{2(0)}=1,(j=1,2,3)$$

…, given the r-th iteration estimates (α^(r), β^(r)), the next (r + 1)-th updating take the following steps

1. $I_{ij}^{(r+1)}=E(I_{ij}\mid {\mathbf{x}}_n,{\mathit{\alpha }}^{(r)},{\mathit{\beta }}^{(r)})={\gamma }_j^{(r)}\phi (x_i\mid {\alpha }_j^{(r)},{\sigma }_j^{2(r)})/{\sum }_{l=1}^3{\gamma }_l^{(r)}\phi (x_i\mid {\alpha }_l^{(r)},{\sigma }_l^{2(r)})$.

2. ${\alpha }_j^{(r+1)}=({\sum }_{i=1}^n{I}_{ij}^{(r+1)}{x}_i+{\alpha }_{j0})/(n_j^{(r+1)}+1)$, and $n_j^{(r+1)}={\sum }_{i=1}^n{I}_{ij}^{(r+1)}$, (j = 1, 2, 3).

3. ${\sigma }_j^{2(r+1)}=({\sum }_{i=1}^n{I}_{ij}^{(r+1)}{(x_i-{\alpha }_j^{(r+1)})}^2+{({\alpha }_j^{(r+1)}-{\alpha }_{j0})}^2)/(n_j^{(r+1)}+1)$, (j = 1, 2, 3).

4. ${\gamma }_j^{(r+1)}=n_j^{(r+1)}/n$.

2.3 Dependence consideration

The genes in the human genome are more or less dependent each other. Generally such dependence varies in spatial location along the genome, from ethnical groups, population stratifications, etc. Although it is impossible to model such dependence accurately, a model with such considerations incorporated may still has some gains. It is known that in the same chromosome, genes closer in genetic distance tend to have stronger dependence with each other. To implement such spatial dependence, for each i, let R_i be the immediate neighbor of x_i which consists of the itself and the two neighboring measurements x_r and x_s in the given chromosome, except the measurements at the ends of the chromosome. Without loss of generality, we assume all the measurements are arranged according to their chromosomal location, thus x₁ is at the left end and x_n is at the right end of the chromosome. Then R_i = {x_i−1, x_i, x_i+1} for (i = 2, …, n − 1), R₁ = {x₁, x₂} and R_n = {x_n−1, x_n}. Larger R_i can be considered similarly. The choice of the size and structure of the neighborhoods R_i’s is a trade-off between goodness-of-fit and applicability/simplicity. Here we used neighborhood size of 3 as experiences suggest that dependence among genes of immediate spatial members are more important generally, and increase the neighborhood structure may not gain much due to the complexity. We assume each of the I_is are spatially dependent only on R_i. Let d(i, j) be the genetic distance, scaled to [0,1], (assumed known) between locus i and j. These distances will be used when spatial dependence is to be incorporated into the model. Here we distinguish such dependence from different chromosomes when there are more than one. Similarly as in Besag et al. (1991), we specify the spatial dependence relationship as (I_i1, I_i2, I_i3)|R_i ~ Multinomial(1, q_i), where q_i = (q_i1, q_i2, q_i3) and

$$q_{ij}=P(I_{ij}=1\mid R_i)=\frac{{\sum }_{x_k\in R_i}{\gamma }_j{e}^{-d(i,k)}\phi (x_k\mid {\alpha }_j,{\sigma }_j^2)}{{\sum }_{j=1}^3{\sum }_{x_k\in R_i}{e}^{-d(i,k)}{\gamma }_j\phi (x_k\mid {\alpha }_j,{\sigma }_j^2)},(i=1,\dots ,n;j=1,2,3).$$ (6)

This can be viewed as posterior assingment of dependence parameters. Note in the above d(i, i) = 0 for all i. When the d(i, j)’s are not available, we just set d(i, i+1) = d(i+1, i) = 1 for all i and all the loci are viewed as equally spaced. Other method of dependence models for this problem can be found, for example in Fridlyand et al. (2004).

With the spatial dependence considered above, (I) in the above should be replaced by

$$I_{ij}^{(r+1)}=E(I_{ij}\mid {\mathbf{x}}_n,{\mathit{\alpha }}^{(r)},{\mathit{\beta }}^{(r)})=\frac{{\sum }_{x_k\in R_i}{\gamma }_j^{(r)}{e}^{-d(i,k)}\phi (x_k\mid {\alpha }_j^{(r)},{\sigma }_j^{2(r)})}{{\sum }_{j=1}^3{\sum }_{x_k\in R_i}{\gamma }_j^{(r)}{e}^{-d(i,k)}\phi (x_k\mid {\alpha }_j^{(r)},{\sigma }_j^{2(r)})}.$$

Lastly, the resulting three clusters are classified as deletion, normal and amplification groups according to the magnitude of their means.

However the above is not the standard EM algorithm, here l(α, β|y_n) is not a proper log-likelihood due to the extra term log π(α). Define

$$S(\mathit{\alpha }\mid {\mathbf{x}}_n,\mathit{\beta })=\pi (\mathit{\alpha })\prod _{i=1}^n\sum _{j=1}^k{\gamma }_j\phi (x_i\mid {\mathit{\alpha }}_j,{\mathbf{\Omega }}_j).$$

The above algorithm still has the ascending property enjoyed by the standard EM algorithm, i.e.

Proposition 3

For all non-negative integer r, we have (Appendix)

$$logS({\mathit{\alpha }}^{(r+1)}\mid {\mathbf{x}}_n,{\mathit{\beta }}^{(r+1)})\ge logS({\mathit{\alpha }}^{(r)}\mid {\mathbf{x}}_n,{\mathit{\beta }}^{(r)}),$$

so that, as for the standard EM algorithm, if there is a unique maximizer (α̌_n, β̂_n) in π(·|x_n, ·), then as r → ∞, we have (α^(r), β^(r)) → (α̌_n, β̂_n).

Remark

Although we state the result only for the mixture model, Proposition 3 is true for general model with general form of missing data.

The algorithm stops at the last iteration R when the convergence criterion is meet, and we set (α̌_n, β̂_n) = (α^(R), β^(R)). Typically, we may choose the relative error criterion. i.e., specify a tolerance error bound ε (usually ≤ 10⁻²), we stop the iteration if

$$\Vert ({\mathit{\theta }}^{(R)}-{\mathit{\theta }}^{(R-1)})/{\mathit{\theta }}^{(R-1)}\Vert :={(\sum _j{({\theta }_j^{(R)}-{\theta }_j^{(R-1)})}^2/{\theta }_j^{2(R-1)})}^{1/2}\le \epsilon$$

or just the Euclidian error, if there are some zero or nearly zero component(s) in θ, as

$$\Vert ({\mathit{\theta }}^{(R)}-{\mathit{\theta }}^{(R-1)})\Vert :=(\sum _j{({\theta }_j^{(R)}-{\theta }_j^{(R-1)})}^2\le \epsilon .$$

We classify each x_i to the j-th state if

$${\widehat{\gamma }}_j\phi (x_i\mid {\stackrel{⌣}{\alpha }}_j,{\widehat{\sigma }}_j^2)=\underset{1\le l\le 3}{max}{\widehat{\gamma }}_l\phi (x_i\mid {\stackrel{⌣}{\alpha }}_l,{\widehat{\sigma }}_l^2).$$ (7)

This is the optimal classification rule in the sense of minimizing the expected loss (Anderson, 2003), it is also the so-called Bayesian assignment.

Let

$$D_j=\{y:y\in R,{\widehat{\gamma }}_j\phi (y\mid {\stackrel{⌣}{\alpha }}_j,{\widehat{\sigma }}_j^2)=\underset{1\le l\le 3}{max}{\widehat{\gamma }}_l\phi (y\mid {\stackrel{⌣}{\alpha }}_l,{\widehat{\sigma }}_l^2)\}.$$

The false discovery rate p_j|l is the probability of mis-classifying a measurement to state j when its true state is l,

$$p_{j\mid l}={\int }_{D_j}\phi (y\mid {\stackrel{⌣}{\alpha }}_l,{\widehat{\sigma }}_l^2)dy,(1\le j\ne l\le 3).$$

This probability may not easy to compute directly in general. But it is easily computed by the following Monte Carlo simulation.

For given number M (typically M ≥ 1000), for m = 1, …, M, do the following steps

1. Sample $y_l^{(m)}$ from the distribution $\phi (y\mid {\stackrel{⌣}{\alpha }}_l,{\widehat{\sigma }}_l^2)$, (l = 1, 2, 3).

2. Define $v_{j\mid l}^{(m)}=1$ if

   $${\widehat{\gamma }}_j\phi (y_l^{(m)}\mid {\stackrel{⌣}{\alpha }}_j,{\widehat{\sigma }}_j^2)=\underset{s}{max}{\widehat{\gamma }}_l\phi (y_l^{(m)}\mid {\stackrel{⌣}{\alpha }}_s,{\widehat{\sigma }}_s^2),$$

   else $v_{j\mid l}^{(m)}=0$ (j = 1, 2, 3).

Then π_j|l is approximated by

$${\widehat{p}}_{j\mid l}=\frac{1}{M}\sum _{j=1}^M{v}_{j\mid l}^{(m)},(j,l=1,2,3;j\ne l).$$

Simulation studies in Yuan (2009) for the independence model case indicate that the hybrid method can improve the inference, especially in the small sample case, when the method applied properly. In the following we illustrate the method with a real CGH data.

3. Analysis of real CGH Array Data

We use the Genetics Analysis Workshop 15 (GAW15) data set with 14 pedigrees of CEPH Utah families, each with three generations and about a dozen normal individuals. Expression level of genes in lymphoblastoid cells of the above subjects were obtained using the Affymetrix Human Focus Arrays that contain probes for 8,500 transcripts. Gene copy number variations in normal people within human genome has been the subject of study (Freeman et al. 2006; Pugh et al. 2008). We analyze the genes, over 7 time interval, of the second individual in this data set. As an illustration, we use the data in the first time interval as a training set to get the prior information about the means, but the information on the variances and the proportion parameters are not used, as it is known that these parameters are difficult to estimate, and the initial estimates may not be reliable on these parameters based on a small sample size. Therefore, the hybrid method is adequate for this problem, and use this partial prior information on the mean, we analyze the 21562 genes on the all the remaining time intervals, and only report the results on the 7-th time interval of the second individual. Results on all the six time intervals and all the observed individuals are similar and omitted due to space limit. The model parameter estimates are compared for the traditional maximum likelihood estimator (MLE), Bayes estimate (BYE) and the proposed hybrid estimator (HBD) for the parameter $\mathit{\theta }=({\gamma }_1,{\gamma }_2,{\gamma }_3,{\alpha }_1,{\alpha }_2,{\alpha }_3,{\sigma }_1^2,{\sigma }_2^2,{\sigma }_3^2)$. In the hybrid estimation, the components (α₁, α₂, α₃) are estimated Bayesian, with a prior π(α) ~ N(α₀, diag(σ²)), where α₀ = (−0.85, 0.22, 1.35) is obtained from the data on the first time interval, and ${\mathit{\sigma }}^2=({\sigma }_1^2,{\sigma }_2^2,{\sigma }_3^2)$. For the full Bayesian estimate, we used π(θ) = π(γ) π(α)π(σ²), with π(α) the same as for the HBD, π(γ) be uniform on (0, 1)³, and π(σ²) = N((0.6, 1.5, 2.5), (0.5, 1, 2)). Here we do not intentionally choose the prior mean too far away from the other estimates to avoid mis-leading results. The commonly used squared error loss is used for the full BYE, so that the estimates are the posterior means. For the hybrid estimate, we use the 0-1 loss on α, so that the Bayesian part estimate on α is the posterior mode, and thus the hybrid estimator θ_n = (α̌_n, β̂_n) is evaluated in closed form given in (5). By the rule of thumb, the minimal effective sample size to estimate one parameter is about 30. As there are nine parameters in θ, the effective sample size should be at least around 300. To see the effects of the sample size on the estimators, we computed the estimates with the three methods using the first 300, 1000, 10000 and all the genes respectively. We run the EM algorithm 500 iterations for all the estimates, with the convergence criterion been well achieved. There are six time intervals, we only display the results on the last time interval, those on the other intervals are similar and omitted. We did the analysis using both the independence gene model and the spatial dependence model via (6). Genetic distance information is not available for this data, so in (6) we just set the distances as a constant.

Since the large number of observations, instead of analyzing the classification result for each of the x_i’s, we discuss the results of the parameter estimations under the three different models, as better parameter estimates correspond to better classification results. The results are reported in Table 1 for the independence model, and Table 2 for dependence model. In the parentheses are the estimated standard errors of the parameters.

Table 1.

Estimation by MLE, BYE and HBD: Independence Model

n	Method	γ1	γ2	γ3	α1	α2	α3	${\sigma }_1^2$	${\sigma }_2^2$	${\sigma }_3^2$
300	MLE	0.128 (0.147)	0.854 (0.062)	0.018 (0.372)	0.051 (0.211)	0.855 (0.141)	3.140 (0.003)	0.006 (15.479)	0.122 (0.305)	5.225 (0.059)
	BYE	0.015 (0.066)	0.970 (0.058)	0.015 (0.409)	0.519 (0.002)	0.757 (0.127)	3.520 (0.003)	0.192 (0.136)	0.188 (0.182)	4.689 (0.070)
	HBD	0.256 (0.071)	0.726 (0.060)	0.018 (0.364)	0.663 (0.064)	1.002 (0.100)	3.005 (0.003)	0.060 (0.634)	0.193 (0.187)	4.531 (0.085)
1000	MLE	0.083 (0.090)	0.911 (0.033)	0.006 (0.320)	0.085 (0.078)	0.839 (0.075)	2.965 (0.001)	0.006 (6.756)	0.140 (0.154)	4.881 (0.065)
	BYE	0.116 (0.075)	0.877 (0.033)	0.006 (0.318)	0.125 (0.051)	0.863 (0.074)	3.087 (0.001)	0.019 (2.094)	0.129 (0.161)	4.087 (0.085)
	HBD	0.133 (0.071)	0.860 (0.033)	0.007 (0.284)	0.146 (0.050)	0.873 (0.075)	2.698 (0.001)	0.024 (1.541)	0.122 (0.171)	3.797 (0.102)
10000	MLE	0.122 (0.021)	0.873 (0.010)	0.005 (0.120)	0.167 (0.020)	0.770 (0.024)	3.192 (0.000)	0.012 (0.858)	0.125 (0.056)	5.246 (0.025)
	BYE	0.128 (0.021)	0.867 (0.010)	0.005 (0.120)	0.173 (0.020)	0.773 (0.024)	3.211 (0.000)	0.013 (0.755)	0.124 (0.057)	5.138 (0.026)
	HBD	0.125 (0.021)	0.870 (0.010)	0.005 (0.120)	0.170 (0.020)	0.772 (0.024)	3.209 (0.000)	0.013 (0.755)	0.124 (0.057)	5.132 (0.026)
21562	MLE	0.233 (0.011)	0.761 (0.007)	0.006 (0.071)	0.200 (0.018)	0.701 (0.015)	2.876 (0.000)	0.016 (0.338)	0.115 (0.045)	4.966 (0.019)
	BYE	0.235 (0.011)	0.759 (0.007)	0.006 (0.071)	0.201 (0.017)	0.702 (0.015)	2.892 (0.000)	0.016 (0.329)	0.115 (0.045)	4.941 (0.019)
	HBD	0.236 (0.011)	0.758 (0.007)	0.006 (0.071)	0.202 (0.017)	0.702 (0.015)	2.869 (0.000)	0.016 (0.327)	0.115 (0.045)	4.924 (0.019)

Table 2.

Estimation by MLE, BYE and HBD: Dependence Model

n	Method	γ1	γ2	γ3	α1	α2	α3	${\sigma }_1^2$	${\sigma }_2^2$	${\sigma }_3^2$
300	MLE	0.139 (0.141)	0.843 (0.062)	0.018 (0.374)	0.058 (0.147)	0.864 (0.143)	3.159 (0.003)	0.011 (7.008)	0.117 (0.314)	5.140 (0.061)
	BYE	0.150 (0.131)	0.833 (0.062)	0.017 (0.382)	0.087 (0.087)	0.874 (0.132)	3.176 (0.003)	0.023 (2.585)	0.122 (0.288)	4.440 (0.083)
	HBD	0.139 (0.141)	0.843 (0.062)	0.018 (0.378)	0.064 (0.143)	0.868 (0.138)	3.158 (0.004)	0.012 (6.658)	0.120 (0.297)	4.337 (0.087)
1000	MLE	0.111 (0.078)	0.883 (0.033)	0.007 (0.303)	0.112 (0.056)	0.859 (0.075)	2.857 (0.001)	0.016 (2.679)	0.129 (0.163)	4.576 (0.072)
	BYE	0.126 (0.072)	0.867 (0.033)	0.007 (0.304)	0.138 (0.047)	0.869 (0.074)	2.886 (0.001)	0.024 (1.529)	0.127 (0.163)	4.013 (0.093)
	HBD	0.110 (0.078)	0.883 (0.033)	0.007 (0.305)	0.114 (0.056)	0.860 (0.074)	2.890 (0.001)	0.015 (2.688)	0.130 (0.160)	3.971 (0.095)
10000	MLE	0.143 (0.020)	0.853 (0.010)	0.005 (0.119)	0.188 (0.018)	0.781 (0.024)	3.167 (0.000)	0.017 (0.558)	0.122 (0.058)	5.210 (0.026)
	BYE	0.146 (0.020)	0.850 (0.010)	0.005 (0.119)	0.191 (0.018)	0.782 (0.023)	3.162 (0.000)	0.018 (0.524)	0.122 (0.058)	5.108 (0.027)
	HBD	0.142 (0.020)	0.853 (0.010)	0.005 (0.119)	0.188 (0.018)	0.781 (0.024)	3.162 (0.000)	0.017 (0.560)	0.122 (0.058)	5.097 (0.027)
21562	MLE	0.258 (0.011)	0.737 (0.007)	0.006 (0.072)	0.215 (0.017)	0.712 (0.015)	2.895 (0.000)	0.020 (0.259)	0.114 (0.046)	5.013 (0.018)
	BYE	0.259 (0.011)	0.735 (0.007)	0.006 (0.072)	0.216 (0.017)	0.713 (0.015)	2.900 (0.000)	0.020 (0.255)	0.114 (0.046)	4.984 (0.019)
	HBD	0.258 (0.011)	0.737 (0.007)	0.006 (0.072)	0.215 (0.017)	0.712 (0.015)	2.897 (0.000)	0.020 (0.259)	0.114 (0.046)	4.974 (0.019)

We can see that for both independence and dependence models, when the sample size is small (300) for this problem, compared to the estimates from the full data as a reference, the accuracy for the MLE, BYE and HBD are not good, but the overall performance of the HBD is better than those of the MLE and BYE, in that estimates from HBD and those of MLE are closer each other, reflecting the fact that good prior information can adjust the results from MLE. While for the BYE, as we do not have reliable prior information on the mixing proportions and variance parameters, its estimates show some larger deviation from those of the MLE, which we regard as more or less mis-leading. When the sample increase to the moderate size of 1000, all the estimators see some improvements, but still the HBD outperforms the MLE and BYE in overall in terms of accuracy. This suggest that for data with small or moderate size, the hybrid method can have some overall advantage than the MLE or Bayesian alone, when we have sound prior information about part of the parameters. When the sample size reaches the full 21562, all estimates show little difference, as expected, a confirmation of the statement in Remark 2 in Yuan (2009), that the Bayesian, MLE and hybrid estimators are asymptotically first order equivalent, normal and efficient. The results from the dependence model seems more reasonable, in that the performances of the three estimators are more homogeneous and stable. This may reflect the fact that the genes are dependent by nature and thus implement such feature improves the fitting, also as in the dependence model the effects are averaged in each gene neighborhood, the outcome is more smooth. Thus our interpretations of results are based on the HBD method with gene-gene spatial dependence model. In contrast to the case-control analysis of this problem, in which the data are collected as log ratios of signal measurements of cases to controls, thus deletion status is associated with negative observations, normal status with near zero observations and amplification with positive ones; here our data are from normal individuals, the means of all the three statuses are positive, with deletion status associated with the smallest means of about 0.215, and those for normal and amplification statuses are about 0.712 and 2.897. The corresponding proportions are 0.258, 0.737 and 0.006, thus about 73.7% genes have no alteration during the past history in the group of normal people under investigation. About 25.8% of the genes have undergone repeat number deletion, but the average amount of deletion is much smaller (with a mean change of −0.497) in scale as compared to that for the amplification (with a mean change of 2.185), but the latter occurs with very small proportion, about 0.6%.

The classification results based on all the genes, using the hybrid method (almost the same by the other two methods for such a large sample size) of the second individual on the last time interval are shown in Figure 1 below without gene-gene spatial dependence, and in Figure 2 for the spatial dependence model. Horizontal axis represents the sequential numbering of genes from 1 to 21562, as separated into nine panels in each figure, and the vertical axis indicates the classified statuses of the genes with 1, 2 and 3 representing deletion, normal and amplification statuses of genes respectively. For this data, the graphical appearances of the independence model and that with gene spatial dependence have no significant differences, as in figures 1 and 2. Each dot in the figures roughly represents two events. Some minor difference can be observed in the region of gene loci 5000–7500, or panel 3 in each figure. Based on our estimates from the dependence model with the HBD method as displayed in Figure 2, more amplifications occurred in the 1–2500 gene loci region, with about 40 amplification events, where deletion is relatively less heavy. Deletions are heavy on loci 17500–21562, but very few amplifications observed in this region. Amplification is sparse in the 2200–2300 region, with about only four amplifications occurred, and with about eight of them occurred in the region 2500–5000. As pointed out before, although the number of amplifications is small compared to that of deletions, the average scale, i.e., the magnitude of copy number changes in each event of the former is much bigger than that of the latter, which is not shown in the figures. Our consensus is that in the analysis of gene copy number variations, along with the proportions of amplifications and deletions, the associated magnitude should also be reported, which is often missing in many such studies. The combined analysis should be more informative than just reporting the number of gains/losses. According to our results on this given normal individual, the average magnitude of gain is about 4.4 times that of losses. If we consider the number of gene alteration of some status times the corresponding average magnitude of mean change (in absolute value) as an index to describe the effect of copy number changes, for the individual under investigation, this index in the deletion status is about 5088 × 0.497 ≈ 2529, while that for amplification is about 40 × 2.185 ≈ 80. Thus it seems that in normal individual, at least for this one under investigation, most gene alterations are in the form of copy number losses, this is in contrast to some other reports in cancer patients (Nymark et al. 2006), in which the proportion of copy number gains is significantly higher than that of losses. As this individual is non-affected, these copy number changes reflect the result of cumulative changes over the generations during the human history.

Figure 1.

Classification results from independence model:1. GCN deletion; 2. Normal; 3. Amplification.

Figure 2.

Classification results from dependence model:1. GCN deletion; 2. Normal; 3. Amplification.

4. Brief Discussion

We considered a Bayesian-frequentist hybrid method to analyze the CGH data, in which the mean parameters are estimated Bayesian and the other parameters frequentist. The method is used to classify a real CGH data, we find that for the individual analyzed, a moderate proportion of gene copy number deletion is observed, while relatively sparse loci with copy number amplification, and around 94.9% of the genes are in normal status, for this normal individual.

The method can be easily extended to multivariate CGH observations, and the spatial dependence specification can be implemented for neighborhood of arbitrary size.

As one reviewer point out, our situation is similar to a hierarchical setting, in which a prior dataset D0 has information on part of the parameters α but not β, and a main dataset D which has information on both α and β. Thus, parameters can be inferred from the joint likelihood P(D, D0|α, β). We think this approach has both pros and cons. The advantage is that it uses both data set so that more information in the data is used. But as D and D0 has different format, this increases the modeling complexity, and the corresponding results may not directly comparable to other methods based on P(D|α, β). However this modeling is of practical interest and will be the among topics in our future research.

Appendix

Proof of Proposition 2

Let θ^(k) = (α^(k), β^(k)), (3) can be formulated as θ^(k+1) = T(θ^(k)) (k = 0, 1, …) for some function T(θ^(k+1)) = (T₁(β^(k)), T₂(α^(k))) determined by (3). Apparently, the convergence of one of the sequences {α^(k)} and {β^(k)} implies the other. So we only need to prove the convergence of {α^(k)}. Denote Ṫ = (Ṫ₁, Ṫ₂) the derivative matrix of T, we have for some θ̃ lies between θ^(k+m) and θ^(k),

$$\begin{array}{c}\Vert {\mathit{\alpha }}^{(k+m)}-{\mathit{\alpha }}^{(k)}\Vert =\Vert {\stackrel{.}{T}}_1(\stackrel{\sim }{\mathit{\beta }})({\mathit{\beta }}^{(k-1+m)}-{\mathit{\beta }}^{(k-1)})\Vert =\Vert {\stackrel{.}{T}}_1(\stackrel{\sim }{\mathit{\beta }}){\stackrel{.}{T}}_2(\stackrel{\sim }{\mathit{\alpha }})({\mathit{\alpha }}^{(k-1+m)}-{\mathit{\alpha }}^{(k-1)})\Vert \\ \le \underset{\mathit{\theta }\in A}{sup}\Vert {\stackrel{.}{T}}_1(\mathit{\beta }){\stackrel{.}{T}}_2(\mathit{\alpha })\Vert ·\Vert {\mathit{\alpha }}^{(k-1+m)}-{\mathit{\alpha }}^{(k-1)}\Vert \le \cdots \le C^k\Vert {\mathit{\alpha }}^{(m)}-{\mathit{\alpha }}^{(0)}\Vert \\ \le C^k\sum _{j=0}^{m-1}\Vert {\mathit{\alpha }}^{(j+1)}-{\mathit{\alpha }}^{(j)}\Vert \le C^k\sum _{j=0}^{m-1}{C}^j\Vert {\mathit{\alpha }}^{(1)}-{\mathit{\alpha }}^{(0)}\Vert ,(k=1,2,\dots ;m=1,2,\dots )\end{array}$$

where C = sup_θ∈A ||Ṫ₁(β)Ṫ₂(α)|| ≥ 0. So if we show C < 1, then

$$C^k\sum _{j=0}^{m-1}{C}^j\Vert {\mathit{\alpha }}^{(1)}-{\mathit{\alpha }}^{(0)}\Vert \le \frac{C^k}{1-C}\Vert {\mathit{\alpha }}^{(1)}-{\mathit{\alpha }}^{(0)}\Vert ,$$

thus {α^(k)} is a Cauchy sequence and the conclusion will be true.

In fact, for fixed β, by Theorem 2 in Yuan (2009) and the given condition, for large n,

$${\stackrel{.}{T}}_1(\mathit{\beta })\approx \frac{\partial G(N({\widehat{\mathit{\alpha }}}_n,I^{-1}(\mathit{\alpha }\mid \mathit{\beta })/n))}{\partial \mathit{\alpha }}.$$

Also, β̂ is determined by the equation 0 = ∂l(α, β|x_n)/∂β, so we have

$$0=\frac{{\partial }^2}{\partial \mathit{\beta }\partial {\mathit{\alpha }}^{\prime }}l(\mathit{\alpha },\mathit{\beta }\mid {\mathbf{x}}_n)+\frac{{\partial }^2}{\partial \mathit{\beta }\partial {\mathit{\beta }}^{\prime }}l(\mathit{\alpha },\mathit{\beta }\mid {\mathbf{x}}_n)\frac{\partial \mathit{\beta }}{\partial {\mathit{\alpha }}^{\prime }},$$

or, by the SLLN,

$${\stackrel{.}{T}}_2(\mathit{\alpha })=-\frac{\partial \mathit{\beta }}{\partial \mathit{\alpha }}=-\left(\frac{{\partial }^2}{\partial \mathit{\alpha }\partial {\mathit{\beta }}^{\prime }}l(\mathit{\alpha },\mathit{\beta }\mid {\mathbf{x}}_n)\right){\left(\frac{{\partial }^2}{\partial \mathit{\beta }\partial {\mathit{\beta }}^{\prime }}l(\mathit{\alpha },\mathit{\beta }\mid {\mathbf{x}}_n)\right)}^{-1}\approx -I_{12}(\mathit{\alpha },\mathit{\beta })I_{22}^{-1}(\mathit{\alpha },\mathit{\beta }),$$

thus by the given condition, C < 1 and the conclusion is true.

Proof of (5), (I), (I′) and (IV)

The proof is for the general case where the observation is a sequence of vectors x_n = (x₁, …, x_n), the corresponding density if $\phi (x_i\mid {\mathit{\alpha }}_j,{\mathbf{\Omega }}_j^{-1})$. Let ${\mathbf{\Lambda }}_j={\mathbf{\Omega }}_j^{-1}$, and d = dim(α). For (5), we have

$$\begin{array}{c}l(\mathit{\alpha },\mathit{\beta }\mid {\mathbf{y}}_n)=\sum _{i=1}^n\sum _{j=1}^3{I}_{ij}(log{\gamma }_j+log\phi (x_i\mid {\mathit{\alpha }}_j,{\mathbf{\Omega }}_j^{-1}))+log\pi (\mathit{\alpha })\\ =\sum _{i=1}^n\sum _{j=1}^k{I}_{ij}\left(log{\gamma }_j-\frac{1}{2}{(x_i-{\mathit{\alpha }}_j)}^{\prime }{\mathbf{\Lambda }}_j(x_i-{\mathit{\alpha }}_j)+\frac{1}{2}\sum _{j=1}^{k}log\mid {\mathbf{\Lambda }}_j\mid -\frac{d}{2}log(2\pi )\right)\\ +\sum _{j=1}^k\left(-\frac{1}{2}{({\mathit{\alpha }}_j-{\mathit{\alpha }}_{j0})}^{\prime }{\mathbf{\Lambda }}_j({\mathit{\alpha }}_j-{\mathit{\alpha }}_{j0})+\frac{1}{2}log\mid {\mathbf{\Lambda }}_j\mid -\frac{d}{2}log(2\pi )\right).\end{array}$$

To get α̌_nj, take partial derivative of l(α, β|y_n) with respect to α_nj, note ∂[(x_i−α_j)′Λ_j(x_i− α_j)]/∂α_j = −2Λ_j(x_i − α_j) and ∂[(α_j − α_j0)′Λ_j(α_j − α_j0)]/∂α_j = 2Λ_j(α_j − α_j0). We have

$$\mathbf{0}=\frac{\partial l(\mathit{\alpha },\mathit{\beta }\mid {\mathbf{y}}_n)}{\partial {\mathit{\alpha }}_j}=\sum _{i=1}^n{I}_{ij}{\mathbf{\Lambda }}_j(x_i-{\mathit{\alpha }}_j)-{\mathbf{\Lambda }}_j({\mathit{\alpha }}_j-{\mathit{\alpha }}_{j0})$$

or, since Λ_j is non-singular, equivalently

$$0=\sum _{i=1}^n{I}_{ij}{x}_i+{\mathit{\alpha }}_{j0}-(n_j+1){\mathit{\alpha }}_j,$$

This gives the expression for α̌_nj.

To get Ω̂, take partial derivative of l(α, β|y_n) with respect to Λ_j, note ∂(x_i−α_j)′Λ_j(x_i− α_j)/∂Λ_j = (x_i − α_j)(x_i − α_j)′ and $\partial log\mid {\mathbf{\Lambda }}_j\mid /\partial {\mathbf{\Lambda }}_j={\mathbf{\Lambda }}_j^{\ast }/\mid {\mathbf{\Lambda }}_j\mid$, where ${\mathbf{\Lambda }}_j^{\ast }$ is the conjugate of Λ_j in the sense ${\mathbf{\Lambda }}_j^{-1}={\mathbf{\Lambda }}_j^{\ast }/\mid {\mathbf{\Lambda }}_j\mid ={\mathbf{\Omega }}_j$. We have

$$\mathbf{0}=2\frac{\partial l(\mathit{\alpha },\mathit{\beta }\mid {\mathbf{y}}_n)}{\partial {\mathbf{\Lambda }}_j}=-\sum _{i=1}^n{I}_{ij}(x_i-{\mathit{\alpha }}_j){(x_i-{\mathit{\alpha }}_j)}^{\prime }-({\mathit{\alpha }}_j-{\mathit{\alpha }}_{j0}){({\mathit{\alpha }}_j-{\mathit{\alpha }}_{j0})}^{\prime }+(n_j+1){\mathbf{\Omega }}_j,$$

plugging in α̌_nj for α_j, this gives (5).

In the above, if π_j(·) = φ(·|α_j0, Ω_j0), with Ω_j0 known, then (α̌_j, Ω̂_j) is the solution of the equations

$$\{\begin{array}{c}{\sum }_{i=1}^n{I}_{ij}(x_i-{\mathit{\alpha }}_j)-{\mathbf{\Omega }}_j{\mathbf{\Omega }}_{j0}^{-1}({\mathit{\alpha }}_j-{\mathit{\alpha }}_{j0})=\mathbf{0}\\ {\sum }_{i=1}^n{I}_{ij}(x_i-{\mathit{\alpha }}_j){(x_i-{\mathit{\alpha }}_j)}^{\prime }-n_j{\mathbf{\Omega }}_j=\mathbf{0}\end{array},$$

which can be obtained by the Newton-Raphson method.

To derive (I), note $P(I_{ij}=1\mid {\mathit{\beta }}^{(r)})={\gamma }_j^{(r)}$ (i = 1, …, n; j = 1, 2, 3), so

$$\begin{array}{l}E(I_{ij}\mid {\mathbf{x}}_n,{\mathit{\alpha }}^{(r)},{\mathit{\beta }}^{(r)})=P(I_{ij}=1\mid {\mathbf{x}}_n,{\mathit{\alpha }}^{(r)},{\mathit{\beta }}^{(r)})=P(I_{ij}=1\mid x_i,{\mathit{\alpha }}^{(r)},{\mathit{\beta }}^{(r)})\\ =\frac{\phi (x_i\mid {\mathit{\alpha }}_j^{(r)},{\mathbf{\Omega }}_j^{(r)})P(I_{ij}=1\mid {\mathit{\beta }}^{(r)})}{{\sum }_{l=1}^3\phi (x_i\mid {\mathit{\alpha }}_l^{(r)},{\mathbf{\Omega }}_l^{(r)})P(I_{il}=1\mid {\mathit{\beta }}^{(r)})}=\frac{{\gamma }_j^{(r)}\phi (x_i\mid {\mathit{\alpha }}_j^{(r)},{\mathbf{\Omega }}_j^{(r)})}{{\sum }_{l=1}^3{\gamma }_l^{(r)}\phi (x_i\mid {\mathit{\alpha }}_l^{(r)},{\mathbf{\Omega }}_l^{(r)})}.\end{array}$$

The derivation of (I′) is similar under the conditional probability for I_ij given in (6), and noting in this case E(I_ij|x_n, α^(r), β^(r)) = P(I_ij = 1|R_i, α^(r), β^(r)).

For (IV), note the maximization over the γ_j’s are under the constraint ${\sum }_{j=1}^3{\gamma }_j=1$. So by the Lagrange method, we are to maximize $l(\mathit{\alpha },\mathit{\beta }\mid {\mathbf{y}}_n)-r{\sum }_{j=1}^3{\gamma }_j$. For j = 1, 2, 3, set

$$0=\frac{\partial }{\partial {\gamma }_j}\left(l(\mathit{\alpha },\mathit{\beta }\mid {\mathbf{y}}_n)-r\sum _{j=1}^3{\gamma }_j\right)=\sum _{i=1}^n{I}_{ij}/{\gamma }_j-r$$

we get ${\sum }_{i=1}^n{I}_{ij}=r{\lambda }_j$. Summing over j on both sides we get $n={\sum }_{i=1}^n{\sum }_{j=1}^3{I}_{ij}=r$, thus

$${\widehat{\lambda }}_j=\sum _{i=1}^n{I}_{ij}/r=n_j/n.$$

Proof of Proposition 3

Define

$$b({\mathbf{y}}_n\mid \mathit{\theta })=\pi (\mathit{\alpha })\prod _{i=1}^n\prod _{j=1}^k{({\gamma }_j\phi (x_i\mid {\mathit{\alpha }}_j,{\mathbf{\Omega }}_j))}^{I_{ij}},a({\mathbf{x}}_n\mid \mathit{\theta })=\pi (\mathit{\alpha })\prod _{i=1}^n\sum _{j=1}^k{\gamma }_j\phi (x_i\mid {\mathit{\alpha }}_j,{\mathbf{\Omega }}_j),$$

and define Q(θ′|θ) = E[log b(y_n|θ′)|x_n, θ], and H(θ′|θ) = E[log g(y_n|θ′)|x_n, θ], where g(y_n|θ) = b(y|θ)/a(x_n|θ), the expectations are over the missing data I_ij’s. Then log S(α, |x_n, θ′) = log a(x_n|θ′) = Q(θ′|θ) − H(θ′|θ). It is seen that g(y_n|θ) is just the conditional density of y_n given x_n, thus Q(·|·) and H(·|·) here play the same roles as they do in the standard EM algorithm (Dempster et al., 1977). Since θ^(r+1) = arg sup_θQ(θ| θ^(r)), we have

$$Q({\mathit{\theta }}^{(r+1)}\mid {\mathit{\theta }}^{(r)})\ge Q({\mathit{\theta }}^{(r)}\mid {\mathit{\theta }}^{(r)}).$$

Also by Lemma 1 in Dempster et al (1977), or just using the property of relative entropy,

$$H({\mathit{\theta }}^{(r+1)}\mid {\mathit{\theta }}^{(r)})\le H({\mathit{\theta }}^{(r)}\mid {\mathit{\theta }}^{(r)}),$$

so we have

$$\begin{array}{c}log\pi ({\mathit{\alpha }}^{(r+1)}\mid {\mathbf{x}}_n,{\mathit{\beta }}^{(r+1)})=loga({\mathbf{x}}_n\mid {\mathit{\theta }}^{(r+1)})=Q({\mathit{\theta }}^{(r+1)}\mid {\mathit{\theta }}^{(r)})-H({\mathit{\theta }}^{(r+1)}\mid {\mathit{\theta }}^{(r)})\\ \ge Q({\mathit{\theta }}^{(r)}\mid {\mathit{\theta }}^{(r)})-H({\mathit{\theta }}^{(r)}\mid {\mathit{\theta }}^{(r)})=loga({\mathbf{x}}_n\mid {\mathit{\theta }}^{(r)})=logS({\mathit{\alpha }}^{(r)}\mid {\mathbf{x}}_n,{\mathit{\beta }}^{(r)}),\end{array}$$

i.e. Proposition 3 is true.