Flexible mixture modeling via the multivariate t distribution with the Box-Cox transformation: an alternative to the skew-t distribution

Abstract

Cluster analysis is the automated search for groups of homogeneous observations in a data set. A popular modeling approach for clustering is based on finite normal mixture models, which assume that each cluster is modeled as a multivariate normal distribution. However, the normality assumption that each component is symmetric is often unrealistic. Furthermore, normal mixture models are not robust against outliers; they often require extra components for modeling outliers and/or give a poor representation of the data. To address these issues, we propose a new class of distributions, multivariate t distributions with the Box-Cox transformation, for mixture modeling. This class of distributions generalizes the normal distribution with the more heavy-tailed t distribution, and introduces skewness via the Box-Cox transformation. As a result, this provides a unified framework to simultaneously handle outlier identification and data transformation, two interrelated issues. We describe an Expectation-Maximization algorithm for parameter estimation along with transformation selection. We demonstrate the proposed methodology with three real data sets and simulation studies. Compared with a wealth of approaches including the skew-t mixture model, the proposed t mixture model with the Box-Cox transformation performs favorably in terms of accuracy in the assignment of observations, robustness against model misspecification, and selection of the number of components.

1 Introduction

In statistics, model-based clustering (McLachlan 1982; Titterington et al. 1985; McLachlan and Basford 1988; Banfield and Raftery 1993; Celeux and Govaert 1995; McLachlan and Peel 2000; Fraley and Raftery 2002; McLachlan et al. 2003; Bouveyron et al. 2007; McNicholas and Murphy 2008; Scrucca 2010; Andrews and McNicholas 2010) is a popular unsupervised approach to look for homogeneous groups of observations. The most commonly used model-based clustering approach is based on finite normal mixture models, which has been shown to give good results in various applied fields, for example, gene expression (Yeung et al. 2001; McLachlan et al. 2002; Pan et al. 2002), image analysis (Wehrens et al. 2004; Fraley et al. 2005; Li et al. 2005), medical diagnosis (Schroeter et al. 1998; Forbes et al. 2006) and astronomy (Kriessler and Beers 1997; Mukherjee et al. 1998).

However, normal mixture models rely on the assumption that each component follows a normal distribution, which is often unrealistic for data with groups asymmetric in shape (Lin et al. 2007b; Lo et al. 2008; Lin 2009a, 2010). A common remedy for the asymmetry issue is to look for transformations of the data that make the normality assumption more realistic. Box and Cox (1964) discussed the power transformation in the context of linear regression, which has also been applied to normal mixture models (Schork and Schork 1988; Gutierrez et al. 1995).

Another line of attempts to resolve the asymmetry observed in data is to enhance the flexibility of the normal distribution by introducing skewness. Azzalini (1985) developed a class of univariate skew-normal distributions with the introduction of a shape parameter to account for skewness, which had been put to use in a mixture modeling context by Lin et al. (2007b). A multivariate version of the skew-normal distributions was first proposed by Azzalini and Dalla Valle (1996), with various generalizations or modifications ensuing. One such modification was found in Sahu et al. (2003), who developed a new class of multivariate skew elliptically symmetric distributions with applications to Bayesian regression models, and included the multivariate skew-normal distribution as a special case. As opposed to Azzalini and Dalla Valle’s (1996) formulation of the skew-normal distribution, the correlation structure in that of Sahu et al. (2003) is not affected by the introduction of skewness in the sense that independence between elements of a random vector is preserved irrespective of changes in the skewness parameters. The latter formulation was adopted by Lin (2009a), who introduced the multivariate skew-normal mixture model, and described a variant of the Expectation-Maximization (EM) algorithm (Dempster et al. 1977), called the Expectation-Constrained-Maximization (ECM) algorithm (Meng and Rubin 1993), for maximum likelihood estimation. However, the implementation of this methodology is fairly computationally intensive. A simplified version of Sahu et al.’s (2003) formulation has recently been suggested by Pyne et al. (2009), who parameterized skewness in the form of a vector in place of a matrix. As a result of this simplification, the computational complexity of parameter estimation has been reduced considerably.

In addition to non-normality, there is also the problem of outlier identification in mixture modeling. Outliers can have a significant effect on the resulting clustering. For example, they will usually lead to overestimating the number of components in order to provide a good representation of the data (Fraley and Raftery 2002). If a more robust model is used, fewer clusters may suffice. Outliers can be handled in the model-based clustering framework, by either replacing the normal distribution with a more robust one (e.g., t; see Peel and McLachlan 2000; McLachlan and Peel 2000) or adding an extra component to accommodate the outliers (e.g., uniform; see Schroeter et al. 1998).

Transformation selection and outlier identification are two issues which can have heavy mutual influence (Carroll 1982; Atkinson 1988). While a stepwise approach in which transformation is preselected ahead of outlier detection (or vice versa) may be considered, it is unlikely to address the problem well, as the preselected transformation may be influenced by the presence of outliers. One possible means of handling the two issues simultaneously is through the application of skew-t distributions (Azzalini and Capitanio 2003; Sahu et al. 2003) in mixture modeling. Such an attempt was given by Lin et al. (2007a), who proposed a skew-t mixture model based on the formulation of Azzalini and Capitanio (2003), but it is confined to the univariate case. Not until recently has a multivariate version of the skew-t mixture model come to light. Lin (2010) and (Pyne et al. 2009) adopted a similar approach to the case of skew-normal in defining the multivariate skew-t distribution, thereby simplifying Sahu et al.’s (2003) formulation with a vector in place of a skewness matrix.

In view of the aforementioned issues, we propose a unified framework based on mixture models using a new class of skewed distributions, namely, multivariate t distributions with the Box-Cox transformation, to handle transformation selection and outlier identification simultaneously. The t distribution provides a robust mechanism against outliers with its heavier tails relative to the normal distribution (Lange et al. 1989). The Box-Cox transformation is a type of power transformation, which can bring skewed data back to symmetry, a property of both the normal and t distributions. Along with the introduction of the mixture model using this new class of distributions, we also describe a convenient means of parameter estimation via the EM algorithm. Whilst the proposed framework is computationally much simpler than mixture modeling using skew-t distributions, it performs well in various scenarios compared to a wealth of competing approaches, as shown in subsequent sections of this article. A simplified form of our proposed framework has been applied to flow cytometry, which shows a favorable performance in identifying cell populations (Lo et al. 2008). This article presents a comprehensive framework that substantially enriches that previous simplified version, including the selection of component-specific transformations, and the provision of a level of robustness adaptive to the data. In addition, the emphasis is laid upon computational development of the proposed methodology. We have also included a large-scale comparison with competing approaches such as those using the skew-normal or skew-t mixture distributions.

The structure of this article is as follows. In Sect. 2 we first introduce the new class of skewed distributions, multivariate t distributions with the Box-Cox transformation. Then we introduce the mixture model using the proposed distributions, and present details including outlier identification, density estimation and the selection of the number of components. In addition, we describe an EM algorithm to simultaneously handle parameter estimation and transformation selection for our proposed mixture model. In Sect. 3, the performance of the proposed framework is examined on real data sets and compared to a wealth of commonly used approaches. Section 4 presents extensive simulation studies to further evaluate our proposed framework relative to the other approaches. Finally, in Sect. 5 we summarize and discuss our findings.

2 Methodology

2.1 Preliminaries

2.1.1 The multivariate t distribution

The multivariate t distribution has found its use as a robust modeling tool in various fields of applied statistics like linear and non-linear regression, time series, and pedigree analysis; see Lange et al. (1989) and Kotz and Nadarajah (2004) for examples. The t distribution is applied in place of the normal distribution when the latter fails to offer long enough tails for the error distribution. Formally, a random vector y of length p is said to follow a p-dimensional multivariate t distribution with mean μ (ν > 1), covariance matrix ν (ν − 2)⁻¹ Σ (ν > 2) and ν degrees of freedom if its density function is given by

$${\varphi }_p(\mathbf{y}\mid \mathit{\mu },\mathbf{\sum },\nu )=\frac{\mathrm{\Gamma }(\frac{\nu +p}{2}){\mid \mathbf{\sum }\mid }^{-1/2}}{{(\pi \nu )}^{p/2}\mathrm{\Gamma }(\frac{\nu }{2}){\{1+{(\mathbf{y}-\mathit{\mu })}^T{\mathbf{\sum }}^{-1}(\mathbf{y}-\mathit{\mu })/\nu \}}^{\frac{\nu +p}{2}}}.$$ (1)

The degrees of freedom ν may be viewed as a robustness tuning parameter, as it controls the fatness of the tails of the distribution. When ν → ∞, the t distribution approaches a p-dimensional multivariate normal distribution with mean μ, covariance matrix Σ, and density function

$${\phi }_p(\mathbf{y}\mid \mathit{\mu },\mathbf{\sum })=\frac{1}{{(2\pi )}^{p/2}{\mid \mathbf{\sum }\mid }^{1/2}}exp\left\{-\frac{1}{2}{(\mathbf{y}-\mathit{\mu })}^T{\mathbf{\sum }}^{-1}(\mathbf{y}-\mathit{\mu })\right\}.$$ (2)

An account of the development of the maximum likelihood estimation of the multivariate t distribution can be found in Liu and Rubin (1995), Liu (1997) and Peel and McLachlan (2000). The estimation involves the use of the EM algorithm or its variants including the ECM and ECME (Liu and Rubin 1994) algorithms. The crux of these algorithms constitutes the fact that we can parameterize a t distribution using a normal-gamma compound distribution. The degrees of freedom ν may be jointly estimated along with other unknown parameters, or fixed a priori when the sample size is small. In the latter case, setting ν = 4 has been found to provide good protection against outliers and work well in many applications (see, for example, Lange et al. 1989; Stephens 2000).

2.1.2 Box-Cox transformation

The power transformation proposed by Box and Cox (1964) was originally introduced to make asymmetric data fulfill the normality assumption in a regression model. The Box-Cox transformation of an observation y is defined as follows:

$$y^{(\lambda )}=\{\begin{array}{cc}\frac{y^{\lambda }-1}{\lambda },& \lambda \ne 0,\\ logy,& \lambda =0,\end{array}$$ (3)

where λ is referred to as the transformation parameter. Note that this function is defined for positive values of y only. In view of the need to handle negative-valued data in some applications, we adopt a modified version (Bickel and Doksum 1981) of the Box-Cox transformation which is also defined for negative values:

$$y^{(\lambda )}=\frac{sgn(y){\mid y\mid }^{\lambda }-1}{\lambda },\lambda >0.$$ (4)

When all data values are positive, this modified Box-Cox transformation reduces to the original version.

2.2 The multivariate t distribution with the Box-Cox transformation

In this article, we propose a new class of distributions, namely, multivariate t distributions with the Box-Cox transformation (t BC), to handle transformation and to accommodate outliers simultaneously. Explicitly, a random vector y of length p following such a distribution has a density function specified by

$${\varphi }_p({\mathbf{y}}^{(\lambda )}\mid \mathit{\mu },\mathbf{\sum },\nu )·\mid J_p(\mathbf{y};\lambda )\mid ,$$ (5)

where $\mid J_p(\mathbf{y};\lambda )\mid =\mid y_1^{\lambda -1}{y}_2^{\lambda -1}\cdots y_p^{\lambda -1}\mid$ is the Jacobian induced by the Box-Cox transformation. Equivalently, the random vector y follows a multivariate t distribution after being Box-Cox transformed. It is difficult to derive the exact mean and variance of the distribution in closed form. However, using first-order Taylor series expansion, approximations for the mean and covariance matrix can be derived. The mean can be approximated by a vector of length p with the j -th element being sgn (λμ_j + 1)|λμ_j + 1|^1/λ, and the variance by ν/(ν − 2)D_p (μ; λ)ΣD_p (μ; λ), where D_p(μ; λ) is a diagonal matrix of order p with the j -th diagonal element being |λμ_j + 1|^1/λ−1. The various shapes that can be represented by the t BC are shown in Fig. 1.

Fig. 1.

Contour plots revealing the shape of bivariate t distributions with the Box-Cox transformation for different values of the transformation parameter. Each distribution has a mean of 10 and unit variance along each dimension. The degrees of freedom parameter is fixed at eight. The values of the transformation parameter λ range from −5 (extremely right-skewed) to 5 (extremely left-skewed)

Analogous to the case of the t distribution without transformation, the t BC approaches a multivariate normal distribution with the Box-Cox transformation (NBC) when ν → ∞. In addition, this class of distributions also includes the untransformed version of the multivariate t or normal distribution. The untransformed t or normal distribution is recovered by setting λ in (5) to one, although there is a translation of one unit to the left in each direction on the original scale (due to the term −1/λ in (4)).

The flexible class of t BC offers robustness against both outliers and asymmetry observed in data. Comparatively, the t distribution alone is deemed robust in the sense that it offers a mechanism to accommodate outliers. As noted by Lange et al. (1989), however, the t distribution is not robust against asymmetric error distributions. When asymmetry is observed, data transformation is desired for the sake of restoring symmetry, and subsequently drawing proper inferences. The introduction of the t BC is therefore in line with the notion of Lange et al. (1989).

2.3 The mixture model of t distributions with the Box-Cox transformation

2.3.1 The model

Making use of the t BC introduced in the last subsection, we now define a G-component mixture model in which each component is described by a t BC. Given data y, with independent p-dimensional observation vectors y_i, i = 1, …, n, the likelihood for the t BC mixture model is given as follows:

$$\begin{array}{l}L(\mathbf{\Psi }\mid \mathbf{y})=\prod _{i=1}^n\sum _{g=1}^G{\pi }_g{\varphi }_p({\mathbf{y}}_i^{({\lambda }_g)}\mid {\mathit{\mu }}_g,{\mathbf{\sum }}_g,{\nu }_g)·\mid J_p({\mathbf{y}}_i;{\lambda }_g)\mid ,\\ \sum _{g=1}^G{\pi }_g=1.\end{array}$$ (6)

The mixing proportion π_g is the probability that an observation belongs to the g-th component. Estimates of the unknown parameters Ψ = (Ψ₁, …, Ψ_G) where Ψ_g = (π_g, μ_g, Σ_g, ν_g, λ_g) can be obtained conveniently using the EM algorithm described in the next subsection. Analogous to the case of t BC, the mixture distribution approaches that of an NBC mixture model with ϕ_p(· |μ_g, Σ_g, ν_g) being replaced by φ_p(· |μ_g, Σ_g) when ν_g → ∞ for all g. Also, the class of t BC mixture models includes the conventional, untransformed t or normal mixture model, obtained by fixing λ_g = 1 for all g. Note that a restricted form of (6) has been previously applied in Lo et al. (2008) to identify cell populations in flow cytometry data, on setting a global transformation parameter λ = λ_g and fixing ν_g = 4 for all g.

2.3.2 Maximum likelihood estimation

In this subsection we illustrate how transformation selection can be handled along with parameter estimation simultaneously via an EM algorithm. We first define two types of missing data, same as the ones used in the maximum likelihood estimation for the t mixture model described in Peel and McLachlan (2000). One is the unobserved component membership z_i = (z_i1, …, z_iG) with

$$z_{ig}=\{\begin{array}{ll}1\hfill & \text{if}{\mathbf{y}}_i\text{belongs}\text{to}\text{the}g-\text{th}\text{component},\hfill \\ 0\hfill & \text{otherwise},\hfill \end{array}$$

associated with each observation y_i. Each vector Z_i follows independently a multinomial distribution with one trial and event properties π = (π₁, …, π_G), denoted as Z_i ~ (1, π). Another type of missing data is the weight u_i, coming from the normal-gamma compound parameterization for the t distribution, such that

$${\mathbf{Y}}_i\mid u_i,z_{ig}=1\sim \text{N}({\mathit{\mu }}_g,{\mathbf{\sum }}_g/u_i)$$ (7)

independently for i = 1, …, n, and U_i ~ Ga(ν_g/2, ν_g/2). The advantage of writing the model in this way is that, conditional upon the U_i ’s, the sampling errors are again normal but with different precisions, and estimation becomes a weighted least squares problem. The complete-data log-likelihood becomes

$$\begin{array}{l}{l}_c(\mathbf{\Psi }\mid \mathbf{y},\mathbf{z},\mathbf{u})\\ =\sum _{i=1}^n\sum _{g=1}^G{z}_{ig}\{log[{\pi }_g{\phi }_p({\mathbf{y}}_i^{({\lambda }_g)}\mid {\mathit{\mu }}_g,{\mathbf{\sum }}_g/u_i)\\ ·\mid J_p({\mathbf{y}}_i;{\lambda }_g)\mid ]+log\text{Ga}(u_i\mid {\nu }_g/2,{\nu }_g/2)\}\\ =\sum _{i=1}^n\sum _{g=1}^G{z}_{ig}\{log{\pi }_g-\frac{p}{2}log(2\pi )-\frac{1}{2}log\mid {\mathbf{\sum }}_g\mid \\ -\frac{u_i}{2}{({\mathbf{y}}_i^{({\lambda }_g)}-{\mathit{\mu }}_g)}^T{\mathbf{\sum }}_g^{-1}({\mathbf{y}}_i^{({\lambda }_g)}-{\mathit{\mu }}_g)\\ +({\lambda }_g-1)\sum _{j=1}^{p}log\mid y_{ij}\mid +\frac{{\nu }_g}{2}log\frac{{\nu }_g}{2}-log\mathrm{\Gamma }\left(\frac{{\nu }_g}{2}\right)\\ +\frac{{\nu }_g}{2}(log{u}_i-u_i)+\left(\frac{p}{2}-1\right)log{u}_i\},\end{array}$$ (8)

where Ga(·| ·) is the density function of u_i. The E-step of the EM algorithm involves the computation of the conditional expectation of the complete-data log-likelihood E_Ψ (l_c|y). To facilitate this, we need to compute z̃_ig ≡ E_Ψ (Z_ig |y_i), ũ_ig ≡ E_Ψ (U_i |y_i, z_ig = 1) and s̃_ig ≡ E_Ψ(log U_i |y_i, z_ig = 1):

$${\stackrel{\sim }{z}}_{ig}\leftarrow \frac{{\pi }_g{\varphi }_p({\mathbf{y}}_i^{({\lambda }_g)}\mid {\mathit{\mu }}_g,{\mathbf{\sum }}_g,{\nu }_g)·\mid J_p({\mathbf{y}}_i;{\lambda }_g)\mid }{{\sum }_{k=1}^G{\pi }_k{\varphi }_p({\mathbf{y}}_i^{({\lambda }_k)}\mid {\mathit{\mu }}_k,{\mathbf{\sum }}_k,{\nu }_k)·\mid J_p({\mathbf{y}}_i;{\lambda }_k)\mid },$$ (9)

$${\stackrel{\sim }{u}}_{ig}\leftarrow \frac{{\nu }_g+p}{{\nu }_g+{({\mathbf{y}}_i^{({\lambda }_g)}-{\mathit{\mu }}_g)}^T{{\mathbf{\sum }}_g}^{-1}({\mathbf{y}}_i^{({\lambda }_g)}-{\mathit{\mu }}_g)}$$ (10)

and

$${\stackrel{\sim }{s}}_{ig}\leftarrow log{\stackrel{\sim }{u}}_{ig}+\psi \left(\frac{{\nu }_g+p}{2}\right)-log\left(\frac{{\nu }_g+p}{2}\right),$$ (11)

where ψ(·) is the digamma function. Note that, if we assume a global transformation parameter λ, (9) used to compute z̃_ig is slightly simplified as

$${\stackrel{\sim }{z}}_{ig}\leftarrow \frac{{\pi }_g{\varphi }_p({\mathbf{y}}_i^{(\lambda )}\mid {\mathit{\mu }}_g,{\mathbf{\sum }}_g,{\nu }_g)}{{\sum }_{k=1}^G{\pi }_k{\varphi }_p({\mathbf{y}}_i^{(\lambda )}\mid {\mathit{\mu }}_k,{\mathbf{\sum }}_k,{\nu }_k)}.$$ (12)

As can be seen in the following, s̃_ig only appears in (18) or (19) for the update of the degrees of freedom ν_g. If we fix ν_g to some predetermined value, then s̃_ig is not needed and so the quantity in (11) does not need to be computed. Upon plugging z̃_ig, ũ_ig and s̃_ig into (8) for z_ig, u_i and log u_i respectively, we obtain the conditional expectation of the complete-data log-likelihood.

In the M-step, we update the parameter estimates with values which maximize the conditional expectation of the complete-data log-likelihood. The mixing proportions are updated with the following formula:

$${\widehat{w}}_g\leftarrow \frac{n_g}{n},$$ (13)

where n_g ≡ Σ_i z̃_ig. The estimation of μ_g and Σ_g needs to be considered along with the transformation parameter λ_g of the Box-Cox transformation. Closed-form solutions for μ_g and Σ_g are available conditional on λ_g as follows,

$${\widehat{\mathit{\mu }}}_g=\frac{{\sum }_{i=1}^n{\stackrel{\sim }{z}}_{ig}{\stackrel{\sim }{u}}_{ig}{\mathbf{y}}_i^{({\lambda }_g)}}{{\sum }_{i=1}^n{\stackrel{\sim }{z}}_{ig}{\stackrel{\sim }{u}}_{ig}}=h_1({\lambda }_g);$$ (14)

$$\begin{array}{l}{\widehat{\mathbf{\sum }}}_g=\frac{{\sum }_{i=1}^n{\stackrel{\sim }{z}}_{ig}{\stackrel{\sim }{u}}_{ig}({\mathbf{y}}_i^{({\lambda }_g)}-{\widehat{\mathit{\mu }}}_g){({\mathbf{y}}_i^{({\lambda }_g)}-{\widehat{\mathit{\mu }}}_g)}^T}{n_g}\\ =h_2({\lambda }_g).\end{array}$$ (15)

No closed-form solution is available for λ_g, but on substituting μ̂_g = h₁(λ_g) and Σ̂_g = h₂(λ_g) into the conditional expectation of the complete-data log-likelihood for μ_g and Σ_g respectively, the problem reduces to a one-dimensional search of λ_g. Explicitly, the optimization is recast as a one-dimensional root-finding problem of the equation ∂E_Ψ (l_c|y)/∂λ_g = 0, in which

$$\begin{array}{l}\frac{\partial E_{\mathbf{\Psi }}(l_c\mid \mathbf{y})}{\partial {\lambda }_g}=\frac{\partial }{\partial {\lambda }_g}\sum _{i=1}^n{\stackrel{\sim }{z}}_{ig}\{-\frac{{\stackrel{\sim }{u}}_{ig}}{2}\left[{({\mathbf{y}}_i^{({\lambda }_g)}-{\mathit{\mu }}_g)}^T{\mathbf{\sum }}_g^{-1}({\mathbf{y}}_i^{({\lambda }_g)}-{\mathit{\mu }}_g)\right]+({\lambda }_g-1)\sum _{j=1}^{p}log\mid y_{ij}\mid \}\\ =\sum _{i=1}^n\left[-{\stackrel{\sim }{z}}_{ig}{\stackrel{\sim }{u}}_{ig}{({\mathbf{y}}_i^{({\lambda }_g)}-{\mathit{\mu }}_g)}^T{\mathbf{\sum }}_g^{-1}\right]\frac{\partial {\mathbf{y}}_i^{({\lambda }_g)}}{\partial {\lambda }_g}+\sum _{i=1}^n{\stackrel{\sim }{z}}_{ig}\sum _{j=1}^{p}log\mid y_{ij}\mid ,\end{array}$$ (16)

where $\partial {\mathbf{y}}_i^{({\lambda }_g)}/\partial {\lambda }_g$ is a vector of length p whose j -th element is ${\lambda }_g^{-2}[sgn(y_{ij})\mid y_{ij}{\mid }^{{\lambda }_g}({\lambda }_{g}log\mid y_{ij}\mid -1)+1]$, and μ_g and Σ_g are replaced with μ̂_g = h₁(λ_g) and Σ̂_g = h₂(λ_g) respectively. The equation may be solved numerically using, for example, Brent’s (1973) algorithm. If we assume a global transformation parameter λ instead, the left hand side of the equation to consider is slightly modified from (16) as

$$\frac{\partial E_{\mathbf{\Psi }}(l_c\mid \mathbf{y})}{\partial \lambda }=\sum _{i=1}^n\left\{\sum _{g=1}^G\left[-{\stackrel{\sim }{z}}_{ig}{\stackrel{\sim }{u}}_{ig}{({\mathbf{y}}_i^{(\lambda )}-{\mathit{\mu }}_g)}^T{\mathbf{\sum }}_g^{-1}\right]\right\}\frac{\partial {\mathbf{y}}_i^{(\lambda )}}{\partial \lambda }+\sum _{i=1}^n\sum _{j=1}^{p}log\mid y_{ij}\mid .$$ (17)

Once a numerical estimate of λ_g has been obtained, we substitute it back into (14–15) to update μ_g and Σ_g respectively.

To complete the M-step, we need to update the estimate of the degrees of freedom ν_g, unless it is fixed a priori. From (8), we see that there are no overlaps between terms involving (μ_g, Σ_g, λ_g) and those involving ν_g. Hence, the incorporation of the Box-Cox transformation does not complicate the estimation of ν_g. Again, since there is no closed-form solution available for ν_g, we turn it into a one-dimensional root-finding problem by considering the equation ∂E_Ψ (l_c | y)/∂ν_g = 0, in which

$$\begin{array}{l}\frac{\partial E_{\mathbf{\Psi }}(l_c\mid \mathbf{y})}{\partial {\nu }_g}=\frac{\partial }{\partial {\nu }_g}\sum _{i=1}^n{\stackrel{\sim }{z}}_{ig}\left\{\frac{{\nu }_g}{2}log\frac{{\nu }_g}{2}-log\mathrm{\Gamma }\left(\frac{{\nu }_g}{2}\right)+\frac{{\nu }_g}{2}({\stackrel{\sim }{s}}_{ig}-{\stackrel{\sim }{u}}_{ig})\right\}\\ \propto n_g\left\{log\frac{{\nu }_g}{2}+1-\psi \left(\frac{{\nu }_g}{2}\right)\right\}+\sum _{i=1}^n{\stackrel{\sim }{z}}_{ig}({\stackrel{\sim }{s}}_{ig}-{\stackrel{\sim }{u}}_{ig}).\end{array}$$ (18)

If we assume a global degrees of freedom ν = ν_g for all g, the derivative ∂E_Ψ (l_c | y)/∂ν is given by

$$\frac{\partial E_{\mathbf{\Psi }}(l_c\mid \mathbf{y})}{\partial \nu }\propto n\left\{log\frac{\nu }{2}+1-\psi \left(\frac{\nu }{2}\right)\right\}+\sum _{i=1}^n\sum _{g=1}^G{\stackrel{\sim }{z}}_{ig}({\stackrel{\sim }{s}}_{ig}-{\stackrel{\sim }{u}}_{ig}).$$ (19)

Alternatively, to improve the convergence, we may exploit the advantage of the ECME algorithm and switch to update ν by optimizing the constrained actual log-likelihood function:

$$\widehat{\nu }\leftarrow \underset{\nu }{argmax}\left\{\sum _{i=1}^{n}log\left(\sum _{g=1}^G{\pi }_g{\varphi }_p({\mathbf{y}}_i^{({\lambda }_g)}\mid {\mathit{\mu }}_g,{\mathbf{\sum }}_g,\nu )·\mid J_p({\mathbf{y}}_i;{\lambda }_g)\mid \right)\right\}.$$ (20)

Apart from an intuitive sense that a faster convergence is expected on disregarding the information of the parameter estimates obtained from the previous iteration (which is carried over by the conditional expectation of the complete-data log-likelihood otherwise) as well as considering the actual likelihood instead of its approximation, it also saves a little computational burden by circumventing the computation of s̃_ig.

The EM algorithm alternates between the E and M-steps until convergence. The quantity z̃_ig may be interpreted as the posterior probability that observation y_i belongs to the g-th component. The maximum a posteriori configuration results from assigning each observation to the component associated with the largest z̃_ig value. The uncertainty corresponding to each assignment may be conveniently quantified as 1 − max_g z̃_ig (Bensmail et al. 1997).

2.3.3 Outlier identification

Just like the case of z̃_ig, the introduction of ũ_ig does not only facilitate the implementation of the EM algorithm, but also aids in the interpretation of the final estimated model. As seen from (14–15), ũ_ig serves as the weight in the weighted least squares estimation of μ_g and Σ_g. It holds a negative relationship with the Mahalanobis distance ${({\mathbf{y}}_i^{({\lambda }_g)}-{\mathit{\mu }}_g)}^T{{\mathbf{\sum }}_g}^{-1}({\mathbf{y}}_i^{({\lambda }_g)}-{\mathit{\mu }}_g)$ between y_i and μ_g on the transformed scale, as given by (10). Hence, a small value of ũ_ig would suggest that the corresponding observation is an outlier, and diminish its influence on the estimation of the parameters. In contrast, in the absence of such a mechanism, a normal mixture model is not robust against outliers, as the constraint Σ_g z̃_ig = 1 for all i restricts all observations to make equal contributions overall towards parameter estimation.

Exploiting such a mechanism, we may conveniently set up a rule of calling an observation with the associated ũ_ig value smaller than a threshold, say, 0.5, an outlier. Such a threshold may be selected on a theoretical basis by considering the one-to-one correspondence between ũ_ig and the Mahalanobis distance which follows some standard, known distribution. On noting that

$${({\mathbf{y}}_i^{({\lambda }_g)}-{\mathit{\mu }}_g)}^T{{\mathbf{\sum }}_g}^{-1}({\mathbf{y}}_i^{({\lambda }_g)}-{\mathit{\mu }}_g)/p\sim F(p,{\nu }_g),$$ (21)

where ${\mathbf{y}}_i^{({\lambda }_g)}$ follows a p-dimensional t distribution with parameters (μ_g, Σ_g, ν_g) and F(·) denotes an F distribution, a threshold c for ũ_ig may be determined by considering the desired threshold quantile level α of the distribution stated in (21):

$$c=\frac{{\nu }_g+p}{{\nu }_g+{pF}_{1-\alpha }(p,{\nu }_g)},$$ (22)

where F_1−α(·) denotes the α quantile of the F distribution such that Pr(F ≥ F_1−α) = 1 − α. For instance, if ν_g = 4, p = 5, and the desired threshold quantile level is α = 0.9, then the corresponding threshold for ũ_ig is c = 0.37 given the 0.9 quantile F_0.1(5, 4) = 4.051. Any observation with the associated ũ_ig < 0.37 will be deemed an outlier.

From (10), we can also see how the degrees of freedom ν_g contributes to the robustness of the parameter estimation process. A smaller value of ν_g tends to downweight outliers to a greater extent, while a large enough value tends to regress all weights to one, approaching the case of the NBC model.

2.3.4 Density estimation

One advantage of mixture modeling based on the normal distribution is that the marginal distribution for any subset of the dimensions is also normally distributed with the mean and covariance matrix extracted from the conformable dimensions (Johnson and Wichern 2002). This favorable property is also observed in the multivariate t distribution (Liu and Rubin 1995; Kotz and Nadarajah 2004), making the estimation of the marginal density for any dimensions available at a very low computational cost. Consider the partition Y = (Y₁, Y₂) as an example. If Y comes from a multivariate t distribution with degrees of freedom and with mean and covariance matrix conformably partitioned as

$$\mathit{\mu }=({\mathit{\mu }}_1,{\mathit{\mu }}_2)\text{and}\frac{\nu }{\nu -2}\mathbf{\sum }=\frac{\nu }{\nu -2}\left(\begin{array}{cc}{\mathbf{\sum }}_{11}& {\mathbf{\sum }}_{12}\\ {\mathbf{\sum }}_{21}& {\mathbf{\sum }}_{22}\end{array}\right)$$

respectively, then its subset Y₁ will follow a t distribution with mean μ₁, covariance matrix ν/(ν − 2)Σ₁₁ and the same ν degrees of freedom. This nice property is easily extended to a t mixture model with more than one component, and, in addition, preserved in our proposed t BC mixture model. One can easily derive the marginal density by extracting the conformable partitions from the means, covariance matrices and the Jacobian. The 90th percentile region of the mixture components shown in Supplementary Fig. 1 (Online Resource 1) is produced by these means.

2.3.5 Selecting the number of components

When the number of mixture components is unknown, we apply the Bayesian Information Criterion (BIC) (Schwarz 1978) to guide the selection. The BIC provides a convenient approximation to the integrated likelihood of a model and, in the context of mixture models, is defined as

$${\text{BIC}}_G=2log{\stackrel{\sim }{L}}_G-K_{G}logn,$$ (23)

where L̃_G is the likelihood value of (6) evaluated at the maximum likelihood estimates of Ψ, and K_G is the number of independent parameters for a G-component mixture model. The BIC would then be computed for a range of possible values for G and the one with the largest BIC (or relatively close to it) would be selected. Although the asymptotic approximation of the integrated likelihood leading to the BIC depends on regularity conditions that are not satisfied in the context of mixture models, theoretical and practical justifications can be found in favor of the application of the BIC on model selection for mixture models. Leroux (1992) shows that the BIC will not underestimate the number of components asymptotically. Keribin (2000) shows that the BIC gives a consistent estimate of the number of components. Empirical evidence justifying the use of the BIC in the context of mixture models can be found in a wealth of literature; see, for example, Fraley and Raftery (1998, 2002), and more recently, McNicholas and Murphy (2008), Wang et al. (2009) and Andrews and McNicholas (2010).

Often, the BIC is applied in line with the principle of parsimony, by which we favor a simpler model if it does not incur a downgrade of the modeling performance. Suppose there are two t BC mixture models with G₁ and G₂ components respectively such that G₁ < G₂. Under the notion of this principle, we would prefer the simpler model, i.e., the one with G₁ components, unless a very strong evidence of improved performance signified by an increase of >10 (Kass and Raftery 1995; Fraley and Raftery 2002) is observed from BIC_G1 over BIC _G1.

3 Application to real data

3.1 Data description

To illustrate our methodology we use the following real data sets.

3.1.1 The bankruptcy data set

This data set was obtained from a study which conducted financial ratio analysis to predict corporate bankruptcy (Altman 1968). The sample consists of 66 manufacturing firms in the United States, of which 33 are bankrupt and the other 33 solvent. The data collected include the ratio of retained earnings (RE) to total assets, and the ratio of earnings before interest and taxes (EBIT) to total assets. They were derived from financial statements released two years prior to bankruptcy, and statements from the solvent firms during the same period.

3.1.2 The crabs data set

Measurements in this data set were collected from a study of rock crabs of genus Leptograpsus (Campbell and Mahon 1974). The sample is composed of 50 crabs for each combination of species (blue and orange color forms) and sex (male and female), resulting in a total of 200 observations. There are five morphological measurements, namely, the frontal lobe size, the width of the rear region of the carapace, the length of the carapace along the midline, the maximum width of the carapace, and the depth of the body, for each crab.

3.1.3 The wine data set

To identify the discriminating factors from a set of chemical and physical characteristics to classify wines by type and origin, a study with a sample of 178 red wines grown in Piedmont, Italy was conducted (Forina et al. 1986). Three cultivars were represented in the sample: Barolo (59), Grignolino (71) and Barbera (48). For each wine 28 measurements were made. Here, we consider a 13-variable data subset available in the gclusR package (Hurley 2004); the 13 continuous measurements include alcohol, malic acid, ash, alkalinity of ash, magnesium, total phenols, flavanoids, non-flavanoid phenols, proanthocyanins, color intensity, hue, OD280/OD315 of diluted wines and proline.

3.2 Implementation

We compare the performance of six mixture modeling approaches using different mixture distributions, namely, t with the Box-Cox transformation (t BC), t, normal with the Box-Cox transformation (NBC), normal, skew-t, and skew-normal. Since all observations in the data sets come with known labels, we can assess and compare the models based on misclassification rates. For the bankruptcy and crabs data sets, we also take the number of components selected by the BIC as an assessment criterion. In the analysis of the wine data set, we examine an interesting problem that how dimension reduction refines the classification performance.

We first fit the data sets using the six aforementioned models in turn, on fixing the number of mixture components at the known values, i.e., two for the bankruptcy data set, four for the crabs data set, and three for the wine data set. The same initialization strategy is applied to the EM algorithm for all the models. Each time, 10 random partitions are generated. Following each partition formed, z̃_ig will be assigned one or zero accordingly, and the model parameters are initialized using the formulae in the M-step. A few EM runs ensue, terminated at a premature tolerance level of 10⁻³ for the relative change in the likelihood values between two successive iterations. Out of the 10 random partitions, the one delivering the highest likelihood value after a few EM runs will be taken as the initial configuration for the eventual EM algorithm. Convergence of the EM algorithm is detected at a tolerance level of 10⁻⁶. At convergence, misclassification rates, i.e., the proportions of observations assigned to the incorrect group, are computed. Each misclassification rate is determined as the minimum considering all permutations of the labels of the components.

To facilitate the comparison of their capability to select the correct number of components, when we apply the aforementioned models, we fit the data and compute the BIC once for each choice of the number of mixture components G = 1, 2, …, M, where M = 6 for the bankruptcy data set and M = 8 for the crabs data set. These values are chosen for M because they are well above the true number of groups (two for the bankruptcy data set and four for the crabs data set) such that little change in the result is expected when we further increase M; numerical problems may arise when M is too large, moreover.

All results presented here were obtained with our software implementation of clustering for flow cytometry, flow-Clust (Lo et al. 2009), except for the skew-normal and skew-t mixture modeling that were obtained with the FLAME software (http://www.broadinstitute.org/cancer/software/genepattern/modules/FLAME/) developed by Pyne et al. (2009).

3.3 Results

3.3.1 Classification

Table 1 shows the misclassification rates for the different models. As can be seen, for the bankruptcy data set, the t BC and NBC mixture models deliver misclassification rates (15.2% and 16.7% respectively) lower than the other methods by a wide margin. By taking a graphical inspection of the results, we find that the poor classification performance of the other four methods is due to the inability to resolve the shape of the two groups of observations properly (Fig. 2(b, d–f)). The challenge likely arises from the scattered group of bankrupt firms, with its most concentrated region located at the upper right corner and in close proximity to the dense group of solvent firms. The sensitivity of normal mixture models to outliers is clearly demonstrated in this example: the obvious outlier at the bottom of the scatter-plot leads to an excessively sparse component representing the bankrupt group. Consequently, most observations in the bankrupt group have been absorbed by the compact component representing the solvent group. The shapes of the components in the t, skew-t and skew-normal mixture models are not all the same, but it appears that for all of them the scattered group of bankrupt firms are split into two components with one absorbing a concentration extending to the left and the other to the bottom. In contrast, both the t BC and NBC mixture models provide a nice representation of both groups of observations (Fig. 2(a, c)). The group of bankrupt firms is resolved quite successfully upon a proper transformation (λ̂≈ 0.5 for both models) of the observations.

Table 1.

Misclassification rates for different models applied to the bankruptcy and crabs data sets

Model	Bankruptcy	Crabs
tBC	0.152 (10)	0.070 (14)
t	0.273 (18)	0.075 (15)
NBC	0.167 (11)	0.345 (69)
Normal	0.318 (21)	0.290 (58)
Skew-t	0.303 (20)	0.085 (17)
Skew-normal	0.394 (26)	0.175 (35)

The best results are shown in bold. The numbers of misclassified cases are given within parentheses

Fig. 2.

Scatterplots revealing the assignment of observations for different models applied to the bankruptcy data set. The black solid lines represent the 90th percentile region of the components in the mixture models. Misclassified observations are drawn in red

As another means of performance assessment, we look into the location of the misclassified observations in a plot of the ordered uncertainties (Fig. 3). On observing that the misclassified observations have spread over the entire range of the uncertainties, it suggests that the t, skew-t and skew-normal mixture models simply provide an incorrect representation of the two groups (Fig. 3(b, e, f)). The quality of the fit using the t BC and NBC mixture models respectively is confirmed by the corresponding uncertainty plots (Fig. 3(a, c)). The observations associated with high uncertainties are also the ones most likely to be misclassified.

Fig. 3.

Plots revealing the location of misclassified observations relative to the ordered uncertainties of all observations for different models applied to the bankruptcy data set. Locations of the misclassified observations are marked with red vertical lines

The results on the crabs data set once again show that the t BC mixture model delivers the best performance in terms of misclassification rate (7%). It is followed closely by the t (7.5%) and skew-t (8.5%) mixture models. The crabs data set has also been recently analyzed by Raftery and Dean (2006), Bouveyron et al. (2007) and Scrucca (2010), who employed dimension reduction techniques or subspace clustering to improve Gaussian mixture modeling on high-dimensional data by seeking a more informative subspace. Their reported misclassification rates range between 5–7.5%. Achieving competitive performance against these approaches, the t BC model certainly appears favorable facilitated by allowing for more flexible shapes of the components on the original data space.

Supplementary Fig. 1 (Online Resource 1) shows a scatterplot of the crabs data set projected onto the first two dimensions, namely, the frontal lobe size and the width of the rear region of the carapace. However, unlike the case for the bankruptcy data set with only two dimensions, a visually clear discrimination of the four groups in the crabs data set cannot be achieved by projecting the observations onto any two out of the five dimensions. Therefore, we opt for displaying the crabs data set on its second versus third principal components which provides a good visually discriminating effect. Figure 4(a, b) suggest that those few misclassified observations in the t BC and t mixture models are all likely in the overlapping region of neighboring groups, justifying that these models provide a good representation of all the four groups in the data set. This is further confirmed by a check on the uncertainty plots, in which the misclassified observations are also among the ones with the highest uncertainties (Fig. 5(a, b)). Meanwhile, from Fig. 4(c, d, f) we find that, for the poorly performed NBC, normal and skew-normal mixture models, misclassified cases are concentrated on one or two of the groups. Supplementary Fig. 1(c, d, f) (Online Resource 1) reveal that these models incorrectly split the assignment of the observations from those groups into other components. As expected, these three poorly performing normal-based models have misclassified observations spreading over the entire range in the uncertainty plots (Fig. 5(c, d, f)).

Fig. 4.

Plots revealing the assignment of observations for different models applied to the crabs data set, displayed via the second and third principal components. Misclassified observations are drawn in red, overriding the original colors used to reveal their true group memberships

Fig. 5.

Plots revealing the location of misclassified observations relative to the ordered uncertainties of all observations for different models applied to the crabs data set. Locations of the misclassified observations are marked with red vertical lines

In the following, we examine whether a pleasant classification performance persists when we apply the t BC mixture modeling methodology to the wine data set with a higher dimension (p = 13). From Table 2, we see that the t BC mixture model delivers a misclassification rate (10.7%) drastically lower than the other models (28.1%–38.2%). The high misclassification rates observed likely arise from the fact that the number of free parameters to estimate increases in the order of p², while the sample size is not very large (n = 178) relative to the data dimension. We therefore attempt to reduce the data dimension via principal component analysis and proceed with the major principal components that account for the majority of the total variance in the data. A turning point at four is observed from the scree plot (Supplementary Fig. 2 (Online Resource 2)), suggesting that the first four principal components are the major contributors to the total variance; the cumulative proportion of the total variance explained by these four principal components is about 80%. We thereby extract the first four principal components, repeat the model-fitting procedure as detailed in Sect. 3.2, and determine the misclassification rates. As reported in Table 2, a significant improvement in classification has been observed for all the six approaches. With the exception of the skew-normal mixture model, each of them succeeds in arriving at a much lower misclassification rate of 5.1%–7.3%. The t BC mixture model still manages to deliver marginally the best classification performance among the six approaches. Its count of nine misclassified observations lags behind the result of Andrews and McNicholas (2010) who correctly classified all but one of the observations, but levels the finding reported in Scrucca (2010); both cited references integrated dimension reduction techniques into Gaussian or t mixture modeling.

Table 2.

Misclassification rates for different models applied to the wine data set and its first four principal components

Model	Wine	Prin. Comp.
tBC	0.107 (19)	0.051 (9)
t	0.320 (57)	0.062 (11)
NBC	0.382 (68)	0.062 (11)
Normal	0.303 (54)	0.062 (11)
Skew-t	0.281 (50)	0.073 (13)
Skew-normal	0.354 (63)	0.124 (22)

The best results are shown in bold. The numbers of misclassified cases are given within parentheses

Figure 6 displays the wine data set projected onto its first and second principal components. The five models represented in Fig. 6(a–e) accordingly by and large recover the group structure of the data, while the skew-normal mixture model poorly identifies the separation between Barolo and Grignolino wines. As in the analysis of the crabs data set, once again, all the misclassified observations are found in the rightmost region of the plot of ordered uncertainties for the t BC and t mixture models (Fig. 7(a–b)). On the contrary, from Fig. 7(f) we see that five of the misclassified observations in the skew-normal mixture model fall in the least-uncertainty region (left end of the plot). This empirically justifies its failure to recover the shape of the groups of the wine data set.

Fig. 6.

Plots revealing the assignment of observations for different models applied to the wine data set, displayed via the first and second principal components. Misclassified observations are drawn in red, overriding the original colors used to reveal their true group memberships

Fig. 7.

Plots revealing the location of misclassified observations relative to the ordered uncertainties of all observations for different models applied to the wine data set. Locations of the misclassified observations are marked with red vertical lines

3.3.2 Selecting the number of components

From the BIC curves shown in Fig. 8, we observe that one single peak is observed for each modeling choice over the range of the number of components attempted. The number of components at which a peak is observed is deemed optimal by the BIC for the respective model. The BIC has selected the correct number of components (two) for all the mixture models except normal when applied to the bankruptcy data set (Table 3). As to the crabs data set in which the separation of the groups is less clear-cut, it poses a challenge of selecting the right number of components to most models. Only the t BC and t mixture models have recovered the correct number of components (four) guided by the BIC. This result further confirms our observation stated in the last subsection that the four-component mixture model using the t BC or t mixture model provides the best representation of the data out of all candidates.

Fig. 8.

Plots of BIC against the number of components for the different models applied to the bankruptcy and crabs data sets

Table 3.

The number of components selected by the BIC for different models applied to the bankruptcy and crabs data sets

Model	Bankruptcy	Crabs
tBC	2	4
t	2	4
NBC	2	3
Normal	3	3
Skew-t	2	2
Skew-normal	2	3

The best results are shown in bold

4 Simulation studies

We have conducted a series of simulations to further evaluate the relative performance of our proposed framework to the other approaches presented in Sect. 3.2. The different approaches are evaluated for their sensitivity against model misspecification, using the following two criteria: the accuracy in the assignment of observations, and the accuracy in selecting the number of components.

4.1 Implementation

To facilitate the comparison, we generate data from the following mixture models: t BC, skew-t, t and normal. To assess the accuracy in the assignment of observations, two settings of parameter values have been adopted: one taken from the estimates obtained when applying each of the aforementioned models to the bankruptcy data set, and the other one from the crabs data set, with the number of components set as the respective known values. As a result, each data set generated from the bankruptcy setting consists of two components and two dimensions, while that from the crabs setting has four components and five dimensions. For data sets generated under the bankruptcy setting we fix the number of observations at 200, while it is set as 500 for the crabs setting. 100 data sets are generated from each of the aforementioned models under each setting. To study the accuracy in selecting the number of components, we focus at the crabs setting. Pertaining to this criterion, the crabs setting offers a better platform to discriminate the relative performance of the different approaches for its larger number of groups and higher dimensions. 1000 observations are generated from the crabs setting instead to avoid numerical problems that may arise from small components formed when the number of components is significantly larger than the true number.

We apply the six approaches presented in Sect. 3.2 in turn to each generated data set. The same implementation details elucidated in Sect. 3.2 apply here, with the exceptions stated below. In order to complete the simulation studies within a reasonable timeframe, whilst guaranteeing a satisfactory level of convergence of the EM algorithm, the premature tolerance level and the convergence tolerance level are changed to 10⁻² and 10⁻⁵ respectively.

In the study of the accuracy in the assignment of observations, model fitting is done by presuming that the number of components is known, i.e., two for the bankruptcy setting and four for the crabs setting. Similar to the way we determined the misclassification rates in our real data analysis, we consider all permutations of the labels of the components and take the lowest one out of all misclassification rates computed. The performance of the different models is compared via the average misclassification rates. To study the accuracy in selecting the number of components, each time when we apply a model to a data set generated under the crabs setting, we set the number of components from one up to eight in turn. The number of components is then selected to be the one which delivers the highest BIC.

On a sidetrack, as suggested by a reviewer of this article, we make a parallel run of the entire simulation study of the accuracy in the assignment of observations by replacing random partitioning with 10 partitions returned by K-means clustering (MacQueen 1967) in initialization. All implementation settings are exactly the same as what have been stated above, otherwise. The purpose of conducting such a parallel run is to see if K-means, a popular choice for initialization in clustering analysis, improves classification by providing apparently more refined partitions than random partitioning.

4.2 Results

4.2.1 Classification

As shown clearly in Table 4 that summarizes the result using random partitioning in initialization, our proposed t BC mixture model is the only model that remains the best or close to the best in all the comparisons made. It delivers the lowest misclassification rates under both settings (7.5% and 1.1% respectively) when data are generated from the t BC mixture model. The flexibility of the t BC mixture model is exhibited when we look into its performance in the scenario of model misspecification. It remains close to the respective true model in those cases, and even delivers the lowest misclassification rate when the true model is t under the bankruptcy setting (10.9%), or normal under the crabs setting (2.7%). Contrariwise, when data are generated from the t BC mixture model, with a lack of mechanisms to handle asymmetric components, both the t and normal mixture models do not perform well. It is worth noting that even the skew-t mixture model, which is intended for data departing from symmetry, also performs poorly; the associated mis-classification rate is as high as 14.2% under the bankruptcy setting, while that for t BC is only 7.5%. When data are generated from the skew-t mixture model, taking advantage of the correct specification the skew-t mixture model performs well. The t BC mixture model also shows a competent performance, however. Meanwhile, the skew-t mixture model performs satisfactorily when the true mixture model is t or normal. The normal mixture model cannot match the others at all when data are generated from any other model included in this study, showing its vulnerability to outliers and asymmetric components. In addition, it is interesting to notice that the normal mixture model gives a rather high mis-classification rate (3.8%) relative to the levels attained by t BC, t and skew-t (2.7%–2.9%) when it itself is the true model for data generation under the crabs setting.

Table 4.

Average misclassification rates for different models applied to data sets generated under the bankruptcy or crabs setting

		Model used to fit data
		tBC	t	NBC	Normal	Skew-t	Skew-normal
Model used to generate data under the bankruptcy setting	tBC	0.075	0.124	0.075	0.126	0.142	0.110
	Skew-t	0.100	0.094	0.225	0.189	0.087	0.191
	t	0.109	0.109	0.126	0.134	0.114	0.144
	Normal	0.032	0.032	0.033	0.030	0.032	0.031
Model used to generate data under the crabs setting	tBC	0.011	0.014	0.057	0.074	0.015	0.016
	Skew-t	0.024	0.023	0.046	0.060	0.020	0.021
	t	0.024	0.021	0.048	0.070	0.023	0.023
	Normal	0.027	0.029	0.042	0.038	0.028	0.028

The best results are shown in bold

The result of a parallel run of this part of study using K-means initialization is summarized in Supplementary Table 1 (Online Resource 3). Compared with Table 4, whilst a slight increase overall in the misclassification rates pertaining to the bankruptcy setting when using K-means is observed, the increase as to the crabs setting is significant and consistent across all the six approaches in comparison. We, however, do not deem this finding counter-intuitive. As can be seen, there is considerable skewness underlying the groups in data sets generated under the bankruptcy or crabs setting. K-means is equivalent to the classification EM algorithm for a normal mixture model assuming a common covariance matrix spherical in shape (Celeux and Govaert 1992; Celeux and Govaert 1995). These assumptions are obviously violated by the asymmetric groups found in the generated data sets. Such a violation of the assumptions incurs K-means to deliver misleading initial partitions, which ultimately converge to sub-optimal local maxima. The result presented here is a mere example revealing how an apparently more informative initialization scheme based upon an incorrect assumption can compromise the result. As such, initialization other than random partitioning that brings in additional assumptions should be used with caveat.

4.2.2 Selecting the number of components

Table 5 summarizes the result of this part of study, giving 90% coverage intervals of the number of components, together with numbers of times the correct number of components is selected for each model out of the 100 repetitions. The t BC mixture model selects the correct number of components (four) in the majority of repetitions, even in case of model misspecification. It is the only model that remains to contain only the true number of components in all the 90% coverage intervals. On the other hand, both the skew-t and skew-normal mixture models fail to distinguish the four groups properly in about 25% of the data sets generated from the t BC mixture model. Besides, both the NBC and normal mixture models, when applied to data sets generated from the t or skew-t mixture model, tend to require an additional component to accommodate the data in an excess of outliers.

Table 5.

90% coverage intervals of the number of components, and numbers of times the correct number of components is selected by the BIC for different models applied to data sets generated under the crabs setting

		Model used to fit data
		tBC		t		NBC		Normal		Skew-t		Skew-normal
		Int.	# corr.	Int.	# corr.	Int.	# corr.	Int.	# corr.	Int.	# corr.	Int.	# corr.
Model used to generate data	tBC	(4, 4)	96	(4, 4)	100	(4, 4)	96	(4, 4)	99	(3, 5)	72	(3, 5)	78
	Skew-t	(4, 4)	99	(4, 4)	96	(4, 5)	88	(4, 5)	90	(4, 4)	100	(4, 4)	100
	t	(4, 4)	97	(4, 4)	97	(4, 5)	94	(4, 5)	85	(4, 4)	100	(4, 4)	100
	Normal	(4, 4)	100	(4, 5)	95	(4, 4)	96	(4, 5)	95	(4, 4)	100	(4, 4)	100

The best results are shown in bold. Int. = Interval;

^#corr = number of times the correct number of components is selected

5 Discussion

In this article, we have introduced a new class of distributions, the t distributions with the Box-Cox transformation, for mixture modeling. The proposed methodology is in line with Lange et al. (1989) notion that transformation selection and outlier identification are two issues of mutual influence and therefore should be handled simultaneously. In our real data applications and simulation studies, we have shown the flexibility of this methodology in accommodating asymmetric components in the presence of outliers, and in coping with model misspecification. The vulnerability of the normal-based models to outliers is exposed in the analysis of the crabs data set, in which the presence of outliers prevents a clear distinction of the four groups. A lack of mechanisms to downsize the influence of remote observations undermines the ability of these approaches to properly locate the cores of the four groups in the data set. On the other hand, the analysis of the bankruptcy data set provides a very good example of demonstrating the importance of incorporating data transformation in clustering. In the absence of a means to accommodate components departing from symmetry, the t mixture model fails to provide a reasonable representation of the data, while the number of groups is known in advance. Our simulation studies have confirmed these findings.

As mentioned in the Introduction, although mixture modeling using our proposed t BC distributions and that using the skew-t distributions follow two lines of development with more or less the same aim, our approach has the appeal of being computationally much simpler to implement. As noted in Lin (2010), difficulties have been encountered in evaluating the conditional expectation of the complete-data log-likelihood in the E-step of the EM algorithm for the skew-t mixture model. The objective function cannot be derived in closed form due to the presence of analytically intractable quantities. Numerical techniques for optimization as well as integration need to be employed extensively to update a vast amount of quantities in both the E and M-steps of the algorithm, undermining the computational stability therein. Besides, the parameterization that accounts for skewness in our proposed model originates from the family of power transformations, which is intuitively interpretable. It is less trivial to interpret the skewness vector parameterized in the skew-t distribution, however. In addition, as presented in Sect. 2.3.3, the way to identify outliers using our approach is straightforward and supported with a theoretical justification: exploiting the relationship between ũ_ig and the quantile of an F distribution through (22), it is almost costless to proceed with outlier identification once the EM algorithm is completed. On the contrary, when the skew-t mixture model is used, we cannot determine such a threshold by recasting it as a known quantity obtained from a standard distribution. Consequently, it demands extra computational effort to identify outliers, especially when the dimension of the data is high. Finally, and perhaps most importantly, as demonstrated from our real data applications and simulation studies, the simplicity of the computational implementation of our proposed methodology is not achieved at the expense of the quality of performance. The results have shown that our proposed approach performs as well as that based on the skew-t mixture model, or even slightly better.

In this article, we present a flexible form of multivariate mixture modeling that simultaneously incorporates outlier identification and data transformation. Nevertheless, we are aware that the modeling performance may not be very satisfactory when we apply the proposed methodology to data sets with a high dimension, say, p ≥ 10. As noted by Fraley and Raftery (2002) and Bouveyron et al. (2007), mixture models in the Gaussian (or related) family suffer from an excessive number of parameters to estimate when the data dimension is high: the number of free parameters increases in the order of p². Incidentally, a high-dimensional data set likely contains variables redundant in providing clustering information. To resolve the aforementioned issues, we need to reduce the number of free parameters. Constrained parameterization of the covariance matrices via eigenvalue decomposition first proposed by Banfield and Raftery (1993) would be a potential direction to investigate at a subsequent stage. Another, but not mutually exclusive, approach is to incorporate dimension reduction techniques. In this article, the analysis of the wine data set has revealed the potential benefit of dimension reduction with a simple application of principal component analysis. More vigorous approaches (see, for example, McNicholas and Murphy 2008; Scrucca 2010; Andrews and McNicholas 2010) that integrate dimension reduction into mixture modeling would be desirable such that our proposed methodology can handle feature selection along with data transformation and outlier identification concurrently.

An open-source software package that facilitates flow cytometry analysis with the methodology proposed in this article has been developed and is available at Bioconductor (Gentleman et al. 2004). It is released as an R package called flowClust (Lo et al. 2009) and addresses the vast demand for software development from the flow cytometry community. flowClust is dedicated to the automated identification of cell populations, and is well integrated into other flow cytometry packages. Meanwhile, we recognize the potential of applying the proposed methodology in other fields, and the importance of developing a general-purpose tool like MCLUST (Fraley and Raftery 2002, 2006), the popular software that performs clustering analysis based on normal mixture models. We are going to work on such a general-purpose, stand-alone software that will serve as a contribution to the general public.