An overview of heavy-tail extensions of multivariate Gaussian distribution and their relations

Abstract

Many extensions of the multivariate normal distribution to heavy-tailed distributions are proposed in the literature, which includes scale Gaussian mixture distribution, elliptical distribution, generalized elliptical distribution and transelliptical distribution. The inferences for each family of distributions are well studied. However, extensions are overlapped or similar to each other, and it is hard to differentiate one extension from the other. For this reason, in practice, researchers simply pick one of many extensions and apply it to the analysis. In this paper, to enlighten practitioners who should conduct statistical procedures not based on their preferences but based on how data look like, we comparatively review various extensions and their estimators. Also, we fully investigate the inclusion and exclusion relations of different extensions by Venn diagrams and examples. Moreover, in the numerical study, we illustrate visual differences of the extensions by bivariate plots and analyze different scatter matrix estimators based on the microarray data.

1. Introduction

A multivariate normal distribution (MVN) has inarguably been the most fundamental and overarching cornerstone in the theory of multivariate statistics. It has been proven for more than centuries that it possesses a bundle of nice theoretical properties, which facilitate developments and applications of multivariate procedures in real-life problems. For example, equality of population mean can be tested through the Hotelling's T ${}^2$ procedure [26] under the Gaussian assumption, or hidden clusters of data can be identified by the model-based clustering [5]. However, it is an admitted fact that the multivariate analyses built upon the MVN assumption mostly do not show acceptable performance when the distributional assumption is violated. In modern sciences, diverse types of data have poured out, showing non-‘normal’ behaviors.

In this context, many alternatives or extensions of MVN are discussed in order to solve complex problems arising from many applications. The family of heavy-tailed distributions covers distributions having tail-decay slower than that of the Gaussian; for example, multivariate t distribution. Research on heavy-tail extension has brought considerable and interesting multivariate distributions. Out of them, the followings are dominantly used in modeling heavy-tail data: (1) scale Gaussian mixture distribution in image processing [22,44], (2) an elliptical distribution in covariance matrix estimation [12] and in mixture modeling [31], (3) a generalized elliptical distribution in scatter matrix estimation under missing data [19], (4) a non-paranormal distribution for estimating a graphical model [36], mutual information [47], and causality [41], and (5) a transelliptical distribution for sparse PCA [23] and separable covariance matrix estimation [42].

No doubt, there exist some overviews or reviews on the extended classes of multivariate distributions (not limited to the preceding examples); Adcock and Azzalini [1] cast their attention on a family of the skewed distributions, and Babic et al. [4] compared and classified the distributions according to their own criteria. As evidenced by the two review works, many extensions have been studied individually, but missing the mutual relations and equivalences. Thus, the goal of this paper is to clarify their inclusion relations and characterize distributional properties. We summarize the inclusions by Venn diagrams and examine whether they are strict by giving concrete examples. We also briefly address scatter matrix estimation in heavy-tailed distributions and relationships between the estimators. Furthermore, through numerical experiments, (1) we show how data from different heavy-tail families look different and (2) we compare the scatter matrix estimators when heavy-tailed gene expression data are given. We hope the account given in this paper will lessen unnecessary confusions that may have arisen in this field and thus serve as a guide for researchers who are concerned about possible remedies when data is against normality.

The remainder of this paper is as follows. In Section 2, we describe heavy-tailed distributions and explain related facts. In Section 3, we investigate the relationships between the class of distributions and between available estimators of a scatter matrix. In Section 4, we show distributions of heavy-tailed distributions and empirically compare the estimators based on the gene expression data. Section 5 ends this paper with summary and comments.

2. Heavy-tail extensions

We briefly introduce definitions of the heavy-tailed distributions mentioned at the beginning of this section. Individual characteristics are to be explained in the subsequent subsections. In this paper, we mainly use a stochastic representation of a random vector that follows a heavy-tailed distribution. For different characterizations such as using a characteristic function or a probability density function, we refer readers to [17].

Assume a p-dimensional random vector $\mathit{X}=(X_1,\dots ,X_p{)}^{\mathrm{T}}$ satisfies the following stochastic representation:

$$\mathit{X}\stackrel{d}{=}\mathit{\mu }+R\mathit{A}{\mathit{U}}_r,$$ (1)

where $\stackrel{d}{=}$ means equality in distribution, ${\mathit{U}}_r$ is a random vector uniformly distributed on the unit $(r-1)$-sphere in ${\mathbb{R}}^r$, R is a scalar random variable in $\mathbb{R}$, $\mathit{\mu }$ is a (deterministic) vector in ${\mathbb{R}}^p$, and $\mathit{A}$ is a (deterministic) $p\times r$ matrix. The class of distributions defined as above contains the location-scale parameters ( $\mathit{\mu }$: a location parameter and $\mathbf{\Sigma }=\mathit{A}{\mathit{A}}^{\mathrm{T}}$ : a scatter matrix) as in the Gaussian distribution. However, an additional parameter R controls the tail behavior, which produces diverse characteristics.

By specifying model parameters $\mathit{\mu }$, R, and $\mathit{A}$ or utilizing transformations on $\mathit{X}$, we can express different heavy-tailed distributions through (1). For instance, the multivariate normal distribution with an identity covariance matrix corresponds to the above definition (1) with

$$R=\sqrt{{\chi }_p^2},\mathit{A}={\mathbf{I}}_p,$$

where R is distributed independently of ${\mathit{U}}_p$ and ${\chi }_p^2$ is a random variable from the chi-square distribution with degrees of freedom p. One should aware that suitable identifiability conditions on R and $\mathit{A}$ are needed in this characterization [23,42]. A necessity of the conditions is because $R^{\prime }=cR$ and ${\mathit{A}}^{\prime }=(1/c)\mathit{A}$ for any positive factor c>0 result in the same representation of $\mathit{X}$. Here, we assume $\underset{j}{max}{\mathbf{\Sigma }}_{jj}=1$ if needed, following [23].

Definition 2.1

Assume $\mathit{X}$ is a multivariate random vector in ${\mathbb{R}}^p$ and has a stochastic representation (1) with $\mathrm{rank}(\mathbf{\Sigma })=r(\le p)$. Then, different classes of heavy-tailed distributions are characterized as follows.

• (SGM) If $R=\lambda \sqrt{{\chi }_p^2}$ with a positive random variable λ, $\mathit{X}$ follows the scale Gaussian mixture distribution. Here, λ, ${\chi }_p^2$, and ${\mathit{U}}_r$ are (jointly) independent.

• (EL) If R is a non-negative random variable, $\mathit{X}$ follows the elliptical distribution. Here, R and ${\mathit{U}}_r$ are independent.

• (GEL) (No further restriction) $\mathit{X}$ follows the generalized elliptical distribution.

• (NPN) Let $\mathit{\mu }=\mathbf{0}$. If there exist the strictly increasing functions $\{f_1,\dots ,f_p\}$ such that $(f_1(X_1),\dots ,f_p(X_p){)}^{\mathrm{T}}$ follows the multivariate Gaussian distribution, $\mathit{X}$ follows the non-paranormal distribution.

• (TEL) Let $\mathit{\mu }=\mathbf{0}$. If there exist the strictly increasing functions $\{f_1,\dots ,f_p\}$ such that $(f_1(X_1),\dots ,f_p(X_p){)}^{\mathrm{T}}$ follows the elliptical distribution, $\mathit{X}$ follows the transelliptical distribution.

Note that R and ${\mathit{U}}_r$ are not necessarily independent when we define GEL. Throughout this paper we will use the above abbreviations (e.g. EL=elliptical distribution).

The stochastic representation proves its utility as a generative model. For example, a random sample of ${\mathit{U}}_r$ can be generated by normalizing independent standard normal samples:

$${\mathit{U}}_r\stackrel{d}{=}\mathit{S}/||\mathit{S}|{|}_2,$$

where each component of $\mathit{S}$ independently follows $N(0,1)$ and $||\cdot |{|}_2$ is a vector 2-norm. If we generate R independently of $\mathit{S}$ (assuming R is from a known distribution), a random sample of $\mathit{X}$ is obtained by multiplying the generated R and $\mathit{S}/||\mathit{S}|{|}_2$, if $\mathit{\mu }=\mathbf{0},\mathit{A}=\mathbf{I}$. Finally, it should be noted that we only consider continuous distributions satisfying $\mathrm{P}(R=0)=0$ for expositional simplicity.

Parameter estimation in heavy-tailed distributions has been extensively studied under different contexts: for example, estimation of multivariate t by Nadarajah and Kotz [39], estimation of multivariate Cauchy by Auderset et al. [3], covariance matrix estimation by Goes et al. [21] and Ke et al. [28], and mean estimation and regression by Lugosi and Mendelson [38]. As the references have already discussed this topic intensively, we will not focus on it, but new comparative results for estimators are addressed.

Given data, the random variable R (or a functional parameter) is not generally estimated, but rather specified by users, and then the other Euclidean parameters ( $\mathit{\mu }$, $\mathbf{\Sigma }$) are estimated. However, there have been attempts to estimate all together [6,8,50]. We discuss this point at the end of this paper.

2.1. Scale Gaussian mixture distribution (SGM)

One of the commonly used families of distributions in SGM is the finite mixture of Gaussians. If discrete λ (λ appears in Definition 2.1) has k levels $\{{\lambda }_i{\}}_{i=1}^k$ with each weight ${\alpha }_i=\mathrm{P}(\lambda ={\lambda }_i)$, then $\mathit{X}$ is the mixture Gaussian random variable with its density function at $\mathit{x}$ given by $\sum _{i=1}^k{\alpha }_i\varphi ((\mathit{x}-\mathit{\mu })/{\lambda }_i;\mathbf{0},\mathbf{\Sigma })$. $\varphi (\cdot ;\mathbf{0},\mathbf{\Sigma })$ is the density of the normal distribution that has its mean zero and non-singular covariance matrix $\mathbf{\Sigma }$. Besides, we can consider abundant examples of the SGM family by choosing a continuous mixing variable λ. To name a few, if we set λ by an exponential variable with its density $f(\lambda ;\eta )={\eta }^{-1}\exp (-\lambda /\eta )$, it yields the multivariate inverse normal distribution. Letting λ be the inverse Gaussian variable having its density $f(\lambda ;\delta ,\gamma )=(2\pi {)}^{-1/2}\delta {\lambda }^{-3/2}\exp (\delta \gamma -({\delta }^2/\lambda +{\gamma }^2\lambda )/2)$ produces the multivariate Laplace distribution [15]. More recently, Punzo and Bagnato [45] introduced the multivariate shifted exponential normal distribution represented by $\lambda =1/\sqrt{W}$ where W is from a shifted exponential distribution.

SGM has been applied to diverse fields (e.g. natural image processing [44,53], biomedical records and time series in astronomy [51], speech signal processing [25], and genetics [14]). Also, it is often set as a prior distribution in Bayesian analysis (with some modifications) [7,40].

2.2. Elliptical distribution (EL)

The most familiar class of distributions in EL is the multivariate normal distribution (R is a square root of the chi-squared random variable). However, not limited to this, various choices of R have been also explored in the literature to address different features in data. Such examples are the multivariate logistic distribution, Pearson Type II, VII distributions, and Kotz-type distribution where specific forms of R can be found in Table 1 of [54].

It is noteworthy that not all EL-distributed random variables possess the density functions, but if exists, it should be of the following form;

Definition 2.2

A p-dimensional random vector $\mathit{X}$ is said to have an elliptic distribution with mean $\mathit{\mu }$, a scatter matrix $\mathbf{\Sigma }$ (positive definite), and Lebesgue measurable generator $g_p$ : ${\mathbb{R}}^{+}\to {\mathbb{R}}^{+}$ if its density function at $\mathit{x}$ has the form

$$f(\mathit{x})=|\mathbf{\Sigma }{|}^{-1/2}{g}_p((\mathit{x}-\mathit{\mu }{)}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}(\mathit{x}-\mathit{\mu })).$$ (2)

A number of known distributions agree with this formulation: a multivariate normal distribution $g_p(x)=c_p\exp (-x^2/2)$ and a multivariate t distribution $g_p(x;\nu )={\tilde{c}}_{p,\nu }(1+x/\nu {)}^{-(p+\nu )/2}$ ( $\nu >0$) where $c_p$, ${\tilde{c}}_{p,\nu }$ are (different) normalizing constants depending on $p,\nu$. See Section 3.2 for more examples.

Once density functions of EL-distributed variables are accessible, parameter estimation becomes obvious: the principle of maximum likelihood. One of the well-known cases is the sample mean and covariance matrix for estimating a mean vector and a covariance matrix under the multivariate normal distribution. Another example is the multivariate t distribution, though an explicit form of estimators is not available and thus one needs dedicated algorithms (see [39] for treatment on estimation under the multivariate t distribution).

A density function is not often tractable or not known a priori, but we can still achieve legitimate estimates for the scatter matrix parameter $\mathbf{\Sigma }$. Lindskog et al. [32] proved that the Kendall's tau estimator is invariant to the choice of R, and thus can be used to detect the correlation structure of any EL-distributed data. The gist of Kendall's tau estimator, or the rank correlation, is that it solely depends on the rank of samples, not the magnitude, thereby bypassing estimation of the scalar random variable R. Recall its definition where the $(i,j)$-th entry of Kendall's tau matrix is given by

$${\hat{\mathbf{\Sigma }}}_{ij}^{\tau }=\sin ({\displaystyle \frac{\pi }{n(n-1)}}\sum _{h<h^{\prime }}\mathrm{sign}({\mathit{x}}_{hi}-{\mathit{x}}_{h^{\prime }i})\mathrm{sign}({\mathit{x}}_{hj}-{\mathit{x}}_{h^{\prime }j})),1\le i,j\le p,$$ (3)

where $\{{\mathit{x}}_h{\}}_{h=1}^n$ is EL-distributed data and ${\mathit{x}}_{hi}$ indicates the ith entry of ${\mathit{x}}_h$. The estimator is also valid for the case of TEL, which will be explained later.

Other known statistics estimating a scatter matrix in EL are a multivariate version of Kendall's tau estimator [13] and Tyler's M estimator [52]. Both estimators are underpinned by the normalization of data, which obviates the need for estimating/specifying R. The multivariate Kendall's tau can be used to retrieve an eigenspace of $\mathbf{\Sigma }$ [24]. We elucidate Tyler's M estimator in the following subsection since the estimator is also valid in a broader class of distributions (GEL).

2.3. Generalized elliptical distribution (GEL)

The generalized elliptical distribution extends EL by allowing dependency between R and ${\mathit{U}}_p$ and/or permitting R to take any values in $\mathbb{R}$. This extension enables to model asymmetry of data that elliptical distributions cannot appropriately deal with. The skew elliptical distribution in ${\mathbb{R}}^p$ is often represented by a random vector conditioning on a sign of a real-valued random variable where their joint distribution is EL. By defining new $R^{\ast }$ depending on a sign of the real-valued variable, we can find appropriate random variables $R^{\ast }$ and $\mathit{U}\in {\mathbb{R}}^{p+1}$ depending on each other to define the skew elliptical random vector (Theorem 28 of [18]). This distribution is suited for the analysis of financial data [2] where data asymmetry is commonly observed.

Aforementioned Tyler's M estimator is attractive under GEL because R is not necessarily specified in the estimation procedure. To introduce its derivation, we assume $\mathit{X}$ defined in (1) is centered, and then we have

$${\displaystyle \frac{\mathit{X}}{||\mathit{X}|{|}_2}}=\mathrm{sign}(R){\displaystyle \frac{\mathit{A}{\mathit{U}}_p}{||\mathit{A}{\mathit{U}}_p|{|}_2}}.$$ (4)

Frahm [18] showed that the normalized vector ignoring the sign has the same density as the angular central Gaussian (ACG) distribution with a scatter matrix $\mathbf{\Sigma }=\mathit{A}{\mathit{A}}^{\mathrm{T}}$.

Fact 2.1 Theorem 1, [19] —

The normalized vector $\mathit{Y}=\mathit{A}{\mathit{U}}_p/||\mathit{A}{\mathit{U}}_p|{|}_2$ is distributed over a unit sphere and distributed as GEL with its density with respect to the uniform measure on the unit sphere being

$$f(\mathit{y})\propto |\mathbf{\Sigma }{|}^{-1/2}({\mathit{y}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}\mathit{y}{)}^{-p/2},$$ (5)

where $\mathbf{\Sigma }=\mathit{A}{\mathit{A}}^{\mathrm{T}}$. (5) is the probability density function of the angular central Gaussian distribution.

Let $\{{\mathit{x}}_h{\}}_{h=1}^n$ be a set of samples from GEL and $\{{\mathit{y}}_h{\}}_{h=1}^n$ be its normalized samples ignoring the signs. We can derive the negative log-likelihood function of $\mathbf{\Sigma }$ (up to additive and scaling factors) based on (5):

$$\ell (\mathbf{\Sigma };\{{\mathit{y}}_h{\}}_{h=1}^n)=n\log |\mathbf{\Sigma }|+p\sum _{h=1}^n\log ({\mathit{y}}_h^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}{\mathit{y}}_h).$$

Its maximum likelihood estimator (MLE) is defined by a fixed-point equation:

$$\hat{\mathbf{\Sigma }}={\displaystyle \frac{p}{n}}\sum _{h=1}^n{\displaystyle \frac{{\mathit{y}}_h{\mathit{y}}_h^{\mathrm{T}}}{{\mathit{y}}_h^{\mathrm{T}}{\hat{\mathbf{\Sigma }}}^{-1}{\mathit{y}}_h}}.$$

It is easy to see due to (4) that the above equation is equivalent to that defines a well-known Tyler's M estimator [52]:

$$\hat{\mathbf{\Sigma }}={\displaystyle \frac{p}{n}}\sum _{h=1}^n{\displaystyle \frac{{\mathit{x}}_h{\mathit{x}}_h^{\mathrm{T}}}{{\mathit{x}}_h^{\mathrm{T}}{\hat{\mathbf{\Sigma }}}^{-1}{\mathit{x}}_h}}.$$ (6)

The solution would be attained through an iterative algorithm. To satisfy the identifiability conditions mentioned earlier, it is sufficient to re-scale the estimator by $\hat{\mathbf{\Sigma }}\leftarrow \hat{\mathbf{\Sigma }}/\underset{jj}{max}{\hat{\mathbf{\Sigma }}}_{jj}$ once after convergence is met.

It is worth noting that Tyler's M estimator given in (6) is not the MLE, if exists, in respect of GEL-distributed $\mathit{X}$, but it is the MLE of ACG-distributed $\mathit{Y}$. Therefore, we need a justification to explain the connection between the estimator and observations $\{{\mathit{x}}_h{\}}_{h=1}^n$. A scatter matrix $\mathbf{\Sigma }$ is often formally defined by a quantity satisfying the following identity [19]:

$${\displaystyle \frac{{\mathbf{I}}_p}{p}}=\mathbb{E}\left[{\displaystyle \frac{{\mathbf{\Sigma }}^{-1/2}{\mathit{x}}_h{\mathit{x}}_h^{\mathrm{T}}{\mathbf{\Sigma }}^{-1/2}}{\Vert {\mathbf{\Sigma }}^{-1/2}{\mathit{x}}_h{\Vert }_2^2}}\right].$$

We can easily check by rearranging (6) that Tyler's M estimator satisfies the sample version of the above equation.

2.4. Non-paranormal (NPN) and transelliptical (TEL) distributions

Non-paranormal and transelliptical distributions adopt an idea of copulas that model marginal structures and dependency of variables individually. For NPN, the mutual dependency is considered by the multivariate normal distribution through its covariance matrix, and then marginal transformations are utilized to normally distributed random variables. In the meantime, TEL considers elliptical random variables instead of MVN.

Lafferty et al. [30] first coined the term ‘non-paranormal’ to refer a nonparametric extension of the normal distribution. Indeed, this extension has a richer history in the field of statistics and was first introduced with being entitled ‘meta-Gaussian’ [29]. Furthermore, as shown in Lemma 1 of [36], NPN is equivalent to the Gaussian copula, which has its own epic starting from [20,48]. According to Han and Liu [23], TEL is traced back to [16] where they first call it a ‘meta-elliptical’ distribution. However, Liu et al. [35] pointed out NPN and TEL are slightly different from meta-Gaussian and meta-elliptical distributions (Lemma 3.5 of [35]). The difference comes from the case when joint and/or marginal density functions of $\mathit{X}$ are not defined (or do not exist). Thus, NPN and TEL are strictly larger classes than meta-Gaussian and meta-elliptical distributions, respectively, while each of them is equivalent to its counterpart when density exists. Liu et al. [35] asserted that TEL is more attractive due to better interpretation and easier theoretical analysis. Hence, we mainly discuss NPN and TEL here.

To estimate a scatter matrix $\mathbf{\Sigma }$ under NPN, Lafferty et al. [30] use the so-called Winsorized estimator which estimates the transformation $\{f_j{\}}_{j=1}^p$ in nonparametric sense. On the other hand, Liu et al. [34] propose to use Kendall's tau and Spearman's rho estimators which do not directly estimate the functions. They can achieve the parametric rate $O_p(n^{-1/4})$ with the two estimators, which is faster than the rate obtained in [30]. Han and Liu [23] further verify in Theorem 3.2 that Kendall's tau statistic proved to work under EL is also valid under TEL. This property is intuitively obvious since two Kendall's tau statistic based on $\{{\mathit{x}}_h{\}}_{h=1}^n$ and $\{f({\mathit{x}}_h){\}}_{h=1}^n$ are exactly the same. We finally remark that identifiability conditions for NPN and TEL are necessary to distinguish the distribution. Liu et al. [36] and Lafferty et al. [30] required the monotone transformations to preserve means and variances.

3. Mutual relations and estimators

3.1. Mutual relations

We first display the Venn diagram (Figure 1) of heavy-tailed distributions mentioned in the previous section to allow readers a good grasp of their relations. The left side of Figure 1 shows one type of direction of extensions from MVN:

$$\mathrm{MVN}\stackrel{(1)}{⊊}\mathrm{SGM}\stackrel{(2)}{⊊}\mathrm{EL}\stackrel{(3)}{⊊}\mathrm{GEL}.$$

From left to right, the families of heavy-tailed distributions are enlarged as the random variable R in (1) becomes more general or less restricted. Another extension depends on a monotone transformation (Figure 1, right):

$$\mathrm{MVN}\stackrel{(4)}{⊊}\mathrm{NPN}\stackrel{(5)}{⊊}\mathrm{TEL}.$$

The two extended cases (NPN and TEL) differ by which underlying distributions the transformation is applied to.

Figure 1.

Venn diagram of the heavy-tailed distributions. Their abbreviations are given in each section title.

We give concrete examples for strict inclusions in the above relations.

1. The multivariate t distribution

2. A random vector uniformly distributed on the p-dimensional unit sphere

3. The skew normal distribution

4. MVN transformed by monotone functions

5. The multivariate t distribution transformed by monotone functions.

Adopted from [10], the second example is the case where a density of an EL-distributed random variable does not exist. However, a random vector $\mathit{X}$ from SGM generally has a density, because the form $\mathit{X}=\lambda \mathit{V}$ implies its absolute continuity where $\mathit{V}$ is MVN with its covariance matrix being an identity matrix and $\lambda \in \mathbb{R}$ is independent of $\mathit{V}$.

3.2. Additional relations

Lu et al. [37] proposed a new family of heavy-tailed distributions called pair-elliptical. A random vector belonging to this family satisfies that all pairs are bivariate elliptical. Pair-normal distribution is also similarly defined. Based on these families, Lu et al. [37] described characterizations of MVN and EL. First, MVN is exactly the same as the intersection of EL and NPN, or that of EL and pair-normal. Also, EL can be characterized by the intersection of TEL and pair-elliptical. The equivalences are summarized as follows:

$$\begin{array}{rl}& \mathrm{EL}\cap \mathrm{NPN}=\mathrm{MVN},\\ & \mathrm{EL}\cap \mathrm{pair}-\mathrm{normal}=\mathrm{MVN},\\ & \mathrm{TEL}\cap \mathrm{pair}-\mathrm{elliptical}=\mathrm{EL}.\end{array}$$

Also, we can find a special characterization for a sub-class of EL by considering the consistency property described below.

Fact 3.1 Theorem 1, [27]¹ —

Assume $\mathit{X}$ follow EL and satisfy two different representations (1) and (2). Then, the two followings are equivalent.

• A density generator $g_p$ possess the consistency property, i.e.

  $${\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{g}_{p+1}\left(\sum _{j=1}^{p+1}{v}_j^2\right)\mathrm{d}{v}_{p+1}=g_p\left(\sum _{j=1}^p{v}_j^2\right),$$

  for any $p\in \mathbb{N}$ and $(v_1,\dots ,v_p)\in {\mathbb{R}}^p$ almost surely (w.r.t. the Lebesgue measure).

• There exists a random variable Q>0, unrelated to dimension p, such that for any $p\in \mathbb{N}$, $R\stackrel{d}{=}\sqrt{{\chi }_p^2}/\sqrt{Q}$, and ${\chi }_p^2$, Q, and ${\mathit{U}}_p$ are mutually independent.

It should be noted that the second statement in Fact 3.1 depends on the scale random variable Q independent of $p\ge 1$. The property would be useful for determining if a family of distributions belongs to SGM, i.e. it is a sufficient condition for a family of distributions to be in SGM. However, it is not necessary since the multivariate logistic distribution does not satisfy the consistency property but [49] proved the mixing variable Q is expressed by the Kolmogorov–Smirnov distribution for an univariate case. The relation is drawn by a Venn diagram in Figure 2.

Figure 2.

Venn diagram of the heavy-tailed distributions. The class of EL satisfying the consistency property denoted by ‘CON’ is a proper subset of SGM.

We name a few of EL families with the consistency property [27]: MVN with $g_p(x)=(2\pi {)}^{-p/2}\exp \{-x/2\}$, the multivariate t distribution with $g_p(x)=\frac{\mathrm{\Gamma }((\nu +p)/2)}{\mathrm{\Gamma }(\nu /2){\nu }^{p/2}{\pi }^{p/2}}\{1+x/\nu {\}}^{-(\nu +p)/2}$ ( $\nu >0$), the stable laws distribution $\psi (t)=\exp (r{t}^{\alpha })$ ( $0<\alpha \le 2$, r<0). By the previous argument, the three examples belong to SGM. On the other hand, EL not satisfying the consistency property has the following distributions: the multivariate logistic distribution with $g_p(x)=c_p{e}^{-x}/(1+e^{-x}{)}^2$, Pearson Type II distribution with $g_p(x)=c_{p,m}(1-x{)}^m\mathrm{I}(0<x<1)$ (m>0), Pearson Type VII distribution with $g_p(x)=c_{p,m}(1+x{)}^{-m}$ ( $m>p/2$), the multivariate Bessel distribution with $g_p(x)=c_{p,n,\alpha ,\beta }(x^{1/2}/\beta {)}^a{K}_a(x^{1/2}/\beta )$ ( $a>-n/2$, $\beta >0$) where $K_a$ is the modified Bessel function of the third kind, and finally the power exponential distribution with $g_p(x)=c_{p,s,\gamma }\exp (-\gamma x^s)$ ( $\gamma ,s>0$).

Note that all mutual relationships are only guaranteed in a finite-dimensional space, but can be different in other spaces. For example, Boente et al. [10] proved that the class of EL is equivalent to that of SGM in the infinite-dimensional (separable Hilbert) space.

3.3. Invariance property of scatter matrix estimators

As mentioned in Section 2, different robust estimators are available for different classes of heavy-tailed distributions. Table 1 summarizes the types of heavy-tailed distributions under which Kendall's tau and/or Tyler's M estimators are legitimate. The common characteristic of two estimators is that they work even if R is not explicitly determined. In other words, they are invariant to the specification of R. Kendall's tau is invariant to the monotone function f and the scale random variable R of TEL, while Tyler's M estimator to R in GEL.

Table 1.

Comparison of scatter matrix estimators for heavy-tailed distributions. ‘O’ means the estimator is valid under the distribution, while ‘–’ implies the absence of such property.

	Estimator
Distribution	Kendall (3)	Tyler (6)
EL	O	O
GEL	–	O
TEL	O	–

To delve into this point, we compare the explicit formulas of estimators based on samples from different distributions but sharing the same scatter matrix. Let us denote by ${\hat{\mathbf{\Sigma }}}^{M,X}$ the estimator (6) obtained from n samples that follow GEL with its scatter matrix $\mathbf{\Sigma }$ (and mean zero). On the other hand, assume ${\mathit{y}}_h$, $h=1,\dots ,n$, is from the angular central Gaussian with its scatter matrix $\mathbf{\Sigma }$ (and mean zero), and define its Tyler's M estimator by

$${\hat{\mathbf{\Sigma }}}^{M,Y}={\displaystyle \frac{p}{n}}\sum _{h=1}^n{\displaystyle \frac{{\mathit{y}}_h{\mathit{y}}_h^{\mathrm{T}}}{{\mathit{y}}_h^{\mathrm{T}}({\hat{\mathbf{\Sigma }}}^{M,Y}{)}^{-1}{\mathit{y}}_h}}.$$

Then, by Fact 2.1, the two Tyler's M estimators are identical in the distributional sense, i.e. ${\hat{\mathbf{\Sigma }}}^{M,X}\stackrel{d}{=}{\hat{\mathbf{\Sigma }}}^{M,Y}$.

We now turn our attention to TEL. Assume $\mathit{Z}$ and $\tilde{\mathit{Z}}$ are from $N_p(0,\mathbf{\Sigma }/2)$ and $\mathit{X}$ and $\tilde{\mathit{X}}$ are from TEL sharing the same scatter matrix. Based on Lemma 5 of [32] and monotonicity of the transformation functions, it is observed that

$$\mathrm{I}((X_k-{\tilde{X}}_k)(X_{\ell }-{\tilde{X}}_{\ell })>0)\stackrel{d}{=}\mathrm{I}((Z_k-{\tilde{Z}}_k)(Z_{\ell }-{\tilde{Z}}_{\ell })>0)$$ (7)

We write by ${\hat{\mathbf{\Sigma }}}_{k\ell }^{\tau ,X}$ the rank correlation estimator (3) computed from n copies of $\mathit{X}$. We also define the estimator using n MVN data $\{{\mathit{z}}_h{\}}_{h=1}^n$ (copies of $\mathit{Z}$) by

$${\hat{\mathbf{\Sigma }}}_{k\ell }^{\tau ,Z}=\sin ({\displaystyle \frac{\pi }{n(n-1)}}\sum _{h<h^{\prime }}\mathrm{sign}({\mathit{z}}_{hk}-{\mathit{z}}_{h^{\prime }k})\mathrm{sign}({\mathit{z}}_{h\ell }-{\mathit{z}}_{h^{\prime }\ell })).$$ (8)

Since the sums of signs in (3) and (8) are identically distributed (∵ (7)), we can conclude that ${\hat{\mathbf{\Sigma }}}_{k\ell }^{\tau ,X}\stackrel{d}{=}{\hat{\mathbf{\Sigma }}}_{k\ell }^{\tau ,Z}$, $\mathrm{\forall }k,\ell$.

Table 1 also implies that a multivariate distribution belonging to EL can have multiple choices for estimating its scatter matrix. One natural question is how different the available estimators are and/or which one is more efficient. It has been shown by Goes et al. [21] and Zhang et al. [56] that Tyler's M estimator and the sample covariance matrix are close to each other under MVN assumption. Assume that ${\mathit{y}}_h$, $h=1,\dots ,n$ are random samples from mean-zero MVN and let us denote the sample covariance matrix by $\mathit{S}=n^{-1}\sum _{h=1}^n{\mathit{y}}_h{\mathit{y}}_h^{\mathrm{T}}$. According to Lemma 2.1 of [56], Tyler's M estimator can be expressed as

$${\hat{\mathbf{\Sigma }}}^{M,Y}={\displaystyle \frac{\sum _{h=1}^n{\hat{w}}_h{\mathit{y}}_h{\mathit{y}}_h^{\mathrm{T}}}{\mathrm{tr}(\sum _{h=1}^n{\hat{w}}_h{\mathit{y}}_h{\mathit{y}}_h^{\mathrm{T}})}},$$

where $\{{\hat{w}}_h{\}}_{h=1}^n$ are the unique solution to the minimization problem

$$\underset{w_h>0,\sum _{h=1}^n{w}_h=1}{min}-\sum _{h=1}^n\log w_h+{\displaystyle \frac{n}{p}}\log \left|\sum _{h=1}^n{w}_h{\mathit{y}}_h{\mathit{y}}_h^{\mathrm{T}}\right|.$$

They also prove that the estimated weights $\{{\hat{w}}_h{\}}_{h=1}^n$ are approximately equal to 1/n, that is to say, $\underset{1\le h\le n}{max}|n{\hat{w}}_h-1|\to 0$ almost surely as $n,p\to \mathrm{\infty }$ when an underlying distribution for ${\mathit{y}}_h$ is the isotropic Gaussian distribution $N(\mathbf{0},{\mathbf{I}}_p)$ [56, Lemma 2.3], which is later generalized to the case of an arbitrary scatter matrix $\mathbf{\Sigma }$ [21, Lemma 7]. Using this fact, Goes et al. [21] showed in Lemma 8 that if $p/n\to \gamma <1$ and $n,p\to \mathrm{\infty }$, then for some constants $C,c,c^{\prime }>0$ depending only on γ we have for all $0<ϵ<c^{\prime }$

$$\mathrm{P}(||\mathit{S}-{\hat{\mathbf{\Sigma }}}^{M,Y}|{|}_{max}\ge ϵ)\le Cn\exp (-cn{ϵ}^2),$$

provided that $\mathbf{\Sigma }$ satisfying $\mathrm{tr}(\mathbf{\Sigma })=p$ is in the class of approximately sparse matrices introduced in [9]. The concentration is useful to derive that of $||{\hat{\mathbf{\Sigma }}}^{M,Y}-\mathbf{\Sigma }|{|}_{max}$ due to the triangle inequality: $||{\hat{\mathbf{\Sigma }}}^{M,Y}-\mathbf{\Sigma }|{|}_{max}\le ||{\hat{\mathbf{\Sigma }}}^{M,Y}-\mathit{S}|{|}_{max}+||\mathit{S}-\mathbf{\Sigma }|{|}_{max}$. The deviation inequality based on the element-wise maximum norm has been of consequence in the estimation of high-dimensional covariance/precision matrices (see [43] for more details).

4. Numerical study

4.1. Comparison of heavy-tail extentions

We consider the following generative models of four distributions with mean zero and covariance matrix $\mathbf{\Sigma }$:

$$\begin{array}{rl}{\mathit{X}}_{MVN}& =R\mathit{A}{\mathit{U}}_p,\\ {\mathit{X}}_{SGM}& =\lambda {\mathit{X}}_{MVN},\\ {\mathit{X}}_{EL}& =R{\mathit{X}}_{MVN},\\ {\mathit{X}}_{GEL}& =S_U{\mathit{X}}_{EL}=S_{U}R{\mathit{X}}_{MVN}\end{array}$$

where $\mathit{A}={\mathbf{\Sigma }}^{1/2}$, R>0 is the random variable following $R^2\sim {\chi }^2(p)$, $\lambda \sim \mathrm{Exp}(1)$, and $S_U=\sum _{j=1}^p{U}_j$. λ is independent of R and ${\mathit{U}}_p$. Note that the three extensions of MVN share the same quantities (R, $\mathit{A}$, ${\mathit{U}}_p$) of ${\mathit{X}}_{MVN}$ and have distinct scalar random variables, which are $\lambda R$ for ${\mathit{X}}_{SGM}$, $R^2$ for ${\mathit{X}}_{EL}$, and $S_U{R}^2$ for ${\mathit{X}}_{GEL}$. Moreover, by applying a monotone transformation $x\mapsto x^3$ to ${\mathit{X}}_{MVN}$ and ${\mathit{X}}_{EL}$, we can define ${\mathit{X}}_{NPN}$ and ${\mathit{X}}_{TEL}$. Based on the generating mechanism, we can compare the differences induced by either the scalar random variables or the marginal transformation. To empirically do it, we generate n samples of ${\mathit{X}}_{MVN}$ and then sequentially produce those of the other extensions. Here, we fix n = 10000, p = 2, and $\mathbf{\Sigma }=[1,0.5;0.5;1]$ (Figure 3).

Figure 3.

Scatterplot of bivariate random variables. n = 10000 data points from MVN are first generated and then those from the others are created by their definitions depending on the MVN samples.

Compared to MVN, other distributions exhibit much larger values ( $>4$ in absolute value). Especially for TEL, the most extreme value reaches about 5000, which is not commonly observable in other distributions. Indeed, the scale of samples in TEL changes up to what transformation we use. SGM and EL keep the elliptical shape of contours, but SGM is more concentrated at the center. GEL is the only distribution among six that displays asymmetric spreading, one of its distinctive characteristics. NPN and TEL show similar non-elliptical dispersion of data with different scales.

4.2. Application to estimation of scatter matrix in genomics data

We provide an illustration of results using different scatter matrix estimators to appreciate their behaviors when heavy-tailed data are given.

The dataset we use here is adopted from the preceding work [55], where gene-expressions of the plant Arabidopsis thaliana are measured by Affymetrix GeneChip microarrays and the gene-regulatory network for the genes related to the isoprenoid pathway is constructed. In contrast to the original work under the multivariate Gaussian assumption, Liu et al. [34,36] assumed the data are from a heavy-tailed distribution and find very different results from the original. In this section, we investigate the difference of three scatter matrix estimators -- Kendall's tau, Tyler's M, and Pearson's R - which are the initial estimators for the estimation of the (sparse) inverse covariance matrix in [34,36].

The gene microarray data contains n = 118 chip records from p = 39 genes,² which are log-transformed and standardized. For an expositional purpose, we apply a monotone function $x\mapsto x^3$ to every observation, making heavy-tailedness more distinct. Since Kendall's tau estimator only finds out a correlation structure, the other two estimators are also converted to the correlation counterparts.

The estimators are given in the top panels of Figure 4. They appear to be similar, but a closer look could find subtle differences in Pearson's R from the other two. To be specific, there exist clear distinctions in the area bounded by a black box, such as different degrees of correlations and their opposite signs. Then, we numerically compare their eigen-structures. In the bottom left panel of Figure 4, eigenvalues of each estimator commonly show an elbow-like pattern. While it is inarguably at the fourth index where the sharp elbow is detected in Kendall's tau and Tyler's M estimators, it is rather obscure to pick a cutoff point in Pearson's R type. Next, we measure the angles between two eigenvectors from different correlation estimators. For example, in the bottom right panel of Figure 4 the green line labeled by ‘K_vs_T’ plots the angle between the first eigenvectors from Kendall's tau and Tyler's M, that of the second eigenvectors, and so on. The first three angles are relatively close to 0, implying each eigenvector is almost parallel to the counterpart. On the other hand, the angles formed by eigenvectors of Pearson's R are not close to zero. Moreover, we can approximate each estimator by only using a few (say, the first 3 here) of eigenvalues and eigenvectors as if the remaining part is a noisy component that might disguise similarity of the three estimators. The middle of Figure 4 shows that the approximated Pearson's R does not look like the other two that are alike. Taken together, the eigen-structure of Pearson's R estimator presents a sharp contrast to those of Kendall's tau and Tyler's M.

Figure 4.

Comparison of three correlation matrix estimators. The first row shows their original heatmaps, while the second the approximated versions using the first three eigenvectors. The bottom left panel describes the eigenvalues of each estimator, while the right one compares angles between eigenvectors from two different estimators.

Considering Kendall's tau and Tyler's M are well-founded estimators for heavy-tailed distributions but Pearson's R is not, the disparity adds a caveat to practitioners that heavy-tailed features should be suitably treated. It also implies that analyses based on heavy-tailed assumptions may end up with different findings. Furthermore, the similarity in eigen-structure between Kendall's tau and Tyler's M may indicate the first three eigen-pairs would be central information of the scatter matrix, which could benefit researchers interested in this dataset.

Next, we use the principal component (PC) scores for outlier detection based on a simple idea as follows [11,46]. Data points with exceptionally large (in absolute value) PC values are regarded as potential outliers. The scores are defined by the inner product between observed data $\{{\mathit{x}}_i{\}}_{i=1}^n$ and eigenvectors $\{{\mathit{e}}_j{\}}_{j=1}^p$ obtained from any of Kendall's tau, Tyler's M, and Pearson's correlation matrix estimators. Since the first four PCs explain most of the variance (see bottom left in Figure 4), we show in Figure 5 the pairwise scatter plot of them.

Figure 5.

Scatterplot matrix of PC scores obtained by three different correlation matrices (labeled on the top). PC1 is on the x-axis, while PC2, PC3, PC4 on the y-axes.

When using the robust estimators (Tyler, Kendall), it can be seen that 3, 13, and 113th are far away from the majority of observations. However, for Pearson's estimator, the third observation substantially deviates from the majority, and as a consequence it may hinder one from capturing the variance of the others that include 13, 133th points. This could result in different scientific conclusions about outliers.

5. Conclusion

MVN is the most fundamental and commonly used distributional assumption in the multivariate analysis due to its attractive theoretical properties, but mostly due to its ease of use and accessibility. Therefore, practitioners or even statistical experts are inclined to overlook a deviation from Gaussianity found in real-world data and innocently take advantage of the good properties of MVN. One of the factors for this phenomenon is researchers are not fully aware that many alternatives to MVN are already in place. For this purpose, this paper addresses a review of heavy-tailed multivariate distributions and conducts an in-depth investigation into their relationships. As a consequence, we hope this work would encourage researchers to use the extensions properly in future research.

We finally conclude the paper with a discussion. One of the appealing characteristics of heavy-tailed families is that they are parametric distributions that can specify the location and scale parameters, which promise interpretability of statistical models To date, when estimating $\mathit{\mu }$ and $\mathbf{\Sigma }$, most researches exploit two schemes. On the one hand, the distribution of R is fully specified so that the maximum likelihood estimators are attained through the likelihood function of $\mathit{X}$. On the other hand, one can pursue an estimator invariant to R so as Kendall's tau estimator. However, the former approach is only applicable to limited (known) distributions, while the latter cannot perform further procedures (e.g. prediction) because a generative model cannot be specified. The limitations lead to a natural question of whether one can directly estimate the distribution of R from data. For example, estimating the degrees of freedom in t distribution matches this problem [33]. We believe it is a practically important problem that would attract many researchers' attention in the field.

Funding Statement

This work was partially supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) [grant number NRF-2021R1A2C1010786].

Disclosure statement

No potential conflict of interest was reported by the authors.