Finite mixtures of matrix variate Poisson-log normal distributions for three-way count data

Abstract

Motivation

Three-way data structures, characterized by three entities, the units, the variables and the occasions, are frequent in biological studies. In RNA sequencing, three-way data structures are obtained when high-throughput transcriptome sequencing data are collected for n genes across p conditions at r occasions. Matrix variate distributions offer a natural way to model three-way data and mixtures of matrix variate distributions can be used to cluster three-way data. Clustering of gene expression data is carried out as means of discovering gene co-expression networks.

Results

In this work, a mixture of matrix variate Poisson-log normal distributions is proposed for clustering read counts from RNA sequencing. By considering the matrix variate structure, full information on the conditions and occasions of the RNA sequencing dataset is simultaneously considered, and the number of covariance parameters to be estimated is reduced. We propose three different frameworks for parameter estimation: a Markov chain Monte Carlo-based approach, a variational Gaussian approximation-based approach, and a hybrid approach. Various information criteria are used for model selection. The models are applied to both real and simulated data, and we demonstrate that the proposed approaches can recover the underlying cluster structure in both cases. In simulation studies where the true model parameters are known, our proposed approach shows good parameter recovery.

Availability and implementation

The GitHub R package for this work is available at https://github.com/anjalisilva/mixMVPLN and is released under the open source MIT license.

1 Introduction

Finite mixture models are popular for clustering applications and are widely used on two-way data (McLachlan and Peel 2000; McNicholas 2016). Three-way data are becoming increasingly commonplace in several fields, including bioinformatics. Three-way data structures are characterized by three entities or modes: the units (rows), the variables (columns), and the occasions (layers). For two-way data, each observation is represented as a vector whereas, for three-way data, each observation can be regarded as a matrix. A random matrix ${\mathbf{T}}_n$ is said to contain $k\in \{1,\dots ,p\}$ responses over $i\in \{1,\dots ,r\}$ occasions and $n=1,\dots ,N$ such units are considered. This provides N independent and identically distributed random matrices ${\mathbf{T}}_1,{\mathbf{T}}_2,\dots ,{\mathbf{T}}_N$.

Matrix variate distributions offer a natural approach for modeling three-way data. Extensions of matrix variate distributions in the context of mixture models have given rise to mixtures of matrix variate distributions, which have been used to cluster three-way data (Viroli 2011; Anderlucci and Viroli 2015; Dogru et al. 2016; Gallaugher and McNicholas 2018). Here, the interest lies in clustering the N observed matrices into G clusters, while utilizing all information from the other two modes (Viroli 2011). It is assumed that matrices ${\mathbf{T}}_1,{\mathbf{T}}_2,\dots ,{\mathbf{T}}_N$ are conditionally independent and identically distributed observations coming from a mixture model with G possible groups in proportions ${\pi }_1,\dots ,{\pi }_G$ (Viroli 2011). The density of the G-component mixture is $f\left({\mathbf{T}}_n\right|{\pi }_1,\dots ,{\pi }_G,{\mathit{\vartheta }}_1,\dots ,{\mathit{\vartheta }}_G)={\sum }_{g=1}^G{\pi }_g{f}^{(r\times p)}({\mathbf{T}}_n\left|{\mathit{\vartheta }}_g\right).$ Here parameters of the distribution function $f^{(r\times p)}(·)$ are represented by ${\mathit{\vartheta }}_g$ and ${\pi }_g>0$, such that ${\sum }_{g=1}^G{\pi }_g=1$, is the mixing proportion of the gth component.

Three-way datasets are common in biological studies, including RNA sequencing (RNA-seq), where gene expression count data are collected for N genes across p conditions at r occasions. However, efficiently analyzing these complex data remains an ongoing challenge. While some early work utilized the Poisson distribution to model such count data (Marioni et al. 2008; Bullard et al. 2010), this was not ideal because of its restrictive mean–variance relationship, and so the negative binomial distribution emerged as the univariate distribution of choice (Love et al. 2014; Dong et al. 2016). However, the multivariate extensions of Poisson (Campbell 1934) and negative binomial distributions (Doss 1979) are seldom used in practice due to their computational complexity (Brijs et al. 2004). Silva et al. (2019) proposed a mixture model-based clustering methodology for overdispersed, multivariate count data based on the multivariate Poisson-log normal (MPLN) distribution for two-way RNA-seq data. For genes $n\in \{1,\dots ,N\}$ and samples $c\in \{1,\dots ,rp\}$, the MPLN distribution is given by

where denotes the Poisson distribution and the ${\mathcal{N}}_{rp}$ is a rp-dimensional normal distribution. To account for the differences in library sizes across each sample c of an RNA-seq study, a fixed, known constant s_c, representing the normalized library sizes, is added to the mean of the Poisson distribution. In this work, mixtures of MPLN distributions and matrix variate normal distributions are extended to give rise to mixtures of matrix variate Poisson-log normal (MVPLN) distributions for clustering three-way count data. Details of parameter estimation are provided, and both real and simulated data illustrations are used to demonstrate the clustering ability.

2 Materials and methods

2.1 Matrix variate Poisson-log normal distribution

Mathematical properties of the matrix variate normal distribution can be found in Gupta and Nagar (2000). By considering a matrix variate structure, the number of free covariance parameters to be estimated is reduced from $(1/2)rp(rp+1)$ to $(1/2)\left[r\right(r+1)+p(p+1\left)\right]$. The matrix variate normal distribution can be extended to give rise to MVPLN distribution using a hierarchical structure. Consider N independent and identically distributed random matrices ${\mathbf{Y}}_n,\hspace{0.17em}n=1,\dots ,N$, each of dimension r × p. In the MVPLN framework, $Y_{\mathit{nik}}|{\theta }_{\mathit{nik}}$ follows a Poisson distribution with mean $\text{exp}\left({\theta }_{\mathit{nik}}\right)$, and ${{\mathit{\theta }}_n}^{}$ follows a r × p matrix variate normal distribution ${\mathcal{N}}_{r\times p}(\mathbf{M},\mathbf{\Phi },\mathbf{\Omega })$, where M is a r × p matrix of means, $\mathbf{\Phi }$ is a r × r covariance matrix containing the variances and covariances between r occasions, and $\mathbf{\Omega }$ is a p × p covariance matrix containing the variance and covariances of the p variables. Figure 1 provides a graphical representation of a mixture of MVPLN distributions.

Figure 1.

Graphical representation of the MVPLN mixture model.

The vectorization of ${{\mathbf{Y}}_{\mathbf{n}}}^{\top }$, denoted $\text{vec}\left({{\mathbf{Y}}_n}^{\top }\right)$ , is rp-dimensional. Given all $\text{vec}\left({{\mathbf{Y}}_n}^{\top }\right)$, i.e. for $n=1,\dots ,N$, the library sizes $\text{vec}\left(\mathbf{s}\right)$ can be calculated. The $\text{vec}\left(\mathbf{s}\right)$ and $\text{vec}\left({{\mathit{\theta }}_n}^{\top }\right)$ are rp-dimensional. The covariance matrix of $\text{vec}\left({{\mathbf{Y}}_n}^{\top }\right)$ is $\mathbf{\Sigma }=\mathbf{\Phi }\otimes \mathbf{\Omega }$, where ⊗ denotes the Kronecker product. Note that $\mathbf{\Sigma }$ has dimension rp × rp, and the probability mass function of the MVPLN distribution is where $\mathit{\vartheta }=(\mathbf{M},\mathbf{\Phi },\mathbf{\Omega }),\hspace{0.17em}f(·)$ is the probability mass function of Poisson distribution, $\mathrm{vec}{\left({{\mathit{Y}}_n}^{\top }\right)}_c$ represent the $c^{th}$ element of $\mathrm{vec}\left({{\mathit{Y}}_n}^{\top }\right),$ and $g^{(r\times p)}(·)$ is the probability density function of matrix variate normal distribution.

$$\begin{array}{l}f({\mathit{Y}}_n,\mathit{s}|\vartheta )=\\ {\int }_{\mathrm{ℝ}}\left\{{\prod }_{c=1}^{rp}f\left(\text{vec}\right({{\mathbf{Y}}_n}^{\top }{)}_c\left|\text{vec}\right({{\mathbf{\theta }}_n}^{\top }{)}_c,\text{vec}\left(\mathbf{s}{)}_c\right)\right\}{g}^{(r\times p)}\left({\mathbf{\theta }}_n\right|\mathit{\vartheta }) d\text{vec}({{\mathbf{\theta }}_n}^{\top }),\end{array}$$

The unconditional mean and covariance of the MPLN distribution can be calculated using the properties of the log-normal distribution and of the conditional expectation (Aitchison and Ho 1989; Tunaru 2002). For the MVPLN distribution, the unconditional mean and covariance are respectively. The MVPLN distribution can account for both the correlations between variables and the correlations between occasions, as two different covariance matrices are used for the two modes. This makes the model ideal for modeling RNA-seq data when expression measurements for different conditions at different occasions, e.g. time-points or replicates, are available.

$$\begin{array}{l}\mathbb{E}\left(Y_{ik}\right)=\mathbb{E}\left[\mathbb{E}\right(Y_{ik}\left|{\theta }_{ik}\right)]=\text{exp}\hspace{0.17em}\left\{{\mu }_{ik}+\frac{1}{2}\left({\mathbf{\Phi }}_{ii}{\mathbf{\Omega }}_{kk}\right)\right\}={\mathcal{M}}_{ik},\hfill \\ \mathbb{V}\text{ar}\left(Y_{ik}\right)=\mathbb{E}\left[\mathbb{V}\text{ar}\right(Y_{ik}\left|{\theta }_{ik}\right)]+\mathbb{V}\text{ar}[\mathbb{E}\left(Y_{ik}\right|{\theta }_{ik}\left)\right]\hfill \\ ={\mathcal{M}}_{ik}+{\mathcal{M}}_{ik}^2\left(\text{exp}\hspace{0.17em}\left\{{\mathbf{\Phi }}_{ii}{\mathbf{\Omega }}_{kk}\right\}-1\right),\hfill \end{array}$$

2.2 Finite mixtures of MVPLN distributions

In the mixture model context, a random matrix ${\mathbf{Y}}_n$ is assumed to come from a population with G subgroups each distributed according to an MVPLN distribution. Then N such matrices ${\mathbf{Y}}_1,{\mathbf{Y}}_2,\dots ,{\mathbf{Y}}_N$ are observed, each of which belongs to one of $g\in \{1,\dots ,G\}$ different sub-populations with mixing proportions ${\pi }_1,\dots ,{\pi }_G$. Then the probability density function of a G-component mixture of MVPLN distributions can be written as where $\mathbf{\Theta }=({\pi }_1,\dots ,{\pi }_G,{\mathbf{M}}_1,\dots ,{\mathbf{M}}_G,{\mathbf{\Phi }}_1,\dots ,{\mathbf{\Phi }}_G,{\mathbf{\Omega }}_1,\dots ,{\mathbf{\Omega }}_G)$, the $f_g(·)$ is the probability mass function of a Poisson distribution and the $g_g^{(r\times p)}(·)$ is the probability density function of matrix variate normal distribution. The cluster membership of all units is assumed to be unknown and z_ng is used to cluster membership, where z_ng = 1 if ${\mathbf{Y}}_n$ is in component g and z_ng = 0 otherwise. The complete data consist of the observed and missing data, i.e. (${\mathbf{Y}}_1,\dots ,{\mathbf{Y}}_N,{\mathit{z}}_1,\dots ,{\mathit{z}}_N,{\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_N$). The complete-data likelihood is and the complete-data log-likelihood is where $n_g={\sum }_{n=1}^N{z}_{ng}$ and $\text{log (vec}{\left({{\mathbf{Y}}_n}^{\top }\right)}_c\text{!)}$ is the log of the factorial of the $c^{th}$ element of $\mathrm{vec}\left({{\mathit{Y}}_n}^{\top }\right).$Compared to the mixtures of MPLN distribution, the number of free parameters to be estimated is reduced by considering a matrix variate structure (see Figs 2 and 3). For the mixtures of MPLN model, the number of free parameters is $K=(G-1)+\left(\mathit{Grp}\right)+\frac{1}{2}Grp[rp+1]$, whereas for mixtures of MVPLN model it is $K=(G-1)+\left(\mathit{Grp}\right)+\frac{1}{2}G\left[r\right(r+1)+p(p+1\left)\right]$.

Figure 2.

Scatter plot illustrating how the number of free parameters K grows with data dimensionality rp for the mixtures of MPLN model and for the mixtures of MVPLN model. Here $G=2,r=2,$ and rp = 4 up to 100.

Figure 3.

Scatter plot illustrating how the number of free parameters K grows with the number of clusters G for the mixtures of MPLN model and for the mixtures of MVPLN model. Here $G=1:100,r=2,p=5$.

$$\begin{array}{l}f(\mathbf{Y};\mathbf{\Theta })=\sum _{g=1}^G{\pi }_g{f}_{\mathbf{Y}}\left(\mathbf{Y}\right|{\mathbf{M}}_g,{\mathbf{\Phi }}_g,{\mathbf{\Omega }}_g)\\ =\sum _{g=1}^G{\pi }_g{\int }_{\mathrm{ℝ}}\left\{\prod _{c=1}^{rp}{f}_g\left(\text{vec}\right({{\mathbf{Y}}_n}^{\top }\left|\text{vec}\right({{\mathbf{\theta }}_{ng}}^{\top }{)}_c,\text{vec}\left(\mathbf{s}{)}_c\right)\right\}\\ \times g_g^{(r\times p)}\left({\mathbf{\theta }}_{ng}\right|{\mathbf{M}}_g,{\mathbf{\Phi }}_g,{\mathbf{\Omega }}_g\left)\mathrm{ }d\text{vec}\right({{\mathbf{\theta }}_{ng}}^{\top })\end{array}$$

$$\begin{array}{l}{L}_c\left(\mathbf{\Theta }\right)=\prod _{n=1}^N\prod _{g=1}^G[{\pi }_g\left\{\prod _{c=1}^{rp}{f}_g\left(\text{vec}\right({{\mathbf{Y}}_n}^{\top }\left|\text{vec}\right({{\mathbf{\theta }}_{ng}}^{\top }{)}_c,\text{vec}\left(\mathbf{s}{)}_c\right)\right\}\\ \times g_g^{(r\times p)}\left({\mathbf{\theta }}_{ng}\right|{\mathbf{M}}_g,{\mathbf{\Phi }}_g,{\mathbf{\Omega }}_g){]}^{z_{ng}},\end{array}$$

$$\begin{array}{l}{l}_c\left(\mathbf{\Theta }\right)=\sum _{g=1}^G{n}_g\hspace{0.17em}\text{log}\hspace{0.17em}{\pi }_g\\ - \sum _{n=1}^N\sum _{g=1}^G\sum _{c=1}^{rp}{z}_{ng}\hspace{0.17em}\text{exp}\hspace{0.17em}\{\text{vec}{\left({{\mathit{\theta }}_{ng}}^{\top }\right)}_c+\text{log}\hspace{0.17em}\text{vec}{\left(\mathbf{s}\right)}_c\}\\ + \sum _{n=1}^N\sum _{g=1}^G{z}_{ng}\left[\text{vec}\left({{\mathit{\theta }}_{ng}}^{\top }\right)+\text{log}\hspace{0.17em}\text{vec}\left(\mathbf{s}\right)\right]\text{vec}{\left({{\mathbf{Y}}_n}^{\top }\right)}^{\top }\\ - \sum _{n=1}^N\left(\sum _{g=1}^G{z}_{ng}\right)\sum _{c=1}^{rp}\hspace{0.17em}\text{log}\hspace{0.17em}\left(\text{vec}{\left({{\mathbf{Y}}_n}^{\top }\right)}_c\text{!}\right)-\frac{\mathit{nrp}}{2}\text{log}(2\pi )\\ - \frac{p}{2}\sum _{g=1}^G{n}_g\hspace{0.17em}\text{log}\hspace{0.17em}\left|{\mathbf{\Phi }}_g\right|-\frac{r}{2}\sum _{g=1}^G{n}_g\hspace{0.17em}\text{log}\hspace{0.17em}\left|{\mathbf{\Omega }}_g\right|\\ - \frac{1}{2}\sum _{n=1}^N\sum _{g=1}^G{z}_{ng}\text{tr}\left[{\mathbf{\Phi }}_g^{-1}({\mathit{\theta }}_{ng}-{\mathbf{M}}_g){\mathbf{\Omega }}_g^{-1}{({\mathit{\theta }}_{ng}-{\mathbf{M}}_g)}^{\top }\right],\end{array}$$

2.3 Parameter estimation

Three different frameworks for parameter estimation for the mixtures of MVPLN models are proposed: one based on Markov chain Monte Carlo (MCMC) methods, one based on variational Gaussian approximation (VGA) as well as a hybrid approach. MCMC-based approaches are computationally intensive; hence, we also provide a computationally efficient variational approximation framework for parameter estimation. Finally, a hybrid approach combines the variational approximation-based approach and MCMC-based approach.

2.3.1 MCMC-based approach

In the MCMC-based approach, the MCMC expectation–maximization (MCMC–EM) algorithm is used to estimate the model parameters (see Silva et al. 2019, for details). Using an MCMC–EM algorithm, the expected value of the ${\mathit{\theta }}_{ng}$ and the Z_ng conditional on the parameter updates from the tth iteration, respectively, are updated in the expectation (E-) step as follows: where and ${\mathit{\theta }}_{ng}^{\left(f\right)}$ is a random sample simulated via the $\mathtt{RStan}$ package for iterations $f=1,\dots ,B$. In the E-step, the expectation is taken conditional on the current parameter estimates; hence, the use of (t) on parameters in (1). As the values from initial iterations are discarded from further analysis to minimize bias, the number of iterations used for parameter estimation is W, where W < B. The conditional expected-value of the complete-data log-likelihood is where C is a constant with respect to ${\mathbf{M}}_g,\hspace{0.17em}{\mathbf{\Phi }}_g$ and ${\mathbf{\Omega }}_g$, and $n_g^{\left(t\right)}={\sum }_{n=1}^N{z}_{ng}^{\left(t\right)}$. During the M-step, the updates for the parameters are obtained as follows:

$$\begin{array}{l}\mathbb{E}\left({\mathit{\theta }}_{ng}\right|{\mathbf{Y}}_n)\simeq \frac{1}{W}\sum _{f=1}^W{\mathit{\theta }}_{ng}^{\left(f\right)}\simeq {\mathit{\theta }}_{ng}^{\left(t\right)},\hfill \\ \mathbb{E}\left(Z_{ng}\right|{\mathbf{Y}}_n,{\mathit{\theta }}_{ng},\mathbf{s})=\frac{q_{ng}}{{\sum }_{h=1}^G{q}_{nh}}:=z_{ng}^{\left(t\right)},\hfill \end{array}$$ (1)

$$\begin{array}{l}{q}_{ng}={\pi }_g^{\left(t\right)}\left\{\prod _{c=1}^{rp}{f}_g\left(\text{vec}{\left({{\mathbf{Y}}_n}^{\top }\right)}_c|\text{vec}{\left({{\mathit{\theta }}_{ng}}^{\top }\right)}_c,\text{vec}{\left(\mathbf{s}\right)}_c\right)\right\}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\times g_g^{(r\times p)}\left({\mathit{\theta }}_{ng}^{\left(t\right)}\right|{\mathbf{M}}_g^{\left(t\right)},{\mathbf{\Phi }}_g^{\left(t\right)},{\mathbf{\Omega }}_g^{\left(t\right)})\end{array}$$

$$\begin{array}{l}\mathcal{Q}\left(\mathbf{\Theta }\right)\simeq \mathbb{E}\left\{l_c\right(\mathbf{\Theta }\left)\right\}\simeq C-\frac{p}{2}\sum _{g=1}^G{n}_g\hspace{0.17em}\text{log}\hspace{0.17em}\left|{\mathbf{\Phi }}_g\right|-\frac{r}{2}\sum _{g=1}^G{n}_g\hspace{0.17em}\text{log}\hspace{0.17em}\left|{\mathbf{\Omega }}_g\right|\\ -\frac{1}{2}\sum _{n=1}^N\sum _{g=1}^G{z}_{ng}^{\left(t\right)}\mathbb{E}\left[\text{tr}\right({\mathbf{\Phi }}_g^{-1}({\mathit{\theta }}_{ng}-{\mathbf{M}}_g){\mathbf{\Omega }}_g^{-1}{({\mathit{\theta }}_{ng}-{\mathbf{M}}_g)}^{\top }\left)\right|{\mathit{\theta }}_{ng}^{\left(t\right)},z_{ng}=1],\end{array}$$

$$\begin{array}{l}{\pi }_g^{(t+1)}=\frac{n_g^{\left(t\right)}}{N},\hspace{1em}{\mathbf{M}}_g^{(t+1)}=\frac{{\sum }_{n=1}^N{z}_{ng}^{\left(t\right)}\mathbb{E}\left({\mathit{\theta }}_{ng}\right)}{n_g},\\ {\mathbf{\Phi }}_g^{(t+1)}=\frac{{\sum }_{n=1}^N{z}_{ng}^{\left(t\right)}\mathbb{E}\left(\right({\mathit{\theta }}_{ng}-{\mathbf{M}}_g^{(t+1)}\left){\left({\mathbf{\Omega }}_g^{\left(t\right)}\right)}^{-1}{({\mathit{\theta }}_{ng}-{\mathbf{M}}_g^{(t+1)})}^{\top }\right)}{p{n}_g},\\ {\mathbf{\Omega }}_g^{(t+1)}=\frac{{\sum }_{n=1}^N{z}_{ng}^{\left(t\right)}\mathbb{E}\left({({\mathit{\theta }}_{ng}-{\mathbf{M}}_g^{(t+1)})}^{\top }{\left({\mathbf{\Phi }}_g^{(t+1)}\right)}^{-1}\right({\mathit{\theta }}_{ng}-{\mathbf{M}}_g^{(t+1)}\left)\right)}{r{n}_g}.\end{array}$$

2.3.2 VGA-based approach

Variational approximations (Wainwright and Jordan 2007) are approximate inference techniques in which a computationally convenient approximating density is used in place of a more complex but “true” posterior density. The approximating density is obtained by minimizing the Kullback–Leibler (KL) divergence between the true and the approximating densities. Suppose we have an approximating density $q\left(\mathit{\theta }\right)$, then the marginal log of the probability mass function can be written where $D_{\text{KL}}\left(q\right|\left|f\right)=\int q\left(\mathit{\theta }\right)\hspace{0.17em}\text{log}\hspace{0.17em}\frac{q\left(\mathit{\theta }\right)}{f\left(\mathit{\theta }\right|\mathbf{Y})}d\mathit{\theta }$ is the KL divergence between $f\left(\mathit{\theta }\right|\mathbf{Y})$ and approximating distribution $q\left(\mathit{\theta }\right)$, and $F(\mathbf{Y},q)=\int [\hspace{0.17em}\text{log}\hspace{0.17em}f(\mathbf{Y},\mathit{\theta })-\text{log}\hspace{0.17em}q\left(\mathit{\theta }\right)\left]q\right(\mathit{\theta })d\mathit{\theta }$ is our evidence lower bound (ELBO). Thus, to minimize the KL divergence, we maximize our ELBO. For VGA, $q\left(\mathit{\theta }\right)$ is assumed to be a Gaussian distribution.

$$\text{log}\hspace{0.17em}{f}_Y\left(\mathbf{Y}\right)=F(q,\mathbf{Y})+D_{\text{KL}}\left(q\right|\left|f_Y\right),$$

The complete-data log-likelihood for the mixtures of MVPLN distributions can be written

$$\begin{array}{l}{l}_c\left(\mathbf{\Theta }\right)=\sum _{g=1}^G\sum _{n=1}^N{z}_{ng}\hspace{0.17em}\text{log}\hspace{0.17em}{\pi }_g+\sum _{g=1}^G\sum _{n=1}^N{z}_{ng}\hspace{0.17em}\text{log}\hspace{0.17em}f\left({\mathbf{Y}}_n\right|{\mathbf{M}}_g,{\mathbf{\Phi }}_g,{\mathbf{\Omega }}_g)\\ =\sum _{g=1}^G\sum _{n=1}^N{z}_{ng}\hspace{0.17em}\text{log}\hspace{0.17em}{\pi }_g+\sum _{g=1}^G\sum _{n=1}^N{z}_{ng}\left[F({\mathbf{Y}}_n,q_{ng})+D_{\text{KL}}\left(q_{ng}\right|\left|f_{ng}\right)\right],\end{array}$$

where $D_{\text{KL}}\left(q_{ng}\right|\left|f_{ng}\right)=\int q\left({\mathit{\theta }}_{ng}\right)\hspace{0.17em}\text{log}\hspace{0.17em}\frac{q\left({\mathit{\theta }}_{ng}\right)}{f\left({\mathit{\theta }}_n\right|{\mathbf{Y}}_n,Z_{ng}=1)}d{\mathit{\theta }}_{ng}$ is the KL divergence between $f\left({\mathit{\theta }}_n\right|{\mathbf{Y}}_n,Z_{ng}=1)$ and approximating distribution $q\left({\mathit{\theta }}_{ng}\right)$. Assuming $q\left({\mathit{\theta }}_{ng}\right)=N_{r\times p}({\mathit{\xi }}_{ng},{\Delta }_{ng},{\mathit{\kappa }}_{ng})$, the ELBO for each observation ${\mathbf{y}}_n$ becomes

$$\begin{array}{l}F(q_{ng},Y_n)=-[{\displaystyle \sum _{i=1}^r{\displaystyle \sum _{j=1}^p\exp }}\{{\left({\mathit{\xi }}_{ng}\right)}_{ij}+\frac{1}{2}{\left({\Delta }_{ng}\right)}_{ii}{\left({\Omega }_{ng}\right)}_{jj}+\log s_{ij}\left\}\right]\\ + {\left[\text{vec}\left({\mathit{\xi }}_{ng}^{\top }\right)+\log \text{vec}(s)\right]}^{\top }\text{vec}\left(Y_n^{\top }\right)\\ - \left[{\displaystyle \sum _{c=1}^{rp}\log }(\text{vec}{\left(Y_n^{\top }\right)}_c!)\right]-\frac{p}{2}\log \left|{\Phi }_g\right|-\frac{r}{2}\log \left|{\Omega }_g\right|\\ - \frac{1}{2}{\left(\text{vec}\left({\mathit{\xi }}_{ng}^{\top }\right)-\text{vec}\left(M_g^{\top }\right)\right)}^{\top }{\Phi }_g^{-1}\otimes {\Omega }_g^{-1}\left(\text{vec}\left({\mathit{\xi }}_{ng}^{\top }\right)-\text{vec}\left(M_g^{\top }\right)\right)\\ + \text{tr}\left\{{\Phi }_g^{-1}{\Delta }_{ng}\right\}\text{tr}\left\{{\Omega }_g^{-1}{\mathit{\kappa }}_{ng}\right\}+\frac{p}{2}\log \left|{\Delta }_{ng}\right|+\frac{r}{2}\log \left|{\mathit{\kappa }}_{ng}\right|+\frac{rp}{2}.\end{array}$$

The variational parameters that maximize the ELBO will minimize the KL divergence between the true posterior and the approximating density. Thus, parameter estimation can be done in an iterative EM-type approach such that the following steps are iterated. At the (t + 1)th step:

1. Conditional on the variational parameters ${\mathit{\xi }}_{ng},\hspace{0.17em}{\Delta }_{ng}$, and ${\mathit{\kappa }}_{ng}$ and on ${\mathbf{M}}_g,\hspace{0.17em}{\mathbf{\Phi }}_g$, and ${\mathbf{\Omega }}_g$, the $\mathbb{E}\left(Z_{ng}\right)$ is computed. Given π_g, ${\mathbf{M}}_g,\hspace{0.17em}{\mathbf{\Phi }}_g$, and ${\mathbf{\Omega }}_g$,

   $$\mathbb{E}\left(Z_{ng}\right|{\mathbf{Y}}_n)=\frac{{\pi }_{g}f\left({\mathbf{Y}}_n\right|{\mathbf{M}}_g,{\mathbf{\Phi }}_g,{\mathbf{\Omega }}_g)}{{\sum }_{h=1}^G{\pi }_{h}f\left({\mathbf{Y}}_n\right|{\mathbf{M}}_h,{\mathbf{\Phi }}_h,{\mathbf{\Omega }}_h)}.$$
   Note that this involves the marginal distribution of Y which is difficult to compute. Hence, we use an approximation of $\mathbb{E}\left(Z_{ng}\right)$, where we replace the marginal density of the exponent of ELBO such that

   $${\widehat{z}}_{ng}^{(t+1)}\stackrel{\text{def}}{=}\frac{{\pi }_g\hspace{0.17em}\text{exp}\hspace{0.17em}\left[F(q_{ng},{\mathbf{Y}}_n)\right]}{{\sum }_{h=1}^G{\pi }_h\hspace{0.17em}\text{exp}\hspace{0.17em}\left[F(q_{nh},{\mathbf{Y}}_n)\right]}.$$

   This approximation is computationally convenient and a similar framework has been previously utilized (Gollini and Murphy 2014; Tang et al. 2015). This approximation works well in simulation studies and real data analysis.

2. Given ${\widehat{z}}_{ng}^{(t+1)}$, variational parameters ${\mathit{\xi }}_{ng},\hspace{0.17em}{\Delta }_{ng}$, and ${\mathit{\kappa }}_{ng}$ are updated conditional on ${\mathbf{M}}_g^{\left(t\right)},\hspace{0.17em}{\mathbf{\Phi }}_g^{\left(t\right)}$, and ${\mathbf{\Omega }}_g^{\left(t\right)}$.

   1. A fixed-point method is used for updating ${\Delta }_n$:

      $$\begin{array}{l}{\Delta }_{ng}^{(t+1)}=p[{\mathbf{I}}_{r\times r}\odot \{\text{diag}{\left({\mathit{\kappa }}_{ng}^{\left(t\right)}\right)}^{\top }\left[\mathbf{e}\mathbf{x}\mathbf{p}\right\{{\mathit{\xi }}_{ng}^{\left(t\right)}+\text{log}\hspace{0.17em}\mathbf{s}\\ {{+\frac{1}{2}\text{diag}({\Delta }_{ng}^{(t)})\text{diag}{\left({\mathit{\kappa }}_{ng}^{\left(t\right)}\right)}^{\top }\left\}\right]}^{\top }\}+{\mathbf{\Phi }}_g^{-1\left(t\right)}\text{tr}\left\{{\mathbf{\Omega }}_g^{-1\left(t\right)}{\mathit{\kappa }}_{ng}^{\left(t\right)}\right\}]}^{-1},\end{array}$$

      where the vector function $\mathbf{e}\mathbf{x}\mathbf{p}\left[\mathbf{a}\right]=(e^{a_1},\dots ,e^{a_r})\prime$ is a vector of the exponential of each element of the r-dimensional vector a, $\text{diag}\left(\mathit{\kappa }\right)=({\mathit{\kappa }}_{11}\dots ,{\mathit{\kappa }}_{pp})$ puts the diagonal elements of the p × p matrix $\mathit{\kappa }$ into a p-dimensional vector, and $\odot$ is the Hadmard product.

   2. A fixed-point method is used for updating ${\mathit{\kappa }}_{ng}$:

      $$\begin{array}{l}{\mathit{\kappa }}_{ng}^{(t+1)}=r[{\mathbf{I}}_{p\times p}\odot \{\text{diag}{\left({\Delta }_{ng}^{(t+1)}\right)}^{\top }[\mathbf{e}\mathbf{x}\mathbf{p}\{{\mathit{\xi }}_{ng}^{\left(t\right)}+\text{log}\hspace{0.17em}\mathbf{s}\\ {+\frac{1}{2}{(\text{diag}\left({\mathit{\kappa }}_{ng}^{\left(t\right)}\right)\text{diag}{({\Delta }_{ng}^{(t+1)})}^{\top })}^{\top }\left\}\right]\}+{\mathbf{\Omega }}_g^{-1\left(t\right)}\text{tr}\left\{{\mathbf{\Phi }}_g^{-1\left(t\right)}{\Delta }_{ng}^{(t+1)}\right\}]}^{-1},\end{array}$$

      where the vector function $\mathbf{e}\mathbf{x}\mathbf{p}\left[\mathbf{a}\right]=(e^{a_1},\dots ,e^{a_p})\prime$ is a vector of exponential each element of the p-dimensional vector a, $\text{diag}(\Delta )=({\Delta }_{11}\dots ,{\Delta }_{rr})$ puts the diagonal elements of the r × r matrix $\Delta$ into a r-dimensional vector, and $\odot$ is the Hadmard product.

   3. Newton’s method is used to update ${\mathit{\xi }}_{ng}$:

      $$\begin{array}{l}\text{vec}({\mathit{\xi }}_{ng}^{\top (t+1)})=\text{vec}({\mathit{\xi }}_{ng}^{\top \left(t\right)})-{\Psi }_{ng}^{-1(t+1)}\{\text{vec}(Y_n^{\top })-exp[\log \text{vec}\left(s^{\top }\right)\\ +\text{vec}({\mathit{\xi }}_{ng}^{\top \left(t\right)})+\frac{1}{2}\text{diag}({\Psi }_{ng}^{-1(t+1)})]\\ -{\Psi }_{ng}^{-1(t+1)}\left(\text{vec}\left({\mathit{\xi }}_{ng}^{\top \left(t\right)}\right)-\text{vec}\left(M_g^{\top \left(t\right)}\right)\right)-\text{vec}\left(Y_n^{\top }\right)\},\end{array}$$

      where ${\mathbf{\Psi }}_{ng}^{(t+1)}={\Delta }_{ng}^{(t+1)}\otimes {\mathit{\kappa }}_{ng}^{(t+1)}$.

3. Given ${\widehat{z}}_{ng}^{(t+1)}$ and the variational parameters ${\mathit{\xi }}_{ng}^{(t+1)},\hspace{0.17em}{\Delta }_{ng}^{(t+1)}$, and ${\mathit{\kappa }}_{ng}^{(t+1)}$, the parameters π_g, ${\mathbf{M}}_g,\hspace{0.17em}{\mathbf{\Phi }}_g$, and ${\mathbf{\Omega }}_g$ can be solved for as

   $$\begin{array}{l}{\pi }_g^{(t+1)}=\frac{n_g^{(t+1)}}{N} \text{where} n_g^{(t+1)}=\sum _{n=1}^N{\widehat{z}}_{ng}^{(t+1)},\\ {\mathbf{M}}_g^{(t+1)}=\frac{{\sum }_{n=1}^N{\widehat{z}}_{ng}^{(t+1)}{\mathit{\xi }}_{ng}^{(t+1)}}{n_g^{(t+1)}},\\ {\mathbf{\Phi }}_g^{(t+1)}=\frac{{\sum }_{n=1}^N{\widehat{z}}_{ng}^{(t+1)}({\mathit{\xi }}_{ng}^{(t+1)}-{\mathbf{M}}_g^{(t+1)}){\mathbf{\Omega }}_g^{-1\left(t\right)}{({\mathit{\xi }}_{ng}^{(t+1)}-{\mathbf{M}}_g^{(t+1)})}^{\top }}{p{n}_g^{(t+1)}}\\ +\frac{{\sum }_{n=1}^N{\widehat{z}}_{ng}^{(t+1)}{\Delta }_{ng}^{(t+1)}\text{tr}\left\{{\mathbf{\Omega }}^{-1\left(t\right)}{\mathit{\kappa }}_{ng}^{(t+1)}\right\}}{p{n}_g^{(t+1)}},\\ {\mathbf{\Omega }}_g^{(t+1)}=\frac{{\sum }_{n=1}^N{\widehat{z}}_{ng}^{(t+1)}{\mathit{\kappa }}_{ng}^{(t+1)}\text{tr}\left\{{\mathbf{\Phi }}_g^{-1(t+1)}{\Delta }_{ng}^{(t+1)}\right\}}{r{n}_g^{(t+1)}}\\ +\frac{{\sum }_{n=1}^N{\widehat{z}}_{ng}^{(t+1)}{({\mathit{\xi }}_{ng}^{(t+1)}-{\mathbf{M}}_g^{(t+1)})}^{\top }{\mathbf{\Phi }}_g^{-1(t+1)}({\mathit{\xi }}_{ng}^{(t+1)}-{\mathbf{M}}_g^{(t+1)})}{r{n}_g^{(t+1)}}.\end{array}$$

2.3.3 Hybrid approach

While the MCMC-based approach can generate exact results, fitting such models can be computationally intensive because we need to evaluate the expected complete-data log-likelihood with respect to the posterior distribution of the latent variables at every iteration of the EM algorithm. For example, for datasets with N = 1000, rp = 6, and G = 2 in Simulation setting 2, the MCMC-based approach took on an average of ∼41 h. On the other hand, the VGA approach is computationally efficient—on the same set of datasets from Simulation 2 with $N=1000,rp=6$, and G = 2 fitting such a model took an average of ∼2 min (see Table 7 of Supplementary File 1 for complete details). However, it does not guarantee an exact posterior (Ghahramani and Beal 1999). Thus, we provide a computationally efficient hybrid approach in which

• Step 1: Fit the model using the VGA-based approach.

• Step 2: Estimate the component indicator variable Z_ng conditional on the parameter estimates from the VGA-based approach.

• Step 3: Using the parameter estimates from Step 1 as the initial values for the parameters and using the classification from Step 2, compute the MCMC-based expectation for the latent variable ${\mathit{\theta }}_{ng}$ and obtain the final estimates of the model parameters.

The hybrid approach comes with a substantial reduction in computational overhead compared to a traditional MCMC-EM but it can generate samples from the exact posterior distribution. Fitting such a model using the hybrid approach on the same set of datasets from Simulation setting 2 with $N=1000,rp=6$, and G = 2 took on average just under 10 min. Complete details of computational times for all three simulation settings are provided in Table 7 of Supplementary File 1. When the primary goal is to detect the underlying clusters (which is the case for the real data analysis), the VGA-based approach is sufficient. However, when the primarily goal is posterior inference, we recommend the hybrid approach as it can better yield an exact posterior similar to the MCMC-EM approach but is computationally efficient.

Details on the convergence criteria, initialization, and parallel implementation for an MCMC-EM approach is provided in Supplementary File 3.

2.4 Identifiability

Model identifiability is vital to obtain unique and consistent parameter estimates. Identifiability of univariate and multivariate finite mixtures of normal distributions has been proved (Teicher 1963; Yakowitz and Spragins 1968). With regard to the mixtures of MVPLN distributions, the estimates for ${\mathbf{\Phi }}_g$ and ${\mathbf{\Omega }}_g$ are only unique up to a strictly positive constant. To eliminate identifiability issues, a constraint needs to be imposed, e.g. the trace of ${\mathbf{\Omega }}_g$ can be set equal to p, the trace of ${\mathbf{\Phi }}_g$ can be set equal to r, or the first diagonal element of ${\mathbf{\Phi }}_g$ can be set equal to 1. The latter solution, which is used by Gallaugher and McNicholas (2018), is used for all analyses in this paper. To obtain final parameter estimates, the resulting ${\mathbf{\Phi }}_g^{\left(t\right)}$ is divided by the first diagonal element of ${\mathbf{\Phi }}_g^{\left(t\right)}$, and ${\mathbf{\Omega }}_g^{\left(t\right)}$ is multiplied by the first diagonal element of ${\mathbf{\Phi }}_g^{\left(t\right)}$.

2.5 Model selection and performance assessment

Four model selection criteria are offered, which include the Akaike information criterion (AIC; Akaike 1973), the Bayesian information criterion (BIC; Schwarz 1978), a variation of the AIC used by Bozdogan (1994) called AIC3, and the integrated completed likelihood (ICL; Biernacki et al. 2000). These criteria are given by $\mathrm{AIC}=-2\hspace{0.17em}\text{log}\hspace{0.17em}\mathcal{L}\left({\mathit{\vartheta }}^{\left(\text{MLE}\right)}\right|\mathit{y})+2K,\hspace{0.17em}\mathrm{BIC}=2\hspace{0.17em}\text{log}\hspace{0.17em}\mathcal{L}({\mathit{\vartheta }}^{\left(\text{MLE}\right)}\left|\mathit{y}\right)-K\hspace{0.17em}\text{log}\hspace{0.17em}N,\hspace{0.17em}\mathrm{AIC}3=-2\hspace{0.17em}\text{log}\hspace{0.17em}\mathcal{L}\left({\mathit{\vartheta }}^{\left(\text{MLE}\right)}\right|\mathit{y})+3K$, and $\mathrm{ICL}\approx \mathrm{BIC}+2{\sum }_{n=1}^N{\sum }_{g=1}^G\text{MAP}\left\{{\hat{z}}_{ng}^{\left(t\right)}\right\}\hspace{0.17em}\text{log}\hspace{0.17em}{\hat{z}}_{ng}^{\left(t\right)}$, respectively, where $\mathcal{L}\left({\mathit{\vartheta }}^{\left(\text{MLE}\right)}\right|\mathbf{y})$ represents maximized log-likelihood, ${\mathit{\vartheta }}^{\left(\text{MLE}\right)}$ is the maximum likelihood estimate of the model parameters $\mathit{\vartheta }$, N is the number of observations, and $\text{MAP}\left\{{\hat{z}}_{ng}^{\left(t\right)}\right\}$ is the maximum a posteriori classification given ${\hat{z}}_{ng}^{\left(t\right)}$.

In situations where the true classes are known but, for clustering purposes, are ignored, the adjusted Rand index (ARI; Hubert and Arabie 1985) can be used for performance assessment. The ARI takes a value 1 under perfect class agreement and has expected value 0 under random classification.

3 Results

3.1 Simulations

Simulation studies were conducted to illustrate the ability to recover the true underlying parameters for the mixtures of MVPLN algorithm. For Simulation 1, datasets with G = 1 component were generated with N = 1000 observations, r = 2 and p = 3. For Simulation 2, datasets with G = 2 components and ${\pi }_1=0.79$ were generated with N = 1000 observations, r = 2 and p = 3. For Simulation 3, datasets with G = 2 components and ${\pi }_1=0.6$ were generated with N = 1000 observations, r = 2 and p = 3. Further, only diagonal covariance structures for both ${\mathbf{\Phi }}_g$ and ${\mathbf{\Omega }}_g$ were considered in Simulation 3. Each of the simulation settings consisted of 25 different datasets. The count range in the simulated datasets closely represented the range observed in the RNA-seq data (Freixas-Coutin et al. 2017). The covariance matrices ${\mathbf{\Phi }}_g$ and ${\mathbf{\Omega }}_g$ for each setting are generated using the $\mathtt{clusterGeneration}$ package (Qiu and Joe 2015). Initialization of z_ng was done using one hundred different runs of the k-means algorithm. Clustering was performed on each dataset for values $G=1,\dots ,5$. We also compared the performance of our approach on count datasets generated from competitive models. For Simulation 4, we generated 25 datasets from a mixture of six independent Poisson distributions with G = 2 components, ${\pi }_1=0.45$ and N = 1000 and we analyzed the data as 2 × 3 matrix when using mixtures of MVPLN distributions. For Simulation 5, we generated 25 datasets from a mixture of six independent negative binomial distributions with G = 2 components, ${\pi }_1=0.79$ and N = 2000 and we analyzed the data as 2 × 3 matrix when using mixtures of MVPLN distributions. For Simulation 6, to show performance on a dataset with similar number of components to real data, we generated 25 datasets from a mixture of MVPLN distributions with G = 8 components. We set N = 1500, r = 2, p = 3, and ${\pi }_1=\cdots {\pi }_8=0.125$.

Comparative studies were also conducted on datasets from all five simulation settings. Because no comparable methods capable of clustering three-way count data were found in the current literature, datasets from Simulations 1, 2, 3, and 6 were vectorized and analyzed with clustering methods designed for two-way data. For comparison purpose, a model-based clustering technique for count data, $\mathtt{HTSCluster}$ (Rau et al. 2011, 2015), and a distance-based method, k-means clustering (MacQueen 1967) were used. For $\mathtt{HTSCluster}$, initialization and clustering ranges were same as those used for mixtures of MVPLN algorithm. In a classification EM framework (CEM; Celeux and Govaert 1992), k-means has been shown to be equivalent to an isotropic Gaussian mixture model with equal variance across all components and with equal mixing proportions. Here, we fitted an equal variance isotropic Gaussian mixture model using $\mathsf{R}$ package mclust (Scrucca et al. 2016) on the log-transformed data which would be similar to fitting a fuzzy version of the k-means on the latent space and allow us to compute ICL that relies on the estimated soft $\widehat{\mathbf{Z}}$. We then utilized model selection criteria to select the optimal number of components.

The clustering results along with ARI values of our proposed method and other comparative methods are provided in Table 1. As evident in Table 1, in all six simulations, our proposed approach was able to recover the underlying cluster in all 25 datasets using both BIC and ICL, including when the Simulations 4 and 5, where the datasets were generated from mixtures of Poisson and negative binomial distributions. However, overfitting was evident with AIC and AIC3 as in various datasets, it favored models with larger number of components (i.e. G = 3, 4, and 5 for Simulations 1–5 and G = 9 and G = 10 for Simulation 6). The AIC and AIC3 penalize log-likelihood only for the number of free parameters in the model and are constant with respect to the sample size. When the number of observations is large, these model selection criteria tend to favor more complex models and are known to overestimate number of components (Shibata 1976; Katz 1981). Note that we only provide the ARI values from the VGA approach. The ARI from the hybrid approach is the same as that from the VGA approach because the cluster membership indicator variable is determined in the VGA step in the hybrid approach. While we provide the estimated parameters for using MCMC approach, due to the extreme computational cost that comes with fitting these models, we only fitted the model with correct number of components, and thus, do not provide the summary in Table 1. For each simulation setting, we also provide a measure of cluster separation. For each simulated dataset, we computed the normalized separation index, $I\in [0,1]$, by Hennig (2019) using the $\mathsf{R}$ package $\mathtt{fpc}$ (Hennig 2020), where higher values indicate that clusters have good separation. As Euclidean distance is used as a distance measure for computing the separation index, and we computed the separation index using the log-transformed data, which would be equivalent to assessing separation in the latent space. The average values of the separation index along with the standard deviation are provided in Table 1. The parameter estimation results for ${\mathbf{M}}_g,\hspace{0.17em}{\mathbf{\Phi }}_g$, and ${\mathbf{\Omega }}_g$ via the mixtures of MVPLN algorithm for Simulations 1–3 using all three approaches are summarized in Supplementary File 1. As can be seen through the simulations, all three approaches can recover the parameter estimates very well. However, there is a slight increase in the precision of these estimations with the hybrid approach. Given the large number of groups, for Simulation 6, the Frobenius norm of the difference between the estimated parameters and true values of the parameters are provided in Supplementary File 1. In Table 7 of Supplementary File 1, we provide the computation time taken to fit the mixtures of MVPLN model with correct number of components using all three components. Although all three approaches provide comparable parameter estimates, the computational time taken to fit MVPLN using MCMC-based approach is substantially larger than for the VGA and the hybrid approaches as seen in Table 7 of Supplementary File 1. Overall, the simulation experiments illustrated that our approach for parameter estimation (Section 2.3) is effective at parameter recovery for the mixtures of MVPLN distributions.

Table 1.

Mean of normalized cluster separation index (standard deviation) for each simulation setting and the number of clusters selected (average ARI, standard deviation) from all 25 datasets for each simulation setting using different model selection criteria.

Method	Simulation setting	Mean of normalized separation index (SD)	BIC	ICL	AIC	AIC3
Mix. of MVPLN	1	–	1 (1.00, 0.00)	1 (1.00, 0.00)	1, 2 (0.88, 0.33)	1 (1.00, 0.00)
(VGA-based	2	0.44 (0.02)	2 (1.00, 0.00)	2 (1.00, 0.00)	2, 3, 4, 5 (0.89, 0.22)	2 (1.00, 0.00)
approach)	3	0.37 (0.02)	2 (1.00, 0.00)	2 (1.00, 0.00)	2, 3, 4, 5 (0.86, 0.21)	2, 3, 4 (0.96, 0.12)
	4	0.69 (0.01)	2 (1.00, 0.00)	2 (1.00, 0.00)	2 (1.00, 0.00)	2 (1.00, 0.00)
	5	0.44 (0.02)	2 (1.00, 0.00)	2 (1.00, 0.00)	2, 5 (0.99, 0.06)	2 (1.00, 0.00)
	6	0.14 (0.01)	8 (0.99, 0.00)	8 (0.99, 0.00)	8, 9, 10 (0.99, 0.01)	8, 9 (0.99, 0.01)
$\mathtt{HTSCluster}$	1	–	5 (0.00, 0.00)	5 (0.00, 0.00)	5 (0.00, 0.00)	5 (0.00, 0.00)
	2	0.44 (0.02)	5 (−0.00, 0.00)	5 (−0.00, 0.00)	5 (−0.00, 0.00)	5 (−0.00, 0.00)
	3	0.37 (0.02)	5 (0.00, 0.01)	5 (0.00, 0.01)	5 (0.00, 0.01)	5 (0.00, 0.01)
	4	0.69 (0.01)	2 (1.00, 0.00)	2 (1.00, 0.00)	2, 3, 4 (0.97, 0.06)	2, 3, 4 (0.97, 0.06)
	5	0.44 (0.02)	5 (0.22, 0.01)	5 (0.22, 0.01)	5 (0.22, 0.01)	5 (0.22, 0.01)
	6	0.14 (0.01)	10 (0.03, 0.13)	10 (0.03, 0.13)	10 (0.03, 0.13)	10 (0.03, 0.13)
Fuzzy-version of	1	–	5 (0.00, 0.00)	4,5 (0.00, 0.00)	5 (0.00, 0.00)	5 (0.00, 0.00)
k-means	2	0.44 (0.02)	5 (0.24, 0.01)	5 (0.24, 0.01)	5 (0.24, 0.01)	5 (0.24, 0.01)
	3	0.37 (0.02)	3, 4, 5 (0.55, 0.06)	2, 3, 4, 5 (0.81, 0.23)	4, 5 (0.54, 0.04)	4, 5 (0.54, 0.04)
	4	0.69 (0.01)	2 (1.00, 0.00)	2 (1.00, 0.00)	2, 4, 5 (0.54, 0.12)	2, 4, 5 (0.54, 0.12)
	5	0.44 (0.02)	2 (1.00, 0.00)	2 (1.00, 0.00)	2, 3, 4, 5 (0.82, 0.27)	2, 3, 4, 5 (0.82, 0.27)
	6	0.14 (0.01)	8, 9, 10 (0.93, 0.03)	8, 9, 10 (0.96, 0.02)	10 (0.90, 0.01)	10 (0.90, 0.01)

In the two simulation settings where the datasets were generated from mixtures of independent Poisson distributions and mixtures of independent negative binomial distributions (Simulation 4 and 5), our approach was able to recover the underlying cluster structure and estimate the mean and variances of samples fairly well. With regard to $\mathtt{HTSCluster}$, a model with G = 5, the highest value for G considered, was selected for in Simulations 1, 2, 3, and 5 and a model with G = 10, the highest value for G considered, was selected for in Simulation 6. Furthermore, the ARI values were low across these simulation settings, indicating that observations were not assigned to the correct clusters. However, in simulation setting 4 where the datasets are generated from mixtures of independent Poisson distribution, $\mathtt{HTSCluster}$ is able to recover the underlying cluster structure in all 25 datasets using BIC and ICL with perfect classification. It is worth noting that several studies have demonstrated that in presence of biological variations, the observed variability in the RNA-seq data is greater than what is predicted by the Poisson model (Gao et al. 2010). Thus, the negative binomial distribution (a hierarchical Poisson model similar to the univariate version of the MPLN model) has emerged as the most predominantly used distribution of choice for univariate analysis involving RNA-seq data (Gao et al. 2010). For k-means clustering, the average ARI values were also low for Simulations 1–3 but the value for Simulation 6 was high and perfect classification was observed in Simulations 4 and 5 using BIC and ICL.

3.2 Clustering transcriptome data

To illustrate the applicability of mixtures of MVPLN distributions for detecting the underlying cluster structure, the VGA-based approach was applied to an RNA-seq dataset. Typically, only a subset of genes from the experiment are used for cluster analysis, in order to reduce noise. For this analysis, only the differentially expressed genes were used for clustering. Freixas-Coutin et al. (2017) used RNA-seq to monitor the transcriptional dynamics in the seed coats of darkening and nondarkening cranberry beans (Phaseolus vulgaris) at three developmental stages: early, intermediate and mature. The aim of the study was to evaluate if the changes in the seed coat transcriptome were associated with proanthocyanidin levels as a function of seed development in cranberry beans. The RNA-seq data are available on the National Center for Biotechnology Information (NCBI) Sequence Read Archive (SRA) under the BioProject PRJNA380220.

The study identified 1336 differentially expressed genes, which were used for clustering. The raw read counts for genes were obtained from Binary Alignment/Map files using samtools (Li et al. 2009) and HTSeq (Anders et al. 2015). The median value from the three replicates per each developmental stage was used. Each observation or gene (the unit) was structured as a 2 × 3 matrix, such that it contained counts for the two varieties (the variables): darkening and nondarkening, across the three developmental stages (the occasions): early, intermediate and mature. On the three-way data of dimensions $1336\times 2\times 3$, a clustering range of $G=1,\dots ,10$ was considered using k-means initialization (100 runs). Furthermore, we repeated the analysis 10 times. Since BIC and ICL both performed well in recovering the underlying cluster structure in the simulated data, here we also used BIC and ICL for model selection. Both BIC and ICL selected a model with G = 8. In this model, Clusters 1–8 were composed of $206 (15.4\%),\hspace{0.17em}163 (12.2\%),\hspace{0.17em}104 (7.8\%),\hspace{0.17em}162 (12.1\%),\hspace{0.17em}126 (9.4\%),\hspace{0.17em}147 (11.0\%),\hspace{0.17em}162 (12.1\%)$, and 266 (19.9%) genes, respectively. See Supplementary File 2 for gene composition of each cluster. Expression patterns across the clusters were visualized using a heatmap. The log-transformed expression patterns of the clusters are illustrated using the heatmap in Fig. 4.

Figure 4.

Heatmap showing, for Cluster 1 through Cluster 8, log-transformed gene expression patterns for the G = 8 model selected by both BIC and ICL for the cranberry bean RNA-seq dataset. The red and blue colors represent the log-transformed expression levels, where red represents high expression and blue represents low expression. The rows represent the genes and the columns represent samples involved in the RNA-seq study. Respectively, the samples are D-E: darkening early, D-I: darkening intermediate, D-M: darkening mature, ND-E: nondarkening early, ND-I: nondarkening intermediate, and ND-M: nondarkening mature cranberry bean.

In simulation studies where the true labels are available and ARI of true and predicted class label can be used to assess clustering performance. Because no true labels are available in real data, we use visualization of the data using heatmap to assess the trends in the clusters obtained on the transcriptome data in Fig. 4. We also visualize the cluster-specific μ_g which relates to mean trends of the expression levels of genes in different clusters in Fig. 5. As evident in Figs 4 and 5, each cluster has its distinctive expression signatures. In some clusters (for example, Cluster 4), the means of the gene expression signatures are similar between the darkening and nondarkening beans but their mean expression pattern varies between developmental stages; whereas, in Cluster 8, the means of the gene expression signatures are similar across the developmental stages but varies slightly between the darkening and nondarkening beans. On the other hand, in Cluster 3, the mean gene expression signatures vary across both the developmental stages as well as between darkening and nondarkening beans.

Figure 5.

Visualization of the cluster-specific ${\mathbf{\mu }}_g$ which relates to the mean trends of the expression levels of genes in Clusters 1–8.

For all simulation and transcriptome data analyses, the normalization factors representing library sizes for samples were obtained using the trimmed mean of M values from calcNormFactors function of edgeR package (Robinson and Oshlack 2010; McCarthy et al. 2012).

A table of mathematical notation is provided in Supplementary File 4 and a table of abbreviations is provided in Supplementary File 5.

4 Discussion

A mixture of MVPLN distributions is introduced for clustering three-way count data, targeted at expression data arising from RNA-seq experiments. This is the first use of a mixture of MVPLN distributions for clustering within the literature. By allowing for a direct analysis of three-way data structures, matrix variate distributions permit the estimation of correlations within and between variables and occasions. This makes them very attractive for analyzing matrix data in the context of clustering. Further, by considering a matrix variate structure, the number of free covariance parameters to be estimated is greatly reduced under high dimensional settings. Herein, three different parameter estimation frameworks are proposed: an approach based on MCMC, one based on VGA, and one based on a hybrid approach. When posterior inference is of interest, the MCMC-based approach is favorable but it can be computationally intensive. On the other hand, the VGA-based approach only approximates the posterior distribution that relies on approximation but it is computationally efficient. Therefore, here we also propose a hybrid approach that is computationally efficient and it samples from the true posterior. Through simulation studies, we show that the VGA approach provides good clustering performance even when the datasets are generated from mixtures of other discrete distributions.

Using simulated data, it was illustrated that the algorithm for mixtures of MVPLN distributions is effective and returned favorable clustering results. The transcriptome data analysis showed the applicability of the mixture model-based clustering method on RNA-seq count data. A possible future direction of this work would be to make use of subspace clustering methods and to develop the matrix variate factor analyzers model. This would permit clustering of data in low-dimensional subspaces as high-dimensional RNA-seq datasets become frequent. Another path is to consider restrictions on the matrices ${\mathbf{\Phi }}_g$ and ${\mathbf{\Omega }}_g$, as done by Viroli (2011). Also, constraints on ${\mathbf{\Phi }}_g$ similar to those introduced by McNicholas and Murphy (2010), and used by Anderlucci and Viroli (2015), could be beneficial when analyzing longitudinal RNA-seq data.

Supplementary data

Supplementary data are available at Bioinformatics online.

Conflict of interest

None declared.

Funding

This work was supported by Discovery Grant from the Natural Sciences and Engineering Research Council of Canada (NSERC) [to S.S. Grant no. RGPIN-2021-03812 and to P.M. Grant no. RGPIN-2023-06030], Canada Research Chairs program [to S.S. Grant no. 2020-00303 and P.M. Grant no. 950-230797, respectively], Queen Elizabeth II Graduate Scholarships in Science & Technology [to A.S.], Ontario Graduate Fellowship [to A.S.], and an E.W.R. Steacie Memorial Fellowship from NSERC [to P.M. Grant no. 537596-2019]. No funding body played a role in the design of the study, analysis and interpretation of data, or in writing the manuscript.

Data availability

The RNA-seq data used in the manuscript are publicly available on the National Center for Biotechnology Information (NCBI) Sequence Read Archive (SRA) under the BioProject PRJNA380220. The GitHub R package for this work is available at https://github.com/anjalisilva/mixMVPLN.