Bayesian Multinomial Logistic Normal Models through Marginally Latent Matrix-T Processes

Abstract

Bayesian multinomial logistic-normal (MLN) models are popular for the analysis of sequence count data (e.g., microbiome or gene expression data) due to their ability to model multivariate count data with complex covariance structure. However, existing implementations of MLN models are limited to small datasets due to the non-conjugacy of the multinomial and logistic-normal distributions. Motivated by the need to develop efficient inference for Bayesian MLN models, we develop two key ideas. First, we develop the class of Marginally Latent Matrix-T Process (Marginally LTP) models. We demonstrate that many popular MLN models, including those with latent linear, non-linear, and dynamic linear structure are special cases of this class. Second, we develop an efficient inference scheme for Marginally LTP models with specific accelerations for the MLN subclass. Through application to MLN models, we demonstrate that our inference scheme are both highly accurate and often 4–5 orders of magnitude faster than MCMC.

1. Introduction

Motivated by the growing need for efficient inference for a wide class of multinomial logistic-normal (MLN) models, in this article we develop two key ideas. First, we introduce the class of Marginally Latent Matrix-T Process (Marginally LTP) models. As the name suggests, Marginally LTP models are defined by a shared canonical marginal form which is a multivariate generalization of Student-t processes (Shah et al., 2014) and allow for non-Gaussian likelihoods. We show that this class is extremely flexible, encompassing many useful models including generalized linear models, generalized Gaussian process models, and generalized dynamic linear models. Second, we develop a general inference scheme for Marginally LTP models (which we term the collapse-uncollapse sampler) with specific accelerations (namely a marginal Laplace approximation) for the subclass of MLN models. Through both simulations and analyses of real datasets using MLN models, we show that our inference schemes are both highly accurate and often 4–5 orders of magnitude faster than MCMC.

MLN models are used for the analysis of compositions measured through multivariate counting. In contrast to multinomial Dirichlet models, MLN models permit both positive and negative covariation between multinomial categories (Aitchison and Shen, 1980). While multinomial logistic-normal topic models have been used in natural language processing for some time (Blei and Lafferty, 2006; Glynn et al., 2019), more recently these models have been adopted for regression and time-series modeling of microbiome data (Grantham et al., 2017; Silverman et al., 2018a; Äijö et al., 2017).

Yet, inference in MLN models is challenging due to lack of conjugacy between the multinomial and the logistic normal. Early work with MLN models used Metropolis within Gibbs samplers (Cargnoni et al., 1997; Billheimer et al., 2001) and could scale to just a small number multinomial categories (i.e., less than 5). Recently, Pólya–Gamma data augmentation was proposed as a means of inference in MLN regression by augmenting Pólya–Gamma random variables between the multinomial and logistic normal components of a model. Yet for MLN models, the number of Gibbs sampling steps scales linearly with the number of multinomial categories (Polson et al., 2013). Numerous authors have found this approach too computationally intensive to scale to large multinomial models and have instead developed augmentation methods based on a stick-breaking representation of the multinomial (Linderman et al., 2015; Zhang and Zhou, 2017). However, this stick breaking representation does not maintain the logistic-normal form of the model and is sensitive to the labeling of multinomial categories (Linderman et al., 2015). Most recently, several authors (Silverman et al., 2018a; Äijö et al., 2017; Grantham et al., 2017) have shown that Hamiltonian Monte Carlo (HMC) provides for a more efficient and scalable approach to inference in MLN models. In particular, Grantham et al. (2017) used a HMC within a Gibbs sampler whereas both Silverman et al. (2018a), and Äijö et al. (2017) found that the No-U-Turn-Sampling algorithm provided by the Stan Modeling language (Gelman et al., 2015), provided more scalable inference. However, both these approaches are still limited in the number of categories or samples that they can handle. Silverman et al. (2018a) analyzed approximately 800 samples each with only 10 multinomial categories; Äijö et al. (2017) analyzed 36 multinomial categories but had to run their model over the dataset using a sliding window of 60 samples at a time; and Grantham et al. (2017) analyzed 166 samples and 2662 categories but had to impose low rank structure on the logistic normal model for computational tractability. In this work we show that our inference methods scale to hundreds to thousands of categories and samples and permit inference for a wide variety of models including non-linear regression models (as in Äijö et al. (2017)), dynamic linear models (as in Silverman et al. (2018a)), and linear regression models (as in Grantham et al. (2017)).

The layout of this article is as follows. In Section 2 we introduce a common motivation for the use of MLN models. In Section 3 we introduce the class of Marginally LTP models which encompasses many useful MLN models. In Section 4 we develop inference methods for Marginally LTP Models as well as developing specific acceleration for MLN models. In Section 5 we demonstrate our approaches through extensive simulation studies of MLN models. Finally, in Section 6 and 7 we demonstrate the utility of our approaches by developing both linear and non-linear regression models for microbiome sequence count data. Finally, we close with a discussion in Section 8.

2. Multinomial Logistic-Normal (MLN) Models

Our primary motivation in this work was to develop efficient inference for a class of models we refer to as multinomial logistic-normal (MLN) models. Consider a dataset $Y$ consisting of $N$ observations of $D$-dimensional count vectors; where the counting process for each observation is modeled as multinomial. For example, in the analysis of microbiome data we may consider $Y_{\cdot j}$ to be a count vector with a total of $n_j={\sum }_i{Y}_{ij}$ counts, representing the number of DNA molecules observed for each of $D$ different bacterial taxa in sample $j$. Yet, in many such datasets, multinomial count variation is just one source important variation. Consider the task of modeling a hypothetical dataset of $N$ political polls each collected in a different year and each counting the number of polled individuals who identify with one of $D$ different political parties. In such a setting we may wish to develop a model of the form:

$$Y_{\cdot t}~\text{Multinomial}\left(n_t,{\pi }_{\cdot t}\right)$$

$$\pi ~f\left(\theta \right)$$

where $f\left(\theta \right)$ represents a time-varying stochastic process with parameters $\theta$. Often, a logistic normal model represents an appealing form for $f$ as it, in contrast to Dirichlet models, allows for both positive and negative covariation between the political parties (Aitchison and Shen, 1980). Furthermore, if $\varphi$ represents a log-ratio transform such as the ${\text{ALR}}_D$ transform, with inverse given by:

$${\text{ALR}}_D^{-1}\left({\eta }_{\cdot j}\right)=\left(\frac{e^{{\eta }_{1j}}}{1+\sum _{i=1}^{D-1}{e}^{{\eta }_{ij}}},\cdots ,\frac{e^{{\eta }_{(d-1)j}}}{1+\sum _{i=1}^{D-1}{e}^{{\eta }_{ij}}},\frac{1}{1+\sum _{i=1}^{D-1}{e}^{{\eta }_{ij}}}\right),$$ (1)

then we can write a multinomial logistic normal (MLN) model as a multinomial transformed-multivariate normal model:

$$Y_{\cdot t}~\text{Multinomial}\left(n_t,{\pi }_{\cdot t}\right)$$

$${\pi }_{\cdot t}={\varphi }^{-1}\left({\eta }_{\cdot t}\right)$$

$$\text{vec}\left(\eta \right)~N\left(\mu ,\Sigma \right).$$

This relationship between the logistic-normal and the multivariate normal demonstrates another appealing property of logistic-normal models: They can often be easily formulated as a transformation of existing multivariate normal models.

In what follows, we develop efficient inference methods for a class of models we term Marginally Latent Matrix-T (Marginally LTP) models. We show that the class of Marginally LTP models encompasses many useful MLN models such as linear regression models, non-linear regression models, and time-series models.

3. Modeling Overview

In this section we will introduce Marginally Latent Matrix-T Process (Marginally LTP) models as a flexible class of models capable of describing a wide variety of linear regression, non-linear regression, and time-series models.

3.1. Matrix-Normal and Matrix-T, Distributions and Processes

To build the class of Marginally LTP models we first review matrix-normal distributions and processes as well as matrix-t distributions and processes, highlighting properties we will make use of in this article.

Matrix-Normal Distribution

The matrix-normal distribution is a generalization of the multivariate normal distribution to random matrices. We describe a random $m\times n$ matrix $X$ as being distributed matrix-normal $Y~N(M,U,V)$ if $\text{vec}\left(Y\right)~N\left(\text{vec}\right(M),V\otimes U)$ where $\otimes$ denotes the Kronecker product, $U$ is a $m\times m$ covariance matrix and $V$ is a $n\times n$ covariance matrix.

Matrix-Normal Process

We define a stochastic process $\mathsf{\text{Y}}$ as a matrix-normal process on the set $𝒳={𝒳}^{\left(1\right)}\times {𝒳}^{\left(2\right)}$ and denoted $\mathsf{\text{Y}}~\mathsf{\text{GP}}(\mathsf{\text{M}},\mathsf{\text{K}},\mathsf{\text{A}})$ if $\text{Y}$ evaluated on any two finite subsets ${\mathbf{x}}^{\left(1\right)}=\left(x_1^{\left(1\right)},\dots ,x_P^{\left(1\right)}\right)\in {𝒳}^{\left(1\right)}$ and ${\mathbf{x}}^{\left(2\right)}=\left(x_1^{\left(2\right)},\dots ,x_N^{\left(2\right)}\right)\in {𝒳}^{\left(2\right)}$ is distributed as $Y~N(M,K,A)$ where $M_{ij}=\mathsf{\text{M}}\left(x_i^{\left(1\right)},x_j^{\left(2\right)}\right),K_{ij}=\mathsf{\text{K}}\left(x_i^{\left(1\right)},x_j^{\left(1\right)}\right),A_{ij}=\mathsf{\text{A}}\left(x_i^{\left(2\right)},x_j^{\left(2\right)}\right)$ for matrix function $\mathsf{\text{M}}$ and kernel functions $\mathsf{\text{K}}$ and $\text{A}$. The requirement that $\mathsf{\text{K}}$ and $\mathsf{\text{A}}$ be kernel functions implies that the matrices $K$ and $A$ are covariance matrices (i.e., they are symmetric positive definite).

Matrix-t Distribution

The matrix-t distribution is a generalization of the multivariate-t distribution to random matrices. Like the multivariate-t, the matrix-t can be defined constructively through its relationship to the matrix-normal and inverse Wishart distributions. Let $\Sigma$ denote a random covariance matrix such that $\Sigma ~IW(\Xi ,v)$ where $\Xi$ represents a positive semi-definite scale matrix and $v>0$. Also suppose that $X~N(0,I,V)$. If $C{C}^T=\Sigma$ then the distribution of $Y=CX$ is denoted as matrix-t such that $Y~T(v,0,\Xi ,V)$. For a random matrix $\eta ~T(v,B,K,A)$ the log density of $\eta$ may be written

$$\text{log}{T}_{P\times N}\left(\eta ∣v,B,K,A\right)=\text{log}{\Gamma }_P\left(\frac{v+N+P-1}{2}\right)-\text{log}{\Gamma }_P\left(\frac{v+P-1}{2}\right)-\frac{NP}{2}\text{log}\pi -\frac{N}{2}\text{log}\left|K\right|-\frac{p}{2}\text{log}\left|A\right|-\frac{v+N+P-1}{2}\text{log}\left|I_p+K^{-1}[\eta -B]A^{-1}[\eta -B{]}^T\right|$$ (2)

where ${\Gamma }_a\left(b\right)$ refers to the multivariate gamma function. These results follows directly from Gupta and Nagar (2018, p. 134).

Matrix-t Process

Through analogy to our definition of matrix normal processes, we define a matrix-t process through its relationship to the matrix-t distribution. We define a stochastic process $\mathsf{\text{Y}}~\mathsf{\text{TP}}(v,\mathsf{\text{B}},\mathsf{\text{K}},\mathsf{\text{A}})$ defined on the set $𝒳={𝒳}^{\left(1\right)}\times {𝒳}^{\left(2\right)}$ as a matrix-t process if $\mathsf{\text{Y}}$ evaluated on any two finite subsets ${\text{x}}^{\left(1\right)}=\left(x_1^{\left(1\right)},\dots ,x_P(1)\right)\in {𝒳}^{\left(1\right)}$ and ${\mathbf{x}}^{\left(2\right)}=\left(x_1^{\left(2\right)},\dots ,x_N^{\left(2\right)}\right)\in {𝒳}^{\left(2\right)}$ is distributed as $Y~T(v,B,K,A)$ where $v$ is a scalar strictly greater than zero, $B_{ij}=\mathsf{\text{B}}\left(x_i^{\left(1\right)},x_j^{\left(2\right)}\right),K_{ij}=\mathsf{\text{K}}\left(x_i^{\left(1\right)},x_j^{\left(1\right)}\right)$, and $A_{ij}=\mathsf{\text{A}}\left(x_i^{\left(2\right)},x_j^{\left(2\right)}\right)$ for matrix function $\mathsf{\text{B}}$, and kernel functions $\mathsf{\text{K}}$ and $\mathsf{\text{A}}$. Matrix-t processes can be alternatively seen as a multivariate generalization of Student-t processes which have found widespread use in statistical analysis (Shah et al., 2014).

3.2. Latent Matrix-t Processes (LTPs)

To generalize matrix-t processes to a more flexible set of data types, e.g., count data, we now define LTPs as a generalization of a matrix-t processes. We accomplish this by defining a stochastic process $\mathsf{\text{Y}}$ as a hierarchical process formed by a process $\mathsf{\text{F}}$ having parameters that, with appropriate transformation $\varphi$, follow a matrix-t process. Additionally, we now explicitly denote dependence on model hyper-parameters which we collectively refer to as $\delta$.

Definition 1 Latent Matrix-t Process We define a stochastic process $\mathsf{\text{Y}}$ as a latent matrix-t process $\mathsf{\text{Y}}~\mathsf{\text{LTP}}(\mathsf{\text{F}},\varphi ,v,\mathsf{\text{B}},\mathsf{\text{K}},\mathsf{\text{A}},\delta )$ on the set $𝒳={𝒳}^{\left(1\right)}\times {𝒳}^{\left(2\right)}$ if $\mathsf{\text{Y}}$ evaluated on any $P$ dimensional finite subset ${\mathit{x}}^{\left(1\right)}\in {𝒳}^{\left(1\right)}$ and any $N$ dimensional finite subset ${\mathit{x}}^{\left(2\right)}\in {𝒳}^{\left(2\right)}$ is distributed

$$Y~f(\pi ,\delta )$$ (3)

$$\pi ={\varphi }^{-1}\left(\eta \right)$$ (4)

$$\eta ~T\left(v,B\left(\delta \right),K\left(\delta \right),A\left(\delta \right)\right).$$ (5)

where $\eta$ denotes a $P\times N$ real valued matrix, $B\left(\delta \right)$ a $P\times N$ dimensional real valued matrix function of parameters $\delta$ defined by $\left[B\right(\delta ){]}_{ij}=\mathsf{\text{B}}\left(x_i^{\left(1\right)},x_j^{\left(2\right)},\delta \right),K(\delta )$ is a $P\times P$ covariance matrix defined as $\left[K\right(\delta ){]}_{ij}=\mathsf{\text{K}}\left(x_i^{\left(1\right)},x_j^{\left(1\right)},\delta \right),A(\delta )$ is an $N\times N$ covariance matrix defined as $\left[A\right(\delta ){]}_{ij}=\mathsf{\text{A}}\left(x_i^{\left(2\right)},x_j^{\left(2\right)},\delta \right),v$ is a scalar subject to $v>0,\pi$ is an element of a space $\Pi$ defined via the one-to-one mapping ${\varphi }^{-1}:{ℛ}^{P\times N}\to \Pi$, and $f$ denotes a probabilistic model for the observed data (a likelihood model), with parameters $\pi$ and $\delta$, which is itself an evaluation of the process $\mathsf{\text{F}}$ evaluated on a finite subset of the set $\Pi$.

3.3. Marginally LTP Models

To allow us to represent latent processes beyond LTPs, we next introduce a generalization of LTPs to a larger class which we term Marginally LTP models. This definition is straightforward, we define Marginally LTP models as those models which have a marginal that is an LTP.

Definition 2 Marginally LTP models If a model described by the joint distribution $p(\eta ,\Psi ,Y)$ may be written as $p(\Psi \mid \eta ,Y)p(\eta ,Y)$ where $p(\eta ,Y)$ is an LTP, we refer to $p(\eta ,\Psi ,Y)$ as a Marginally LTP model and $p(\eta ,Y)$ as the model’s collapsed representation.

In the next three subsections we demonstrate that Marginally LTP models provide a rich class of models. We give three examples of Marginally LTP models: (1) a class of multivariate generalized linear models; (2) a flexible class of models for inference in multivariate non-Gaussian time-series; and (3) a flexible class of multivariate generalized non-linear regression models.

3.3.1. Generalized Multivariate Conjugate Linear (GMCL) Models

First we develop generalization of Bayesian multivariate linear regression with conjugate priors which permits non-Gaussian observations (Rossi et al., 2012, p. 32). As in Section 2, let us consider $Y$ to represent $N$ independent $D$-variate measurements and consider $X$ to represent $N$ sets of $Q$-dimensional covariates. We define generalized multivariate conjugate linear (GMCL) models as

$$Y_{\cdot j}~f\left({\pi }_{\cdot j}\right)$$ (6)

$${\pi }_{\cdot j}={\varphi }^{-1}\left({\eta }_{\cdot j}\right)$$ (7)

$${\eta }_{\cdot j}~N(\Lambda {X.}_j,\Sigma )$$ (8)

$$\Lambda ~N(\Theta ,\Sigma ,\Gamma )$$ (9)

$$\Sigma ~IW\left(\Xi ,v\right).$$ (10)

We may describe the joint density of this model as $p(\Lambda ,\Sigma ,\eta ,Y)$ which can be factored as $p(\Lambda ,\Sigma \mid \eta ,Y)p(\eta ,Y)$. Therefore, to parallel to the definition of Marginally LTP models we may equate $\Psi =\{\Lambda ,\Sigma \}$. In Appendix A we prove that $p(\eta ,Y)$ is an LTP with parameters

$$B=\Theta X$$

$$K=\Xi$$

$$A=I_N+X^T\Gamma X$$

and with $\{\Theta ,\Gamma ,\Xi \}\in \delta$. This result demonstrates that all GMCL models are Marginally LTP models. Further, by letting $f$ denote the multinomial distribution and ${\varphi }^{-1}$ denote the inverse ALR transform, we can build multinomial logistic-normal linear models as a special case of GMCL models.

3.3.2. Generalized Multivariate Dynamic Linear Models (GMDLMs)

We develop a flexible class of multivariate time-series models for non-Gaussian observations. We term this class of models generalized multivariate dynamic linear models (GMDLMs). GMDLMs represent an extension of the multivariate dynamic linear models introduced in Quintana and West (1987) and developed further in West and Harrison (1997, Ch. 16) to non-Gaussian observations. Using notation from West and Harrison (1997, Ch. 16), let ${\eta }_t^T$ denote a row-vector (i.e., the transpose of the $t$-th column of $\eta$). We define the GMDLM as

$$Y_{\cdot j}~f\left({\pi }_{\cdot j}\right)$$ (11)

$${\pi }_{\cdot j}={\varphi }^{-1}\left({\eta }_{\cdot j}\right)$$ (12)

$${\eta }_t^T=F_t^T{\Theta }_t+{\nu }_t^T,{\nu }_t~N\left(0,{\gamma }_t\Sigma \right)$$ (13)

$${\Theta }_t=G_t{\Theta }_{t-1}+{\Omega }_t,{\Omega }_t~N\left(0,W_t,\Sigma \right)$$ (14)

$${\Theta }_0~N\left(M_0,C_0,\Sigma \right)$$ (15)

$$\Sigma ~IW(\Xi ,v)$$ (16)

where ${\Theta }_t$ represents a $Q\times P$ matrix describing the state of the time-series at time $t,G_t$ denotes the $Q\times Q$ state transition matrix at time $t,F_t$ denotes a $Q\times 1$ vector describing a linear model relating the latent space to the parameters ${\eta }_t,\Sigma$ is a $P\times P$ covariance matrix specifying the covariation between the $P$ dimensions of the time-series, $W_t$ is a $Q\times Q$ covariance matrix describing the covariation of the perturbations affecting latent states, and ${\gamma }_t$ is a scalar allowing an analyst to weight the importance of select observations (${\gamma }_t$ is typically equal to 1).

The joint model for the GMDLM can be written $p(\Theta ,\Sigma ,\eta ,Y)$ which can be factored as $p(\Theta ,\Sigma \mid \eta ,Y)p(\eta ,Y)$. To parallel the definition of Marginally LTP models, here we have $\Psi =\{\Theta ,\Sigma \}$. In Appendix B we prove that $p(\eta ,Y)$ is an LTP with parameters

$$B=\left[\begin{array}{ccccc}\mid & & \mid & & \mid \\ {\alpha }_1& \cdots & {\alpha }_t& \cdots & {\alpha }_T\\ \mid & & \mid & & \mid \end{array}\right]$$

$${\alpha }_t={\left(F_t^T{𝒢}_{t:1}{M}_0\right)}^T$$

$$K=\Xi$$

$$A_{t,t-k}=\{\begin{array}{l}{\gamma }_t+F_t^T\left[W_t+\sum _{\ell =t}^2{𝒢}_{t:\ell }{W}_{\ell -1}{𝒢}_{\ell :t}^T+{𝒢}_{t:1}{C}_0{𝒢}_{1:t}^T\right]F_t\text{if}k=0\hfill \\ F_t^T\left[{𝒢}_{t:t-k+1}{W}_{t-k}+\sum _{\ell =t-k}^2{𝒢}_{t:\ell }{W}_{\ell -1}{G}_{\ell :t-k}^T+{𝒢}_{t:1}{C}_0{G}_{1:t-k}^T\right]F_{t-k}\text{if}k>0\hfill \end{array}$$

where we have introduced ${𝒢}_{t:\ell }$ as a short hand notation for the product $G_t\cdots G_{\ell }$ and where we have hyper-parameters $\left\{\Xi ,M_0,C_0,W_1,\dots ,W_T,{\gamma }_1,\dots ,{\gamma }_T,G_1,\dots ,G_T,F_1,\dots ,F_T\right\}\in \delta$. This result demonstrates that GMDLMs are a special case of Marginally LTP models.

3.4. Generalized Multivariate Gaussian Process (GMGP) Models

Finally, we develop a flexible class of generalized multivariate non-linear models based on the matrix normal processes discussed in Section 3.1. These models utilize a separable kernel structure to allow modeling of vector valued data as seen, for example, in coregionalization models Álvarez et al. (2012). As a motivating example, suppose that we wish to model a microbiome time-series. In particular, suppose we wish to predict the relative abundance of an unobserved taxa at an unobserved time-point. Let us consider $X$ to encompass available temporal metadata for observed samples, e.g., time-indices as well as other relevant covariates influencing composition at each observed time-point. Further, let us consider $Z$ to encompass available metadata regarding each observed bacterial taxa, e.g., 16S sequence as well as whether the bacteria is aerobic or anaerobic. In this section we describe a flexible class of models which we term Generalized Multivariate Gaussian Process (GMGP) models which are capable of performing this, as well as many other, analysis tasks.

To enable GMGP models to make predictions regarding unobserved multinomial categories (e.g., unobserved taxa) we must first define Inverse Wishart Processes. These processes can be defined constructively in a similar manner to the matrix normal and matrix-t processes we defined in Section 3.1. Given a set $\text{Z}$ with $P$-dimentional finite subset $Z=\left[Z_{\cdot 1},\dots ,Z_{\cdot P}\right]$, a scalar $\nu >0$, and a kernel function $\Xi$ such that ${\Xi }_{ij}=\Xi \left(Z_{\cdot i},Z_{\cdot j}\right)$, we define a stochastic process $\Sigma ~\text{IWP}(\Xi ,\nu )$ as an Inverse Wishart Process on the set $\text{Z}$ if $\Sigma$ evaluated on any subset $Z$ is distributed as $\Sigma ~IW(\Xi ,\nu +p)$. In words, an Inverse Wishart Process is a probability distribution over kernel functions.

Using the above construction of Inverse Wishart Processes, we can now define the GMGP model form:

$$Y_{\cdot j}~f\left({\pi }_{\cdot j}\right)$$ (17)

$${\pi }_{\cdot j}={\varphi }^{-1}\left({\eta }_{\cdot j}\right)$$ (18)

$${\eta }_{\cdot j}~N\left(\Lambda \left(X_{\cdot j}\right),\Sigma \left(Z\right)\right)$$ (19)

$$\Lambda ~\text{GP}(\Theta ,\Sigma ,\Gamma )$$ (20)

$$\Sigma ~\text{IWP}(\Xi ,\nu )$$ (21)

where $\Theta$ is a mean function and $\Gamma$ as well as $\Xi$ are kernel functions.

We may describe the joint density of the above model as $p(\Lambda ,\Sigma ,\eta ,Y,X)$ which can be factored as $p(\Lambda ,\Sigma \mid \eta ,Y,X)p(\eta \mid Y,X)$. In Appendix C, we prove that $p(\eta \mid Y,X)$ is an LTP with parameters $\text{B}=\Theta ,\text{K}=\Xi$, and $\text{A}=\text{I}+\Gamma$ where $\text{I}$ represents the identity kernel defined by:

$$\text{I}\left(x_i,x_j\right)\{\begin{array}{ll}1\hfill & \text{if}{x}_i=x_j\hfill \\ 0\hfill & \text{otherwise}\hfill \end{array}.$$

It should be noted that the LTP form of GMGP models is very similar to that of GMCL models; the major difference between GMGP and GMCL models being the use of mean and kernel functions in place of mean and covariance matrices. Still, we discuss these models separately as they will often be used in very different ways – GMCL models for inferring linear effects of covariates, GMGP models for non-linear smoothing and prediction. We demonstrate examples of both of these models using real data in Sections 6 and 7.

4. Inference in Marginally LTP Models

Our overarching goal was to develop efficient and accurate posterior inference for MLN models, many of which are a special case of Marginally LTP models. In this section, we demonstrate how the canonical LTP form of Marginally LTP Models can be exploited for efficient inference of this larger model class. types of parameters, $\eta$ which are distributed matrix-t and of a model to produce a LTP form. The sampling $\eta$ on $\eta$ and observed data $\left(p\right(\Psi \mid \eta ,Y\left)\right)$. In Section 4.1 we introduce a sampling scheme for Marginally LTP models which we refer to as the collapse-uncollapse (CU) sampler which exploits the hierarchical structure of Marginally LTP models to improve computational efficiency for various types of inference. In Section 4.2 we further build on the CU sampler by introducing a Laplace approximation as a means of accelerating a bottleneck step in the CU sampler. In Sections 4.3 we discuss the CU sampler in the context of the GMCL, GMDLM and GMGP models introduced in the last section. In Section 4.4, we discuss error bounds for the Laplace approximation. In Section 4.5, we discuss inference of hyperparameters. Finally, in Section 4.6 we discuss the fido software package which implements a number of MLN models using the CU sampler with Laplace approximation based on these models Marginally LTP form.

4.1. The Collapse-Uncollapse (CU) Sampler

Consider the task of sampling from the posterior distribution of a Marginally LTP model with joint density $p(\Psi ,\eta ,Y)$. The corresponding posterior density can be decomposed as

$$p(\eta ,\Psi \mid Y)=p(\Psi \mid \eta ,Y)\frac{p(\eta ,Y)}{p\left(Y\right)}.$$

This decomposition implies that, given a Marginally LTP model with joint probability $p(\eta ,\Psi ,Y)$, we may sample from the posterior by first sampling from the posterior of the collapsed (LTP) model $p(\eta ,Y)$ and then given that sample of $\eta$ and the observed $Y$ we may then sample $\Psi$ from the conditional $p(\Psi \mid \eta ,Y)$. Together the sample of $\eta$ and $\Psi$ then represents a single sample from the posterior of the Marginally LTP model, $p(\Psi ,\eta \mid Y)$ (Algorithm 1).

Our rationale for focusing on the CU sampler for inference in Marginally LTP models is as follows. We expect that many Marginally LTP models (such as those introduced in Section 3) have partial conjugacy. Exploiting this partial conjugacy is central to many popular methods such as Metropolis-within-Gibbs (Cargnoni et al., 1997). Yet, by embedding MCMC steps within a Gibbs sampler techniques such as adaptation (Gelman et al., 2015) or approximate methods such as Laplace approximations may not make sense as they would have to be recomputed at each step. In contrast, the CU sampler allows the non-conjugate sampling to occur up front (in the sampling of $p(\eta \mid Y)$) so that such techniques can be used. Moreover, after multiple samples of $\eta$ have been produced, uncollapsing the model can be done in parallel for each sample of $\eta$. Therefore, the CU sampler may be advantageous as it permits the use of adaptive or approximate methods for sampling the non-conjugate model components and permits a degree of parallelism not allowed by Metropolis-within-Gibbs.

The CU Sampler for Marginally LTP Models therefore requires two features for efficient inference. First, we require an efficient means of producing samples from the collapsed (LTP) form $p(\eta \mid Y)$. As we will show in Section 5, sampling $p(\eta \mid Y)$ can be more efficient than sampling $p(\Psi ,\eta \mid Y)$ just by virtue of the fact that the former has fewer dimensions. Therefore the CU sampler alone can be more efficient than sampling the full (uncollapsed) model. Still, in Section 4.2 we develop a Laplace approximation for $p(\eta \mid Y)$ which can further improve efficiency. Second, we require an efficient means of sampling from the posterior conditional $p(\Psi \mid \eta ,Y)$. In Section 4.3 we discuss efficient means of sampling $p(\Psi \mid \eta ,Y)$ for the GMCL, GMDLM, and GMGP model classes.

4.2. Laplace Approximation for the Collapsed Form

Sampling $p(\eta \mid Y)$ is often the major computational bottleneck when inferring Marginally LTP models via the CU sampler. To accelerate this step, we developed a Laplace approximation for the density $p(\eta \mid Y)$. This approximation is defined as $q(\eta \mid Y)=N\left(\text{vec}\hat{\eta },H^{-1}\left(\text{vec}\hat{\eta }\right)\right)$ where $\hat{\eta }$ denotes the maximum a posteriori (MAP) estimate of $p(\eta \mid Y)$ and $H^{-1}\left(\text{vec}\hat{\eta }\right)$ denotes the inverse Hessian matrix of $\text{log}p(\eta \mid Y)$ evaluated at the point vec $\hat{\eta }$. That is, $\hat{\eta }$ is defined as the solution to the following optimization problem

$$\hat{\eta }=\underset{\eta \in {ℛ}^{P\times N}}{\text{argmin}}\left[-\text{log}p\left(\eta ∣Y\right)\right].$$ (22)

The solution to this optimization problem is discussed in Appendix F.

While the accuracy of our Laplace approximation will depend on a number of factors including the choice of likelihood, prior, and link function, we hypothesized that such a Laplace approximation would provide an accurate approximation to an LTP posterior in a number of common settings. First, all exponential family likelihoods are log-convex with respect to their natural parameters (Jordan, 2010). Therefore, we expect the Laplace approximation to be particularly useful with any choice of likelihood $f$ from the exponential family (e.g., the multinomial distribution) and with a corresponding choice of $\varphi$ such that $\eta$ represents the natural parameters of $f$ (e.g., the ALR transform). Second, with regards to the matrix-t prior, the matrix-normal can provide a good approximation for the matrix-t for suitably large $v$ (Gupta and Nagar, 2018, p. 137) as it is both globally symmetric and log-convex about the MAP estimate. We hypothesized that even though the matrix-t is not globally log-convex except as $v\to \infty$, in practice the log-convexity about the MAP estimate coupled with its global symmetry would be enough to provide a useful approximation even for small values of $v$. We note that both our simulation studies in Section 5 and analyses of real data in Section 6 suggest this hypothesis is reasonable. Finally, specifically for models parameterized by probabilities (such as the Multinomial logistic-normal), MacKay (1998) showed that the softmax parameterization can produce more accurate Laplace approximations than the more traditional simplex basis. Notably, the inverse ALR parameterization we choose is a linear transformation of the softmax transform (Pawlowsky-Glahn et al., 2015) and therefore has identical accuracy to a Laplace approximation using the softmax parameterization (MacKay, 1998). Together, these features led us to hypothesize that a Laplace approximation could provide a useful and accurate approximation for the the posterior of an LTP.

Developing an efficient Laplace approximation for LTP models required closed form solutions for the gradient and Hessian of LTPs. To develop these tools note that, by Bayes rule, we may write

$$-\text{log}p(\eta \mid Y)\propto -\text{log}f\left(Y\mid {\varphi }^{-1}\left(\eta \right)\right)-p\left(\eta \right).$$ (23)

By linearity of the derivative operator we may write the gradient and Hessian of $-\text{log}p(\eta \mid Y$) as

$$-\frac{d\text{log}p(\eta \mid Y)}{d\text{vec}\left(\eta \right)}=-\frac{d\text{log}f\left(Y\mid {\varphi }^{-1}\left(\eta \right)\right)}{d\text{vec}\left(\eta \right)}-\frac{d\text{log}p\left(\eta \right)}{d\text{vec}\left(\eta \right)}$$ (24)

$$-\frac{d^2\text{log}p\left(\eta \mid Y\right)}{d\text{vec}\left(\eta \right)d\text{vec}{\left(\eta \right)}^T}=-\frac{d^2\text{log}f\left(Y\mid {\varphi }^{-1}\left(\eta \right)\right)}{d\text{vec}\left(\eta \right)d\text{vec}{\left(\eta \right)}^T}-\frac{d^2\text{log}p\left(\eta \right)}{d\text{vec}\left(\eta \right)d\text{vec}{\left(\eta \right)}^T}.$$ (25)

Thus we find that calculating the gradient and Hessian of LTPs reduces to calculating the gradient and hessian of $\text{log}f\left(Y\mid {\varphi }^{-1}\left(\eta \right)\right)$ and the matrix-t density $\text{log}p(\eta \mid X)$ separately. The additive structure of the gradient and Hessian are central to generalizing this approach to a variety of different observation distributions $f$ and transformations ${\varphi }^{-1}$. In Appendix D we provide the gradient and Hessian for the matrix-t density. With these results, to derive a flexible class of multinomial logistic-normal models, we only need to provide the gradient and Hessian for the logit-parameterized multinomial which we give in Appendix E. We describe the implementation of the Laplace Approximation for an LTP in Appendix F.

4.3. Efficient Sampling from Posterior Conditionals

The second step of the CU sampler involves sampling from the density $p(\Psi \mid \eta ,Y)$. While the density of $p(\Psi \mid \eta ,Y)$ is specific to the particular Marginally LTP model, we develop efficient means of sampling from this density for the GMCL, GMDLM, and GMGP models in Appendices A, B, and C respectively. In particular, for all three of these model classes we make use of the fact that $\Psi$ is conditionally independent of $Y$ given $\eta$, that is $p(\Psi \mid \eta ,Y)=p(\Psi \mid \eta )$. This conditional independence also reduces sampling from the conditionals to computing the posterior distribution of standard Bayesian multivariate linear regression for GMCL and GMGP model and conjugate multivariate dynamic linear models for the GMDLM model. That is, for all three of these model classes, sampling the conditionals reduces to posterior inference for equivalent Bayesian Gaussian models that have been well described previously and have efficient closed form solutions.

4.4. Error Rate for a Laplace Approximation to the Collapsed Form

The inference scheme we propose above for Marginally LTP models has two parts: First, sample from the collapsed LTP representation of the model; Second, uncollapse those samples to produce samples from the full Marginally LTP model. If, as we discuss above, we use a Laplace approximation to sample from the collapsed LTP representation, then the only error induced by this inference scheme is due to the Laplace approximation. We wanted to develop intuition regarding the error rate of this approximation when the observation distribution is a logit-parameterized multinomial. In Appendix K, we prove that for large $v$ this error rate is $O_p\left((D-1)\sum _{j=1}^N{n}_j^{-1}\right)$. That is, the error is stochastically bounded by the sum of the inverse of the average number of counts in each sample. This result follows from theory recently proposed by Ogden (2018) and provides a more general error bound than those used by Kass and Steffey (1989) or Rue et al. (2009). In particular, this bound accounts for not only the number of observed multinomial samples $\left(N\right)$ but the number of counts in each multinomial sample $\left(n_j\right)$ and the dimension over which those counts are spread $\left(D\right)$.

This error bound is intuitive in a number of ways. First, a multinomial sample $j$ with $n_j$ counts can be thought of as $n_j$ independent observations; it is therefore intuitive that our error bound is proportional to $n_j^{-1}$. Second, the number of dimensions in the Laplace approximation to a multinomial sample grows linearly with one minus the number of multinomial categories; intuitively, our error bound is proportional to $D-1$. Third, the number of dimensions in the Laplace approximation to the collapsed LTP form grows linearly with the number of observed samples; intuitively, our error bound grows (approximately) linearly with the number of observed samples. Finally, based on the observation that the multinomial parameterized by log-ratio coordinates is globally log-convex (Jordan, 2010) whereas the matrix-t distribution is only log-convex near the mean; it makes intuitive sense that a stronger likelihood (implied by larger values of $n_j$) would decrease the error of the Laplace Approximation.

This error bound also sheds light on when this Laplace approximation in the CU sampler will provide a useful, accurate inference method for MLN models. For example, this error bound suggests that an ideal dataset for this Laplace approximation is one that has many non-zero counts and lower data-sparsity. In contrast, it suggests that the Laplace approximation should not be used for high-dimensional classification problems, where there are many multinomial categories but only a single non-zero entry per sample. Still, as we demonstrate in the next section, the Laplace approximation can handle substantial data sparsity and many small counts with only minimal error.

4.5. Hyperparameter Inference

Until this point we have not considered the presence of unknown hyperparameters in the LTP form (i.e. we have considered $\delta$ or $\nu$ as given). Yet, for a number of Marginally LTP models, we expect estimation of such hyperparameters will be of interest. For example, within the GMDLM model we anticipate researchers may want to allow the terms $W_t$ to be subject to their own stochastic model. This would in turn require that some portion of $\delta$ is unknown. Alternatively, for GMCL models, analysts may want to infer the degree-of-freedom parameter $\nu$ empirically rather than setting it based on subjective prior information. Overall, we leave inference of $\nu$ and $\delta$ as future work but note a few potential avenues for practitioners looking to infer these parameters. When the hyper-parameter set $\{\nu ,\delta \}$ is small, these parameters may be efficiently selected by cross-validation (Rasmussen, 2003). In contrast, when the set is large (i.e., when $\delta$ is high-dimensional), alternative approaches are likely needed. In particular, we note that Type-II MAP estimation can provide an efficient means of empirically setting hyper-parameters in a variety of hierarchical Bayesian models (Riihimäki et al., 2014).

4.6. Software for Marginally LTP models with Multinomial Observations

For inference of Marginally LTP models with multinomial observations and log-ratio link functions, we developed the R package fido (Silverman, 2019). Fido implements the CU sampler with Laplace approximation described above using optimized C++ code. Estimation of $\hat{\eta }$ is performed using the L-BFGS optimizer which we have found provides efficient and stable numerical results. Additionally all code required to reproduce the results of the next two sections, including the alternative implementations of multinomial logistic-normal linear models discussed in Section 5 is available as a GitHub repository at github.com/jsilve24/fido_paper_code.

5. Simulations

We performed a series of simulation studies to evaluate the CU sampler both with and without the Laplace Approximation in terms of accuracy and efficiency of posterior inference of multinomial logistic-normal models. The only portion of our inference algorithm that is approximate is the Laplace approximation to the LTP form. As this form is shared by all marginally LTP models, we focus our simulations only on multinomial logistic-normal linear models for simplicity (e.g., Equations (6)–(10) where $f$ is the multinomial distribution and $\varphi$ is the ${\text{ALR}}_D$ transform). To evaluate the utility of the CU sampler we compared Hamiltonian Monte Carlo (HMC) of the full model (HMC Uncollapsed) to the CU sampler where sampling of the collapsed (LTP) form was performed using HMC (HMC Collapsed). Both HMC implementations were inferred using the highly optimized No-U-Turn-Sampler provided in the Stan modeling language (Gelman et al., 2015) which has been frequently used for the analysis of MLN models (Äijö et al., 2017; Silverman et al., 2018a). To further evaluate the utility of the Laplace approximation to the collapsed form in the CU sampler (LA Collapsed), we used the function pibble from the fido software package described in Section 4.6. Finally, to compare LA Collapsed to an alternative scheme for approximate inference, we included two mean-field automatic-differentiation Variational Bayes (VB) implementations (Kucukelbir et al., 2015). The first was a VB approximation to the full form (VB Uncollapsed), the second was a VB approximation to just the collapsed form of the CU sampler (VB Collapsed). VB Uncollapsed was unstable in practice and often resulted in error during optimization (likely due to the increased number of parameters in the uncollapsed model). As a result, only the results form VB Collapsed could be shown below.

In order to compare these implementations, we created a series of simulations based on the corresponding likelihood model (Appendix H). We identified three key parameters, the sample size $\left(N\right)$, the observation dimension $\left(D\right)$, and the number of model covariates $\left(Q\right)$ which we varied in order to test each implementation over a wide range of conditions. By varying these parameters in different simulations we were able to vary the error bound for the Laplace approximation introduced in Section 4.4 (Figure S1). We designed these simulations to span a wide range of sparsity (Figure 1, Column 1). We choose the tuple $\{N=100,D=30,Q=5\}$ as our base condition and independently varied each simulation parameter from that base condition ($N$ from 10 to 1000, $D$ from 3 to 500, and $Q$ from 2 to 500). Each panel in Figure 1 shows a different simulation metric (e.g., percent of data matrix $Y$ that were zero counts or the performance of a given inference method on each simulation) for a given tuple when a particular element of the tuple $(N,D,Q)$ is varied from the base condition. For example, the top left panel shows the sparsity of each dataset for $N=100,Q=5$, and where $D$ is varied (x-axis). Additionally, to account for the stochastic nature of the simulations, three simulations were performed for each tuple $\{N,D,Q\}$. For each simulation, each of the five implementations were fit and allowed a maximum of 48 hours to produce 2000 samples. Prior hyper-parameters were chosen to reflect common default choices, e.g., mean parameters set to zero, and covariance parameters set to the identity matrix. The prior degrees-of-freedom parameter $\nu$ is defined on the range $\nu >D$, this parameter was chosen as $\nu =D+10$. Further details of the simulation and model fitting procedure can be found in Appendix H.

Figure 1: Simulation study comparing multinomial logistic normal linear model implementations.

Each row of plots depicts simulation results for varying a different simulation parameter ($D$, the number of multinomial categories; $N$, the number of samples; and $Q$, the number of covariates). The percent of counts that were zero in each simulation is given in the first column. The error bound of the Laplace approximation, which was developed in Section 4.4, is shown in Figure S1. Implementations were compared in terms of efficiency (measured SpES), accuracy of point estimation (measured by RMSE of Regression Coefficients), and accuracy of uncertainty quantification (measured by RMSE of Standard Deviation of Regression Coefficients). For VB Collapsed and LA Collapsed, the number of effective samples is taken to be equal to the total number of samples as both methods produce independent samples from an approximation to the posterior.

To quantify the accuracy and efficiency of each implementation we defined the following performance metrics. As a measure of efficiency, we calculated the average number of seconds needed for the implementation to produce one independent sample from the target posterior (i.e., Seconds per Effective Sample - SpES). Specifying independent samples is important as HMC samplers produce autocorrelated samples. In contrast, both LA Collapsed and VB Collapsed produce independent samples from the approximate posterior, as a result, for these two methods, SpES equals the number of samples per second. To quantify the accuracy of point estimates from each implementation (i.e., either the posterior mean or MAP estimates) we used the root mean squared error of the point estimate for $\Lambda$ from its true simulated value. Notably, given finite $N$ we do not expect that any implementation will be able to perfectly reconstruct the true simulated value for $\Lambda$; rather, this metric provides a means of comparing the relative performance of each implementation. Finally, to quantify the accuracy of uncertainty quantification from each implementation we compared posterior intervals against those of the HMC Collapsed model which was taken as a gold standard. In particular, we define the root mean squared error of standard deviations as the average difference between the estimated posterior standard deviations, $\mathit{\text{sd}}\left({\Lambda }_{ij}\right)$, compared to the estimates produced by HMC Collapsed.

Beyond our error bound for the Laplace approximation, we hypothesized that the proportion of zero values in the dataset would impact the accuracy of both the Laplace and variational approximations. In particular, we hypothesized that datasets with higher than 30% zero values would see a substantial degradation in approximation accuracy. As hypothesized we found that the proportion of zero values (the sparsity) of the dataset closely resembled approximation accuracy (Figure 1). Yet, we found that in practice, LA collapsed performed far better than expected: LA Collapsed provided nearly identical estimates of posterior uncertainty to both HMC implementations up to over 90% data sparsity. Additionally, LA Collapsed provided nearly identical point estimates to both HMC implementation over the full spectrum of simulations. Finally, LA Collapsed was often up to 5 orders of magnitude faster than HMC and often 1–2 orders of magnitude faster than VB.

5.1. Computational Efficiency

Overall, the CU sampler with a Laplace approximation (LA Collapsed) provided the most efficient inference across all tested conditions. More specifically LA Collapsed displayed speed-ups of between 1 to 5 orders of magnitude in comparison to HMC Collapsed and HMC Uncollapsed and often between 1–2 order of magnitude compared to VB Collapsed. Notably, HMC Uncollapsed failed to complete sampling within 48 hours for $D>100$.

Beyond the high efficiency of LA Collapsed, our results also demonstrate that the CU sampler can improve inference in HMC without the use of approximate inference methods. These results likely stem from the smaller number of dimensions in HMC for the collapsed versus uncollapsed implementations. Most noticeably, the collapsed representation completely removes dependency on $Q$ from HMC run-times as $\Lambda$ is marginalized out of the collapsed representation. However, for large $N$ the HMC Uncollapsed is more efficient than HMC Collapsed. This later result may reflect that the heavy tails of the matrix-t distribution produce a more challenging geometry for HMC than the expanded matrix normal and inverse Wishart forms. Such a finding has been well described previously for both univariate and multivariate-t distributions (Stan Development Team, 2018, Section 20).

5.2. Point Estimation

Overall point estimation using LA Collapsed (i.e., MAP estimates) was nearly identical to point estimation using either HMC Collapsed or HMC Uncollapsed (i.e., mean estimates). In contrast, point estimation using VB Collapsed produced substantially larger errors, especially for large values of $D$. Overall these results demonstrate that the CU sampler maintains accuracy in point estimation and that MAP estimation provides an excellent approximation to the mean in multinomial logistic normal models.

5.3. Uncertainty Quantification

Beyond accuracy of point estimates, we also wanted to study the accuracy of estimates of uncertainty from each implementation. We consider the HMC Collapsed implementation to be the gold standard on which we based our performance metric (RMSE of standard deviations). Except for values of $Q$ greater or equal to 250 (where the proportion of zero values is >90%), the uncertainty estimates of LA Collapsed were nearly identical to those of both HMC implementations. Yet, at larger values of $Q$, when sparsity is >90%, we observed differences not only between LA Collapsed and HMC but between the two HMC implementations themselves. There are two possible explanations for this. First, that LA Collapsed had a slightly better point estimation accuracy in these same large $Q$ simulations could point to the fact that LA Collapsed is correct and instead HMC estimates of uncertainty were incorrect due to the often small effective sample size for large $Q$. Alternatively, this could support our previous hypothesis that the Laplace approximation had higher error in uncertainty quantification with higher data sparsity. Given the ergodicity of HMC it seems more likely that the Laplace approximation is in error in these regions of high sparsity. Yet, that the approximation only began to show substantial error when sparsity is >90% is notable. Beyond LA Collapsed and the HMC implementations, VB Collapsed consistently demonstrated higher error in uncertainty quantification as compared to the other implementations.

Finally, to provide context regarding the size of the differences in uncertainty quantification, we provide direct visualizations of posterior intervals for all four implementations in Figure S3 and S4. These two simulations were chosen to highlight a case in which LA Collapsed was highly accurate (Figure S3) in terms of uncertainty quantification and a case in which it differed from HMC estimates (Figure S4). Notably, visualization of posterior intervals consistently demonstrated that the posterior mean was centered symmetrically in the 95% credible regions. This symmetry suggested that our metric RMSE of standard deviations captures much of the discrepancies in uncertainty quantification without higher order moments. Additionally, for context, we include a fifth implementation, PCLM (pseudo-count augmented linear model). The PCLM uses a pseudo-count based estimate of $\eta$ which ignores the multinomial count variation. Such approximations are common in the analysis of microbiome sequence count data (Silverman et al., 2017; Gloor et al., 2016). Unsurprisingly, this PCLM implementation demonstrated substantially higher error rates than any of the other implementations (Figure S5).

6. Identifying Biomarkers of Crohn’s Disease Using Microbiome Data

Crohn’s Disease (CD) is a type of inflammatory bowel disease that has been linked to aberrant immune response to intestinal microbiota (Jostins et al., 2012; Khor and Hibberd, 2012; Gevers et al., 2014). To demonstrate that LA Collapsed (from the R package fido) provides an accurate and efficient means of modeling real microbiome data, we reanalyzed a previously published study comparing microbial composition in the terminal ileum of subjects with CD to healthy controls (Gevers et al., 2014). Only LA Collapsed could efficiently scale to this data size (49 taxa, 250 samples, 4 covariates). To allow us to compare to alternative implementations we randomly subset the data to contain 83 samples. On this subset HMC Uncollapsed and VB Collapsed repeatedly failed to run due to numerical instability. In addition, LA Collapsed produced posterior estimates nearly identical to HMC Collapsed but more than 1000 times faster.

Using the four model implementations introduced in Section 5, a Bayesian multinomial logistic normal linear model was fit to investigate the relationship between bacterial composition and CD. For both the full data-set and the subset, our regression model was defined for the $j$-th sample by the covariate vector

$$x_j={\left[1,x_{j\left(\text{CD}\right)},x_{j(\text{Inflamed)}},x_{j(\text{Age)}}\right]}^T$$

where $x_{j\left(\text{CD}\right)}$ is a binary variable denoting whether the $j$-th sample was from a patient with CD or a healthy control, $x_{j\text{(Inflamed)}}$ a binary variable denoting inflammation at time of sample collection, $x_{j\text{(Age)}}$ denoting age of the subject, and the preceding 1 represents a constant intercept. To evaluate the impact of using small values for the degree-of-freedom parameter $\nu$ in model priors, we set $\nu =D+3$. A full description of our prior assumptions is given in Appendix I and results of posterior predictive checks are shown in Figure S6.

Even though all four implementations were initialized identically, both the HMC Uncollapsed and VB Collapsed implementations repeatedly resulted in errors due to numerical instability. Thus only LA Collapsed and the HMC Collapsed implementations could fit this model for even the subset dataset. Whereas the HMC Collapsed model took approximately 30 minutes, LA Collapsed took only 3 seconds. Thus LA Collapsed is over 1000 times faster than HMC Collapsed on real data. Additionally, posterior estimates of $\Lambda$ produced by both the HMC Collapsed and LA Collapsed implementations are nearly identical (Figure S7). These results demonstrate that in real data scenarios LA Collapsed can provide efficient and accurate posterior inference.

By modeling the full dataset we found the centered log-ratio (CLR) coordinates corresponding to 12 genera to be associated with CD status (95% credible interval not covering 0; Figure 2). These results are in general agreement with prior analyses (Gevers et al., 2014). As in prior analyses, we find a substantial increase in the abundance of proteobacteria in CD versus healthy controls. Similarly, we find that the families Pasteurellaceae and Enterobacteriacaeae, Gemellaceae, and Fusobacteriaceae are highly enriched and that the class Clostridia are depleted in CD. Notably, Fusobacteria has been independently suggested as a marker of IBD (Strauss et al., 2011; Kostic et al., 2012). These findings serve to validate our results and build confidence in our methods.

Figure 2: Posterior mean and credible intervals for $\Lambda$ of fido::pibble (LA Collapsed) applied to Crohn’s disease data.

Only the 12 families found to be associated with Crohn’s Disease (CD) (i.e., Posterior 95% credible region not covering zero) are shown. Taxa are denoted as clr_[class]_[family]. $\Lambda$ is represented in centered log-ratio (CLR) coordinates rather than additive log-ratio (ALR) so that each coordinate could be identified with a different bacterial taxa.

In contrast, our results differ from prior analyses of this data in certain respects. We find that the family Peptostreptococcaceae is likely decreased in CD versus healthy controls and we find no association for Veillonellaceae. Three factors support our results. First, our analysis accounts for count variation and compositional constraints whereas prior analyses have not. It is well known that the handing of count variation and compositional constraints can have substantial impact on conclusions in the analysis of sequence count data (McMurdie and Holmes, 2014; Silverman et al., 2020; Gloor et al., 2017). Second, Peptostreptococcaceae has been found to be decreased in CD based on the analysis of independent data (Imhann et al., 2018). Third, in visualizing the count data for Peptostreptococcaceae and Veillonellaceae (Figure S8) we find no visual difference in Veillonellaceae but a notable difference in Peptostreptococcaceae. Therefore, we conclude that our approach has revealed novel associations in this data and excluded potentially spurious conclusions.

7. Inferring Microbial Trajectories in an Artificial Gut Model

Artificial gut models provide a powerful in vitro approach to studying microbial communities. To demonstrate the generality of our inference methods for Marginally LTP models, we reanalyzed a previously published high-resolution longitudinal study of 4 artificial gut models using a GMGP model (Silverman et al., 2018a). Each of the 4 artificial gut models represent a closed system that were maintained in nearly identical conditions and inoculated with an identical fecal slurry. Following Silverman et al. (2018a), We therefore chose to model each of the four vessels $(r\in 1,\dots ,4)$ as independent but with a shared covariance structure using the following GMGP model:

$$Y_{\cdot \mathit{\text{tr}}}~\text{Multinomial}\left(n_{\mathit{\text{tr}}},{\pi }_{\cdot \mathit{\text{tr}}}\right)$$

$${\pi }_{\cdot \mathit{\text{tr}}}={\text{ALR}}_D^{-1}\left({\eta }_{\cdot \mathit{\text{tr}}}\right)$$

$${\eta }_{\cdot \mathit{\text{tr}}}~N\left(\Lambda \right({X.}_{\mathit{\text{tr}}}),\Sigma (Z\left)\right)$$

$$\Lambda ~\text{GP}\left(\Theta ,\Sigma ,{\Gamma }_{\text{vessel}}\circ {\Gamma }_{\mathit{\text{time}}}\right)$$

$$\Sigma ~\text{IWP}(\Xi ,\nu )$$

where $\Xi$ is a kernel based on sequence similarity between bacterial taxa, ${\Gamma }_{\text{time}}$ is a squared exponential kernel based on the time between samples, ${\Gamma }_{\text{vessel}}$ is a block identity kernel, and $\circ$ denote the element-wise multiplication of kernel functions. To evaluate the impact of using small values for the degree-of-freedom parameter $\nu$ in model priors, we set $\nu =D+2$. Details on these kernels as well as the matrix functions $\Theta$ are described further in Appendix J. The above GMGP model was inferred using the function basset from the R package fido. While there are differences between the generalized dynamic linear model used in Silverman et al. (2018a) and the above GMGP model, it is notable that the model used in Silverman et al. (2018a) took on the order of 5 hours while the GMGP model above, using CU sampler with Laplace approximation, ran in just 4 seconds.

Our results are in general agreement with those of Silverman et al. (2018a). Notably, we found a distinct decrease in the relative amount of the family Bacteroidaceae immediately after the introduction of a $B$. ovatus probiotic at hour 60. Still, our analyses revealed a number of features unappreciated in prior analyses. Most notably, our GMGP analyses suggests that the degree to which the community was undergoing sub-daily oscillations was far greater than was appreciated in Silverman et al. (2018a). Silverman et al. (2018a) noticed that the relative amount of Enterobacteriaceae displayed distinct, unsyncronized, sub-daily oscillations in all four artificial gut vessels. We too found evidence of unsyncronized sub-daily oscillations in all four vessels. However, we also found such oscillatory dynamics in numerous other dimensions including the CLR coordinates corresponding to the Lachnospiraceae, Desulfovibrionaceae, and Synergistaceae. We suspect that the flexibility provided by the non-linear GMGP model allowed these oscillatory patterns to be more easily revealed than the random walk dynamics originally modeled in Silverman et al. (2018a).

8. Conclusion

In this work we have developed efficient inference for the analysis of a large class of multinomial logistic-normal models through the use of a shared marginal representation. We demonstrated that, in comparison to HMC, the CU sampler with a marginal Laplace approximation improved sampler efficiency by up to 5 orders of magnitude while preserving accuracy of point estimation and uncertainty quantification. Yet, the performance of our Laplace approximation under observation distributions beyond the log-ratio parameterized multinomial is more uncertain. We hypothesize that our results could generalize to other exponential family distributions parameterized by natural parameters since such distributions are globally log-concave. Yet, we expect that there are other observation distributions where a Laplace approximation to the LTP form may be sub-optimal. Rather than resorting to using MCMC to infer the collapsed model form, we suggest that methods of particle refinement of the initial Laplace approximation (e.g., parallel MCMC steps for each sample from the LTP form, or sequential importance resampling) may be more efficient. We believe such extensions are prime areas for future work.

Here we have compared the CU sampler with marginal Laplace approximation against HMC and VB for inference of MLN models, yet many other comparisons are possible. In particular, both the Integrated Nested Laplace Approximation (INLA) (Rue et al., 2009) and Pólya-gamma data-augmentation (Polson et al., 2013) are popular approaches for inferring Bayesian logistic models. Like INLA, our approach uses Laplace approximations to posterior marginals; yet INLA’s requires that the number of hyper-parameters is small (e.g., ≤ 6) which proves limiting in inferring MLN models with potentially dense co variation between multinomial categories as we address here. In contrast to INLA, Pólya-gamma data augmentation uses a Gibbs sampling algorithm with augmented Pólya-gamma random variables. Yet numerous authors have found that Pólya-gamma data augmentation is too slow for scalable inference of MLN models due to two key limitations (Linderman et al., 2015; Glynn et al., 2019; Zhang and Zhou, 2017). First, MLN models do not permit block updates to Pólya-gamma random variables and as a result, the number of Gibbs steps required for each posterior sample scales linearly with the number of multinomial categories (Polson et al., 2013; Linderman et al., 2015; Zhang and Zhou, 2017). Second, when the number of multinomial categories is large, sampling Pólya-gamma random variables can become rate limiting (Glynn et al., 2019). Rather than INLA or Pólya-gamma data augmentation, we believe that the most fruitful comparisons involve alternative approximations for sampling the collapsed representation of Marginally LTP models. Notably, for inference of hierarchical Bayesian Gaussian processes, multiple authors have found expectation propagation to be more accurate, albeit an order of magnitude slower than, Laplace approximation (Jylänki et al., 2011; Nickisch and Rasmussen, 2008; Kuss et al., 2005). Overall, further comparisons will both help to clarify the use cases for the CU sampler with marginal Laplace approximation and point to potential future improvements.

One limitation of this work is that our derived error bound required the assumption that $\nu \to \infty$. This assumption was required so that the Matrix-t distribution was globally log-convex – a requirement of the theory introduced in (Ogden, 2018). In practice however, we expect practitioners to use finite values of $\nu$ and in these cases our error bound serves only as a tool for building intuition regarding the error rate of our Laplace approximation. Despite this limitation, our analyses of both simulated and real data suggest that the Laplace approximation provides accurate inference even when $\nu$ is small. Still, we expect some degradation of the accuracy of the Laplace approximation for smaller values of $\nu$ compared to larger values of $\nu$.

Figure 3: Posterior mean and credible intervals for $\Lambda$ of fido::basset applied to artificial gut data.

Following Silverman et al. (2018a) we analyze the data from 4 independent artificial guts at the bacterial family level. The posterior mean as well as 50% and 95% credible regions for $\Lambda$ are depicted. The posterior is depicted with respect to centered log-ratio (CLR) coordinates so that each coordinate could be identified with a different bacterial family.