Integrating sample similarities into latent class analysis: a tree-structured shrinkage approach

Abstract

This paper is concerned with using multivariate binary observations to estimate the probabilities of unobserved classes with scientific meanings. We focus on the setting where additional information about sample similarities is available and represented by a rooted weighted tree. Every leaf in the given tree contains multiple samples. Shorter distances over the tree between the leaves indicate a priori higher similarity in class probability vectors. We propose a novel data integrative extension to classical latent class models with tree-structured shrinkage. The proposed approach enables (1) borrowing of information across leaves, (2) estimating data-driven leaf groups with distinct vectors of class probabilities, and (3) individual-level probabilistic class assignment given the observed multivariate binary measurements. We derive and implement a scalable posterior inference algorithm in a variational Bayes framework. Extensive simulations show more accurate estimation of class probabilities than alternatives that suboptimally use the additional sample similarity information. A zoonotic infectious disease application is used to illustrate the proposed approach. The paper concludes by a brief discussion on model limitations and extensions.

1 |. INTRODUCTION

1.1 |. Motivating application

The fields of infectious disease epidemiology and microbial ecology need better tools for tracing the transmission of microbes between humans and other vertebrate animals (i.e., zoonotic transmissions), especially for colonizing opportunistic pathogens (COPs). Unlike frank zoonotic pathogens (e.g., Salmonella, SARS-CoV-2), the epidemiology of COPs, such as Escherichia coli (E. coli), Staphylococcus aureus (S. aureus), and Enterococcus spp., can be particularly cryptic due to their ability to asymptomatically colonize the human body for indefinite periods prior to initiating an infection, transmitting to another person, or being shed without a negative outcome (e.g., Price et al., 2017). Some COPs can colonize many different vertebrate hosts and cross-species transmissions can go unrecognized. Estimating the probability of zoonotic origin for a population of isolates and for each isolate would provide important insights into the natural history of infections and inform more effective intervention strategies, such as eliminating high-risk clones from livestock via vaccination.

Scientists often have two complementary sources of information: (i) a phylogenetic tree constructed based on a few single nucleotide polymorphisms (SNPs) in the core genome shared by all isolates, where the leaves represent distinct core-genome multi-locus sequence types (STs, Maiden et al., 1998); the tree is useful for identifying a recent common ancestor for isolates that comprise an infectious disease outbreak; (ii) presence or absence of multiple mobile genetic elements (MGE) that provide selective advantages in particular hosts and may be lost and gained as COPs transmit among hosts (e.g., Lindsay and Holden, 2004).

Recent research on two COP species, E. coli and S. aureus, has demonstrated the utility of complementing core-genome phylogenetic trees with host-associated MGEs to resolve host origins (e.g., Liu et al., 2018). However, in both cases only a single host-associated MGE was used. Analyses were largely limited to visual inspection of how each element fell on the scaffold of the evolutionary tree. For this approach to reach its full potential, we would need a statistical model that can (1) integrate phylogenetic information with the presence and absence of multiple host-associated MGEs, and (2) estimate the probability with which the isolates were derived from a particular host in each ST-specific population and for each individual isolate.

1.2 |. Integrating sample similarities into latent class analysis

Based on multivariate binary data (e.g., presence or absence of multiple MGEs), we use latent class models (LCMs; e.g., Lazarsfeld, 1950; Goodman, 1974) to achieve the scientific goal of estimating the probabilities of unobserved host origins and perform individual-level probabilistic assignment of host origin. LCMs are examples of latent variable models that assume the observed dependence among multivariate discrete responses is induced by variation among unobserved or “latent” variables. It is well known that any multivariate discrete data distribution can be approximated arbitrarily closely by an LCM with a sufficiently large number of classes (Dunson and Xing, 2009, corollary 1). The most commonly used LCMs assume the class membership indicators for the observations are drawn from a population with the same vector of class probabilities.

Trees or hierarchies are useful and intuitive for representing and reasoning about similarity or relation among objects in many real-world domains. We assume known entities at the leaves. In our context, each leaf may contain multiple observations or samples, each associated with the multivariate binary responses that are then combined to form the rows of a binary data matrix $\mathbf{Y}$. In the motivating application, the latent class indicates the unobserved host origin (human or nonhuman) to be inferred by the presence or absence of multiple MGEs. The additional sample similarity information is represented by a maximum likelihood phylogenetic tree (e.g., Scornavacca et al., 2020). The leaves represent distinct contemporary core-genome E. coli STs.

To integrate tree-encoded sample similarity information into a latent class analysis, ad hoc groupings of the leaves may be adopted. From the finest to the coarsest leaf grouping, one may (1) analyze data from distinct lineages one at a time, (2) manually form groups of at least one leaf node and fit separate LCMs, or (3) fit all the data by a single LCM. However, all these methods pose significant statistical challenges. First, separate latent class analyses may have low accuracy in estimating latent class probabilities and other model parameters for rare lineages. Second, observations of similar lineages may have similar propensities in host jump resulting in similar host origin class probabilities. Modeling these similarities could lead to gain in statistical efficiency. Third, approaches based on coarse ad hoc groupings may obscure the study of the variation in the latent class probabilities across different parts of the tree. Finally, based on a single LCM or other approaches that use ad hoc leaf groupings, individual-specific posterior class probabilities can be averaged within in each leaf to produce a local estimate of the vector of class probabilities. However, the ad hoc post-processing cannot fully address the issue of assessment of posterior uncertainty nor data-driven grouping of leaves, necessitating development of an integrative probabilistic modeling framework for uncertainty quantification and adaptive formation of leaf groups.

In this paper, we focus on integrating the tree-encoded sample similarity information into latent class analysis. We assume the tree information is given and not computed from the multivariate binary measurements. Observations in nearby leaves are assumed to have a priori similar propensities of being members of a particular class as characterized by the latent class probabilities. For example, higher similarities are indicated by shorter pairwise distances between observations. More generally, classical covariate-dependent latent class models (e.g., Formann, 1992; Bandeen-Roche et al., 1997) let the latent class probabilities vary explicitly as functions of observed covariates so that observations with more similar covariate values are assumed to have more similar latent class probabilities. Fully probabilistic tree-integrative methods have appeared in machine learning literature (e.g., Roy et al., 2006; Ghahramani et al., 2010; Ranganath et al., 2015) or in statistics for modeling hierarchical topic annotations (e.g., Airoldi and Bischof, 2016) or hierarchical outcome annotations based on given trees (e.g., Thomas et al., 2019). In epidemiology, Avila et al. (2014) proposed a two-stage approach to link patient clusters estimated from the tree and by the LCM results, which, however, remains ad hoc. The existing literature does not address probabilistic tree-integrative latent class analysis or adaptive formation of leaf groups for dimension reduction.

1.3 |. Primary contributions

In this paper, we propose an unsupervised, tree-integrative LCM framework to (1) discover groups of leaves where multivariate binary measurements in distinct leaf groups have distinct vectors of latent class probabilities; and observations nested in any leaf group may belong to a pre-specified number of latent classes; (2) accurately estimate the latent class probabilities for each discovered leaf group and assign probabilities of an individual sample belonging to the latent classes; (3) leverage the relationship among the observations as encoded by the tree to boost the accuracy of the estimation of latent class probabilities. Without pre-specifying the leaf groups, the automatic data-driven approach enjoys robustness by avoiding potential mis-specification of the grouping structure. On the other hand, the discovered data-driven leaf groups dramatically reduce the dimension of leaves into fewer subgroups of leaves hence improving interpretation. In addition, the proposed approach shows better accuracy in estimating the latent class probabilities in terms of root mean squared errors, indicating the advantage of the shrinkage. On posterior computation, we derive a scalable inference algorithm based on variational inference (VI).

The rest of the paper is organized as follows. Section 2.2 defines tree-related terminologies and formulates LCMs. Section 3 proposes the prior for tree-structured shrinkage in LCMs. Section 4 derives a variational Bayes algorithm for inference. Section 5 compares the performances of the proposed and alternative approaches via simulations. Section 6 illustrates the approach by analyzing an E. coli data set. The paper concludes with a brief discussion.

2 |. MODEL

We first introduce necessary terminologies and notations to describe a rooted weighted tree. LCMs are then formulated for data on the leaves of the tree.

2.1 |. Rooted weighted trees

A rooted tree is a graph $𝒯=(𝒱,E)$ with node set $𝒱$ and edge set $E$ where there is a root $u_0$ and each node has at most one parent node. Let $p=\left|𝒱\right|$ represent the total number of leaf and nonleaf nodes. Let ${𝒱}_L\subset 𝒱$ be the set of leaves, and $p_L=\left|{𝒱}_L\right|<p$. We typically use $u$ to denote any node $(u\in 𝒱)$ and $v$ to denote any leaf $\left(v\in {𝒱}_L\right)$. Each edge in a rooted tree defines a clade: the group of leaves below it. Splitting the tree at an edge creates a partition of the leaves into two groups. For any node $u\in 𝒱$, the following notations apply: $c\left(u\right)$ is the set of offspring of $u,pa\left(u\right)$ is the parent of $u,d\left(u\right)$ is the set of descendants of $u$ including $u$, and $a\left(u\right)$ is the set of ancestors of $u$ including $u$. In Figure 3(a), if $u=2$, then $c\left(u\right)=\{\mathrm{6,7},8\},pa\left(u\right)=\left\{1\right\}$, $d\left(u\right)=\{\mathrm{2,6},\mathrm{7,8}\}$, and $a\left(u\right)=\left\{\mathrm{1,2}\right\}$. The phylogenetic tree in our motivating application is a nested hierarchy of 133 STs for $N=2663$ observations, where the $p_L=\left|{𝒱}_L\right|=133$ leaves represent distinct STs and the $p-p_L=132$ internal (nonleaf) nodes represent ancestral E. coli strains leading up to the observed leaf descendants.

FIGURE 3.

Simulation studies show the proposed model produces grouped estimates ${\hat{\pi }}_v^{\text{dgrp }}$ with similar or smaller RMSEs compared to alternatives (see Section 5). This figure appears in color in the electronic version of this article, and any mention of color refers to that version

Edge-weighted graphs appear as a model for numerous problems where nodes are linked with edges of different weights. In particular, the edges in $𝒯$ are attached with weights where $w:E\to {\mathbb{R}}^{+}$ is a weight function. Let ${𝒯}_w=(𝒯,w)$ be a rooted weighted tree. A path in a graph is a sequence of edges that joins a sequence of distinct vertices. For a path $P$ in the tree connecting two nodes, $w\left(P\right)$ is defined as the sum of all the edge weights along the path, often referred to as the “length” of $P$. The distance between two vertices $u$ and $u^{\prime }$, denoted by ${\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}}_{{𝒯}_w}\left(u,u^{\prime }\right)$ is the length of a shortest (with minimum length) $\left(u,u^{\prime }\right)$-path. $\mathit{\text{dist}}{ }_{{𝒯}_w}$ is a distance: it is symmetric and satisfies the triangle inequality. In our motivating application, the edge length represents the number of nucleotide substitutions per position; the distance between two nodes provides a measurement of the similarity or divergence between any two core-genome sequences of the input set. In this paper, we use $w_u$ to represent the edge length between a node $u$ and its parent node $pa\left(u\right).w_u$ is fully determined by ${𝒯}_w$. For the root $u_0$, there are no parents, that is, $pa\left(u_0\right)=\varnothing$; we set $w_{u_0}=1$.

2.2 |. Latent class models for data on the leaves

Although LCMs can deal with multiple categorical responses in general, for simpler presentations in this paper, we focus on presenting the model and algorithm using multivariate binary responses and their application to the motivating data.

Notations

Let ${\mathit{Y}}_i^{\left(v\right)}={\left(Y_{i1}^{\left(v\right)},\dots ,Y_{iJ}^{\left(v\right)}\right)}^{\top }\in \{\mathrm{0,1}{\}}^J$ be the vector of binary responses for observation $i\in \left[n_v\right]$ that is nested within leaf node $v\in {𝒱}_L$, where $n_v$ is the number of observations in leaf $v$. Throughout this paper, let $\left[Q\right]≔\{1,\dots ,Q\}$ denote the set of positive integers smaller than or equal to $Q$, where $Q$ is a positive integer. Let ${\mathbf{Y}}^{\left(v\right)}={\left({\mathit{Y}}_1^{\left(v\right)},\dots ,{\mathit{Y}}_{n_v}^{\left(v\right)}\right)}^{\top }$ be the data from observations in leaf $v$. Let $\mathbf{Y}={\left({\left({\mathbf{Y}}^{\left(1\right)}\right)}^{\top },\dots ,{\left({\mathbf{Y}}^{\left(p_L\right)}\right)}^{\top }\right)}^{\top }$ represent the binary data matrix with $N={\sum }_{v\in {𝒱}_L} n_v$ rows and $J$ columns. Let $ℒ={\left(v_1,\dots ,v_N\right)}^{\top }$ be the “sample-to-leaf indicators” that map every row of data $\mathbf{Y}$ into a leaf in ${𝒯}_w$. Sample similarities are then characterized by between-leaf distances in ${𝒯}_w$. In this paper, we assume $ℒ$ and ${𝒯}_w$ are given and focus on incorporating $\left(ℒ,{𝒯}_w\right)$ into a statistical model for $\mathbf{Y}$.

LCM for Data on the Leaves

The LCM is specified in two steps:

$$\text{class indicator :}{I}_i^{\left(v\right)}\mid {\pi }_v~{\text{ Categorical }}_K\left\{{\pi }_v\right\},{\pi }_v\in S_{K-1},$$ (1)

$$\begin{gathered} \text{ data :}{Y}_{ij}^{\left(v\right)}\mid I_i^{\left(v\right)}~\text{ Bernoulli }\left\{{\theta }_{j,I_i^{\left(v\right)}}\right\}, \\ \text{independently for feature }j\in \left[J\right], \end{gathered}$$ (2)

and independently for observation $i\in \left[n_v\right]$ and leaf node $v\in {𝒱}_L$. Here $K$ is a pre-specified number of latent classes in the context of the application, that is, $K=2$ for unobserved human and nonhuman hosts; see Section 4.1 for a simple strategy in applications where data-driven $K$ is desired. In addition, $\mathbf{I}=\left\{I_i^{\left(v\right)}:i\in \left[n_v\right];v\in \right.\left.{𝒱}_L\right\}$ represent the latent class indicators and $Z_{ik}^{\left(v\right)}=1\left\{I_i^{\left(v\right)}=\right. k\},k\in [K]$, where $1\left\{A\right\}$ is an indicator function that equals 1 if statement $A$ is true and 0 otherwise; Let $\mathbf{Z}=\left\{Z_{ik}^{\left(v\right)}\right\}$. We have assumed observations in different leaves have potentially different vectors of class probabilities ${\mathit{\pi }}_v={\left({\pi }_{v1},\dots ,{\pi }_{vK}\right)}^{\top }\in S_{K-1},v\in {𝒱}_L$, where $S_{K-1}=\{\mathit{r}\in \left.[\mathrm{0,1}{]}^K:{\sum }_{k=1}^K r_k=1\right\}$ is the probability simplex. ${\theta }_{jk}\in \left[\mathrm{0,1}\right]$ is the positive response probability for feature $j\in \left[J\right]$ in class $k\in \left[K\right]$. In our motivating application, the MGEs adapt to the unobserved type of host origin (i.e., latent class) that can be characterized by class-specific response probability profiles ${\theta }_{\cdot k}={\left({\theta }_{1k},\dots ,{\theta }_{Jk}\right)}^{\top },k\in \left[K\right]$; let $\mathbf{\Theta }={\left({\theta }_{\cdot 1},\dots ,{\theta }_{\cdot }\right)}^{\top }$. Because the latent class indicators $I_i^{\left(v\right)}$’s are assumed to be unobserved, the observed data likelihood for $N$ observations is ${\prod }_{v\in {𝒱}_L} {\prod }_{i=1}^{n_v} {\sum }_{k=1}^K {\pi }_{vk}\mathbb{P}\left({\mathit{Y}}_i^{\left(v\right)}\mid I_i^{\left(v\right)}=\right. \left.k,{\theta }_{\cdot k}\right)$.

Throughout this paper, we assume that we wish to classify individuals into $K$ classes with the same set of $\left({\theta }_{\cdot 1},\dots ,{\theta }_{\cdot K}\right)$ so classes have coherent interpretation. However, we do not assume that observations are drawn from a population with a single vector of latent class probabilities. Figure 1 provides a schematic of the data generating mechanism given ${\mathit{\pi }}_v$ for three leaves.

FIGURE 1.

Schematic representation of a hypothetical rooted weighted tree with three leaves and data generated based on the proposed model with $K=3$ latent classes, $n_{v_1}=2,n_{v_2}=4$ and $n_{v_3}=2,J=8$. This figure appears in color in the electronic version of this article, and any mention of color refers to that version

3 |. PRIOR DISTRIBUTION

We first specify a prior distribution for $\left\{{\mathit{\pi }}_v:v\in {𝒱}_L\right\}$. Because leaf-specific sample sizes may vary, we propose a tree-structured prior to borrow information across nearby leaves. The prior encourages collapsing certain parts of the tree so that observations within a collapsed leaf group share the same vector of latent class probabilities. In particular, we extend Thomas et al. (2019) to deal with rooted weighted trees in an LCM setting. The prior specification is completed by priors for the class-specific response probabilities $\mathbf{\Theta }$.

Tree-structured prior for latent class probabilities ${\mathit{\pi }}_v$

We specify a spike-and-slab Gaussian diffusion process prior along a rooted weighted tree based on a logistic stick-breaking parameterization of ${\mathit{\pi }}_v$. We first reparameterize ${\mathit{\pi }}_v$ with a stick-breaking representation: ${\pi }_{vk}=V_{vk}{\prod }_{s<k} \left(1-V_{vs}\right)$, for $k\in \left[K\right]$, where $0\le V_{vk}\le 1$, for $k\in [K-1]$ and $V_{vK}=1$.

We further logit-transform $V_{vk},k\in [K-1]$, to facilitate the specification of a Gaussian diffusion process prior without range constraints. In particular, let ${\eta }_{vk}={\sigma }^{-1}\left(V_{vk}\right),k\in [K-1],v\in {𝒱}_L,$ where $\sigma \left(x\right)=1/\{1+\mathrm{e}\mathrm{x}\mathrm{p}(-x\left)\right\}$ is the sigmoid function. The logistic stick-breaking parameterization is completed by

$${\pi }_{vk}={\left\{\sigma \left({\eta }_{vk}\right)\right\}}^{1\{k<K\}}\prod _{s<k}\sigma \left(-{\eta }_{vs}\right),k\in \left[K\right],$$ (3)

which affords simple and accurate posterior inference via variational Bayes (see Section 4).

For a leaf $v\in {𝒱}_L$, let

$${\eta }_{vk}=\sum _{u\in a\left(v\right)}{\xi }_{uk},k\in \left[K-1\right].$$ (4)

Here ${\eta }_{vk}$ is defined for leaves only and ${\xi }_{uk}$ is defined for all the nodes. Suppose $v$ and $v^{\prime }$ are leaves and siblings in the tree such that $pa\left(v\right)=pa\left(v^{\prime }\right)$, setting ${\xi }_{vk}={\xi }_{v^{\prime }k}=0$ implies ${\eta }_{vk}={\eta }_{v^{\prime }k}$ for $k\in [K-1]$, and hence ${\mathit{\pi }}_v={\mathit{\pi }}_{v^{\prime }}$. More generally, a sufficient condition for $M$ leaves ${\eta }_{vk},v\in \left\{v_1,\dots ,v_M\right\}$ to fuse is to set ${\xi }_{uk}=0$ for any $u$ that is an ancestor of any of $\left\{v_1,\dots ,v_M\right\}$ but not common ancestors for all $v_m$. That is, to achieve grouping of observations that share the same vector of latent class probabilities, in our model, it is equivalent to parameter fusing. In the following, we specify a prior on the ${\xi }_{uk}$ that a priori encourages sparsity, so that closely related observations are likely grouped to have the same vector of class probabilities. The fewer distinct ancestors two nodes have, the more likely the parameters ${\eta }_{vk}$ are fused, because the prior would encourage fewer auxiliary variables ${\xi }_{uk}$ to be set to zero. In particular, we specify

$${\xi }_{uk}=s_u{\alpha }_{uk},\forall u\in 𝒱,$$ (5)

$$\begin{gathered} {\alpha }_{uk}~N\left(0,{\tau }_{1k{\ell }_u}{w}_u\right), \\ \text{independently for }k\in \left[K-1\right],\forall u\in 𝒱, \end{gathered}$$ (6)

$$\begin{gathered} s_{u_0}=1,\text{ and }{s}_u~\text{Bernoulli}\left({\rho }_{{\ell }_u}\right), \\ \text{independently for }u\in 𝒱\setminus u_0, \end{gathered}$$ (7)

$${\rho }_{\ell }~\text{Beta}\left(a_{\ell },b_{\ell }\right),\text{ independently for }\ell \in \left[L\right],$$ (8)

where $N(m,s)$ represents a Gaussian density function with mean $m$ and variance $s.{\tau }_{1k\ell }$ is the unit-length variance and controls the degree of diffusion along the tree that may differ by dimension $k$ and node level ${\ell }_u$ where ${\ell }_u\in \left[L\right]$ represents the “level” or “hyperparameter set indicator” for node $u$. For example, in simulations and data analysis, we will assume that the root for the diffusion process has a prior unit-length variance distinct from other non-root nodes. For the root $u_0$ with $s_{u_0}=1,{\alpha }_{u_{0}k}$ initializes the diffusion of ${\eta }_{uk}$.

Leaf groups are formed by selecting a subset of nodes in $𝒱:𝒰=\left\{u\in 𝒱:s_u=1\right\}$. Except a probability-zero set, two leaves $v$ and $v^{\prime }$ are grouped, or “fused,” if and only if $a\left(v\right)\cap 𝒰=a\left(v^{\prime }\right)\cap 𝒰$. In particular, the null set is $\left\{{\eta }_{vk}={\eta }_{v^{\prime }k},k\in [K-1]\right\}\cap \left\{{\sum }_{u\in \left[a\right(v)\cap 𝒱]\setminus \left[a\left(v^{\prime }\right)\cap {𝒱}^{\prime }\right]} {\alpha }_{uk}=\right.\left.{\sum }_{u\in \left[a\left(v^{\prime }\right)\cap 𝒞\right]\setminus \left[a\right(v)\cap 𝒱]} {\alpha }_{uk}\right\}$ where the latter has probability zero. In Section 4.1, we will estimate $𝒰$ using the posterior median model.

Remark 1. Equations (4)–(8) define a Gaussian diffusion process initiated at ${\alpha }_{u_{0}k}$:

$$\begin{gathered} {\eta }_{uk}\mid \text{ others }~N\left(\sum _{u^{\prime }\in a\left(u\right)}{\xi }_{u^{\prime }k},s_u{\tau }_{1k{\ell }_u}{w}_u\right), \\ \text{independentlyfork }k\in \left[K-1\right], \end{gathered}$$ (9)

for any nonroot node $u\ne u_0$; also see the seminal formulation by Felsenstein (1985). To aid the understanding of this Gaussian diffusion prior, it is helpful to consider a special case of $s_u=1$ and ${\ell }_u=1,\forall u\in 𝒱$. For two leaves $v,v^{\prime }\in {𝒱}_L$, the prior correlation between ${\eta }_{vk}$ and ${\eta }_{v^{\prime }k}$ is

$$\text{Corr}\left({\eta }_{vk},{\eta }_{v^{\prime }k}\right)=\frac{{\sum }_{u\in a\left(v\right)\cap a\left(v^{\prime }\right)}{w}_u}{{\left\{{\text{dist}}_{{𝒯}_w}\left(u_0,v\right){\text{dist}}_{{𝒯}_w}\left(u_0,v^{\prime }\right)\right\}}^{1/2}}.$$ (10)

When $v$ and $v^{\prime }$ have the same number of ancestors $\left(\left|a\right(v\left)\right|=\left|a\left(v^{\prime }\right)\right|\right)$ and all edges have identical weight $w_u=c,\forall u$, the prior correlation is the fraction of common ancestors. Note that ${\mathit{\eta }}_v$ fully determines ${\mathit{\pi }}_v$ in (3) and induces correlations among $\left\{{\mathit{\pi }}_v,v\in {𝒱}_L\right\}$.

Remark 2. One reviewer raised an important question on the choice of encouraging prior correlation among $\left\{{\mathit{\pi }}_v\right\}$ rather than among the latent class indicators $\left\{I_i^{\left(v\right)}\right\}$. In the present prior distribution, by integrating out $\left\{{\mathit{\pi }}_v\right\}$, we have induced prior marginal correlation among $\left\{I_i^{\left(v\right)}\right\}$ for observation in nearby leaves. Additional prior correlation among the $\left\{I_i^{\left(v\right)}\right\}$ can be introduced via an additional layer of prior over the $\left\{I_i^{\left(v\right)}\right\}$ conditional on $\left\{{\mathit{\pi }}_v\right\}$, for example, through clustered samples. The absence of such clustered sampling structure in the motivating application points us toward the former simpler strategy.

Priors for class-specific response probabilities

Let ${\gamma }_{jk}=\mathrm{l}\mathrm{o}\mathrm{g}\left\{{\theta }_{jk}/\left(1-{\theta }_{jk}\right)\right\}$. We specify

$$\begin{gathered} {\gamma }_{jk}~N\left(0,{\tau }_{2jk}\right), \\ \text{independently for feature }j\in \left[J\right]\text{ and class }k\in \left[K\right]. \end{gathered}$$ (11)

Joint distribution

Let $\mathit{\beta }=(\mathbf{Z},\mathit{s},\mathit{\gamma },\mathit{\alpha },\mathit{\varrho })$ collect all the unknown parameters where

$$\begin{gathered} s=\left\{s_u:u\in 𝒱\right\}, \\ \gamma =\left\{{\gamma }_{jk},j\in \left[J\right];k\in \left[K\right]\right\}, \\ \alpha =\left\{{\alpha }_{uk}:u\in 𝒱,k\in \left[K-1\right]\right\}, \\ \mathit{\varrho }={\left({\rho }_1,\dots ,{\rho }_L\right)}^{\top },\mathit{a}={\left(a_1,\dots ,a_L\right)}^{\top }, \end{gathered}$$

and $\mathit{b}={\left(b_1,\dots ,b_L\right)}^{\top }$. Hereafter we use $\mathrm{p}\mathrm{r}(A\mid B)$ to denote a probability density or mass function of quantities in $A$ with parameters $B$; when $B$ represents hyperparameters or given information in this paper, we simply use $\mathrm{p}\mathrm{r}\left(A\right)$, for example, we will use $\mathrm{p}\mathrm{r}(\mathbf{Y},\mathit{\beta })$ to represent $\mathrm{p}\mathrm{r}\left(\mathbf{Y},\mathit{\beta }\mid {\mathit{\tau }}_1,{\mathit{\tau }}_2,\mathit{a},\mathit{b},{𝒯}_w,ℒ\right)$. The joint distribution of data and unknown quantities can thus be written as

$$\text{pr}\left(Y\mid \beta \right)\text{pr}\left(\beta \right)=\prod _{v\in {𝒱}_L}\prod _{i=1}^{n_v}\prod _{k=1}^K[{\left\{\sigma \left({\eta }_{vk}\right)\right\}}^{1\{k<K\}}\prod _{s<k}\times \left\{1-\sigma \left({\eta }_{vs}\right)\right\}\prod _{j=1}^J\sigma \left(X_{ij}^{\left(v\right)}{\gamma }_{jk}\right)]Z_{ik}^{\left(v\right)}$$ (12)

$$\begin{gathered} \times \prod _{u\in 𝒱}\prod _{k=1}^{K-1}\left(\frac{1}{\sqrt{2\pi {\tau }_{1k{\ell }_u}{w}_u}}\text{exp}\left\{-\frac{1}{2{\tau }_{1k{\ell }_u}{w}_u}{\alpha }_{uk}^2\right\}\right) \\ \times \prod _{j=1}^J\prod _{k=1}^K\left(\frac{1}{\sqrt{2\pi {\tau }_{2jk}}}\text{exp}\left\{-\frac{1}{2{\tau }_{2jk}}{\gamma }_{jk}^2\right\}\right)\times \prod _{u\in 𝒱}{\rho }_{{\ell }_u}^{s_u} \\ \times {\left(1-{\rho }_{{\ell }_u}\right)}^{1-s_u}\cdot \prod _{\ell =1}^L\frac{1}{\text{Beta}\left(a_{\ell },b_{\ell }\right)}{\rho }_{\ell }^{a_{\ell }-1}{\left(1-{\rho }_{\ell }\right)}^{b_{\ell }-1}, \end{gathered}$$ (13)

where $X_{ij}^{\left(v\right)}=2Y_{ij}^{\left(v\right)}-1$. Tree information ${𝒯}_w$ enters the joint distribution in the definition of ${\mathit{\eta }}_v$ (Equations 4); sample-to-leaf indicators $ℒ$ choose among $\left\{{\mathit{\eta }}_v,v\in {𝒱}_L\right\}$ for every observation in Equation (12). By setting $s_u=0$ for all the nonroot nodes in Equation (5), the classical LCM with a single $\mathit{\pi }={\mathit{\pi }}_{u_0}$ results. Figure 2 shows a directed acyclic graph (DAG) that represents the model likelihood and prior specifications.

FIGURE 2.

The directed acyclic graph (DAG) representing the structure of the model likelihood and priors. The quantities in squares are either data or hyperparameters; the unknown quantities are shown in the circles. The arrows connecting variables indicate that the parent parameterizes the distribution of the child node (solid lines) or completely determines the value of the child node (double-stroke arrows). The rectangular “plates” where the variables are enclosed indicate that a similar graphical structure is repeated over the index; The index in a plate indicate nodes, hyperparameter levels, leaves, subjects, classes, and features. This figure appears in color in the electronic version of this article, and any mention of color refers to that version

4 |. VARIATIONAL INFERENCE ALGORITHM

Calculating a posterior distribution often involves intractable high-dimensional integration over the unknowns in the model. Traditional sequential sampling approaches such as Markov chain Monte Carlo (MCMC) remains a widely used inferential tool based on approximate samples from the posterior distribution. They can be powerful in evaluating multidimensional integrals. However, they do not guarantee closed-form posterior distributions. Variational inference (VI) is a popular alternative to MCMC for approximating the posterior distribution and has been widely used in machine learning and gaining interest in statistics (e.g., Ormerod and Wand, 2010; Blei et al., 2017). In particular, VI has also been used for fitting the classical LCMs (e.g., Grimmer, 2011). VI requires a user-specified family of distributions that can be expressed in tractable forms while being flexible enough to approximate the true posterior; the approximating distributions and their parameters are referred to as “variational distributions” and “variational parameters”, respectively. VI algorithms find the best variational distribution that minimizes the Kullback-Leibler (KL) distance between the variational family and the true posterior distribution. VI has been widely applied in Gaussian (Titsias and Lázaro-Gredilla, 2011; Carbonetto and Stephens, 2012) and binary likelihoods (e.g., Jaakkola and Jordan, 2000; Thomas et al., 2019). Also see Blei et al. (2017) for a detailed review. We use VI because it is fast, bypasses infeasible analytic integration or data augmentation that is otherwise needed for MCMC under Dirac spike components and prior-likelihood nonconjugacy (Tüchler, 2008), and enables data-driven selection of hyperparameters via approximate empirical Bayes (Equation (S8), Supporting Information). These advantages of VI are achieved at a cost of slight variance-covariance under-estimation, the degree of which we assess in Section 5.

We use VI algorithm to conduct inference using variational distributions factorized as

$$q\left(\beta \right)=q\left(\gamma \right)\cdot \underset{q\left(s,\alpha \right)}{\underbrace{\prod _{u\in 𝒱}q\left(s_u,{\alpha }_u\right)}}\cdot \underset{q\left(Z\right)}{\underbrace{\prod _{v\in {𝒱}_L}\prod _{i=1}^{n_v}q\left(Z_i^{\left(v\right)}\right)}}\cdot \underset{q\left(\mathit{\varrho }\right)}{\underbrace{\prod _{\ell =1}^{L}q\left({\rho }_{\ell }\right)}},$$ (14)

where $q\left({\mathit{Z}}_i^{\left(v\right)}\right)$ is a multinomial distribution with variational parameters ${\mathit{r}}_i^{\left(v\right)}={\left(r_{i1}^{\left(v\right)},\dots ,r_{iK}^{\left(v\right)}\right)}^{\top }$, and $r_{ik}^{\left(v\right)}$ represents the approximate posterior probability of observation $i$ in leaf $v$ belonging to class $k$ and ${\sum }_{k=1}^K r_{ik}^{\left(v\right)}=1$. Importantly, we make no other assumptions about the particular parametric form of variational distributions, which by the VI updating rules can be shown to take familiar distributional forms (see Section A1, Supporting Information).

VI finds $q$ that minimizes the KL distance between the variational family and the true posterior distribution: $\mathrm{K}\mathrm{L}\left(q\right(\mathit{\beta }\left)\right||\mathrm{p}\mathrm{r}(\mathit{\beta }\mid \mathbf{Y}\left)\right)=-\int q\left(\mathit{\beta }\right)\mathrm{l}\mathrm{o}\mathrm{g}\left\{\frac{\mathrm{p}\mathrm{r}(\beta \mid \mathbf{Y})}{q\left(\mathit{\beta }\right)}\right\}d\mathit{\beta }$. However, the KL distance depends on the intractable posterior distribution is not easily computed. Fortunately, based on a well-known equality, $\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{p}\mathrm{r}\left(\mathbf{Y}\right)=ℰ\left(q\right)+\mathrm{K}\mathrm{L}\left(q\right(\mathit{\beta })\parallel \mathrm{p}\mathrm{r}(\mathit{\beta }\mid \mathbf{Y})$), where $ℰ\left(q\right)=\int q\left(\mathit{\beta }\right)\mathrm{l}\mathrm{o}\mathrm{g}\frac{\mathrm{p}\mathrm{r}(\mathbf{Y},\mathit{\beta })}{q\left(\mathit{\beta }\right)}d\mathit{\beta }$ is referred to as evidence lower bound (ELBO) because $\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{p}\mathrm{r}\left(\mathbf{Y}\right)\ge ℰ\left(q\right)$. Because $\mathrm{p}\mathrm{r}\left(\mathbf{Y}\right)$ is a constant, minimizing the KL divergence is equivalent to maximizing $ℰ\left(q\right)$. The VI algorithm updates each component of $q\left(\mathit{\beta }\right)$ in turn while holding other components fixed. However, because of the nonlinear sigmoid functions in Equation (12), generic VI updating algorithms for $q\left(s_u,{\mathit{\alpha }}_u\right)$ and $q\left(\mathit{\gamma }\right)$ involve integrating over random variables in the sigmoid function hence lack closed forms. To make the updates analytically tractable, we replace Equation (12) with an analytically tractable lower bound. In particular, we use a technique introduced by Jaakkola and Jordan (2000) that bounds the sigmoid function from below by a Gaussian kernel with a tuning parameter, hence affords closed-form VI updates; also see Durante et al. (2019) for a modern view of this technique as a bona fide mean-field approximation with Pòlya Gamma data augmentation. In particular, we will use the inequality

$$\sigma \left(x\right)\ge \sigma \left(\psi \right)\text{exp}\left\{\left(x-\psi \right)/2-g\left(\psi \right)\left(x^2-{\psi }^2\right)\right\}≔h\left(x,\psi \right),$$ (15)

with $g\left(\psi \right)=\frac{1}{2\psi }\left[\sigma \left(\psi \right)-\frac{1}{2}\right]$ where $\psi$ is a tuning parameter.

We approximate $\mathrm{E}\mathrm{L}\mathrm{B}\mathrm{O}ℰ\left(q\right)$ by ${ℰ}^{*}\left(q\right):$

$${ℰ}^{*}\left(q\right)≔\int q\left(\beta \right)\text{log}\times \frac{h^{*}\left(X,\psi ,\gamma ,Z\right)h^{**}\left(\varphi ,s,\alpha ,Z\right)\text{pr}\left(s,\gamma ,\alpha ,\mathit{\varrho }\right)}{q\left(\beta \right)}\text{d}\beta \le ℰ\left(q\right),$$ (16)

where

$$h^{*}\left(X,\psi ,\gamma ,Z\right)=\prod _{v\in {𝒱}_L}\prod _{i=1}^{n_v}\prod _{k=1}^K{\left\{{\prod }_{j=1}^{J}h\left(X_{ij}^{\left(v\right)}{\gamma }_{jk},{\psi }_{jk}\right)\right\}}^{Z_{ik}^{\left(v\right)}},$$

and

$$h^{**}\left(\varphi ,s,\alpha ,Z\right)=\prod _{v\in {𝒱}_L}\prod _{i=1}^{n_v}\prod _{k=1}^K\{{\left\{h\left({\eta }_{vk};{\varphi }_k^{\left(v\right)}\right)\right\}}^{\mathbf{1}\{k<K\}}{\times \prod _{m<k}h\left(-{\eta }_{vm};{\varphi }_m^{\left(v\right)}\right)\}}^{Z_{ik}^{\left(v\right)}}.$$

The VI algorithm iterates until convergence to find the optimal variational distribution $q$ that maximizes ${ℰ}^{*}\left(q\right)$. Because ${ℰ}^{*}\left(q\right)\le \mathrm{l}\mathrm{o}\mathrm{g}\pi \left(\mathbf{Y}\right)$, it can be viewed as an approximation to the marginal likelihood. We maximize over $\mathit{\psi }$ and $\mathit{\varphi }$ to obtain the best approximation. In addition, we adopt an approximate empirical Bayes approach by optimizing the VI objective function ${ℰ}^{*}\left(q\right)$ over the hyperparameters ${\mathit{\tau }}_1$ and ${\mathit{\tau }}_2$. Relative to specifying weakly informative but often nonconjugate hyperprior for the variance parameters, optimizing hyperparameter is more practically convenient (e.g., Thomas et al., 2019). Because updating the hyperparameters changes the prior, we need to update $q,\mathit{\psi }$, and $\mathit{\varphi }$ again. This leads to an algorithm that alternates between maximizing ${ℰ}^{*}\left(q\right)$ in $(q,\mathit{\psi },\mathit{\varphi })$ and in $\left({\mathit{\tau }}_1,{\mathit{\tau }}_2\right)$ until convergence. We update the hyperparameters every $d$ complete VI iterations. Pseudocode in Algorithm 1 outlines the VI updates; Section A1 in the Supporting Information details the exact updating formula.

4.1 |. Posterior summaries

Two sets of point and interval estimates for $\left\{{\mathit{\pi }}_v:v\in {𝒱}_L\right\}$ are available from the VI algorithm: (1) data-driven grouped (“fused”) estimates (${\widehat{\mathit{\pi }}}_v^{\text{dgrp }}$) that are formed by setting a subset of $\mathit{s}$ to one and the rest to zero, and (2) leaf-specific estimates $\left({\stackrel{\mathit{ˆ}}{\mathit{\pi }}}_v^{\text{leaf }}\right)$. For (1), we select the posterior median model by setting $s_u=1$ for nodes in $\widehat{𝒰}=\left\{u:E_{q_t}\left[s_u\right]>0.5\right\}$ (see Step 1b, Section A1, Supporting Information). For leaves $v$ and $v^{\prime }$, with probability one, ${\widehat{\mathit{\pi }}}_v^{\mathrm{d}\mathrm{g}\mathrm{r}\mathrm{p}}={\widehat{\mathit{\pi }}}_{v^{\prime }}^{\mathrm{d}\mathrm{g}\mathrm{r}\mathrm{p}}$ if and only if $a\left(v\right)\cap \widehat{𝒰}=a\left(v^{\prime }\right)\cap \widehat{𝒰}$. Because no closed-form posterior distributions for ${\mathit{\pi }}_v$ are readily available under logistic stick-breaking representation, we compute the approximate posterior mean and approximate 95% credible intervals (CrIs) by a Monte Carlo procedure after convergence of Algorithm 1. For $u\in \widehat{𝒰}$, we first draw $B={10}^5$ random independent samples of ${\alpha }_{uk}$ from $N\left(E_{q_t}\left[{\alpha }_{uk}\mid s_u=1\right],V_{q_t}\left[{\alpha }_{uk}\mid s_u=1\right]\right)$, for $k\in [K-1]$. We then compute $B$ corresponding ${\mathit{\pi }}_v$ vectors based on Equations (3)–(5) with $s_u=1\{u\in \widehat{𝒰}\}$ in (5). Finally, we compute the empirical means and 95% CrIs marginally for ${\pi }_{vk},k\in \left[K\right]$. The above Monte Carlo procedure is extremely fast given only independent Gaussian samples are drawn. As a comparison, for (2), we define leaf-specific estimates ${\stackrel{\mathit{ˆ}}{\mathit{\pi }}}_v^{\text{leaf }}$ by the mean of (3) where ${\eta }_{vk}\stackrel{d}{~}N\left({\sum }_{u\in a\left(v\right)} E_{q_t}\left[s_u{\alpha }_{uk}\right],{\sum }_{u\in a\left(v\right)} V_{q_t}\left[s_u{\alpha }_{uk}\right]\right)$, for $k\in \left[K\right]$. We also use Monte Carlo simulation to approximate the posterior means and 95% CrIs. In general, ${\stackrel{\mathit{ˆ}}{\mathit{\pi }}}_v^{\text{leaf }}$ differ across the leaves. In contrast, the data-driven grouped estimates $\left\{{\stackrel{\mathit{ˆ}}{\mathit{\pi }}}_v^{\mathrm{d}\mathrm{g}\mathrm{r}\mathrm{p}}\right\}$ induce dimension reduction.

Prediction

The out-of-sample predictive probability of class $k$ for a new observation nested in leaf $v$ is $r_{i^{\prime }k}^{\left(v\right)}≔\text{pr}\left(I_{i^{\text{'}}}^{\left(v\right)}=k\mid \right.\left.Y_{i^{\prime }}^{\left(v\right)},𝒟\right)$, where $𝒟=\left(\mathbf{Y},{𝒯}_w,ℒ,\mathit{a},\mathit{b},{\mathit{\tau }}_1,{\mathit{\tau }}_2\right)$. We have

$$r_{i^{\prime }k}^{\left(v\right)}=\int \underset{\left(i\right)}{\underbrace{\text{pr}\left(I_{i^{\prime }}^{\left(v\right)}=k\mid {\theta }_{\cdot k},{\pi }_v,Y_{i^{\prime }}^{\left(v\right)},𝒟\right)}}\underset{\left(ii\right)}{\underbrace{\text{pr}\left({\theta }_{\cdot k},{\pi }_v\mid Y_{i^{\prime }}^{\left(v\right)},𝒟\right)}}\text{d}{\theta }_{\cdot k}\text{ d}{\pi }_v.$$ (17)

We approximate (17) by plug-in estimators: ${\widehat{r}}_{i^{\prime }k}^{\left(v\right)}\propto \text{pr}\left(Y_{i^{\text{'}}}^{\left(v\right)}\mid \right.\left.I_{i^{\prime }}^{\left(v\right)}=k,{\hat{\theta }}_{\cdot k},{𝒯}_w\right)\cdot {\hat{\pi }}_{vk},k\in \left[K\right]$. This can be seen by noting that term $\left(i\right)\propto \mathrm{p}\mathrm{r}\left(Y_{i^{\prime }}^{\left(v\right)}\mid I_{i^{\prime }}^{\left(v\right)}=k,{\theta }_{\cdot k},{𝒯}_w\right)\cdot {\pi }_{vk}$, and term $\left(ii\right)\approx \mathrm{p}\mathrm{r}\left({\theta }_{\cdot k},{\mathit{\pi }}_v\mid D\right)$ that we approximate by a Dirac measure at $\left({\hat{\theta }}_{\cdot k},{\stackrel{\mathit{ˆ}}{\mathit{\pi }}}_v\right)$. Here ${\stackrel{\mathit{ˆ}}{\mathit{\pi }}}_v={\widehat{\mathit{\pi }}}_v^{\text{dgrp }}$.

Choice of $\mathit{K}$

In applications where data-driven selection of $K$ is more desirable, we may follow Bishop (2006) and use criterion ${ℰ}_K^{*}\left(q\right)+\mathrm{l}\mathrm{o}\mathrm{g}(K!)$, where ${ℰ}_K^{*}\left(q\right)$ is the lower bound of $\mathrm{l}\mathrm{o}\mathrm{g}$ marginal data likelihood for a $K$-class model and the correction term is to make different models comparable (e.g., Grimmer, 2011, section 5.2).

5 |. SIMULATION

5.1 |. Design and performance metrics

We conducted a simulation study to evaluate the performance of the proposed tree-integrative LCM. We compare our model to a few alternatives with ad hoc grouping of observations in terms of accuracy in estimating $\left\{{\mathit{\pi }}_v,v\in \right. \left.V_L\right\}$. Data were generated under two scenarios with different class-specific response profiles $\mathbf{\Theta }$. Section A2 in the Supporting Information details the true parameter settings of the simulations. Figure 3(a) visualizes the tree ${𝒯}_w$ with equal edge weights and true leaf groups used in the simulation with $p_L=11$ leaves and $G=3$ groups.

We simulated $R=200$ independent replicate data sets for different total sample sizes $(N=\mathrm{1000,4000})$. For each $N$, we set $n_v\approx N/p_L$ for $\nu \in {𝒱}_L$ (with rounding where needed) to investigate balanced leaves and set $n_v$ to be approximately $\frac{1}{5}N/p_L$ or $\frac{4}{5}N/p_L$ with equal chance for mimicking unbalanced observations across leaves. For observations in a leaf $v$, we simulate ${\mathit{Y}}_i^{\left(v\right)}$ according to an LCM with class probabilities ${\mathit{\pi }}_v$ and class-specific response probabilities $\mathbf{\Theta }$. We simulated data for different dimensions $J=\mathrm{21,84}$, for $K=3$ classes.

For each simulated data set, we fitted the proposed model, based on which we compute ${\widehat{\mathit{\pi }}}_v^{\text{dgrp }}$ and ${\stackrel{\mathit{ˆ}}{\mathit{\pi }}}_v^{\text{leaf }}$ (see Section 4.1). Our primary interest is in $\left\{{\widehat{\mathit{\pi }}}_v^{\text{dgrp }}\right\};\left\{{\widehat{\mathit{\pi }}}_v^{\text{leaf }}\right\}$ are for comparisons. In addition, we also tested a few approaches based on ad hoc leaf node groupings: (1) True grouping analysis (fit separate LCMs to obtain estimates in each of the true groups); (2) Single group LCM analysis (omit sample-to-leaf indicators $ℒ$, hence the tree information); (3) Ad hoc grouping 1 (manual grouping coarser than the true grouping); 4) Ad hoc grouping 2: classical LCMs for data on each leaf. All analyses assume $\mathbf{\Theta }$ does not vary by leaves.

We used three model performance metrics. First, we computed the root mean squared errors (RMSE) for an estimate ${\stackrel{\mathit{ˆ}}{\mathit{\pi }}}_v$ where $\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}\left({\hat{\pi }}_v\right)=\sqrt{{\left(K{p}_L\right)}^{-1}{\sum }_{k=1}^K {\sum }_{v\in V_L} {\left\{{\hat{\pi }}_{vk}-{\pi }_{vk}\right\}}^2}$. Second, we compared the true and the estimated leaf groupings via adjusted Rand Index (ARI, Hubert and Arabie, 1985). ARI is a chance-corrected index that takes value between −1 and 1 with values closer to 1 indicating better agreement. Finally, we estimated the coverage probability of the approximate 95% CrIs. For each true group g, we compute the frequency of the approximate 95% CrI (computed along with ${\stackrel{\mathit{ˆ}}{\mathit{\pi }}}_v^{\mathrm{d}\mathrm{g}\mathrm{r}\mathrm{p}}$) containing the truth, conditional on the event that an estimated partition of the leaf nodes includes $g$.

5.2 |. Simulation results

Figure 3 shows comparisons among the RMSEs for different models under different scenarios. For sample sizes $N=1000$ and $N=4000$, the proposed methods with data-driven grouping $\left({\stackrel{\mathit{ˆ}}{\mathit{\pi }}}_v^{\text{dgrp }}\right)$ produced similar or better RMSE than analyses based on ad hoc leaf groupings, which restrict leaves into incorrect groupings that are coarser (single LCM and ad hoc grouping 1) or finer (ad hoc grouping 2) than the truth. The proposed approach $\left({\stackrel{\mathit{ˆ}}{\mathit{\pi }}}_v^{\mathrm{d}\mathrm{g}\mathrm{r}\mathrm{p}}\right)$ achieved similar RMSE as ${\widehat{\mathit{\pi }}}_v^{\text{leaf }}$, indicating little accuracy was lost in exchange for dimension reduction. The RMSEs of ${\widehat{\mathit{\pi }}}_v^{\text{dgrp }}$ were similar to estimates of ${\pi }_v,v\in {𝒱}_L$ obtained from analyses based on the true leaf grouping. Indeed, the accuracy of group discovery increased with sample sizes with other settings fixed. Average ARIs across replications for each scenario were high (0.94 to 0.99) indicating good recovery of the true leaf groups. Although the groups discovered were not perfect, the comparable RMSEs suggest desirable adaptability of the proposed approach in effective collapsing of the leaves. The RMSE for ${\widehat{\mathit{\pi }}}_v^{\text{dgrp }}$ was smaller than analyses based on a refined leaf-level grouping: smaller sample sizes in the leaves resulted in loss of efficiency in separate estimations of ${\mathit{\pi }}_v$ across leaves. RMSEs were further reduced under a larger $J$ or balanced sample sizes in the leaves. However, we again observed similar relative advantage of the proposed ${\widehat{\mathit{\pi }}}_v^{\text{dgrp }}$. The relative comparisons of RMSEs under less discrepant true class-specific response profiles remained similar (see Figure S2, Supporting Information).

The observed coverage rates of the approximate 95% CrIs achieved the nominal level satisfactorily (see Figure S1, Supporting Information). Slight under-coverage occurred under smaller $N$, unbalanced sample sizes, smaller $J$ and leaf groups with smaller number of observations. This is partially a consequence of VI as an inner approximation to the posterior distribution that may underestimate the posterior uncertainty (e.g., Bishop, 2006, chapter 10).

Finally, we also considered scenarios where only a single group of leaves is present in truth for which the classical LCM is perfectly appropriate. Figure S3 in the Supporting Information shows, by learning the posterior node-specific slab-versus-spike selection probabilities, the proposed model produces similar RMSEs as the classical LCM.

6 |. E. COLI DATA APPLICATION

6.1 |. Background and data

E. coli infections cause millions of urinary tract infections (UTIs) in the United States each year (e.g., Johnson and Russo, 2002). Many studies have shown that extraintestinal pathogenic E. coli (ExPEC) strains routinely colonize food animals and contaminate the food supply chain serving as a likely link between food-animal E. coli and human UTIs (e.g., Johnson et al., 2005). The scientific team adopted a novel strategy of augmenting fine-scale core-genome phylogenetics with interrogation of accessory host-adaptive MGEs (see Section 1.1). The scientific goal is to accurately estimate the probabilities of E. coli isolates with human and nonhuman host-origins across genetically diverse but related E. coli sequence types (STs).

We restrict our analysis to $N=2663$ E. coli isolates in a well-defined collection from humans and retail meat obtained over a 12-month period in Flagstaff, Arizona, United States. Each isolate belongs to one of $p_L=133$ different STs (leaves in the phylogenetic tree) that are identified via a multilocus sequence typing scheme based on short-read DNA sequencing. A total of $J=17$ MGEs were curated and associated with functional annotations. Each ST was represented by at least four isolates. We constructed rooted, maximum-likelihood phylogenies using core-genome SNP data for the 133 STs. Figure 4 shows the estimated phylogenetic tree for the STs where the edge lengths represents the substitution rate in the conserved core genome. Every ST is overlaid in the same row with the empirical frequencies of (1) $J=17$ MGEs and (2) the observed sources (human clinical or meat samples) that may differ from the true host origin. The observed frequencies of the MGEs vary greatly across lineages. We apply the proposed tree-integrative LCM to (1) estimate the probabilities of unobserved human and non-human host-origins for all E. coli STs with data-driven groupings of the STs for dimension reduction; and (2) to produce isolate-level probabilistic host-origin assignment. The context of the study restricts us to assume the host origin of each isolate is in one of two unobserved class of human versus food animals. A subset of preliminary data is analyzed in this paper for illustrating the proposed method. Inclusion of additional samples and/or MGEs may change findings. The final results and the detailed workflow of MGE discovery will be reported elsewhere.

FIGURE 4.

The empirical frequencies for $J=17$ MGEs within each ST mapped in the core-genome phylogenetic tree. The red scale bar represents the substitution rate in the conserved core genome. The bars on the right indicate the total number isolates of each ST; the gray and blue bars represent the number of isolates obtained from apparent nonhuman and human sources, respectively. The core-genome phylogenetic tree on the left margin maps $N=2663$ E. coli isolates into $p_L=133$ STs (leaves). This figure appears in color in the electronic version of this article, and any mention of color refers to that version

6.2 |. Data results

The proposed approach produces estimated class-specific response profiles $\left({\widehat{\theta }}_{.k},k=\mathrm{1,2}\right)$ that exhibit differential enrichment of MGEs (Figure 5b). For example, MGEs 3, 10 to 17 are estimated with probability of between 0.15 and 0.71 being present in class 1, with log odds ratios (LORs; class 1 vs. class 2: $\left.LOR\left({\widehat{\theta }}_{j1},{\widehat{\theta }}_{j2}\right)\right)$ greater than 1. The functional annotations of these MGEs reveal that class 1 is likely associated with food-animal hosts. In contrast, MGEs 4–9 are estimated to be present in class 2 with probability between 0.35 to 0.82 with LORs greater than 1 relative to the corresponding estimated response probabilities in class 1. The results suggest the MGEs are highly associated with different types of host-origins.

FIGURE 5.

(a) Data results with estimated leaf groups and latent class probabilities by group. ST names (ST_#_isolates) are aligned to the tips of the circular tree, which are colored by discovered leaf groups. The scale bar represents the substitution rate in the conserved core genome. The circular heatmap shows the estimated latent class probabilities (${\stackrel{\mathit{ˆ}}{\mathit{\pi }}}_v^{\text{dgrp }},v\in {𝒱}_L$); (b) and (c) see the captions of the subfigures. This figure appears in color in the electronic version of this article, and any mention of color refers to that version

The proposed approach discovered 21 ST leaf groups, for which distinct estimated vectors of the latent class probabilities ${\stackrel{\mathit{ˆ}}{\mathit{\pi }}}_v^{\text{dgrp }}$ are shown in Figure 5(a). For many estimated ST groups, the class probabilities are almost entirely dominated by one type of host-origin. For example, the estimated ST Group 1 (38 leaves; 649 samples; class 1 probability 0.98, 95% CrI: (0.97, 0.99)) and Group 3 (31 leaves; 422 samples; class 1 probability 0.97, 95% CrI: (0.96, 0.98)) showed high probabilities of nonhuman (class 1) host-origin of E. coli. The results suggest recent cross-species transmissions were rare among multiple nearby lineages.

We also compared against results based on two fixed and more restrictive leaf groups, (a) classical LCM (one leaf group); (b) four leaf groups selected by the scientific team (Figure S4, Supporting Information). The single LCM (a) estimated the probability of class 1 to be 0.60, 95% CrI: (0.58, 0.62). The ad hoc leaf grouping (b) produced coarser estimates relative to the proposed ${\widehat{\mathit{\pi }}}_v^{\text{dgrp }}$ that identified four local leaves (ST1141, ST10, ST744 and ST5996) comprising 116 samples that have estimated probability of class 1: 0.74(0.66, 0.82)). This highlights the inability of potentially misspecified leaf groups to uncover subtle local variations in the latent class probabilities. We compared these models via 10-fold cross-validation based on the mean predictive log-likelihood (MPL) of the test data, which is computed by plugging in the estimated latent class probabilities and response probability profiles. Of note, because of small sample sizes in some leaves, a naive cross-validation may by chance result in a training set without any observation in some leaves. We therefore randomly keep two observations per leaf and use one random fold of the remaining samples as test data. The proposed approach (with posterior median node selection) achieves the highest MPL (−2015.48) compared to (a) (−2030.15) and (b) (−2162.45) (Figure S6, Supporting Information). The estimates of response probability profiles are similar.

On an individual isolate level, the proposed model can estimate the probability that an isolate was derived from a particular host. For example, by incorporating additional observed sample source information, we can compute “posterior concordance probability (PCP)” for each observation. In particular, PCP, $r_{i,S_i^{\left(v\right)}}^{\left(v\right)}$, is defined as the approximate posterior probability of the true host origin agreeing with the observed sample source category $S_i^{\left(v\right)}$ of the same E. coli isolate (e.g., $S_i^{\left(v\right)}=1$ for meat and 2 for human clinical samples). Figure 5(c) shows the histogram of PCPs for all the isolates. Small PCPs, for example, below a user-specified threshold of 0.5, indicate likely recent host jumps that may subject to further examination to estimate the timing of host transmissions based on in vitro stability data of each MGE.

7 |. DISCUSSION

In this paper, we proposed a tree-integrative LCM for analyzing multivariate binary data. We formulated the motivating scientific question in terms of inferring latent class probabilities that may vary in different parts of a tree. We proposed a Gaussian diffusion prior for logistic stick-breaking parameterized latent class probabilities and designed a scalable approximate algorithm for posterior inference. Our E. coli data analysis revealed that multiple MGEs are disproportionately associated with specific host origins. Combined with external sample source information, the model can help identify isolates that underwent recent host jump, paving the way for further isolate-level host origin validation.

Our study has some limitations. First, the MGE data we analyzed may represent a fraction of the host-associated accessory elements. By design, additional accessory elements identified in future studies can be readily integrated and evaluated in the proposed framework. Second, host-associated accessory elements are lost and gained over time as E. coli strains transition across hosts. For infections that were zoonotic in nature, we did not observe how much time had lapsed between the cross-species host jump and the actual infection. Our model partly accounted for these uncertainties by the imperfect positive response probabilities. However, the timings may drive the presence or absence of multiple MGEs, resulting in potential statistical dependence given the true class of host-origin. Deviations from local independence assumption may impact model-based inference (e.g., Albert and Dodd, 2004; Pepe and Janes, 2006). In practice, a subset of samples with ascertained host-origins may provide critical information to estimate the conditional dependence structure.

Further model extensions may improve model applicability. First, when a subset of observations is not mapped in the tree at random, the algorithm can add additional unobserved leaf indicators to be inferred along with other parameters. Second, it is important to note that the tree integrated into LCM in general is estimated with uncertainty in the topological structure. Methods that use an additional layer of prior over the tree space centered around the estimated tree may account for the upstream uncertainty (e.g., Willis and Bell, 2018). Third, E. coli isolates may vary in additional factors such as the hosts’ clinical characteristics. Regression extensions may refine the understanding of variation in latent class probabilities and positive response probabilities that are driven by covariates (e.g., Huang and Bandeen-Roche, 2004). Fourth, LCM is an example of probability tensor decomposition methods (e.g., Johndrow et al., 2017), the tree-integrative LCM motivates extensions to general graph-guided probability tensor decomposition methods. Finally, the truncated stick-breaking formulation in Equation (3) motivates connections to a broader class of covariate-indexed dependent process priors as $K$ approaches infinity (e.g., Ren et al., 2011; Rodriguez and Dunson, 2011). Extensions along this line may also relax the present assumption of identical number of realized classes at additional computational cost.

On computation, without relying on prior-likelihood conjugacy, neuronized priors for Bayesian sparse linear regression has been proposed (Shin and Liu, 2021). Comparative studies against spike-and-slab priors are warranted. In addition, one known drawback of the mean field VI is that it tends to underestimate the marginal posterior variances of parameters. In our simulations, we showed near nominal coverages of the true $\mathit{\pi }$ with slight undercoverages happening mostly for leaf groups with very small sample sizes. It is an interesting line of work to incorporate the methods of Giordano et al. (2015) to correct the variance-covariance matrices used in the component variational distributions. We leave these topics for future work.

DATA AVAILABILITY STATEMENT

An R package “lotR” is freely available at https://github.com/zhenkewu/lotR. The data that support the findings in this paper are available from the corresponding author on reasonable request.