Bayesian Additive Regression Trees for Multivariate Skewed Responses

Summary

This paper introduces a nonparametric regression approach for univariate and multivariate skewed responses using Bayesian additive regression trees (BART). Existing BART methods use ensembles of decision trees to model a mean function, and have become popular recently due to their high prediction accuracy and ease of use. The usual assumption of a univariate Gaussian error distribution, however, is restrictive in many biomedical applications. Motivated by an oral health study, we provide a useful extension of BART, the skewBART model, to address this problem. We then extend skewBART to allow for multivariate responses, with information shared across the decision trees associated with different responses within the same subject. The methodology accommodates within-subject association, and allows varying skewness parameters for the varying multivariate responses. We illustrate the benefits of our multivariate skewBART proposal over existing alternatives via simulation studies and application to the oral health dataset with bivariate highly skewed responses. Our methodology is implementable via the R package skewBART, available on GitHub.

1 |. INTRODUCTION

Highly skewed multivariate responses are commonly observed in many biomedical and clinical research problems. For example, in a preliminary analysis (see Figure 1) of periodontal disease (PD) status¹ among a population of Type-2 diabetic (T2D) Gullah-speaking African-Americans (henceforth, the GAAD study) two popular PD endpoints — the average periodontal pocket depth (PPD) and average clinical attachment level (CAL) — are highly skewed. The data analytic goal here is to assess the cross-sectional associations of important covariates, say, age and diabetes level, on the bivariate responses (average PPD and average CAL of all measured tooth-sites) of any patient from a highly diabetic population.

FIGURE 1.

GAAD data: Scatter plot of PPD and CAL responses, averaged for each subject, with marginal density estimates.

To model the relationship between a k-dimensional multivariate observed response y_i and a p-dimensional vector of covariates x_i without imposing any pre-determined functional form, we consider the nonparametric regression of y_i on the corresponding x_i, given by

$$y_i=f\left(x_i\right)+{\mathit{ϵ}}_i,$$ (1)

where the errors ϵ_i are iid with common multivariate density h_ϵ(·) for i = 1, …, n. The unknown function f returns k-dimensional vectors corresponding to x_i. For univariate case with k = 1, a popular strategy for modeling f is to use an ensemble of decision trees, such as random forests² or boosted decision trees³.

In this paper, we focus on the Bayesian additive regression trees (BART) model introduced by Chipman et al.⁴. Unlike the parametric regression functions utilized to model skewed multivariate responses⁵, decision tree ensembles efficiently capture both nonlinear and interaction effects of covariates in f(x), automatically. Additionally, compared to other machine learning algorithms, BART models provide a number of appealing advantages. First, they are typically robust to the choice of tuning parameters, and yield high prediction accuracy. Second, unlike many competing machine learning algorithms, BART allows for uncertainty quantification by providing a full posterior distribution for the predictions. Recent work has also established attractive theoretical properties for BART^6,7. BART models have successfully been applied to many statistical problems, including classification⁸, survival analysis^9,10, density estimation¹¹, and high dimensional sparse regression¹².

Existing BART models come with various limitations, however. For example, in studies with highly skewed responses, the error density h_ϵ(·) being a multivariate Gaussian density may not be appropriate. Bhingare et al.,⁵ showed that the mean PPD and mean CAL are highly skewed, and methods which adapt to skewed responses tend to outperform methods based on Gaussian errors regarding precision of the estimated parameters and predictions of future outcomes. Second, as illustrated by Linero and Yang⁷, the estimate of f(·) obtained from the usual BART models lack smoothness, leading to suboptimal performance when the true f(x) varies smoothly with x. To improve performance of the estimate when the true f(·) is believed to be smooth, Linero and Yang⁷ introduced the soft BART (SBART) model.

In this paper, we introduce the skewBART model, which takes the error density h_ϵ(·) in equation (1) to be a multivariate skewed normal density to accommodate non-Gaussian responses. We also extend the SBART model for univariate response to obtain smooth estimates of f(·) under skewed errors. We develop an efficient Gibbs sampler based on the Bayesian backfitting algorithm⁴ to perform practical nonparametric Bayesian inference. In addition to accommodating skewness of the error, we introduce a skewed multivariate response model, with multivariate decision trees used for regression functions and a multivariate skewed distribution for the error vector for each subject. This is motivated from the fact that most epidemiological studies/trials on PD model CAL, thereby quantifying the (past) disease history and progression, in contrast to the current disease status measured by PPD¹³. Hence, a thorough assessment of PD may suggest modeling PPD and CAL considering a multivariate framework. The use of multivariate decision trees has been used both to handle mixed-type responses such as zero-inflated log-normal/gamma distributions¹⁴, as well as to implement targeted smoothing over space or time¹⁵. To the best of our knowledge, this is the first attempt to bring multivariate decision tree methods to bear in the BART framework to model multivariate skewed response data. To increase the precision of the inference, our multi-skewBART model takes into account the association of the multivariate responses within each subject, borrowing information across the different responses by using the same decision trees for the mean vector. We further illustrate the advantages of our skewBART and multi-skewBART proposals in practice on the GAAD study.

The rest of the paper is organized as follows. After briefly reviewing both the BART model and the important properties of the skew normal distribution, Section 2 introduces the skewBART and multi-skewBART models, respectively for univariate and multivariate responses. Details on the Bayesian inference, including prior specifications and the related Markov chain Monte Carlo (MCMC) algorithm for posterior computation are outlined in Section 3. In Section 4, we compare the finite sample performances (including out-of-sample predictions) of our proposals to existing alternatives using synthetic data. In Section 5, we demonstrate the application to the motivating GAAD oral health dataset. Section 6 concludes with a discussion. Details on the proposed MCMC algorithm appear in the Appendix.

2 |. THE SKEWBART AND MULTI-SKEWBART MODELS

2.1 |. Review of the BART and SBART models

For the scalar response, the BART framework introduced by Chipman et al.⁴ models f(x_i) in (1) as a sum of m trees

$$f\left(x_i\right)=\sum _{t=1}^{m}g\left(x_i;{𝒯}_t,{ℳ}_t\right),$$ (2)

where 𝒯_t is a binary tree structure with n_t leaf nodes and ${ℳ}_t=\left\{{\mu }_{t1},\dots ,{\mu }_{t{n}_t}\right\}$ is a collection of parameters for n_t leaves of the tth tree 𝒯_t with h_ϵ being a Gaussian density. The function g(x_i; 𝒯_t, ℳ_t) returns ${\sum }_{\ell =1}^{n_t}{\mu }_{t\ell }\eta \left(x_i;{𝒯}_t,\ell \right)$ where η(x_i; 𝒯_t, ℓ) is the indicator that x_i is associated with leaf node ℓ in 𝒯_t, i.e., g(x_i; 𝒯_t, ℳ_t) = μ_tℓ if-and-only-if x_i is associated to leaf ℓ of tree 𝒯_t; for example, if x = (0.62, 0.45) in Figure 2 (bottom-left), then g(x, 𝒯_t, ℳ_t) = μ_t1. The tree 𝒯_t consists of interior splitting rules of the form [x_j ≤ C_b] for each branch node b, which induces a partition on the covariate space. At each interior node, the splitting rule uses predictor j with prior probability s_j, where s = (s₁, …, s_p) is a probability vector. Throughout this manuscript, we place a Dirichlet(a∕p, …, a∕p) hyperprior on s to allow the model to filter out irrelevant predictors¹². The cutpoint C_b of the splitting rule is assigned a uniform prior over its possible values.

FIGURE 2.

Top: an example of a binary tree 𝒯_t where the terminal nodes are labeled with the corresponding parameters μ_tℓ. Bottom: the corresponding partition of the predictor space 𝒳 = [0, 1]² from BART (left panel) and SBART (right panel) when κ = 0.1.

To avoid overfitting, each tree is given a regularization prior which shrinks the individual leaf parameters to 0 so that each tree contributes only a small portion of the overall fit. It is regularized by the depth of each tree and the prior distribution of leaf parameters. The probability that each node at depth d is non-terminal is given by γ(1 + d)^−β for hyperparameters γ ∈ (0, 1) and β ∈ [0, ∞). The default prior specification⁴ sets γ = 0.95 and β = 2, which encourages the trees to be shallow (rarely exceeding depth 2–3). The leaf parameters μ_tℓ are given iid $𝒩\left(0,{\sigma }_{\mu }^2/m\right)$ priors, ensuring that the prior variance of f(x) is constant as a function of m.

To define a BART model (2) of the non-parametric regression in (1) for the univariate response y_i, we must specify a likelihood for the observed data D = {(y_i, x_i) : i = 1, ⋯ , n}, a prior on the unknown model parameters related to the tree structures, and a prior on the parameters σ² of the unknown univariate error distribution h_ϵ(· | σ²). Let 𝓣 = {𝒯₁, …, 𝒯_m} denote the set of all trees and let 𝓜 = {ℳ₁, …, ℳ_m} denote the set of leaf parameters for all trees. Bayesian computation then proceeds by using MCMC based samples from the joint posterior p(𝓣, 𝓜| D), which is proportional to the likelihood times the joint prior p(θ) of all the unknown parameters θ. When the error distribution is 𝒩(0, σ²), the prior consists of a conjugate inverse gamma prior p(σ²) for σ², independent priors p(𝒯_t) for the tree structures 𝒯_t, and conditionally-independent priors for the terminal node parameters. For the leaf node set ℳ_t = {μ_tℓ : 1 ≤ ℓ ≤ n_t} conditional on the tree structure 𝒯_t, a conjugate normal distribution $𝒩\left(0,{\sigma }_{\mu }^2/m\right)$ is widely used as the common independent prior p(μ_tl). The resulting joint prior distribution factors as

$$\begin{gathered} p\left(𝓣,𝓜,{\sigma }^2\right)=\prod _{t=1}^m\left[p\left({𝒯}_t,{ℳ}_t\right)\right]p\left({\sigma }^2\right) \\ =\prod _{t=1}^m\left[p\left({ℳ}_t\mid {𝒯}_t\right)p\left({𝒯}_t\right)\right]p\left({\sigma }^2\right) \\ =\prod _{t=1}^m\left[\prod _{\ell =1}^{n_t}p\left({\mu }_{t\ell }\mid {𝒯}_t\right)\right]p\left({𝒯}_t\right)p\left({\sigma }^2\right). \end{gathered}$$ (3)

By denoting the (n × 1) vector of partial residuals excluding the tth tree as R_t = (R_t1, …, R_tn)^⊤ where $R_{ti}=y_i-{\sum }_{k\ne t}g\left(x_i;{𝒯}_k,{ℳ}_k\right)$, the integrated likelihood function of 𝒯_t is

$$p\left({𝒯}_t\mid R_t,\sigma \right)\propto p\left({𝒯}_t\right)\int p\left(R_t\mid {ℳ}_t,{𝒯}_t,\sigma \right)p\left({ℳ}_t\mid {𝒯}_t,\sigma \right)d{ℳ}_t$$ (4)

given 𝒯_−t = {𝒯_k : k ≠ t} and ℳ_−t = {ℳ_k : k ≠ t}. For brevity, in equation (4) and all following expressions of posterior and conditional posteriors, we suppress dependence on covariate vector x. Because we have a conjugate prior for ℳ_t, the left-hand side of (4) is available in closed form. This allows the use of the Bayesian backfitting algorithm¹⁶ to update (𝒯_t, ℳ_t) sequentially for t = 1, …, m by using the Metropolis-Hastings algorithm⁴ to sample 𝒯_t using (4) and then sampling ℳ_t from its full conditional.

Despite the recent popularity of BART, the estimates obtained from BART have some drawbacks; in particular, BART lacks smoothness due to the distinct partitions as illustrated in Figure 2 (bottom-left). To overcome this lack of smoothness, Linero and Yang⁷ proposed SBART, which forms predictions by averaging over many random paths down the tree. In the splitting rule in SBART at branch b, x goes left at branch b with probability $\psi \left(x_{j_b};C_b,{\kappa }_b\right)=\psi \left(\frac{x_j-C_b}{{\kappa }_b}\right)$, where κ_b > 0 is a bandwidth parameter controlling the sharpness of the decision. By averaging over all possible paths down the tree, each leaf from SBART has a global impact on f which allows the model to share information across different covariate regions. Figure 2 (bottom-right) illustrates the smoothed partition of the predictors space in SBART with the logistic ψ(u) = (1 + e^−u)⁻¹. The splitting rules of BART induce a distinct partition of the predictor space 𝒳 = [0, 1]². On the other hand, SBART smooths the BART partition. Such smoothness is useful when the underlying f(x) is believed to be smooth. We note that if the bandwidth parameter κ approaches 0, the splitting rule of SBART converges to the splitting rule of the original BART model.

Throughout, we write f ~ BART_m(α, β, σ_μ) and f ~ BART_m if the scalar valued multivariable function f follows the BART prior respectively with the specified hyperparameter values of (α, β, m, σ_μ) and with the default values of Chipman et al.⁴. Similarly, we write f ~ SBART_m(α, β, σ_μ) and f ~ SBART_m to denote SBART priors respectively with pre-specified hyperparameters and with the default values recommended by Linero and Yang⁷.

2.2 |. Review of the skew-normal distribution

We use 𝒮𝒩(ξ, σ², α) to denote a univariate skew-normal (SN) distribution¹⁷ with probability density function $\frac{2}{\sigma }\varphi \left(\frac{y-\xi }{\sigma }\right)\Phi \left(\alpha \frac{y-\xi }{\sigma }\right)$, where ϕ and Φ are the standard normal density and cumulative distribution functions, respectively. The parameters ξ and σ are location and scale parameters, while α is a shape parameter which allows for the density to exhibit skewness. The stochastic representation of the univariate skew-normal distribution, such that E ~ 𝒮𝒩(ξ, σ², α), is

$$E\stackrel{d}{=}W+\lambda Z$$ (5)

where W and Z are respectively independent 𝒩(ξ, τ²) and 𝒩₊(0, 1) random variables, with $\lambda =(\alpha \sigma )/\sqrt{1+{\alpha }^2}$ and $\tau =\sigma /\sqrt{1+{\alpha }^2}$; here, 𝒩₊(μ, σ²) denotes a 𝒩(μ, σ²) truncated with the support of (0, ∞).

Different extensions^18,19,20 of the univariate SN distribution to the multivariate setting have been proposed. For extending skew-BART to multivariate responses, we consider the generalized multivariate skew-normal (MSN) distribution^19,21, which is both practical and flexible. A (k × 1) random vector E follows an MSN distribution, denoted by E ~ 𝒮𝒩_k(μ, Σ, Λ), if its probability density function is given by

$$2^k{\varphi }_k(e\mid \mu ,\Omega ){\Phi }_k\left({\Lambda }^{\top }{\Omega }^{-1}(e-\mu )\mid 0,{\Delta }^{-1}\right)$$ (6)

where Λ = diag(λ₁, …, λ_k) is the (k × k) skewness matrix, $\mu \in {ℝ}^k$ is the location vector, Σ is a (k × k) positive definite scale matrix, Ω = Σ + ΛΛ^⊤ and Δ = I_k + Λ^⊤Σ⁻¹Λ. Also, ϕ_k(y|μ, Σ) and Φ_k(y|μ, Σ) are the density and cumulative distribution functions of a multivariate normal distribution, denoted by 𝒩_k(μ, Σ), with mean $\mu \in {ℝ}^k$ and (k×k) positive definite covariance Σ. Note that (6) belongs to the class of fundamental skew distributions considered by Arellano-Valle and Genton¹⁹. From Proposition 1 of Arellano-Valle et al.²¹, E can be represented stochastically as

$$E\stackrel{d}{=}W+\Lambda Z,$$ (7)

where W ~ 𝒩_k(μ, Σ) and Z ~ 𝒩_k+(0, I_k) are independent k-variate normal and truncated normal random vectors. The family of MSN densities has the desirable and practically useful property of being closed under marginalization. Additionally, each of the individual components of the MSN distribution of (6) has a univariate skew-normal marginal of (5). Sahu et al.²⁰ and Bhingare et al.⁵ present physical interpretations of the univariate skew-normal density of (5), and the multivariate skew-normal density (7), in terms of convolution of a skewing shocks Z to a symmetric distribution.

2.3 |. Univariate skewBART

We first introduce the univariate skewBART model to extend BART to accommodate skewed responses when k = 1. We model the scalar-valued nonparametric regression function f(·) in (1) using a sum-of-trees model with an SN distributed error as

$$\{\begin{array}{l}{y}_i=f\left(x_i\right)+{ϵ}_i\text{\hspace{1em} with \hspace{1em}}{ϵ}_i\stackrel{iid}{~}𝒮𝒩\left(0,{\sigma }^2,\alpha \right)\hfill \\ f\left(x_i\right)={\sum }_{t=1}^{m}g\left(x_i;{𝒯}_t,{ℳ}_t\right).\hfill \end{array}$$ (8)

Using the stochastic representation of (5), we can rewrite (8) as

$$y_i=\sum _{t=1}^{m}g\left(x_i;{𝒯}_t,{ℳ}_t\right)+\lambda Z_i+W_i$$ (9)

where W_i ~ 𝒩(0, τ²), Z_i ~ 𝒩₊(0, 1), $\lambda =\alpha \sigma /\sqrt{1+{\alpha }^2}$ and $\tau =\sigma /\sqrt{1+{\alpha }^2}$. For the scalar y_i, we use the SBART of Linero and Yang⁷ with f ~ SBART_m to incorporate smoothness in f(·). The prior specifications described in Section 3.1 along with the reparameterization in (9) allows us to construct a simple data augmentation step to fit the skewBART model of (8).

2.4 |. Multivariate skewBART

We extend the sum-of-trees model (8) using the MSN distribution (7) to accommodate multivariate outcomes. This extension improves prediction accuracy by incorporating the correlation between responses, as well as the dependency in error estimation. Our proposed multivariate skewBART model, called the multi-skewBART model, for the multivariate response $Y={\left(y_1,\dots ,y_n\right)}^{\top }\in {ℝ}^{n\times k}$ models the nonparametric multi-variable f(·) in (1) as

$$\{\begin{array}{l}{y}_i=f\left(x_i\right)+{ϵ}_i\text{\hspace{1em}with\hspace{1em}}{ϵ}_i\stackrel{iid}{~}𝒮{𝒩}_k(0,\Sigma ,\Lambda )\hfill \\ f\left(x_i\right)={\sum }_{t=1}^{m}g\left(x_i;{𝒯}_t,{ℳ}_t\right),\hfill \end{array}$$ (10)

where g(x_i; 𝒯_t, ℳ_t) returns a k-dimensional μ_tℓ if x_i is associated with the leaf node ℓ in 𝒯_t for ℓ = 1, …, n_t. For nonparametric f(·) in (10), we say that f has a multivariate BART prior, denoted by f ~ MultiBART_m(α, β, Σ_M), where (α, β, m) are the usual BART hyperparameters for the trees and (k × k) matrix Σ_M is the prior covariance of μ_tℓ ~ 𝒩_k(0, Σ_M). Further details of this prior specification are presented in Section 3. The tree 𝒯_t in (10) has a binary tree structure like univariate skewBART, except that ${ℳ}_t=\left\{{\mu }_{t1},\cdots ,{\mu }_{t{n}_t}\right\}$ is now a set of n_t vectors leaf parameters, each of dimension k, of the t-th tree 𝒯_t. Using the stochastic representation of the MSN distribution in (7), the multi-skewBART model of (10) is now expressed as

$$y_i=\sum _{t=1}^{m}g\left(x_i;{𝒯}_t,{ℳ}_t\right)+\Lambda Z_i+W_i,$$ (11)

where $Z_i={\left(Z_{i1},\cdots ,Z_{ik}\right)}^{\top }\stackrel{iid}{~}{𝒩}_{k+}\left(0,I_k\right),W_i={\left(W_{i1},\dots ,W_{ik}\right)}^{\top }\stackrel{iid}{~}{𝒩}_k\left(0,\Sigma \right)$, and the scale-matrix Σ and the skewness matrix Λ are defined as in (6). For the multivariate model, we use the BART framework to consider the association among k response components without adapting to the smoothness of f.

3 |. PRIOR SPECIFICATION AND POSTERIOR COMPUTATION

Previous works have repeatedly observed that the default priors proposed by Chipman et al.⁴ work remarkably well in practice; for example, both Chipman et al.⁴ (for BART) and Linero and Yang⁷ (for SBART) observe that, across a large number of benchmark datasets, the default priors are often competitive with more computationally expensive methods based on tuning the hyperparameters via cross-validation, with the default priors even occasionally outperforming the cross-validation methods. For the tree structures, we use the default prior for the BART model and provide priors for the rest of the parameters of the error distribution in (1). We then derive Gibbs samplers for fitting the models.

3.1 |. skewBART: Prior choices and MCMC computation

As a preprocessing step, we apply a quantile normalization to each of p components of the covariate vectors so that each covariate is approximately uniform on (0, 1). Additionally, following Chipman et al.⁴, the dependent variable y_i is also standardized. The leaf parameters μ_tℓ of the standardized y_i are assumed independent and identically distributed $𝒩\left(0,{\sigma }_{\mu }^2/m\right)$ thus regularizing the effect of the individual tree components to contribute only a small part in the overall fit.

For choosing the bandwidth κ_b, Linero and Yang⁷ recommend using tree specific κ_t’s shared across branches in a fixed tree, with κ_t ~ Exp(0.1). We specify a half-Cauchy prior τ ~ Cauchy₊(0, τ₀), where τ₀ is chosen empirically. We obtain τ₀ by fitting the lasso using the glmnet package in R. We use a conjugate univariate normal 𝒩(0, δ) prior for λ to allow both positive and negative skewness in (8), where a large value of δ corresponds to a non-informative prior opinion about the amount of skewness. In summary, for our univariate response model in (9), the priors are given by f ~ SBART_m and

$$\begin{array}{cc}\hfill \lambda ~𝒩(0,\delta )\hfill & \hfill Z_i~{𝒩}_{+}(0,1),\hfill \\ \hfill W_i~𝒩\left(0,{\tau }^2\right)\hfill & \hfill \tau ~{\text{Cauchy}}_{+}\left(0,{\tau }_0\right).\hfill \end{array}$$

Below we describe our MCMC algorithm with a data augmentation step for fitting skewBART. Our strategy is to augment the latent variables Z = (Z₁, …, Z_n) within the Bayesian backfitting algorithm¹⁶ in order to sample from the posterior distribution

$$p(𝓣,𝓜,\lambda ,\tau ,Z\mid Y)$$

using MCMC, where Y = (y₁, …, y_n)^⊤ for univariate case. Our Gibbs sampling algorithm uses a Metropolis-within-Gibbs strategy to iteratively update (𝒯_t, ℳ_t) for t = 1, …, m in a fashion that leaves the full conditional

$$p\left({𝒯}_t,{ℳ}_t\mid {𝒯}_{-t},{ℳ}_{-t},Z,\lambda ,\tau ,Y\right),$$ (12)

invariant. Then, samples of τ, λ and Z are drawn from the conditional posterior distributions

$$\begin{gathered} p(\tau \mid 𝓣,𝓜,Z,\lambda ,Y), \\ p(\lambda \mid 𝓣,𝓜,Z,\tau ,Y), \\ p(Z\mid 𝓣,𝓜,\lambda ,\tau ,Y). \end{gathered}$$ (13)

Sampling from (𝒩_t, ℳ_t) in (12) is equivalent to sampling from

$$p\left({𝒯}_t,{ℳ}_t\mid R_t,\tau \right)$$ (14)

where R_t = (R_t1, ⋯ , R_tn)^⊤ with $R_{ti}=y_i-{\sum }_{k\ne t}g\left(x_i;{𝒯}_k,{ℳ}_k\right)-\lambda Z_i$. This allows us to use the existing Bayesian backfitting algorithm to update the (𝒯_t, ℳ_t).^4,22 The conditional posterior p(λ | 𝓣, 𝓜, Z, τ, Y) of λ given the rest of the parameters is also Gaussian because it has a form similar to the posterior of the regression parameters of the linear regression model of Y on Z under a Gaussian prior for λ. The full conditionals of the components of Z can be derived in closed form as a collection of independent truncated Gaussian random variables. The Bayesian backfitting algorithm for skewBART is described in Algorithm 1 below, with the exact conditional posterior distributions corresponding to (13) relegated to the Appendix.

3.2 |. multi-skewBART: Prior choices and MCMC computation

To extend the univariate skewBART model to multivariate k-dimensional response y_i, we need to replace the univariate Z_i, μ_tℓ, λ and W_i with corresponding vectors Z_i = (Z_i1, ⋯ , Z_ik)^⊤, μ_tℓ = (μ_1tℓ, ⋯ , μ_ktℓ)^⊤, (λ₁, ⋯ , λ_k) and W_i = (W_i1, ⋯ , W_ik)^⊤. We also need to specify the appropriate priors and hyperpriors for these vectors. For prior p(μ_tℓ | 𝒯_t) we use the conjugate multivariate normal distribution, 𝒩_k(μ_M, Σ_M), which allows the (4) to have a closed form expression as in the univariate BART model. The priors on μ_M and Σ_M are chosen to assign high probability to the range observed values of each of k responses. Similar to the univariate case with hyperparameter τ of W_i, we use an empirical estimate of Σ to choose the center $S_0^{-1}$ and the scale ν₀ of the inverse-Wishert hyper-prior of (k × k) matrix Σ. For skewness parameters (λ₁, …, λ_k), we use a conjugate multivariate normal prior 𝒩_k(0, Σ_λ). In summary, for our proposed multivariate response model in (11), the priors are given by f ~ MultiBART(0.95, 2, Σ_M) and

$$\begin{array}{ll}\hfill \left({\lambda }_1,\dots ,{\lambda }_k\right)~{𝒩}_k\left(0,{\Sigma }_{\lambda }\right),& Z_i~{𝒩}_{k+}\left(0,I_k\right),\hfill \\ \hfill W_i~{𝒩}_k(0,\Sigma ),& \Sigma ~\text{inverse-Wishart}\left(v_0,S_0^{-1}\right).\hfill \end{array}$$

We take Σ_M to be a diagonal matrix with entries given by the default parameters from the respective univariate BART model of Chipman et al.⁴ The Gibbs sampler draws (𝒯_t, ℳ_t) given (𝒯_−t, ℳ_−t, Z, Λ, Σ) from

$$p\left({𝒯}_t,{ℳ}_t\mid R_t,\Sigma \right),$$

where R_t = (R_t1, …, R_tn)^⊤ with $R_{ti}=y_i-{\sum }_{k\ne t}g\left(x_i;{𝒯}_k,{ℳ}_k\right)-\Lambda Z_i$. Then, we sample (λ₁, …, λ_k), Σ and Z from each full conditional distribution given as

$$p\left({\lambda }_1,\cdots ,{\lambda }_k\mid 𝓣,ℳ,Z,\Sigma ,Y\right),$$

$$p\left(\Sigma \mid 𝓣,ℳ,Z,{\lambda }_1,\cdots ,{\lambda }_k,Y\right),$$

$$p\left(Z\mid 𝓣,ℳ,{\lambda }_1,\cdots ,{\lambda }_k,\Sigma ,Y\right).$$

where ℳ = {ℳ₁, …, ℳ_m} and each ℳ_t is a collection of k-dimensional leaf vectors associated with tree 𝒯_t. The MCMC algorithm for this model is similar to the original MCMC scheme for univariate skewBART model, but now extended to the multivariate setting.

4 |. SIMULATION STUDY

In this section, we conduct simulation studies to compare skewBART and multi-skewBART to existing methods. The simulations are designed to evaluate the estimation accuracy of the regression function and out-of-sample predictive performance.

4.1 |. Univariate responses

We first examine the inferences obtained by skewBART compared to BART and SBART when the true error distribution takes on four different skewness levels. For each replicated dataset we simulate n = 250 observations using the nonparametric regression model of (1) with the known population mean function

$$y_i=10\text{ sin}\left(\pi x_{i1}{x}_{i2}\right)+20{\left(x_{i3}-0.5\right)}^2+10x_{i4}+5x_{i5}+{ϵ}_i$$ (15)

introduced by Friedman²³, where x_i ~ Uniform([0, 1]⁵) and ${ϵ}_i\stackrel{iid}{~}𝒮𝒩(0,1,\alpha )$. For each experiment we use 200 trees, 5000 MCMC samples, and a burn-in of 2500 draws. Figure 3 presents density estimates of the distribution of the residuals from fitting the three competing methods overlaid with the true error density function, corresponding to the various choices of α, the skewness parameter. The plots on the left (upper and lower) panels correspond to moderate skewness, i.e., α = −1 and α = +1, while the two on the right (upper and lower) panels represent a high magnitude of skewness, i.e., α = −10 and α = +10. Under both moderate and heavy skewness, the empirical distribution of the residuals from our proposed skewBART model appears closest to the true error distribution.

FIGURE 3.

Simulation study (univariate): Error densities corresponding to four different skewness parameters, α = (1, −1, 10, −10), overlaid with the density estimates from the skewBART, SBART and BART fits.

Next, we compare skewBART to BART when the data generating process varies with skewness levels. We simulate n = 250 observations, with σ² = 1 and use m = 200 trees. The skewness values are equally spaced points in the range (−10, 10), with increment of 1. We set hyperparameters for the priors of trees in BART and SBART following recommendations by Chipman et al⁴, and Linero and Yang⁷. To compare model performance, we use the Conditional Predictive Ordinates (CPO)^24,25, where CPO_i = p(y_i | Y_−i) is the predictive density of the i-th observation y_i given the cross-validated data Y_−i = (y₁, …, y_i−1, y_i+1, …, y_n). For notational convenience, we suppress the dependence of these response distributions on covariates X. A natural summary statistic of the CPO_i’s is the log pseudo marginal likelihood (LPML), given by $\text{LPML}={\sum }_{i=1}^n\text{log }\left({\text{CPO}}_i\right)$. Two competing models ℳ₁ and ℳ₂ can be compared using the pseudo-Bayes factor ${\prod }_{i=1}^n{\text{CPO}}_i^{{ℳ}_1}/{\prod }_{i=1}^n{\text{CPO}}_i^{{ℳ}_2}=\text{exp}\left({\text{LPML}}^{{ℳ}_1}-{\text{LPML}}^{{ℳ}_2}\right)$, where the superscript indicates the model under which these quantities are calculated²⁴. Pseudo-Bayes factors are related to the more well-known Bayes factor^24,26, which may not be appropriate under noninformative priors²⁷. The evidence in the literature supporting the use of pseudo-Bayes factors with BART models is limited; however, LPML is still a very practical tool because (i) unlike Bayes factors, LPML is based on leave-one-out cross-validation (making it more robust than Bayes factors under nonparametric and/or noninformative priors), and (ii) it can be conveniently and reliably computed using MCMC samples from the full posterior.

Using the loo package in R, the LPML can be conveniently computed after obtaining the MCMC samples from the joint posterior. Details on the computation of LPML appear in the Appendix. Results corresponding to each skewness level are presented in Figure 4 with 10 replications per simulation setting. Larger values of LPML indicate a better fit of the model.

FIGURE 4.

Simulation study (univariate): The average values of LPML over 10 replications of the simulation experiment with skewness α ∈ (−10, 10). Each dot represents the averaged LPML corresponding to each skewness level and smoothing splines are added for each method to ease visualization.

Among the methods considered, skewBART performs the best, demonstrating improvement over SBART as skewness increases. For sufficiently large |α|, we observe a substantial increase in the LPML for skewBART, since skewBART can capture the excess skewness through the non-Gaussian error assumption. Also, the performances of the skewBART and SBART models are indistinguishable for low to negligible skewness, implying that skewBART does not suffer in terms of performance when the responses are not skewed.

We note that, counter-intuitively, the LPMLs for all models increase as skewness increases; this occurs because the error variance is proportional to 1∕(1+α²), so that increasing the skewness decreases the overall noise. Generally, skewBART performs much better than the alternatives as skewness increases.

4.2 |. Bivariate responses

In this section, we examine the benefits of adapting skewness for bivariate responses using synthetic data. We generate y_i from the bivariate version of model (1) with k = 2 and p = 5 where, x_i ~ Uniform([0, 1]⁵) and ϵ_i ~ 𝒮𝒩₂(0, Σ, Λ), with the scale matrix Σ and skewness matrix Λ. The function f returns 2-dimensional vector for each 5-dimensional x_i. We also use Friedman’s example for f with N = 250 and σ₁ = σ₂ = 1. Following Arellano-valle et al²⁸, we consider 9 different settings varying with λ₁, λ₂, the skewness parameters, and ρ, the correlation parameter in the bivariate specification, as displayed in Figure 5. We compare multi-skewBART to multi-BART, the multivariate version of the standard BART model, via LPML. For both models we use 200 trees and 5,000 MCMC draws. Results of this simulation are presented in Table 1. The result shows that multi-skewBART outperforms multi-BART with larger LPML under all settings. It appears that both skewness and correlation have an impact on the performance. Overall, higher correlation leads to higher LPML, with the highest LPML attained under (λ₁, λ₂) = (0, 3), with fixed correlation for both models. Thus, we conclude that the multi-skewBART fit is better, and its predictive performance remains highly competitive, regardless of the magnitude of skewness and correlation.

FIGURE 5.

Simulation study (bivariate): The contour plots of the bivariate skew-normal distribution for μ = (0, 0) and σ₁ = σ₂ = 1 with different values of (λ₁, λ₂) and ρ.

TABLE 1.

Simulation study (bivariate): LPML corresponding to the fits of the multi-skewBART and Multi-BART, for various settings of λ₁, λ₂ and ρ.

		multi-skewBART	Multi-BART
(λ1, λ2) = (0, 3)	ρ = 0	−356.677	−360.145
	ρ = 0.5	−330.328	−346.016
	ρ = 0.9	−287.791	−308.521
(λ1, λ2) = (2, 3)	ρ = 0	−396.685	−427.258
	ρ = 0.5	−382.333	−412.068
	ρ = 0.9	−386.478	−400.468
(λ1, λ2) = (−2, 2)	ρ = 0	−369.528	−396.559
	ρ = 0.5	−312.813	−384.575
	ρ = 0.9	−311.486	−325.217

Next, we examined how the LPMLs for skewBART and multi-skewBART are impacted by the number of predictors varying from p =5 to 200. Using Friedman’s example, we simulate n = 250 observations with σ² = 1 and α=3 for skewBART and with ${\sigma }_1^2={\sigma }_2^2=1$, (λ₁, λ₂) = (−2, 2) and ρ = 0.5 for multi-skewBART. With the m = 200 trees, results are given in the top panel of Figure 6, with 10 replications per simulation setting. We see that, as the the number of predictors increases, multi-skewBART is less sensitive to irrelevant predictors than skewBART. This is because the multi-skewBART model shares information about the relevant predictors across the different outcomes via the sharing of the decision trees¹⁴. However, both skewBART and multi-skewBART appear typically insensitive to the number of irrelevant predictors. We note that this is in large part due to our use of the Dirichlet prior proposed by Linero¹², and the robustness to the inclusion of irrelevant predictors we observe is consistent with other simulations done on this problem.

FIGURE 6.

Simulation study: The average values of LPML over 10 replications with p ∈ (5, 200) (top) with fixed m = 200 trees (top) for skewBART and multi-skewBART. The average values of LPML over 10 replications with m ∈ (10, 200) with fixed p = 5 (bottom). Smoothing splines (with corresponding 95% confidence bands) are added for each method to ease visualization.

The number of trees is an important tuning parameter in the BART model. To examine how the number of trees impacts the LPML for skewBART and multi-skewBART, we fix p = 5 and vary the number of trees from 10 to 200. Results are given in the bottom panel of Figure 6, with 10 replications per simulation setting. We see a slight increase in LPML as the number of trees increases to the optimal choice. This result is consistent with what has been observed in other studies^4,14: predictive performance of BART models is typically robust to the number of trees provided we include sufficiently many. We find that this behavior holds for both the skewBART and multi-skewBART models.

5 |. APPLICATION: THE GAAD STUDY

PD primarily results from the inflammation of the gums and bone that surround and support the teeth²⁹. The motivating GAAD study used two correlated biomarkers, the PPD and CAL, of PD status. The CAL is defined as the distance (in mm) down a tooth’s root that is no longer attached to the surrounding bone by the periodontal ligament³⁰. The PPD is defined as the distance (also in mm) from the gingival margin to the bottom of the gingival pocket. The CAL and PPD are measured at six pre-specified tooth sites of each available tooth, excluding the third molars/wisdom teeth³¹. When no teeth are missing, CAL (PPD) measurements are available from 168 sites of the subject. The Joint EU/USA Periodontal Epidemiology Working Group guidelines recommend the average CAL and the average of PPD as standard measurements for studying the prevalence and severity of PD, and for overall tooth level periodontal status in epidemiological studies³². Thus, we consider the bivariate response (PPD_i, CAL_i) for subject i to be the average of the PPD and the average of CAL values across all available sites (maximum of 168 sites). For sake of brevity, we will omit the word average hereafter and instead just use, say, CAL to mean average CAL of the subject. The study also includes various subject-level covariates, such as age (in years), body mass index (BMI, in kg∕m²), glycated haemoglobin level (HbA1c, in %), gender (1 for female, 0 for male), and smoking status (1 for past or present smoker, 0 for never). Our analysis use 288 subjects with complete covariate information, mostly females (about 76%), mean age of about 55 years (range 26–87 years), 31% smokers, and about 68% obese subjects (defined as BMI ≥ 30). The predominance of female subjects in our data is not spurious, and resonates with proportions of Gullah females recruited in other studies³³. Furthermore, the substantial evidence of adverse effects of T2D on PD has been extensively explored in oral health³⁴, and the current study is no exception. The HbA1c is considered a standard of care for testing and monitoring T2D. The American Diabetes Association recommends a target HbA1c level of < 7% (well-controlled), ideally between 4–6, for people with T2D^35,36. In our data, we have 60% subjects having poorly controlled T2D (i.e., with HbA1c ≥ 7), while the rest 40% are well-controlled (not T2D free). Unlike previous approaches based on parametric regression functions³⁰ and semiparametric formulations with skewed errors⁵, our approach incorporates tree-based unknown nonparametric regression function to capture both nonlinear and interaction effects of these covariates. Code for implementing these models is available on GitHub at https://github.com/Seungha-Um/skewBART.

We first present a univariate analysis of the GAAD study by fitting three competing methods (skewBART, BART, and SBART) to the PPD and CAL responses separately. We also consider fitting the models after applying a log transformation to the outcomes, as this is often a popular transformation to ameliorate the skewness and improve the fit of the normal error models. All methods use 5,000 MCMC draws with 200 trees, and we compare their fits via LPML. Results of the model comparison are summarized in Table 2. For both responses, the estimated LPMLs reveal that skewBART outperforms the competing approaches, with or without the log transform — we notice that performing a log transform does result in better LPML, however the skewness of error distributions are still important for model fit even after the log-transform. This suggests that the assumption of a marginal skew-normal error instead of Gaussian error is very appropriate for the analysis of GAAD study. The performance of SBART and BART are similar here; SBART enabling smooth regression function provides only a slight boost compared to BART.

TABLE 2.

GAAD data analysis: Model fit summaries (LPML values) obtained from fitting skewBART, SBART and BART models, separately to the PPD and CAL. The “Transformation” column indicates whether bivariate responses were log-transformed.

	Transformation	skewBART	SBART	BART
CAL	None	−380	−440	−446
	Log	−358	−360	−364
PPD	None	−284	−340	−346
	Log	−272	−277	−279

Now, we apply multi-skewBART to jointly model the bivariate skewed responses, CAL and PPD. We compare the fit of multi-skewBART with multi-BART via LPML, using 5,000 MCMC draws. As expected, the multi-skewBART model allowing different skewness levels for CAL and PPD as well as dependence of PPD and CAL within same subject outperforms (LPML = −614 vs −642) the multi-BART for the bivariate response. Same conclusion holds even for log-transformed responses (LPML = −527 vs −529). We conclude that the multivariate skew-normal assumption improves the modeling of the GAAD study.

Next, we examine how the multivariate model multi-skewBART impacts the prediction accuracy even though both skewBART and multi-skewBART assume SN errors. The multi-skewBART also allows sharing of information across decision trees and captures within-subject association. We compare the skewBART and multi-skewBART models using root mean squared error (RMSE), ${\text{RMSE}}^2={\left(n^{\star }\right)}^{-1}{\sum }_{i=1}^{n^{\star }}{\left\{y_i^{\star }-\widehat{{\zeta }_i^{\star }})\right\}}^2$, of each response. Here $\left\{\left(y_i^{\star },x_i^{\star }\right):i=1,\dots ,n^{\star }\right\}$ denotes a collection of held-out observations of a particular response, say, PPD, and $\widehat{{\zeta }_i^{\star }}$ is the posterior predicted value of these n^⋆ PPD responses based on rest of the data. To make a direct comparison, the responses are standardized and the results are averaged over 20 splits into training and testing sets. Results are presented in Table 3. Although multi-skewBART doesn’t aim for smooth regression functions, it gives the best performance in terms of RMSE. This result implies that the association between CAL and PPD within each subject plays a more essential role than the smoothness levels of the regression function of the GAAD study. Also, multi-skewBART shares information across the different responses by using the same decision trees to generate predictions for both responses. The central message conveyed here is that separately modeling PPD and CAL responses ignoring their bi-directional dependence fails to share information across two responses and compromises the prediction accuracy.

TABLE 3.

GAAD data analysis: Root mean squared error (RMSE) computed over 20 replications for the responses (PPD and CAL) from the skewBART and multi-skewBART fits.

	skewBART	multi-skewBART
CAL	1.345	0.838
PPD	0.626	0.423

As there exists strong evidence of a bi-directional association between T2D and PD^36,37, we now focus on evaluating the association of varying HbA1c levels on PPD and CAL. The marginal effect of HbA1c levels is evaluated by Friedman’s partial dependence function³⁸ which provides a summary of the effect due to the covariates of interest by averaging over the other covariates. If x_i can be partitioned as x_i = (x_iI, x_iC), where x_iI is the set of covariates of interest (HbA1c, Gender, smoking status) and x_iC is the complement set, then the partial dependence of the response y_i at a value $x_I^{*}$ of the covariate of interest is defined as

$${\mu }_I\left(x_I^{*}\right)=\mathbb{E}\left[y_i\mid x_{iI}=x_I^{*}\right]\approx n^{-1}\sum _{i=1}^n\mu \left(x_I^{*},x_{iC}\right)$$

where $\mathbb{E}(\cdot )$ denotes the expectation operator and $\mu (x)=f(x)+\sqrt{2/\pi }{\left({\lambda }_1,\dots ,{\lambda }_k\right)}^{\top }$ is the conditional mean of y_i given covariate vector ($x_I^{*},x_{iC}$) (instead of observed x_i) under skew-normal error. We then compare the posterior mean predictions obtained from fitting skewBART and multi-skewBART models within the four subgroups defined by the combination of gender and smoking status. The results are displayed in Figure 7 with both skewBART and multi-skewBART fits. We observe that with increasing HbA1c levels both responses exhibit an overall increasing trend regardless of the subgroups or the model used. This reconfirms the overall (positive) association between HbA1c and PD. We can also see the effect of smoothing, with the multi-skewBART fit being much more rugged compared to the univariate skewBART fits. For both the skewBART and multi-skewBART fits for PPD, once gender is fixed, the fits corresponding to the smokers and non-smokers appear close; males are predicted to have a much higher PPD than females, regardless of their smoking status or HbA1c levels. We also observe that the effect of smoking is homogeneous with respect to the level of HbA1c. This implies that males are more likely to be prone to active/current PD (PPD representing current/active disease status) than females²⁹, irrespective of smoking status. However, this is not the story from the univariate fit of CAL, where the differences between the smokers and non-smokers appear prominent within genders. The prediction curves are very close between male non-smokers and female smokers for higher levels (say, ≥ 11) of HbA1c for both models.

FIGURE 7.

GAAD data analysis: Plots of the posterior means of the partial dependence functions for CAL and PPD responses as a function of HbA1c corresponding to the 4 subgroups (varying with gender and smoking status) from fitting the skewBART (upper panel), and multi-skewBART (bottom panel) models; age and BMI are averaged over.

In terms of differences between the models, we note that, while skewBART is capable of leveraging smoothness, it produces higher RMSE on test data (see Table 3). This implies that considering the association between responses for the GAAD data is more important in generating good predictions than trying to leverage smoothness in the regression function.

6 |. CONCLUSION

In this paper, we proposed the skewBART model, and extended it to the multi-skewBART model which handles multivariate outcomes. The main idea is to use either a univariate SN density or an MSN density as the error density within the BART framework. We showed that, when the error distribution is skewed, skewBART and multi-skewBART provide better model fits than the original BART model; also, multi-skewBART enjoys additional benefits for multivariate responses due to its ability to account for within-subject dependence of the outcomes and because it uses multivariate decision trees to share information across regression functions. We showed that multi-skewBART produces better model fits on the GAAD data than fitting the two outcomes separately using skewBART.

As pointed out by a reviewer, George et al.³⁹ also considered very flexible extensions to BART by assuming the error distribution to be a Dirichlet process mixture of normals (DPM). This model is capable of addressing skewness, as well as other non-Gaussian features, such as heavy-tailedness of the error. For our purposes, the use of a skew-normal distribution is convenient as it (i) is more parsimonious, allowing for simpler comparisons to non-skewed alternatives; (ii) is less flexible, and hence potentially less prone to over-fitting; and (iii) is easier to extend to multivariate outcomes.

The current work considers two continuous responses, PPD and CAL. However, in practice, data responses (elements in the multivariate response vector) can be of mixed types, such as binary ‘bleeding on probing’ outcomes in PD modeling. In such situations, considering a latent variable, or factor modeling, framework with BART specifications may be worthwhile. Also, the number of available (non-missing) site-level responses (within each subject) can be correlated with the PPD, or CAL responses, leading to the informative cluster size (ICS) scenario⁴⁰, and exploration of BART under ICS is non-existent. Computationally, we have relied on the Gibbs sampler and Metropolis-Hastings for tree updates. In big data settings (such as in observational PD databases), scalable Bayesian methods will likely be required. Finally, multi-skewBART could also use SBART-style smoothing to enforce smoothness. Although we have shown that accounting for the association between responses is more important for our conclusions than trying to leverage smoothness, it may be possible to improve inferences by extending multi-skewBART to incorporate the SBART framework. All of these are important avenues for future work, and will be considered elsewhere.

◻

APPENDIX

A. MCMC FOR SKEWBART

We provide details on the Bayesian backfitting algorithm for the skewBART model. Recall that the model is given by

$$y_i=\sum _{t=1}^{m}g\left(x_i;{𝒯}_t,{ℳ}_t\right)+\lambda Z_i+W_i,W_i\stackrel{iid}{~}𝒩\left(0,{\tau }^2\right),$$

$$\tau ~{\text{Cauchy}}_{+}\left(0,{\tau }_0\right),$$

$$\lambda ~𝒩(0,\delta ).$$

Throughout, we write [V | •] to denote the full-conditional distribution of V. Since Z_i ~ 𝒩₊(0, 1), we update Z_i as

$$\left[Z_i\mid •\right]~{𝒩}_{+}\left(\frac{\lambda R_i^{*}}{{\lambda }^2+{\tau }^2},\frac{{\tau }^2}{{\lambda }^2+{\tau }^2}\right)$$

where $R_i^{*}=y_i-f\left(x_i\right)$.

Define $R^{*}=\left(R_1^{*},\dots ,R_n^{*}\right)$ and Z = (Z₁, …, Z_n). By standard results for Bayesian linear regression, the update of λ is

$$[\lambda \mid •]~𝒩\left(\frac{Z^{\top }{R}^{*}}{Z^{\top }Z+1/\delta },{\left(Z^{\top }Z/{\tau }^2+1/\delta \right)}^{-1}\right)$$

The full conditional distribution for τ is proportional to

$${\left({\tau }^2\right)}^{-n/2}\text{ exp}\left[-\frac{1}{2{\tau }^2}{\left(R^{*}-\lambda Z\right)}^{\top }\left(R^{*}-\lambda Z\right)\right]\frac{1}{\pi \left({\tau }_0+{\tau }^2/{\tau }_0\right)},$$

which can be sampled using (for example) slice sampling⁴¹.

B. MCMC FOR MULTI-SKEWBART

The hierarchical model for multi-skewBART is

$$y_i=\sum _{t=1}^{\top }g\left(x_i,{𝒯}_t,{ℳ}_t\right)+\Lambda Z_i+W_i,W_i\stackrel{iid}{~}{𝒩}_k(0,\Sigma )$$

$${\left({\lambda }_1,\cdots ,{\lambda }_k\right)}^{\top }~{𝒩}_k\left(0,{\Sigma }_{\lambda }\right)$$

$$\Sigma ~\text{inverse-Wishart}\left(v_0,S_0^{-1}\right)$$

where Λ = diag(λ₁, ⋯ , λ_k). Since Z_i ~ 𝒩_k+(0, I_k) by the property of skew Normal distribution, we update Z_i as

$$\left[Z_i\mid •\right]~{𝒩}_{k+}{\left({\left(\Lambda {\Sigma }^{-1}\Lambda +I_k\right)}^{-1}\Lambda {\Sigma }^{-1}{R}_i^{*},\Lambda {\Sigma }^{-1}\Lambda +I_k\right)}^{-1}),$$

where $R_i^{*}=y_i-{\sum }_{t}g\left(x_i,{𝒯}_t,M_t\right)$. Here 𝒩_k+(μ, Σ) is a k-dimensional normal distribution with location vector μ, covariance matrix Σ and truncated in the positive k-dimensional quadrant ${ℝ}^{k+}$. The conditional posterior to update (λ₁, ⋯ , λ_k) is

$$\left[\left({\lambda }_1,\cdots ,{\lambda }_k\right)\mid •\right]~{𝒩}_k\left({\left(\sum _{i=1}^n{M}_i{\Sigma }^{-1}{M}_i+{\Sigma }_{\lambda }^{-1}\right)}^{-1}\sum _{i=1}^n{M}_i{\Sigma }^{-1}{R}_i^{*},{\left(\sum _{i=1}^n{M}_i{\Sigma }^{-1}{M}_i+{\Sigma }_{\lambda }^{-1}\right)}^{-1}\right)$$

where M_i = diag(Z_i1, ⋯ , Z_ik). The conditional posterior to update Σ is

$$\Sigma ~\text{inverse-Wishart}\left(v_0+n,{\left(B^{\top }B+S_0\right)}^{-1}\right)$$

where B is a matrix whose kth row is ${\left(R_i^{*}-\Lambda Z_i\right)}^{\top }$. The Bayesian backfitting algorithm for fitting multi-skewBART is now essentially the same as the algorithm for skewBART but with the updates of (Σ, {Z_i}, Λ) replacing the updates for (τ, {Z_i}, λ). The only remaining consideration is computing the integrated likelihood, which we do in the following section.

C. THE INTEGRATED LIKELIHOOD FOR MULTI-SKEWBART

To update 𝓣, we require the conditional posterior of 𝒯_t given the rest of the parameters. The derivation of this conditional posterior for multi-skewBART is actually a generalization of the derivation of the conditional posterior of 𝒯_t for univariate skewBART, and we provide only a sketch of the derivation of this conditional posterior for multi-skewBART. Let ℒ_t denote the collection of leaf nodes of tree t and let [x ⇝ ℓ] mean that the covariate value x is associated to leaf node ℓ of tree t. The conditional posterior is

$$\begin{gathered} p\left({𝒯}_t\mid R_t,\Sigma \right)=p_{𝒯}\left({𝒯}_t\right)\int \prod _{i=1}^n{\varphi }_k\left(Y_i\mid f\left(X_i\right)+\Lambda Z_i,\Sigma \right)\left[\prod _{\ell \in {ℒ}_t}{\varphi }_k\left({\mu }_{tl}\mid 0,D\right)\right]d{\mu }_{t\ell } \\ =p_{𝒯}\left({𝒯}_t\right)\prod _{\ell \in {ℒ}_t}\int \prod _{i:X_i\to \ell }\left\{{\varphi }_k\left(R_{ti}\mid {\mu }_{t\ell },\Sigma \right)\right\}{\varphi }_k\left({\mu }_{t\ell }\mid 0,D\right)d{\mu }_{t\ell }, \end{gathered}$$

where ϕ_k(U | μ, Σ) is the multivariate 𝒩_k(μ, Σ) density and μ_tℓ has 𝒩_k(0, D) prior. This integrated likelihood can be computed in closed form easily by using standard properties of the multivariate Gaussian distribution. Additionally, by conjugacy of the multivariate normal distribution, we have the full conditionals

$${\mu }_{t\ell }~{𝒩}_k\left({\left(D^{-1}+N_{\ell }{\Sigma }^{-1}\right)}^{-1}{\Sigma }^{-1}\sum _i{\mathit{R}}_{ti},{\left(D^{-1}+N_{\ell }{\Sigma }^{-1}\right)}^{-1}\right),$$ (C1)

where N_ℓ is the number of observations associated to leaf ℓ of tree t. We adopt the same Metropolis-Hastings steps for modifying the tree structure form as with skewBART.

D. COMPUTATION OF LPML

The logarithm of the pseudo-marginal likelihood (LPML)^24,42 is given by $\text{LPML}={\sum }_{i=1}^n\text{log}\left({\text{CPO}}_i\right)$, where CPO_i = p(y_i | Y_−i) is the predictive density of observed y_i given rest of the observed data Y_−i (suppressing dependence on X). Due to a simplifying result²⁴, a Monte Carlo approximation of CPO_i can be obtained using samples θ¹, …, θ^S from the joint posterior p(θ | Y) given full observed data Y as

$$p\left(y_i\mid Y_{-i}\right)={\left[\int \frac{p(Y\mid \theta )p(\theta )}{p(Y)p\left(y_i\mid \theta \right)}d\theta \right]}^{-1}\approx \frac{1}{\frac{1}{S}{\sum }_{s=1}^S{\left\{p\left(y_i\mid {\theta }^s\right)\right\}}^{-1}},$$

where θ denotes the collection of all model parameters. The above approximation of CPO_i is a type of harmonic mean estimator, which are generally known to be unstable (possibly having infinite variance). This approximation of CPO_i can be improved by Pareto smoothed importance sampling (PSIS)⁴³. PSIS applies a smoothing procedure to the importance weights by replacing the largest importance sample ratios with the expected order statistics of the fitted generalized Pareto distribution. The reliability of the approximations can be assessed by the estimated shape parameter $\widehat{k}$ of the generalized Pareto distribution. The loo package suggests that approximations of our CPO_i with $\widehat{k}<0.7$ are reliable.