Multinomial probit Bayesian additive regression trees

Abstract

This article proposes multinomial probit Bayesian additive regression trees (MPBART) as a multinomial probit extension of BART - Bayesian additive regression trees. MPBART is flexible to allow inclusion of predictors that describe the observed units as well as the available choice alternatives. Through two simulation studies and four real data examples, we show that MPBART exhibits very good predictive performance in comparison to other discrete choice and multiclass classification methods. To implement MPBART, the R package mpbart is freely available from CRAN repositories.

1. Introduction

Multinomial probit (MNP) model for discrete choice modeling is often used in economics, market research, political sciences and transportation. It models the choices made by agents given their demographic characteristics and/or the features of the available choice alternatives. In this article, we focus on cases where there are at least three choices. Examples include the study of consumer purchasing behavior (McCulloch et al., 2000; Imai & van Dyk, 2005); voting behavior in multi-party elections (Quinn et al., 1999); and choice of different modes of transportation (Bolduc, 1999). Details of the MNP model in which choices depend on predictors in a linear fashion is studied in McFadden et al. (1973); McFadden (1989); Keane (1992); McCulloch & Rossi (1994); Nobile (1998); McCulloch et al. (2000); Imai & van Dyk (2005); Train (2009); Burgette & Nordheim (2012) among others.

Among widely used multinomial choice modeling procedures are the multinomial logit model (McFadden et al., 1973; Train, 2009) and multinomial probit model (McFadden, 1989; McCulloch & Rossi, 1994; Imai & van Dyk, 2005). The former relies on an assumption that a choice outcome is independent of removal (or introduction) of an irrelevant choice alternative while the latter including MPBART does not make this restrictive assumption. In the multinomial probit regression framework, it is assumed that each decision maker faced with K ≥ 3 alternatives uses a (K – 1) vector of latent variables in order to arrive at their choice. Alternative k is chosen if the k^th entry of the latent vector is positive and greater than the other entries, for k = 1, . . . , (K – 1). If none of the entries of the latent vector are positive, then the “reference” alternative K is chosen.

MPBART can also be used as a multiclass classification procedure to classify units into one of K ≥ 3 classes based on their observed characteristics. Multiclass classification is common in many disciplines. In biology, tumors are classified into tumor sub-types based on their gene expression profiles (Khan et al., 2001). In environmental sciences, clouds are classified as clear, liquid clouds, or ice clouds based on their radiance profiles (Lee et al., 2004). Other areas of multiclass classification application include text recognition, spectral imaging, chemistry, and forensic science (Li et al., 2004; Fauvel et al., 2006; Evett & Spiehler, 1987; Vergara et al., 2012).

The effect of predictors on the response may be linear or non-linear, of much or little significance, and at times magnified with interactions. When such complicated relationships exist, models that use ensemble of trees often provide appealing framework since variable selection and inclusion of interactions are intrinsic in construction of trees. Some popular “tree-based” classification methods include CART (Breiman et al., 1984; Quinlan, 1986), Bayesian CART (Chipman et al., 1998), random forests (Breiman, 2001), and gradient boosting (Friedman, 2001). There is a gap in the literature for “tree based” statistical procedures that directly deal with the MNP model in which choice specific predictors can readily be incorporated. This article, thus, seeks to fill that void using Bayesian tree ensembles for multinomial probit regression.

A newcomer to the “tree-based” family is the Bayesian additive regression trees (BART) (Chipman et al., 2010). The innovative idea of BART is to approximate an unknown function f(x) for predicting a continuous variable z given values of input x using a sum-of-trees model

$$f\left(\mathbf{x}\right)\approx \sum _{j=1}^{n_{\mathrm{T}}}g(\mathbf{x},T_j,M_j),$$

where g(x, T_j, M_j) is the j^th tree that consists of sets of partition rules T_j and parameters M_j associated with its terminal nodes. Conceptually, the sum-of-trees structure makes BART adaptive to complicated nonlinear and interaction effects, and the use of Bayesian regularization prior on regression trees minimizes the risk of over-fitting. Empirically, a variety of experiments and applications of BART has confirmed that it has robust and accurate out-of-sample prediction performance (Liu & Zhou, 2007; Chipman et al., 2010; Abu-Nimeh et al., 2008; Bonato et al., 2011). The standard BART further extends to binary classification problems and shows competitive classification performance (Zhang & Härdle, 2010; Chipman et al., 2010).

The success of BART on predicting continuous and binary variables naturally motivates the question of whether the sum-of-trees structure also helps in predicting multinomial choices and classes, thus, we are interested in the utility of the sum-of-trees for discrete choice modeling. We utilize a Bayesian probit model formulation in Albert & Chib (1993); McCulloch & Rossi (1994); McCulloch et al. (2000); Imai & van Dyk (2005) in conjunction with the idea of sum-of-trees regression to propose multinomial probit Bayesian additive regression trees (MPBART). Through a comprehensive simulation study with various data generating schemes, we find that it is a serious contender in its predictive performance to existing multinomial choice models and multiclass classification methods and that it usually ranks among the topmost when a nonlinear relationship exists between the predictors and choice alternatives.

A related work to this article is Agarwal et al. (2013), which utilizes BART for the purpose of satellite image classification. Their multiclass classification procedure combines binary BART and one-versus-all technique of transforming a multiclass problem to a series of binary classification problems. Our work is different from theirs in that we consider the problem within the traditional multinomial probit regression framework rather than the one-versus-all framework.

The article proceeds as follows. Section 2 formally outlines the multinomial probit model in general and MPBART in particular along with the associated data structure, Section 3 delves into the prior specifications and posterior computation for MPBART. Sections 4 and 5 use simulated data sets and real data examples, respectively to illustrate the predictive performance of MPBART. Section 6 closes the article with concluding remarks.

2. MPBART: Multinomial probit Bayesian additive regression trees

Suppose we have a data set (y_i, X_i) for i = 1, . . . , n, where y_i ∈ {1, . . . , K} denotes the available choice alternatives and X_i the predictors for the i^th observation. We are interested in estimating the conditional choice probability p(y_i = k | X_i) for k = 1, . . . , K. The observed choice y_i can be viewed as arising from a vector of latent variables ${\mathbf{z}}_i\in {\Re }^{K-1}$ as in Albert & Chib (1993); Geweke et al. (1994); Imai & van Dyk (2005) via

$$y_i\left({\mathbf{z}}_i\right)=\{\begin{array}{cc}k\hfill & \text{if}\max \left({\mathbf{z}}_i\right)=z_{ik}>0,\hfill \\ K\hfill & \text{if}\max \left({\mathbf{z}}_i\right)<0,\hfill \end{array}\phantom{\}}$$ (1)

for k = 1, . . . , (K – 1), where max (z_i) denotes the largest element of z_i = (Z_i1, . . . , Z_i,K–1)′. The latent vector z_i depends on X_i as follows:

$${\mathbf{z}}_i=\mathbf{G}({\mathbf{X}}_i;\mathbf{T},\mathbf{M})+{ϵ}_i\text{for}i=1,\dots ,n,$$ (2)

where G(X_i; T, M) = (G₁(X_i; T, M), . . . , G_K–1(X_i; T, M))′ is a vector of K – 1 regression functions of ∈_i = (∈_i1, . . . , ∈_i,K–1)′ ~ N(0, 𝚺).

The predictors for the i^th observation are comprised of two compoments v_i and W_i (i.e., X_i = (v_i, W_i) ). The first component is a vector of q - demographic variables ${\mathbf{v}}_i\in {\Re }^q$ that describe the subject. The second component W_i = (w_i1, . . . , w_i(K–1))), where ${\mathbf{w}}_{ik}\in {\Re }^r$, is a matrix of r predictors that vary along the choice alternatives in relation to the reference choice. For example, in a market research scenario, the price of the choices faced by individuals in a study is a choice specific predictor that varies along alternatives and the difference between the prices of k^th choice and the reference choice K will be part of w_ik, for k = 1, . . . , (K – 1).

The tree splitting rules of the k^th sum of trees

$$G_k({\mathbf{X}}_i;\mathbf{T},\mathbf{M})=\sum _{j=1}^{n_{\mathrm{T}}}g({\mathbf{X}}_i,T_{kj},M_{kj}),\text{for}k=1,\dots ,(K-1)$$ (3)

depend on X_i through x_ik = (v_i, w_ik). The j^th tree of the k^th sum of trees, g(·, T_kj, M_kj), consists to T_kj, a set of partition rules based on the predictor space, and M_kj = {μ_kjl, l = 1, . . . , b_kj}, a set of parameters associated with the terminal nodes. The partition rules T_kj are recursive binary splits of the form {x < s} versus {x ≥ s}, where x is one of the predictors that make up x_ik, and s is a value in the range of x. The complete set of parameters of MPBART (1)–(3) is thus

$$\left\{{(T_{kj},M_{kj})}_{k=1,\dots ,(K-1),j=1,\dots ,n_T},\Sigma \right\},$$

where M_kj denotes the collection of terminal nodes of the j^th tree in the k^th sum-of-trees.

3. Prior specifications and posterior computation

3.1. Prior specifications

3.1.1. The 𝚺 prior

The MNP model specification in (2) exhibits a well documented identifiability issue, for example the multiplication of both sides of (2) by a positive constant does not alter the implied choice outcome (Keane, 1992; McCulloch & Rossi, 1994; McCulloch et al., 2000; Nobile, 1998). To circumvent this issue, McCulloch & Rossi (1994); McCulloch et al. (2000); Imai & van Dyk (2005) among others restrict the first diagonal element of 𝚺 to equal one, while Burgette & Nordheim (2012) restricts the trace of 𝚺 to equal K. We implement the latter.

Consider an augmented latent model

$${\tilde{\mathbf{z}}}_i=\mathbf{G}({\mathbf{X}}_i;\mathbf{T},\tilde{\mathbf{M}})+{\widetilde{ϵ}}_i,$$ (4)

where ${\tilde{\mathbf{z}}}_i=\alpha {\mathbf{z}}_i$, ${\widetilde{ϵ}}_i=\alpha {ϵ}_i$, ${\widetilde{ϵ}}_i\sim \mathrm{N}(0,\tilde{\Sigma })$, $\tilde{\Sigma }={\alpha }^2\Sigma$ and ${\tilde{M}}_{kj}=\{\alpha {\mu }_{kjl},l=1,\dots ,b_{kj}\}$. Following Imai & van Dyk (2005); Burgette & Nordheim (2012), we place the prior $p\left(\Sigma \right)=\int p(\Sigma ,{\alpha }^2)p({\alpha }^2\mid \Sigma )d{\alpha }^2\propto {\mid \Sigma \mid }^{-\frac{(v+K)}{2}}{\left(\mathrm{tr}\left[\mathrm{S}{\Sigma }^{-1}\right]\right)}^{-\frac{v(K-1)}{2}}$, with a restriction tr(𝚺) = K; a constrained inverse Wishart distribution induced by $\tilde{\Sigma }\sim$ Inv-Wish $(\nu ,{\alpha }_0^{2}S)$ and ${\alpha }^2\mid \Sigma \sim {\alpha }_0^2\mathrm{tr}\left[S{\Sigma }^{-1}\right]∕{\chi }_{v\left(K\right)}^2$.

3.1.2. The T_kj prior

As in Chipman et al. (1998) and Chipman et al. (2010), the prior on a single tree T_kj is specified through a “tree-generating stochastic process” apriori independent of 𝚺. The tree prior consists of (i) the probability of splitting a terminal node, (ii) the distribution of the splitting variable if the node has to split, and (iii) the distribution of the splitting rule given the splitting variable. For step (i), the probability that a terminal node η splits is given by

$$\frac{\gamma }{{(1+d_{\eta })}^{\beta }},\gamma \in (0,1),\beta \in [0,\infty ),$$

where d_η is the depth of the node. A small γ and a big β result in a tree with small number of terminal nodes. In other words, influence of individual trees in the sum can be controlled by carefully choosing γ and β. For step (ii), the splitting variable is uniformly selected from all possible predictors, representing a prior belief of equal level of importance placed on each predictor. For step (iii), given a splitting predictor, the splitting value s is taken to be a random sample from discrete uniform distribution of the set of observed values of the selected predictor, provided that such a value does not result in an empty partition.

3.1.3. The μ_kjl|T_kj prior

Given a tree T_kj with b_kj terminal nodes, the prior distribution on the terminal node parameters is taken to be

$${\mu }_{kjl}\mid T_{kj}\stackrel{iid}{\sim }\mathrm{N}({\mu }_k,{\tau }_k^2)\text{for}k=1,\dots ,(K-1).$$

For binary classification probelems (i.e., K = 2), Chipman et al. (2010) propose choosing μ₁ = 0 and ${\tau }_1=3∕\left(r\sqrt{n_T}\right)$ so that the sum-of-tree effect ${\sum }_{j=1}^{n_{\mathrm{T}}}g(\mathbf{x},T_{1j},{\mu }_{1j})$ assigns high probability to the interval (−3,3). We extend their method to the multinomial probit setting by assuming μ_k = 0 and ${\tau }_k=3∕\left(r\sqrt{n_T}\right)$ for all k. The hyper-parameters r and n_T play the role of adjusting the level of shrinkage on the contribution of each individual tree. Default values r = 2 and n_T =200 are recommended by Chipman et al. (2010) which we also find reasonable in the multinomial probit setup.

3.2. Posterior computation

Our posterior sampling scheme relies on the partial marginal data augmentation strategy van Dyk (2010). Marginal data augmentation (MDA) and partial marginal data augmentation (Meng & van Dyk, 1999; Imai & van Dyk, 2005; van Dyk, 2010; Burgette & Nordheim, 2012) introduce a “working parameter” that is identifiable given an augmented data, but not identifiable given the observed data. By strategically augmenting the data, MDA and partial MDA result in a computationally tractable posterior distribution and an MCMC chain with improved convergence.

Our posterior computing is accomplished via cycling through the following three steps (for convenience the intermediate draws are flagged with an asterisk).

• (i)Sample from (z, α²) | T, M, 𝚺, y by obtaining random draws of p{(z_i)_i=1,...,n | T, M, ∑, y}, and (α*)² ~ p{α² | 𝚺, M, T, (z_i)_i=1,...,n} = p{α² | 𝚺} followed by transforming to obtain ${\tilde{\mathbf{z}}}_i^{\ast }={\alpha }^{\ast }{\mathbf{z}}_i$ for all i.
• (ii)Sample from $(\mathbf{T},{\tilde{\mathbf{M}}}^{\ast })\sim p\{\mathbf{T},\tilde{\mathbf{M}}\mid {\left({\tilde{\mathbf{z}}}_i^{\ast }\right)}_{i=1,\dots ,n},\Sigma ,{\left({\alpha }^{\ast }\right)}^2,\mathbf{y}\}$ followed by recording $\mathbf{M}={\tilde{\mathbf{M}}}^{\ast }∕{\alpha }^{\ast }$.
• (iii)Sample from $(\Sigma ,{\alpha }^2)\sim p\{\Sigma ,{\alpha }^2\mid \mathbf{T},{\tilde{\mathbf{M}}}^{\ast },{\left({\tilde{\mathbf{z}}}_i^{\ast }\right)}_{i=1,\dots ,n},\mathbf{y}\}$ by random draws of $p\{{\tilde{\Sigma }}^{\ast }\mid \mathbf{T},{\tilde{\mathbf{M}}}^{\ast },{\left({\tilde{\mathbf{z}}}_i^{\ast }\right)}_{i=1,\dots ,n},\mathbf{y}\}$ followed by transforming ${\tilde{\Sigma }}^{\ast }$ to (𝚺, α²).

Our algorithm utilizes a “partial marginalization” strategy since the working parameter α² is updated in steps (i) and (iii), but not in (ii) (cf. the marginalization strategy in Imai & van Dyk (2005) where the working parameter is updated in every step).

The first part of obtaining a sample from (i) is iterative random draws of truncated normals from the conditional distribution z_ik | z_i(–k), T, M, 𝚺 ~ N (m_ik, ψ_ik) with max {0, max(z_i(–k))} as a lower truncation point if y_i = k and as an upper truncation point of if y_i ≠ k. The conditional first moment and variance m_ik, and ψ_ik are given by

$$\begin{array}{cc}\hfill m_{ik}=& {\mathbf{G}}_k({\mathbf{X}}_i;\mathbf{T},\mathbf{M})+{\sigma }_{\mathbf{k}(\mathbf{-}\mathbf{k})}{\Sigma }_{(\mathbf{-}k)(\mathbf{-}\mathbf{k})}^{-1}[{\mathbf{z}}_{i(-k)}-{\mathbf{G}}_{\left(-\mathbf{k}\right)}({\mathbf{X}}_i;\mathbf{T},\mathbf{M})],\text{and}\hfill \\ \hfill {\psi }_{ik}=& {\sigma }_{kk}-{\sigma }_{k(-k)}{\Sigma }_{(-k)(-k)}^{-1}{\sigma }_{k(-k)}^{\prime },\hfill \end{array}$$ (5)

where σ_k(–k) is the k^th column of 𝚺 that excludes σ_kk and 𝚺_(–k)(–k) is the matrix 𝚺 that excludes the k^th column and row.

For (ii), we sample $(T_{kj},{\tilde{M}}_{kj}^{\ast })$ for k = 1, . . . , (K – 1), j = 1, . . . , n_T via the following. Given all the trees and their terminal node parameters but the j^th tree in the k^th sum of trees, $\tilde{\Sigma },{\tilde{\mathbf{z}}}_{i(-K)}^{\ast }$ and ${\left({\alpha }^{\star }\right)}^2$, we observe that

$${\tilde{\mathbf{z}}}_{ij}^{†}=g({\mathbf{X}}_i,T_{kj},{\tilde{M}}_{kj})+{\widetilde{ϵ}}_{ik}^{†},{\widetilde{ϵ}}_{ik}^{†}\sim \mathrm{N}(0,{\tilde{\psi }}_{ik}),$$ (6)

where ${\tilde{\mathbf{z}}}_{ik}^{†}={\tilde{\mathbf{z}}}_{ik}^{\ast }-{\sum }_{l\ne j}^{n_{\mathrm{t}}}g({\mathbf{X}}_i,T_{kl},{\tilde{M}}_{kl})-{\tilde{\sigma }}_{k(-k)}{\tilde{\Sigma }}_{(-k)(-k)}^{-1}[{\tilde{\mathbf{z}}}_{i(-k)}^{\ast }-{\mathbf{G}}_{(-k)}({\mathbf{X}}_i;\mathbf{T},\tilde{\mathbf{M}})]$ and ${\tilde{\psi }}_{ik}={\left({\alpha }^{\ast }\right)}^2{\psi }_{ik}$. We use the back-fitting algorithm, also used in Chipman et al. (2010), to obtain posterior samples of $(T_{kj},{\tilde{M}}_{kj}^{\ast })$ by considering (6) as the single tree model of Chipman et al. (1998). Finally, the posterior sample in (iii) is ${\tilde{\Sigma }}^{\ast }\sim \text{Inv-Wish}(\nu +n,\tilde{S}+{\sum }_{i=1}^n[{\tilde{\mathbf{z}}}_i^{\ast }-\mathbf{G}({\mathbf{X}}_i;\mathbf{T},{\tilde{\mathbf{M}}}^{\ast })][{\tilde{\mathbf{z}}}_i^{\ast }-\mathbf{G}({\mathbf{X}}_i;\mathbf{T},{\tilde{\mathbf{M}}}^{\ast })])$ then taking α² as $\mathrm{tr}\left({\tilde{\Sigma }}^{\ast }\right)∕K$ and transforming to obtain $\Sigma ={\tilde{\Sigma }}^{\ast }∕{\alpha }^2$.

3.3. Posterior-based prediction

In our Bayesian setting, predictions of future observations y^⋆ at new values X^⋆ are based upon the posterior predictive distribution $p(y^{\star }\mid \mathbf{y})=\int p(y^{\star }\mid \mathbf{X},\ominus ,\mathbf{y})p(\ominus ,\mid \mathbf{y})\mathrm{d}\ominus$, where $\ominus$ consists of full unknown parameters of MPBART. For a given loss function, predictions of y^⋆ are made using the optimal choice a ∈{1, . . . , K} that minimizes the expected posterior predictive loss

$$E_{y^{\star }\mid \mathrm{y}}L(y^{\star },a)=\int L(y^{\star },a)p(y^{\star }\mid \mathbf{y})\mathrm{d}{y}^{\star },$$

where L (y^⋆, a) is the loss function of using class a to predict the unknown choice outcome y^⋆. We assume that the loss function L(y, a) assigns a pre-specified non-negative loss to every combination of action a ∈ {1, . . . , K} and true choice y ∈ {1, . . . , K}. These pre-specified loss combinations are described in Table 1 and can equivalently be expressed as

$$L(y,a)=\sum _{l=1}^K\sum _{m=1}^K{C}_{lm}I(y=l,a=m),$$ (7)

where I (·) is the usual indicator function.

Table 1.

Pre-specified costs for the loss function L(y, a).

		Prediction a
Loss		1	2	...	K
True Choice y	1	C 11	C 12	...	C 1K
	2	C 21	C 22	...	C 2K
	⋮	⋮	⋮	⋮	⋮
	K	CK 1	CK 2	...	CKK

Under the loss function (7), the expected posterior predictive loss is given by

$$E_{y^{\star }\mid \mathrm{y}}L(y^{\star },a)=\sum _{l=1}^K{C}_{la}p(y^{\star }=l\mid \mathbf{y}).$$ (8)

We assume that the costs associated with a wrong prediction are all equal to the constant C and correct prediction costs equal to 0 (i.e., C_lm = C > 0 for l ≠ m, and C_ll = 0). Then the expected posterior predictive loss (8) simplifies to E_y^⋆|yL(y^⋆, a) = C{1 – p(y^⋆ = a|y)}, which is minimized at

$$a=\underset{k}{\arg \max }\{p(y^{\star }=k\mid \mathbf{y}),k=1,\dots ,K\}.$$ (9)

The posterior predictive distribution p(y^⋆ = l | y) does not have closed form representation and is thus approximated using Monte Carlo samples drawn from the posterior distributions p(Θ | y). Once computed, they enable the estimation of the predictions 9 through a search over the space a ∈ {1, . . . , K}.

4. Synthetic data examples

4.1. A simulation study in the multinomial choice framework

In this three choice simulation study, we use a function similar to the one used in Friedman (1991) to induce a nonlinear relationship between five choice specific predictors ${\mathbf{w}}_k\in {\Re }^5,k=1,2,3$ and the choice alternatives. The choice specific predictors are from i.i.d Unif[0,1]. In addition, we include a predictor $v\stackrel{iid}{\sim }\mathrm{Unif}[0,2]$ that describes the observed unit. Suppose that f(u) = 20sin(πu₁u₂) – 20(u₃ – 0.5)² + 10u₄ + 5u₅, g(v) = 8v, and

$$\left[\begin{array}{c}\hfill z_1\hfill \\ \hfill z_2\hfill \end{array}\right]=\left[\begin{array}{c}\hfill f({\mathbf{w}}_1-{\mathbf{w}}_3)+g\left(v\right)\hfill \\ \hfill f({\mathbf{w}}_2-{\mathbf{w}}_3)+g\left(v\right)\hfill \end{array}\right]+ϵ,ϵ\sim \mathrm{N}\left(0,\left[\begin{array}{cc}\hfill 1\hfill & \hfill 0.5\hfill \\ \hfill 0.5\hfill & \hfill 1\hfill \end{array}\right]\right).$$ (10)

The response variable is then recorded using

$$y\left(\mathbf{z}\right)=\{\begin{array}{cc}k\hfill & \text{if}\max \left(\mathbf{z}\right)=z_k>0,\hfill \\ 3\hfill & \text{if}\max \left(\mathbf{z}\right)<0,\hfill \end{array}\phantom{\}}\text{for}k=1,2.$$

This true model contains linear, nonlinear, and interaction effects, making it interesting benchmark data set. We are mainly interested in how well MPBART is able to predict the choices on a test data. Hence, we simulate training and test data sets of 500 observations each and compare the predictive performance on the test data for MPBART, Bayesian multinomial probit model (Bayes-MNP) in Imai & van Dyk (2005), the Multinomial logit (MNL) model in Train (2009); McFadden et al. (1973), and the following multiclass classification procedures: support vector machines with linear (SVM-L) and radial (SVM-R) kernels in Cortes & Vapnik (1995); Vapnik (2013), random forest (RF) in Breiman (2001), linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA) in Duda et al. (2012); Friedman et al. (2001), multinomial logistic regression (MNL) in McFadden et al. (1973), classification and regression trees (CART) in Breiman et al. (1984); Quinlan (1986), neural networks (NNET) in Ripley (2007), K-nearest neighbors (KNN) in Cover & Hart (1967) and One vs. All BART (OvA-BART) in Agarwal et al. (2013). We note that for the multiclass classification procedures, a choice specific predictor makes up three separate predictors in this simulation study, one describing each of the choices, putting the total number of predictors for this simulation study at sixteen. For each competing procedure and MPBART, we selected the tuning parameters via a 10-fold cross-validation based on the training data. Table 2 lists the names of these competing procedures, the corresponding R packages utilized and tuning parameters.

Table 2.

List of competing classifiers, the R packages utilized, and tuning parameters that are chosen by cross-validation. The abbreviations in the first column stand for the procedures mentioned in the second paragraph of Section 4.2.

Procedure	R Package	Tuning parameter(s)
RF	randomForest	mtry
CART	rpart	no tuning parameters
SVM-L	kernlab	C
SVM-R	kernlab	C and σ
QDA	MASS	no tuning parameters
LDA	MASS	no tuning parameters
NNET	nnet	size and decay
MNL	mlogit	no tuning parameters
KNN	caret	k
OvA-BART	dbarts	k, power, base

The comparison metric we use in this example and all that follow is test error rate

$$\frac{1}{m}\sum _{i=1}^{m}I({\widehat{y}}_i\ne y_i),$$ (11)

where y_i and ŷ_i are the actual and predicted classes for the i^th observation in a given test data set of size m. This metric makes use of the loss function in (7) with a misclassification cost of C_lm = 1 and a cost of C_ll = 0 for a correct prediction. As can be seen from Table 3, MPBART exhibits a very good out-of-sample predictive accuracy. This is not surprising given the data generating scheme with nonlinear effects.

Table 3.

Comparison of MPBART, and the procedures listed in Table 2 on the first simulation study generated via (10) and the waveform recognition example (12). Training and test data sets of each 500 observations are used for the first simulation study. Training and test data sets of 300 and 500 observations, respectively are used for the waveform recognition example. Average test error rates (with standard errors in parentheses) are reported on 20 replications.

Procedure	Simulation Study - I		Waveform Recognition
	Test Error Rate	Rank	Test Error Rate	Rank
MPBART	0.2725 (0.0060)	1	0.1589 (0.0047)	2
Bayes-MNP	0.3976 (0.0065)	7	0.2167 (0.0197)	11
MNL	0.3921 (0.0064)	6	0.1721 (0.0052)	5
RF	0.4023 (0.0059)	8	0.1676 (0.0043)	3
CART	0.4791 (0.0080)	12	0.3113 (0.0068)	12
SVM-L	0.4072 (0.0058)	9	0.1844 (0.0043)	6
SVM-R	0.3254 (0.0057)	3	0.1708 (0.0053)	4
LDA	0.4095 (0.0064)	10	0.1997 (0.0048)	8
QDA	0.3381 (0.0045)	4	0.2125 (0.0043)	10
NNET	0.2917 (0.0065)	2	0.2012 (0.0071)	9
KNN	0.4195 (0.0070)	11	0.1847 (0.0048)	7
OvA-BART	0.3908 (0.0059)	5	0.1550 (0.0035)	1

4.2. A simulation study for multiclass classification

In this simulation study the waveform recognition problem in Breiman et al. (1984), often used as a benchmark artificial data in multiclass classification studies (Gama et al., 2003; Hastie & Tibshirani, 1996; Keerthi et al., 2005) is employed. The model has 21 predictors and a multiclass response with 3 classes. For each observation, the i^th predictor x_i is generated from

$$x_i=\{\begin{array}{cc}u{h}_1\left(i\right)+(1-u)h_2\left(i\right)+{ϵ}_i,\hfill & \text{if}y=1,\hfill \\ u{h}_1\left(i\right)+(1-u)h_3\left(i\right)+{ϵ}_i,\hfill & \text{if}y=2,\hfill \\ u{h}_2\left(i\right)+(1-u)h_3\left(i\right)+{ϵ}_i,\hfill & \text{if}y=3,\hfill \end{array}\phantom{\}}$$ (12)

where i = 1, . . . , 21, u ~ Unif[0, 1], ∈_i ~ N(0, 1), and h_i are three waveform functions: h₁(i) = max(6 – |i – 11|, 0), h₂(i) = h₁(i – 4), and h₃(i) = h₁(i + 4).

We generate 20 replications of training and test data sets with 300 and 500 observations, respectively from (12) and compare MPBART with classifiers listed in Table 2. Our choice of sample sizes is the same as those in Hastie & Tibshirani (1996) so the results can be compared with them. Table 3 summarizes the average error rates and standard errors in parentheses based on 20 simulations. For LDA, QDA and CART, the error rates are consistent with those reported in Table 1 of Hastie & Tibshirani (1996). MPBART is among best for this data generating scheme exhibiting low average test error rate. Note that Hastie & Tibshirani (1996) report an error rate of 0.157 on test data sets achieved by penalized mixture discriminant analysis.

5. Real data examples

5.1. Multinomial choice data examples

Two discrete choice data sets, dealing with fishing and travel mode choices, are used to illustrate MPBART. Fishing mode choice data is a survey of 1,182 individuals who reported their most recent saltwater fishing modes as either “beach”, “pier”, “boat” or “charter”. The choice specific variables in this data set are expected catch rates per hour and price for each mode of fishing, while the individual specific predictor is monthly income. Details of this data are in Kling & Thomson (1996); Herriges & Kling (1999) and we use the version of data available in the R package mlogit. The second data records the choice of travel mode between Sydney and Melbourne, Australia as either “air”, “train”, “bus” or “car” (Greene, 2003; Kleiber & Zeileis, 2008). It includes 210 individuals’ choice of travel and the following choice specific predictors: general cost associated with the travel mode choice, waiting time at a terminal (with zero recorded for a travel choice of “car”), cost of travel mode and travel time. In addition, the individual specific predictors logarithms of household income, and traveling party size are used. We use the version of the data set in the R package AER (Kleiber & Zeileis, 2008).

After splitting the fishing mode data into ten and the travel mode data into five nearly equal random folds, we implement the procedures MPBART, Bayesian multinomial probit model (Bayes-MNP), the Multinomial logit (MNL) and the multiclass classification procedures listed in Table 2 with one fold set aside as a test data and the remaining folds utilized for training the models. Table 4 reports the average test error rates along with their standard errors. MPBART is again among the procedures with the lowest error rates.

Table 4.

Comparison results on the fishing mode and choice of travel mode data sets. Classification error rates (with standard errors in parentheses) are reported.

Procedure	Fishing Mode		Travel Mode
	Test Error Rate	Rank	Test Error Rate	Rank
MPBART	0.3960 (0.0160)	1	0.0571 (0.0086)	2
Bayes-MNP	0.5546 (0.0171)	10	0.3286 (0.0394)	10
MNL	0.5600 (0.0160)	11	0.3143 (0.0332)	9
RF	0.4746 (0.0148)	3	0.0429 (0.0089)	1
CART	0.5372 (0.0147)	8	0.1048 (0.0161)	3
SVML	0.5034 (0.0139)	6	0.2143 (0.0345)	7
SVMR	0.4882 (0.0194)	4	0.1381 (0.0254)	5
LDA	0.4975 (0.0193)	5	NA	NA
NNET	0.5211 (0.0064)	7	0.3048 (0.0739)	8
KNN	0.5406 (0.0189)	9	0.1810 (0.0358)	6
OvA-BART	0.4434 (0.0144)	2	0.1143 (0.0158)	4

5.2. Multiclass classification data examples

Forensic glass and vertebral column classification data sets, both of which are publicly available at the University of California at Irvine (UCI) machine learning data repository (Bache & Lichman, 2013), are used to illustrate MPBART as a multiclass classification procedure in comparison to the multiclass classification procedures listed in Table 2. The forensic glass classification data set consists of 9 features collected on 214 glass samples classified as one of the 6 glass types: building windows float processed, building windows non-float processed, vehicle windows float processed, containers, tableware, or headlamps. The vertebral column data contains 310 patients diagnosed either as normal, having Disk Hernia or Spondylolisthesis. This data set records the pathology of the human vertebral column, whose main function is the protection of the spine, and its dependence on the characteristics of the pelvis and lumbar spine. Further detail on the data set is available in da Rocha Neto et al. (2011); Calle-Alonso et al. (2013).

In our analysis, we split the forensic glass and vertebral column data sets into five and ten nearly equal random folds, respectively. One fold is set aside as test data and the classification methods in Table 2 and MPBART are trained on the remaining folds. Table 5 shows the average classification error rates with standard errors in parenthesis. QDA could not be implemented in this data set since the representation of observations classified as tableware is very small. For the same reason, we only considered five-fold partitioning of the forensic glass data. MPBART, RF and OvA-BART are the top performing procedures in terms of having the lowest classification error.

Table 5.

Classification error rates and standard errors (in parentheses) for vertebral column and forensic glass data sets.

Procedure	Vertebral Column		Forensic Glass
	Test Error Rate	Rank	Test Error Rate	Rank
MPBART	0.1466 (0.0324)	1	0.2946 (0.0182)	2
RF	0.1645 (0.0265)	4	0.2056 (0.0089)	1
CART	0.1839 (0.0160)	8	0.3272 (0.0356)	5
SVML	0.1484 (0.0285)	2	0.3741 (0.0294)	8
SVMR	0.1742 (0.0216)	6	0.3086 (0.0222)	4
LDA	0.1968 (0.0335)	0	0.3833 (0.0145)	9
QDA	0.1548 (0.0254)	3	NA	NA
NNET	0.2161 (0.0259)	10	0.3740 (0.0172)	7
MNL	0.6129 (0.0304)	11	0.3834 (0.0269)	10
KNN	0.1806 (0.0334)	7	0.3506 (0.0316)	6
OvA-BART	0.1645 (0.0282)	5	0.3083 (0.0196)	3

6. Conclusion

We have proposed and tested through simulation studies and real data examples the utility of Bayesian ensemble of trees for multinomial probit regression and multiclass classification. Regression trees and their ensembles are widely used for the purpose of classification. However, their use in multinomial probit regression which allows the introduction of choice specific predictors is less explored. MPBART fills that gap in the literature. It exhibits very good predictive performance in a range of examples and is among the best when the relationship between the predictors and choice response is nonlinear. The software implementation of MPBART is freely available as an R package mpbart. For the simulation studies and real data examples, the MPBART tuning parameters selected via cross-validation are available at https://github.com/bpkindo/mpbart_cv_selection/.