Parameter-expanded data augmentation for analyzing multinomial probit models

Abstract

The multinomial probit model has been a prominent tool to analyze nominal categorical data, but the computational complexity of maximum likelihood functions presents challenges in the usage of this model. Furthermore, the model identification is extremely tenuous and usually necessitates the covariance matrix of the latent multivariate normal variables to be a restricted covariance matrix, which brings a rigorous task for both likelihood-based estimation and Markov chain Monte Carlo (MCMC) sampling. We tackle this issue by constructing a non-identifiable model and developing parameter-expanded data augmentation. Our proposed methods circumvent sampling a restricted covariance matrix commonly implemented by a painstaking Metropolis-Hastings (MH) algorithm and enable to sample a covariance matrix without restriction through a Gibbs sampler. Therefore, our proposed methods advance the convergence and mixing of the MCMC components considerably. We investigate our proposed methods along with the method based on the identifiable model through simulation studies and further illustrate their performance by an application to consumer choice on liquid laundry detergents data.

1. Introduction

Discrete choice models have been commonly explored to analyze categorical data, especially in the fields of econometrics and transportation systems (Hausman and Wise 1978; Daganzo 1979). The multinomial logit (MNL) model and the multinomial probit (MNP) model are among the most popular discrete choice models. The MNL model assumes independence of irrelevant alternatives, which limits its use for categorical data with correlated levels (McFadden 1974, 1987). As an alternative, the MNP model allows correlated structure among latent utility scores underlying different categories and therefore has extended its use in the categorical data analysis. However, the MNP model is challenging to be utilized due to the computational complexity of maximum likelihood functions.

Various maximum likelihood estimation methods for analyzing the MNP model have been developed and investigated during the last couple of decades. Berndt, Hall, Hall, and Hausman (1974) proposed a minimum distance method; Clark (1961), Horowitz et al. (1982) and Kamakura (1989) explored approximation methods; McFadden (1989) and Pakes and Pollard (1989) developed the method of simulated moments estimator; and Geweke (1991), Hajivassiliou and McFadden (1990) and Keane (1992, 1994) conducted seminal work in simulated maximum likelihoods. Other methodical work including model estimability and identification can be referred to Bunch (1991), Bunch and Kitamura (1991), Geweke et al. (1994), Hajivassiliou et al. (1996), and Train (2009).

With the theoretical development of Gibbs sampler by Gelfand and Smith (1990) and the maturation of Markov chains theory (Tanner and Wong 1987; Smith and Roberts, 1993; Tierney, 1994; Gilks et al. 1996; Gelman et al. 2013), MCMC methods have been popularized and become a general computation tool in Bayesian inference. Pioneer contributions using MCMC to estimate the MNP model include Albert and Chib (1993), Geweke (1991), McCulloch and Rossi (1994), Geweke et al. (1994) and Chib et al. (1998). Albert and Chib (1993), Chib et al. (1998), Zhang et al. (2008) and Chan and Jeliazkov (2009) employed MH algorithm to sample restricted covariance matrix in the MNP model, while McCulloch and Rossi (1994) and Geweke et al. (1994) considered Gibbs sampling to sample a covariance matrix without restriction. Nobile (1998) added one Metropolis step along a direction of constant likelihood based on the work of McCulloch and Rossi (1994) to improve the convergence and mixing of the MCMC components. Chen and Kuo (2002) and Edwards and Allenby (2003) also considered the same model and conducted the MCMC analysis. McCulloch et al. (2000) specified a prior on the identified parameters and enabled to develop Gibbs sampling for the identifiable MNP model. The Bayesian methods avoid direct evaluation of the likelihoods and show advantages compared with the frequentist methods (Albert and Chib 1993; McCulloch and Rossi 1994; Geweke et al. 1994).

Liu and Wu (1999) proposed parameter-expanded data augmentation by expanding the parameter space through including a working/artificial parameter or parameter vector, which makes the identifiable model to be non-identifiable. They proved that under mild conditions the expanded data augmentation based on the non-identifiable model converges no slower than that based on the identifiable model. Another major work regarding extending the parameter space to accelerate the convergence of data augmentation is offered by Meng and van Dyk (1999). Liu et al. (1998) explored parameter expansion for the expectation-maximization algorithm; MacEachern (2007) illustrated this issue using the mixture of Dirichlet process model; and Lawrence et al. (2008) and Zhang (2020, 2022) investigated the matter using multivariate probit models.

Imai and van Dyk (2005a, b) developed the parameter-expanded data augmentation for the MNP model. Based on the work of Imai and van Dyk (2005a), Burgette and Nordheim (2012) and Burgette et al. (2021) developed the MCMC algorithms through trace restriction and symmetric prior for the MNP model, respectively. Jiao and van Dyk (2015) pointed out errors in the work of Imai and van Dyk (2005b), and Burgette et al. (2021) claimed to carefully correct the errors in their work. However, all these works are based on the same non-identifiable MNP model, and the derived data augmentation ignores the working parameter(s) entangled in both the regression parameters and the covariance matrix, and therefore, produces error.

In this investigation, we construct one non-identifiable MNP model and develop the correct and efficient parameter-expanded data augmentation for analyzing the MNP model. We point out the error in the work of Imai and van Dyk (2005a, 2005b) and those related to their work. The article is organized as follows. The MNP model and the constructed nonidentifiable model are presented in Section 2. Section 3 contains two proposed parameter-expanded data augmentation algorithms. We then illustrate our methods through simulation studies and compare them with the one based on the identifiable MNP model in Section 4. An application to the consumer choice on liquid laundry detergents data is demonstrated in Section 5. Finally, discussion and conclusions are offered in Section 6.

2. The MNP model

Let $i=1,\cdots ,n$ index subjects and $j=1,\cdots ,k+1$ index levels of a multinomial/categorical outcome having $k+1$ levels. We let $y_{ij}=1$ if subject $i$ chooses category level $j$ and $y_{ij}=0$ if subject $i$ chooses category level other than $j$, and $y_i=\left(y_{i1},\cdots ,y_{i(k+1)}\right)$ represents a multinomial $1\times (k+1)$ vector. More compactly, we define $d={\left(d_1,\cdots ,d_n\right)}^T$ where $d_i$ contains the index of the chosen category level for subject $i$, i.e., $d_i=j$ if $y_{ij}=1$. For example, there is one three-categorical variable, if subject $i$ chooses category level 2, then $y_i=\left(y_{i1},y_{i2},y_{i3}\right)=(0,1,0)$ and $d_i=2$.

The MNP model assumes that there is a latent vector $u_i=\left(u_{i1},\dots ,u_{i(k+1)}\right)$ underlying each multinomial vector $y_i$, such that the multinomial outcome is determined by the maximum $u_{ij}$. That is

$$d_i=j⟺u_{ij}\ge \underset{1\le l\le \left(k+1\right)}{\text{max}}{u}_{il}.$$ (1)

Then MNP model further assumes that the vector $u_i$ follows a multivariate normal distribution with mean equal to $A_i\beta$ and covariance matrix equal to $V$, where $A_i$ is a $(k+1)\times p$ covariate matrix for subject $i$ and $\beta$ is a $p\times 1$ regression parameter vector. With this notation,

$$u_i=A_i\beta +{\delta }_i$$ (2)

where ${\delta }_i~N_{k+1}(0,V)$. Detailed specification regarding covariate matrix $A_i$ can be referred to Keane (1992) and Geweke et al. (1994).

There are two identification problems in the above MNP model (Bunch 1991; Keane, 1992; Train 2009). Note that the model will not be changed if a constant is added to each element of $u_i$ in (1). This problem is usually solved by subtracting the $(k+1)$-th row of equation (2) from the first $k$ rows. The model becomes

$$Z_i=X_i\beta +{ϵ}_i$$

where $Z_{ij}=u_{ij}-u_{i(k+1)},X_{ij}=A_{ij}-A_{i(k+1)},{ϵ}_{ij}={\delta }_{ij}-{\delta }_{i(k+1)},{ϵ}_i~N_k\left(0,{\Sigma }_r\right)$, and ${\Sigma }_r=\left[I_k,-1_k\right]V{\left[I_k,-1_k\right]}^T$, with $I_k$ being identity matrix and $1_k$ a vector of 1’s. Then the model can be described as:

$$d_i=\left\{\begin{array}{ll}0& \text{if}\underset{1\le l\le k}{\text{max}}{Z}_{il}<0\\ j& \text{if}\underset{1\le l\le k}{\text{max}}{Z}_{il}=Z_{ij}>0\end{array}\right.$$

We notice that the model is still not changed if $Z_{il},l=1,\cdots ,k$, are multiplied by a non-zero constant. The usual way to solve this issue is to restrict the first element of ${\Sigma }_r$, to be equal 1. Then the model can be defined as follows:

$$d_i=\left\{\begin{array}{ll}0& \text{if}\underset{1\le l\le k}{\text{max}}{Z}_{il}<0\\ j& \text{if}\underset{1\le l\le k}{\text{max}}{Z}_{il}=Z_{ij}>0\end{array}\right.$$ (3)

where $Z_i~N_k\left(X_i\beta ,{\Sigma }_r\right)$ and the first diagonal element of ${\Sigma }_r$ being equal to 1.

Based on Liu and Wu (1999), Meng and van Dyk (1999) and MacEachern (2007), the non-identifiable models may improve the convergence and mixing of MCMC components compared with the identifiable models. Wherefore, we construct a non-identifiable MNP model as

$$d_i=\left\{\begin{array}{ll}0& \text{if}\underset{1\le l\le k}{\text{max}}{Z}_{il}<0\\ j& \text{if}\underset{1\le l\le k}{\text{max}}{Z}_{il}=Z_{ij}>0\end{array}\right.$$ (4)

with $Z_i~N\left({\sigma }_{11}^{-1/2}{X}_i\beta ,{\Sigma }_r\right)$. As can be seen, the model is non-identifiable due to introducing the redundant/artificial parameter ${\sigma }_{11}$. Let $W_i=\sqrt{{\sigma }_{11}}{Z}_i$, and then $W_i~N\left(X_i\beta ,{\sigma }_{11}{\Sigma }_r\right)$. Accordingly, the model can be defined as

$$d_i=\left\{\begin{array}{ll}0& \text{if}\underset{1\le l\le k}{\text{max}}{W}_{il}<0\\ j& \text{if}\underset{1\le l\le k}{\text{max}}{W}_{il}=W_{ij}>0\end{array}\right.$$

with $W_i~N_k\left(X_i\beta ,{\sigma }_{11}{\Sigma }_r\right)$. Notice that ${\sigma }_{11}{\Sigma }_r$ is a covariance matrix without restriction and we denote $\Sigma ={\sigma }_{11}{\Sigma }_r$.

3. Parameter-expanded data augmentation

Continue with the notation of Section 2, the joint posterior density of $\beta ,\Sigma$, and $W=\left(W_1,\cdots ,W_n\right)$ given $d=\left(d_1,\cdots ,d_n\right)$ can be written as $P(\beta ,\Sigma ,W\mid d)\propto p\left(\beta \right)\times p\left(\Sigma \right)\times \prod _{i=1}^n\left[I_i\times \varphi \left(W_i;X_i\beta ,\Sigma \right)\right]$, where $\varphi$ is the standard normal density function, and $I_i=1_{\left(d_i=0,w_{il}<0,l=1,\cdots ,k\right)}+1_{\left(d_i=h,w_{ih}={\text{max}}_{1\le l\le h}\left(w_{il,}0\right)\right)}$. Then the MCMC sampling steps can be derived as follows:

• Step 1: $\beta \mid \Sigma ,W,d~N_p\left(\hat{\beta },V_{\beta }\right),$ where $V_{\beta }={\left(\sum _{i=1}^n{X}_i^T{\Sigma }^{-1}{X}_i+C^{-1}\right)}^{-1}$ and $\hat{\beta }=V_{\beta }\left(\sum _{i=1}^n{X}_i^T{\Sigma }^{-1}{W}_i+C^{-1}b\right)$, assuming the prior of $\beta$ follows $N_p(b,\text{C})$.

• Step 2: $P\left(W_{ij}\mid \beta ,\Sigma ,d_i,W_{i(-j)}\right)\propto I_i\times P\left(W_{ij}\mid \beta ,\Sigma ,W_{i(-j)}\right)$, being a truncated univariate normal distribution.

• Step 3: $\Sigma \mid \beta ,W,d~{\text{Inverse}-\text{Wishart}}_k\left(\sum _{i=1}^n\left(W_i-X_i\beta \right){\left(W_i-X_i\beta \right)}^T+V,n+m+k+\right.1)$, assuming a conjugate prior, $\Sigma ~{\text{Inverse}-\text{Wishart}}_k(V,m)$.

We denote this algorithm as parameter-expanded Gibbs sampling (PX-GS) algorithm. To further improve the convergence and mixing of the Gibbs sampling (Liu 1994; Liu and Wu 1999), we marginalize ${\sigma }_{11}$ by jointly sampling $(W,{\sigma }_{11})$ given $(\beta ,{\Sigma }_r,d)$ and then $(\beta ,{\Sigma }_r,{\sigma }_{11})$ given $(W,d)$:

• Sampling $(\left.W,{\sigma }_{11}\right)\mid \left(\beta ,{\Sigma }_r,d\right)$ by first sampling ${\sigma }_{11}\left|\left(\beta ,{\Sigma }_r,d\right)\propto {\sigma }_{11}\right|{\Sigma }_r$, which is the conditional prior of ${\sigma }_{11}$ given ${\Sigma }_r$ and can be derived as ${\sigma }_{11}\left|{\Sigma }_r~\frac{\text{tr}\left(V{{\Sigma }_r}^{-1}\right)}{{\chi }_{mk}^2}\right.$, with tr being a trace of a matrix and ${\chi }_{mk}^2$ being a Chi-squared distribution with $mk$ degrees of freedom; then followed by $W\left|\left(\beta ,{\Sigma }_r,{\sigma }_{11},d\right)\propto W\right|(\beta ,\Sigma ,d)$ which is Step 2.

• Sampling $\left(\beta ,{\Sigma }_r,{\sigma }_{11}\right)\mid (W,d)$ by first sampling $\beta \left|\left({\Sigma }_r,{\sigma }_{11},W,d\right)\propto \beta \right|(\Sigma ,W,d)$ which is Step 1, then followed by $\left({\Sigma }_r,{\sigma }_{11}\right)\left|\right(\beta ,W,d)\propto \Sigma |(\beta ,W,d)$ which is Step 3.

We call this algorithm as parameter-expanded Gibbs sampling with marginalization for the MNP model (PX-GMN).

Imai and van Dyk (2005a, b) proposed marginal data augmentation by assuming that $\sqrt{{\sigma }_{11}}{Z}_i=\sqrt{{\sigma }_{11}}{\left(Z_{i1},\cdots ,Z_{ik}\right)}^T$ follows $N_k\left(\sqrt{{\sigma }_{11}}{X}_i\beta ,{\sigma }_{11}{\Sigma }_r\right)$, for $i=1,\cdots ,n$. The joint posterior density of $\beta ,{\Sigma }_r,{\sigma }_{11}$, and $W=\left(W_1,\cdots ,W_n\right)$ given the observed $d$ can be derived as follows:

$$P\left(\beta ,{\Sigma }_r,{\sigma }_{11},W\mid d\right)\propto \left({\prod }_{i=1}^n{I}_i\right)\times P\left(\beta \right)\times P\left({\Sigma }_r,{\sigma }_{11}\right)\times {\sigma }_{11}^{-n/2}\times {\left|{\Sigma }_r\right|}^{-\frac{n}{2}}\times \text{exp}\left[-\frac{1}{2}{\sum }_{i=1}^n{\left(W_i-\sqrt{{\sigma }_{11}}{X}_i\beta \right)}^T{\sigma }_{11}^{-\frac{1}{2}}{\Sigma }_r^{-1}{\sigma }_{11}^{-\frac{1}{2}}\left(W_i-\sqrt{{\sigma }_{11}}{X}_i\beta \right)\right].$$

As can be seen, the posterior conditional distribution for $\beta$ is a multivariate normal distribution, and the posterior conditional distribution of each $W_{ij}$ follows truncated normal distribution. The posterior density of $\Sigma$ can be written as follows:

$$P(\Sigma \mid \beta ,W,d)\propto P\left(\Sigma \right)\times |\Sigma {|}^{-\frac{n}{2}}\times \text{exp}\left[-\frac{1}{2}{\sum }_{i=1}^n{\left(W_i-\sqrt{{\sigma }_{11}}{X}_i\beta \right)}^T{\Sigma }^{-1}\left(W_i-\sqrt{{\sigma }_{11}}{X}_i\beta \right)\right].$$

As is shown, although the prior of $\Sigma ,P\left(\Sigma \right)$, can be an Inverse-Wishart distribution, however, the posterior density of $\Sigma$ cannot be from an Inverse-Wishart distribution due to ${\sigma }_{11}$ being entangled in the terms of $\sqrt{{\sigma }_{11}}{X}_i\beta$. Therefore, the derived MCMC sampling algorithms in Imai and van Dyk (2005a, b) and those related to their work (Burgette and Nordheim 2012; Burgette et al. 2021; Jiao and van Dyk 2015) contain errors. Algorithm 1 in Imai and van Dyk (2005a) can be restated in the following steps:

• Sampling $\left(W,{\sigma }_{11}\right)\mid \left(\beta ,{\Sigma }_r,d\right)$ by sampling $\widetilde{W\mid }\tilde{\beta },\Sigma ,d$, followed by sampling ${\sigma }_{11}\mid {\Sigma }_r$, then set $W=\sqrt{{\sigma }_{11}}\tilde{W}$.

• Sampling $\left(\beta ,{\sigma }_{11}\right)\left|\right(W,\Sigma ,d)$ by sampling ${\sigma }_{11}\mid \left(W,{\Sigma }_r,d\right)$ then $\beta \mid (W,\Sigma ,d)$.

• Sampling $\Sigma \mid (\beta ,W,d)$.

Here the notation “~” indicates the identifiable parameters. As can be seen, ${\sigma }_{11}$ is marginalized in the above three steps instead of two steps specified in the PX-GMN algorithm. However, due to the fail to derive the correct joint posterior distribution, the derivation of ${\sigma }_{11}\mid \left(W,{\Sigma }_r,d\right)$ is erroneous. We continue to point out the theoretical flaw in Algorithm 2 in Imai and van Dyk (2005a). Specifically, Algorithm 2 can be rewritten as:

• Sampling $\tilde{W}\mid \tilde{\beta },\Sigma ,d$, followed by sampling ${\sigma }_{11}\mid {\Sigma }_r$, then set $W=\sqrt{{\sigma }_{11}}\tilde{W}$.

• Sampling $\Sigma \mid (\beta ,W,d)$.

• $\tilde{\beta }\mid \left(\tilde{W},{\Sigma }_r,d\right)$ with $\tilde{\beta }=\beta /\sqrt{{\sigma }_{11}}$.

As noted, in essence, the algorithm jointly samples $(W,\Sigma ,\tilde{\beta })$ given $d$. However, this is invalid since it samples the identifiable $\tilde{\beta }$ and nonidentifiable $(W,\Sigma )$ jointly instead of either joint sampling the identifiable $(\tilde{W},{\Sigma }_r,\tilde{\beta })$ based on the identifiable model or joint sampling the nonidentifiable $(W,\Sigma ,\beta )$ based on nonidentifiable model, which is the PX-GS algorithm. The main cause for those errors to occur in both algorithms is that their nonidentifiable model is incapable to produce Gibbs sampling for all parameters, and moreover the construction of the sampling steps is not based on the joint posterior distribution, which is failed to be derived.

4. Simulation studies

We conduct thorough simulation studies to illustrate the performance of the PX-GS and PX-GMN algorithms proposed in Section 3 based on the nonidentifiable model and compare them with the PX-MH algorithm based on the identifiable model which entails a MH algorithm for sampling the restricted covariance matrix (Zhang et al. 2008).

The simulation studies are designed based on the simulated setting by McCulloch and Rossi (1994). Specifically, we generate six-choice categorical outcome with one independent covariate $X$ drawn from a uniform distribution over (−2.0, 2.0) and the identified regression parameter $\beta =2/\sqrt{5}\approx 0.894$. The covariance matrix $\Sigma$ has a compound symmetry (CS) correlation structure with equal correlation being 0.5, i.e., CS (0.5), and four identified variances being 0.8, 0.6, 0.4, 0.2 (the first variance or the first element of $\Sigma$ is fixed at 1). We consider four prior scenarios to reflect from weak to strong prior information as follows: the first is denoted as PID to represent non-informative prior with non-informative prior for $\beta$ and $\Sigma ~{\text{Inverse}-\text{Wishart}}_k\left(\right(m-k-1)I,m)$ and $I$ being the identity matrix; the second is denoted as PCS1 with the prior of $\beta$ following $N(1.0,1.0)$ and $\Sigma ~{\text{Inverse}-\text{Wishart}}_k\left(\right(m-k-1\left)\text{CS}\right(0.4),m)$; the third is as PCS2 based on PCS adding the four identified variances all being 0.5; and the fourth is as PCS3 based on PCS adding the four identified variances being the same as the true values. Here, $k$, being 5, is the dimension for latent variables and $m$, the degrees of freedom, is set to be 10. We investigate sample sizes 500 and 2,000 and generate 100 datasets for each simulated scenario. We run the MCMC sampling for 20,000 iterations with 5,000 burn-in and check the MCMC convergence using R packages coda (Plummer et al. 2015) and boa (Smith, 2016).

Table 1, Figure 1 and Figure 2 are based on PID prior for illustration. Table 1 shows the averaged posterior estimated means for all three algorithms are closer to the true values with smaller standard deviations and better 95% credible interval coverage probabilities (CP) for sample size 2,000 than for 500. The PX-GS and PX-GMN algorithms produce larger standard deviations than the PX-MH algorithm does in general, accordingly with better CP. The PX-GMN algorithm has the largest standard deviations among those three algorithms for regression parameter $\beta$, hence yields the best CP. The PX-GS and PX-GMN algorithms have similar standard deviations for correlations and variances. It is noticeable that all three algorithms produce overestimated values for the variance ${\sigma }_{55}$ and underestimated CP, especially for the PX-MH algorithm with sample size 500 (30% CP) and the PX-GS algorithm for sample size 2,000 (61% CP).

Table 1.

Averaged posterior means (mean), standard deviations (SD) and 95% credible interval coverage probabilities (CP%) with sample sizes 500 and 2000 based on 100 generated datasets using PID.

$\mathit{N}=500$	True Values	PX-MH			PX-GS			PX-GSM
		Mean	SD	CP%	Mean	SD	CP%	Mean	SD	CP%
$\beta$	0.89	1.02	0.09	67	1.01	0.14	91	1.01	0.18	97
$r_{12}$	0.5	0.36	0.15	87	0.36	0.17	91	0.38	0.17	92
$r_{13}$	0.5	0.35	0.14	79	0.41	0.16	93	0.41	0.17	95
$r_{14}$	0.5	0.32	0.14	78	0.37	0.17	96	0.38	0.17	95
$r_{15}$	0.5	0.31	0.15	76	0.34	0.18	95	0.36	0.19	96
$r_{23}$	0.5	0.45	0.14	94	0.41	0.16	92	0.42	0.17	94
$r_{24}$	0.5	0.41	0.16	96	0.38	0.16	94	0.39	0.17	95
$r_{25}$	0.5	0.38	0.15	89	0.37	0.16	96	0.39	0.17	97
$r_{34}$	0.5	0.39	0.15	94	0.42	0.15	99	0.45	0.16	99
$r_{35}$	0.5	0.40	0.14	94	0.41	0.16	98	0.43	0.17	99
$r_{45}$	0.5	0.37	0.16	95	0.39	0.16	99	0.4	0.17	98
${\sigma }_{22}$	0.8	0.97	0.28	88	0.91	0.35	97	0.93	0.37	98
${\sigma }_{33}$	0.6	0.75	0.21	88	0.77	0.30	94	0.77	0.31	94
${\sigma }_{44}$	0.4	0.52	0.17	95	0.54	0.22	97	0.54	0.24	98
${\sigma }_{55}$	0.2	0.41	0.12	30	0.42	0.19	75	0.39	0.18	84
$N=2000$
$\beta$	0.89	0.89	0.04	94	0.94	0.06	92	0.93	0.12	100
$r_{12}$	0.5	0.48	0.07	95	0.45	0.08	97	0.49	0.09	98
$r_{13}$	0.5	0.46	0.07	89	0.43	0.09	87	0.47	0.09	95
$r_{14}$	0.5	0.48	0.08	93	0.44	0.09	94	0.46	0.1	96
$r_{15}$	0.5	0.49	0.1	95	0.42	0.1	95	0.45	0.11	94
$r_{23}$	0.5	0.47	0.08	87	0.46	0.08	94	0.49	0.09	92
$r_{24}$	0.5	0.47	0.08	93	0.44	0.09	91	0.48	0.1	95
$r_{25}$	0.5	0.45	0.08	89	0.44	0.09	96	0.47	0.11	98
$r_{34}$	0.5	0.44	0.08	80	0.44	0.09	90	0.47	0.1	93
$r_{35}$	0.5	0.44	0.08	91	0.45	0.09	96	0.47	0.1	96
$r_{45}$	0.5	0.42	0.09	91	0.45	0.09	99	0.47	0.1	98
${\sigma }_{22}$	0.8	0.76	0.11	88	0.85	0.16	95	0.86	0.17	95
${\sigma }_{33}$	0.6	0.54	0.08	81	0.63	0.13	94	0.61	0.13	93
${\sigma }_{44}$	0.4	0.40	0.07	96	0.45	0.09	93	0.42	0.09	94
${\sigma }_{55}$	0.2	0.26	0.05	79	0.31	0.07	61	0.25	0.06	88

Figure 1.

Relative biases for regression parameter, correlations and variances for sample sizes being 500 (1^st column) and 2,000 (2^nd column) based on 100 simulated datasets using PID for the PX-MH (orchid), PX-GS (blue), and PX-GMN (gray) algorithms.

Figure 2.

ACF plots for regression parameter $\beta$, correlations $(r_{23},r_{45})$ and variances $({\sigma }_{22},{\sigma }_{44})$ for sample sizes being 500 (1^st row) and 2,000 (2^nd row) based on 100 simulated datasets using PID for the PX-MH (orchid-dotdash), PX-GS (blue-longdash), and PX-GMN (black-solid) algorithms.

Figure 1 shows for sample size 500, all algorithms mainly have similar averaged relative biases for all parameters. For sample size being 2,000, the PX-GS algorithm has larger relative biases mostly than the PX-MH and PX-GMN algorithms, that is, for regression parameter $\beta$, correlations (except for $r_{35}$ and $r_{45}$) and variances (except for ${\sigma }_{33}$).

Figure 2 displays the autocorrelation function (ACF) plots for selected parameters with the purpose of illustration. As can be observed, for regression parameter $\beta$, the ACF values of the PX-GMN algorithm decrease much faster than those of the PX-GS and PX-MH algorithms, and the ACF values of the PX-GMN decrease faster for sample size 2,000 than those for sample size 500 while those values of the PX-MH and PX-GS algorithms are similar for both sample sizes 500 and 2,000. As for correlations and variances, all three algorithms show similar ACF plots for sample size 500 in comparison with sample size 2,000, specifically, the PX-GS and PX-GMN algorithms have similar ACF plots although the PX-GS algorithm shows marginal advantage over the PX-GMN algorithm for $r_{45}$ and ${\sigma }_{44}$, and both are superior to the PX-MH algorithm.

Figure 3 exhibits prior sensitivity analysis for selected parameters using relative biases and sample size 2,000 for exemplification. As shown, the PX-MH algorithm is utmost affected by the priors and performs best using PID other than any informative prior. On the other hand, the PX-GS and PX-GMN algorithms display better performance with stronger prior information. However, it is noticeable that the PX-GMN algorithm is more robust to the prior specification than the PX-GS algorithm.

Figure 3.

Relative biases for regression parameter $\beta$, correlation $r_{15}$ and variance ${\sigma }_{44}$ for sample sizes being 2,000 based on 100 simulated datasets using PID (bright gray), PCS (light gray), PCS1 (medium gray) and PCS2 (dark gray) for the PX-MH, PX-GS and PX-GMN algorithms.

Through the above examination, we conclude that the PX-GMN algorithm evinces superiority in parameter estimation, MCMC convergence and mixing, and robust to the prior specification in comparison with the PX-GS and PX-MH algorithms, suggesting that parameter-expanded data augmentation with marginalization should be considered to improve the performance of MCMC sampling. In addition, the PX-GS and PX-GMN algorithms based on the non-identifiable model perform better than the PX-MH algorithm based on the identifiable model in convergence and mixing of MCMC components; the PX-GS algorithm may produce biased estimation for large sample size, such as 2000; the PX-MH algorithm has the slowest convergence and hence may produce biased estimation with smallest standard deviations and inferior credible coverages, and is sensitive to the prior specification.

5. Application to consumer choice on liquid laundry detergents

The data of consumer choice on liquid laundry detergents was investigated by Chintagunta and Prasad (1998), Imai and van Dyk (2005a, b) and Burgette et al. (2021). It involves 2657 households in the Sioux Falls, South Dakota market, and six chosen brands- Tide, Wisk, EraPlus, Surf, Solo and All, with the log price for each brand. We analyze this data using a six-choice MNP model with covariate matrix including five intercept terms and one for the log price of each brand (Tide, Wisk, EraPlus, Surf, Solo) minus the log price of All (chosen as the reference category).

We consider non-informative prior for the regression parameter vector $\beta ={\left({\beta }_1,\cdots ,{\beta }_6\right)}^T$, and $\Sigma ~{\text{Inverse}-\text{Wishart}}_k\left(\right(m-k-1)I,m)$, with $k=5,m=10$, and $I$ being the identity matrix. We run the PX-MH, PX-GS and PX-GMN algorithms, each with 200,000 iterations and 100,000 burn-in, and present the results in Table 2. We can see that all three algorithms produce significantly negative estimated regression parameter ${\beta }_6$ for price (95% CIs excluding 0), suggesting that the brand choice is negatively related to the price of the brand; the PX-GMN algorithm has the largest standard deviation (12.00) while the PX-MH algorithm has the smallest one (5.07). The PX-MH and PX-GS algorithms have much larger estimated correlations with smaller estimated standard deviations than the PX-GMN algorithm, however, the 95% CIs for all three algorithms are basically consistent regarding the coverages of 0. The same occurrence can also be observed for the estimation of variances. As shown, the PX-GMN algorithm gives largest estimated ${\sigma }_{22}$ and ${\sigma }_{33}$ with largest standard deviations being 0.76 and 0.43, and the PX-GS algorithm gives largest estimated ${\sigma }_{44}$ and ${\sigma }_{55}$ with largest standard deviations being 0.36 and 0.54. This phenomenon, that is, large estimated standard deviations/variances, is also noticed and commented on by Keane (1992). Hence, we also calculate the values for the log-likelihood function for each algorithm, which illustrate that the three algorithms have similar log-likelihood values with the PX-GMN algorithm having the largest one and the PX-MH algorithm the smallest one. This finding, as elaborated by Keane (1992), is related to model identification and estimability issue.

Table 2.

Posterior means (mean) (only show the covariate price ${\beta }_6$, not including the intercepts ${\beta }_1$ up to ${\beta }_5$), standard deviations (SD), 95% credible intervals (CI) and Log-likelihood (LL) for data of consumer choice on liquid laundry detergents.

LL	PX-MH −3776.784			PX-GS −3715.483			PX-GMN −3643.716
	Mean	SD	CI	Mean	SD	CI	Mean	SD	CI
${\beta }_6$	−70.12	5.07	(−80.06, −59.84)	−76.74	9.28	(−95.71, −59.81)	−86.00	12.00	(−111.17, −64.31)
$r_{12}$	0.63	0.10	(0.432, 0.827)	0.56	0.12	(0.280 0.764)	0.38	0.14	(0.082, 0.615)
$r_{13}$	0.25	0.14	(−0.052, 0.497)	0.15	0.15	(−0.151, 0.427)	−0.12	0.14	(−0.399, 0.157)
$r_{14}$	0.33	0.12	(0.083, 0.546)	0.28	0.13	(0.003, 0.513)	−0.03	0.13	(−0.285, 0.224)
$r_{15}$	0.63	0.07	(0.493, 0.750)	0.58	0.09	(0.385, 0.741)	0.33	0.11	(0.106, 0.532)
$r_{23}$	0.38	0.15	(0.038, 0.662)	0.27	0.17	(−0.092, 0.582)	0.01	0.20	(−0.400, 0.386)
$r_{24}$	0.38	0.17	(0.077, 0.743)	0.31	0.16	(−0.002, 0.605)	0.00	0.18	(−0.333, 0.355)
$r_{25}$	0.66	0.07	(0.516, 0.797)	0.60	0.09	(0.409, 0.764)	0.41	0.12	(0.136, 0.623)
$r_{34}$	0.55	0.09	(0.351, 0.703)	0.53	0.09	(0.336, 0.698)	0.38	0.11	(0.160, 0.577)
$r_{35}$	0.53	0.09	(0.346, 0.696)	0.50	0.10	(0.291, 0.681)	0.27	0.11	(0.035, 0.485)
$r_{45}$	0.62	0.09	(0.421, 0.785)	0.60	0.09	(0.411, 0.749)	0.34	0.10	(0.134, 0.539)
${\sigma }_{22}$	2.12	0.58	(1.350, 3.646)	2.30	0.60	(1.318, 3.664)	2.56	0.76	(1.393, 4.334)
${\sigma }_{33}$	1.28	0.30	(0.790, 1.902)	1.46	0.45	(0.765, 2.494)	1.51	0.43	(0.825, 2.479)
${\sigma }_{44}$	0.97	0.22	(0.602, 1.445)	1.19	0.36	(0.611, 2.019)	1.01	0.29	(0.554, 1.698)
${\sigma }_{55}$	2.15	0.49	(1.346, 3.251)	2.37	0.54	(1.458, 3.562)	1.84	0.43	(1.117, 2.789)

6. Discussion

In this article, we propose two parameter-expanded data augmentation algorithms – PX-GS and PX-GMN based on the constructed nonidentifiable MNP model and investigate their performance with the PX-MH algorithm which is based on the identifiable MNP model. Our investigation shows that the PX-GMN algorithm, which is with marginalization for the redundant parameter, is superior in MCMC estimation, convergence and mixing compared with the PX-GS algorithm without marginalization and the PX-MH algorithm based on the identifiable model.

The PX-GS algorithm produces biased estimation for large sample size, such as 2,000. This matter also happens for analyzing multivariate ordinal data using multivariate probit model (Zhang 2023). In comparison with the PX-MH algorithm based on the identifiable model, the PX-GS and PX-GMN algorithms manifest advantages in 95% CI coverage probabilities, convergence and mixing of MCMC components and robust to the prior specification. However, they tend to produce large estimated standard deviations for estimated quantities, and this phenomenon is related to the tenuous identification issue in the MNP model itself.

It is generally acknowledged that the posterior means are close to the maximum likelihood estimates for relatively weak priors. However, with strong or informative priors specified, maximum a posteriori estimation may be explored to reduce the possible biases associated with the posterior means. This may become one of our future research projects.

Our investigation indicates that construction of non-identifiable models and development of parameter-expansion data augmentation with marginalization should be considered to improve the convergence and mixing of the MCMC components. Concurrently, the frail identification and estimability issue is worth our attention in the application of the MNP model.