Estimation of Multivariate Probit Models via Bivariate Probit

Abstract

This paper suggests the utility of estimating multivariate probit (MVP) models using a chain of bivariate probit estimators. The proposed approach is based on Stata’s biprobit and suest procedures and is driven by a Mata function. Two potential advantages over Stata’s mvprobit procedure are suggested: significant reductions in computation time; and essentially unlimited dimensionality of the outcome set. The time savings arise because the proposed approach does not rely simulation methods; the dimension advantage arises because only pairs of outcomes are considered at each estimation stage. Importantly, the proposed approach provides a consistent estimator of all the MVP model’s parameters under the same assumptions required for consistent estimation via mvprobit, and simulation exercises reported below suggest no loss of estimator precision relative to mvprobit.

1. Introduction

This paper suggests the utility of estimating multivariate probit (MVP) models using a chain of bivariate probit estimators. It will be seen that the proposed approach, based on Stata’s biprobit and suest procedures and driven by a Mata function bvpmvp(…), affords two potential advantages over Stata’s mvprobit procedure: significant reductions in computation time; and essentially unlimited dimensionality of the outcome set (mvprobit’s limit is M=20 outcomes).¹ The time savings arise because, unlike mvprobit, bvpmvp(…) does not rely simulation methods; the dimension advantage arises because only pairs of outcomes are considered at each estimation stage. Importantly, the proposed bvpmvp(…) approach provides a consistent estimator of all the MVP model’s parameters under the same assumptions required for consistent estimation via mvprobit, and simulation exercises reported below suggest no loss of estimator precision relative to mvprobit.

The approach suggested here was inspired by the goal of embedding MVP estimation in a large-replication bootstrap exercise. The simulation results presented in Section 5 suggest that the computation time savings afforded by the bvpmvp(…) method relative to mvprobit can be significant while numerical differences in the respective point estimates and estimated standard errors are trivial. Since the potential applicability of MVP models is broad, it is valuable in practice that such potential not be thwarted by computational challenges.

The plan for the remainder of the paper is as follows. Section 2 describes the MVP model. Section 3 describes the bvpmvp(…) method. Section 4 describes the comparison empirical exercises. Section 5 presents the comparative results. Section 6 considers parallel issues involved in estimation of multivariate ordered probit models. Section 7 summarizes.

2. The Multivariate Probit Model

The multivariate probit model as typically specified is:

$${\text{y}}_{\text{ij}}^{*}={\mathbf{\text{x}}}_{\text{i}}{\mathbf{\text{β}}}_{\text{j}}+{\text{u}}_{\text{ij}}$$ (1)

$${\text{y}}_{\text{ij}}=1\left({\text{y}}_{\text{ij}}^{*}>0\right)$$ (2)

$${\mathbf{u}}_{\text{i}}\text{=}\left[{\text{u}}_{\text{i}1}{,\dots ,\text{u}}_{\text{iM}}\right]~\text{MVN}(\mathbf{0},\mathbf{R})\text{or}{\mathbf{\text{y}}}_{\text{i}}^{*}=\left[{\text{y}}_{\text{i}1}{,\dots ,\text{y}}_{\text{iM}}\right]~\text{MVN}\left({\mathbf{x}}_{\text{i}}\mathbf{B},\mathbf{R}\right)$$ (3)

where i=1,…,N indexes observations, j=1,…,M indexes outcomes, x_i is a K-vector of exogenous covariates, the u_i are assumed to be iid independent across i but correlated across j for any i, and “MVN” denotes the multivariate normal distribution. (Henceforth the “i” subscripts will be suppressed.) The standard normalization sets the diagonal elements of R equal to 1 so that R is a correlation matrix with off-diagonal elements ρ_pq, {p,q}∈{1,…,M}, p ≠ q.² With standard full rank conditions on the x’s and each $\left|{\text{ρ}}_{\text{pq}}\right|<1\text{then}\mathbf{B}=\left[{\mathbf{\text{β}}}_1,\dots ,{\mathbf{\text{β}}}_{\text{M}}\right]\text{and}\mathbf{R}$ will be identified and estimable with sufficient sample variation in the x’s.

3. Estimation and Inference

Estimation of the M-outcome multivariate probit model using mvprobit requires simulation of the MVN probabilities (Cappellari and Jenkins, 2003), with mvprobit computation time increasing in M, K, N, and simulation draws (D).³ It turns out, however, that all the parameters (B,R) can be estimated consistently using bivariate probit -- implemented as Stata’s biprobit procedure -- while consistent inferences about all these parameters are afforded via Stata’s suest procedure. Since the proposed approach will be seen to be significantly faster in terms of computation time with no obvious disadvantages, this strategy may merit consideration in applied work.

The key result for the proposed estimation strategy is that the multivariate normal distribution is fully characterized by the mean vector xB and correlation matrix R. For present purposes, the key feature of the multivariate (conditional) normal distribution $\text{F}\left({\text{y}}_1^{*},\dots ,{\text{y}}_{\text{M}}^{*}|\mathbf{x}\right)$ all its bivariate marginals $\text{F}\left({\text{y}}_{\text{j}}^{*},{\text{y}}_{\text{m}}^{*}|\mathbf{x}\right)$ are bivariate normal with mean vectors and correlation matrixes corresponding to the respective submatrixes of xB and R (Rao, 1973, 8a.2.10).

Under the normalization that the diagonal elements of R are all one, the B parameters are identified based on knowledge of all M (conditional) univariate marginals $\text{F}\left({\text{y}}_{\text{j}}^{*}|\mathbf{x}\right);$ there is no need to appeal to the multivariate features of $\text{F}\left({\text{y}}_1^{*},\dots ,{\text{y}}_{\text{M}}^{*}|\mathbf{x}\right)$ to identify B. The .5M(M-1) bivariate marginals provide the additional information about the ρ_pq parameters. As such, identification of the parameters of all the bivariate marginals implies identification⁴ of the parameters of the full multivariate joint distribution so that consistent estimation of all the bivariate marginal probit models $\mathrm{Pr}\left({\text{y}}_{\text{p}}={\text{t}}_{\text{p}},{\text{y}}_{\text{q}}={\text{t}}_{\text{q}}|\mathbf{x}\right)$ provides consistent estimates of all the parameters (B,R) of the full multivariate probit model $\mathrm{Pr}\left({\text{y}}_1={\text{t}}_1,\dots ,{\text{y}}_{\text{M}}={\text{t}}_{\text{M}}|\mathbf{x}\right){\text{for t}}_{\text{j}}\in \{0,1\},\text{j}=1,\dots ,\text{M}$.

Estimation via Bivariate Probit

The proposed approach, which can be implemented using the Mata function bvpmvp(…) described below, is as follows. First, corresponding to each possible outcome pair, .5M(M-1) bivariate probit models are estimated using biprobit yielding a single estimate⁵ of each ρ_pq and M-1 estimates of each β_j, j=1,…,M. Each of the M-1 estimates of β_j is itself consistent since each biprobit specification uses the same normalization on the relevant submatrixes of R. Each of these estimates ${(\widehat{{\mathbf{\text{β}}}_{\text{p}}},\widehat{{\mathbf{\text{β}}}_{\text{q}}},\widehat{{\text{ρ}}_{\text{pq}}})}_{\text{b}}$, b=1,…,.5M(M-1), is stored and then combined using Stata’s suest procedure, which provides a consistent estimate of the joint variance-covariance matrix of all M(M-1)(.5 + K) parameters estimated with the .5M(M-1) biprobit estimates. Denote this vector of parameter estimates and its estimated variance-covariance matrix as $\widehat{\mathbf{\alpha }}\text{and}\widehat{\mathbf{\Omega }},$ respectively.⁶

Second, the simple averages ${\widehat{\mathbf{\text{β}}}}_{\text{jA}}=\left(\frac{1}{\text{M}-1}\right){\sum }_{\begin{array}{l}\text{m}=1\\ \text{m}\ne \text{j}\end{array}}^{\text{M}}{\widehat{\mathbf{\text{β}}}}_{\text{jm}}$ are computed. This gives a k × M matrix of estimated averaged coefficients, denoted $\widehat{{\mathbf{B}}_{\text{A}}}=\left[\widehat{{\mathbf{\text{β}}}_{1\text{A}}},\dots ,\widehat{{\mathbf{\text{β}}}_{\text{MA}}}\right].$. Since a weighted average of consistent estimators is in general a consistent estimator, the resulting $\widehat{{\mathbf{B}}_{\text{A}}}$ will itself be consistent for B. This averaging arises because the B parameters in the proposed approach are overidentified, i.e. there are M-1 consistent estimates of each β_j, j=1,…,M. One could use some other rule to compute a single consistent estimate of each β_j from among the M-1 candidates, but unless alternative strategies could boast significant precision gains, computational simplicity recommends the simple average as an obvious solution. See the Appendix for further discussion.

Finally, let Q denote the .5M(M-1) vector of the ${\text{tanh}}^{-1}\left({\text{ρ}}_{\text{jk}}\right)$ estimated in each biprobit specification, and define the M(.5(M-1)+ K) × 1 vector $\widehat{\mathbf{\Theta }}={\left[\mathrm{vec}{(\widehat{{\mathbf{B}}_{\text{A}}})}^{\text{T}},{\widehat{\mathbf{Q}}}^{\text{T}}\right]}^{\text{T}}.$ Define H as the M(.5(M-1) + K) × M(M-1)(.5 + K) averaging and selection matrix that maps $\widehat{\mathbf{\alpha }}\text{to}\widehat{\mathbf{\Theta }},\text{i.e.}\widehat{\mathbf{\Theta }}=\mathbf{H}{\widehat{\mathbf{\alpha }}}^{\text{T}};$ the elements of H are 1/(M-1), one, or zero.⁷ The estimated variance-covariance matrix of $\widehat{\mathbf{\Theta }}$, useful for inference, is given by $\widehat{\mathrm{var}}(\widehat{\mathbf{\Theta }})=\mathbf{H}\widehat{\mathbf{\Omega }}{\mathbf{H}}^{\text{T}}.$

bvpmvp(…): A Mata Function to Implement the Proposed Estimation Approach

The function bvpmvp(…) returns the M(k + .5(M-1))×(M(k + .5(M-1))+ 1) matrix whose first column is ${\widehat{\mathbf{\Theta }}}^{\text{T}}$ and whose remaining elements are the M(k + .5(M-1))-dimension symmetric square matrix $\widehat{\mathrm{var}}(\widehat{\mathbf{\Theta }}).$ bvpmvp(…) takes six arguments: (1) a string containing the names of the M outcomes; (2) a string containing the names of the K-1 non-constant covariates; (3) a (possibly null) string containing any “if” conditions for estimation; (4) a scalar indicating whether or not to display the interim estimation results; (5) a scalar indicating the rounding level of presented results; and (6) a scalar indicating whether or not to display the final results. For example:

$$\begin{array}{l}\mathit{b}\mathit{v}\mathit{1}=\mathit{b}\mathit{v}\mathit{p}\mathit{m}\mathit{v}\mathit{p}\mathit{(}"\begin{array}{llll}\mathit{y}\mathit{1}\hfill & \mathit{y}\mathit{2}\hfill & \mathit{y}\mathit{3}\hfill & \mathit{y}\mathit{4}\hfill \end{array}","\begin{array}{llll}\mathit{x}\mathit{1}\hfill & \mathit{x}\mathit{2}\hfill & \mathit{x}\mathit{3}\hfill & \mathit{x}\mathit{4}\hfill \end{array}","\mathit{i}\mathit{f}\_\mathit{n}<=\mathit{10000}",\mathit{0},.\mathit{001},\mathit{1}\mathit{)}\\ \mathit{b}\mathit{v}\mathit{2}=\mathit{b}\mathit{v}\mathit{p}\mathit{m}\mathit{v}\mathit{p}\mathit{(}\mathit{y}\mathit{n},\mathit{x}\mathit{n},\mathit{i}\mathit{c},\mathit{0},.\mathit{001},\mathit{1}\mathit{)}\end{array}$$

bvpmvp(…)’s summary report displays the $\widehat{{\mathbf{B}}_{\text{A}}}$ estimates, their estimated standard errors, and the estimated correlation matrix $\widehat{\mathbf{R}};$ an example is provided in Exhibit 1. Of course, suppression of these results may be useful, for instance, in simulation or bootstrapping exercises. The do file containing the Mata code for bvpmvp(…) is available with this paper’s supplementary materials.

Exhibit 1:

Sample Output from bvpmvp(...) (N=10,000, M=4, K=5)

4. Simulation Exercises

To assess the relative performance of the proposed approach and the approach based on mvprobit a simulation exercise was conducted. Three sample sizes (N=2,000, N=10,000, N=50,000) are considered. The data structure corresponding to (1)-(3) has either K=5 or K=9 covariates x (four or eight independently distributed uniform variates plus a constant) and M=8 binary outcomes y_ij (only four of which are used in some specifications) corresponding to latent ${\text{y}}_{\text{ij}}^{*}$ having cross-outcome correlations ρ_jk variously in $\left\{.2,1/\sqrt{10},.5\right\}$ for all j ≠ k, specifically

$$\mathbf{R}=\left[\begin{array}{cccccccc}1& & & & & & & \\ {10}^{-.5}& 1& & & & & & \\ .5& {10}^{-.5}& 1& & & (\text{symm.})& & \\ {10}^{-.5}& .2& {10}^{-.5}& 1& & & & \\ .5& {10}^{-.5}& .5& {10}^{-.5}& 1& & & \\ {10}^{-.5}& .2& {10}^{-5}& .2& {10}^{-.5}& 1& & \\ .5& {10}^{-.5}& .5& {10}^{-.5}& .5& {10}^{-.5}& 1& \\ {10}^{-.5}& .2& {10}^{-.5}& .2& {10}^{-.5}& .2& {10}^{-.5}& 1\end{array}\right]$$

For mvprobit, the draws(.) option was set both at 10 and 20. The simulations are performed using Stata/SE Version 13.1 on an iMac 3.4GHz Intel Core i7 processor and OS X v10.8.⁸ The do files containing the code used to generate the data and perform the simulations are available on request.

5. Simulation Results

Key results of the simulations are summarized in Tables 1–3. Table 1 displays the absolute and relative computation times for mvprobit and bvpmvp(…) estimation across the various combinations of the N, M, K, and D parameters. Enormous differences in computation time are seen between the two estimation methods across all the different parameter combinations (for reference, it may be useful to recall that there are 86,400 seconds in one day). Tables 2 and 3 present a side-by-side comparison of the point estimates of B and R obtained in one select specification (N=10,000, M=4, K=5). For both B and R the differences between the mvprobit and bvpmvp(…) point estimates and corresponding estimated standard errors are trivial.

Table 1.

Estimation Time Comparisons (in Seconds)

Parameters				Computation Time		Relative Difference (Ratio)
N	M	K	D	mvprobit	bvpmvp(…)
2,000	4	5	10	29	1	29
			20	53		53
		9	10	28	1	28
			20	54		54
	8	5	10	1,219	5	244
			20	2,041		408
		9	10	1,036	8	130
			20	2,044		256
10,000	4	5	10	142	2	71
			20	263		132
		9	10	137	3	46
			20	258		86
	8	5	10	4,628	14	331
			20	10,469		748
		9	10	4,669	19	246
			20	9,833		518
50,000	4	5	10	986	12	82
			20	1,937		161
		9	10	995	18	55
			20	1,970		109
	8	5	10	35,833	65	551
			20	72,406		1114
		9	10	36,647	86	426
			20	73,204		851

Legend

N: Number of sample observations

M: Number of outcomes

K: Number of covariates (including constant term)

D: Number of draws for mvprobit

Note: Stata’s matsize parameter is set at 600 for all specifications.

Table 3.

mvprobit and bvpmvp(…) Comparison: $\widehat{\mathbf{R}}$ Point Estimates, One Example (N=10,000, M=4, K=5; Estimated Standard Errors in Parentheses)

R	Mvprobit (draws=20)	bvpmvp(…)
ρ12	.3190 (.0158)	.3308 (.0159)
ρ13	.4942 (.0134)	.5073 (.0134)
ρ14	.2766 (.0160)	.2872 (.0161)
ρ23	.3356 (.0156)	.3424 (.0158)
ρ24	.2000 (.0163)	.2034 (.0167)
ρ34	.3059 (.0157)	.3086 (.0160)

Table 2.

mvprobit and bvpmvp(…) Comparison: $\widehat{\mathbf{B}}\text{and}\widehat{{\mathbf{B}}_{\mathbf{A}}}$ Point Estimates, One Example (N=10,000, M=4, K=5; Estimated Standard Errors in Parentheses)

Outcome	Covariate	Mvprobit (draws=20)	bvpmvp(…)
y1	x1	0.3265 (.0448)	.3279 (.0446)
	x2	−0.3301 (.0447)	−.3314 (.0447)
	x3	0.3184 (.0447)	.3198 (.0449)
	x4	−0.3902 (.0448)	−.3916 (.0447)
	Constant	0.3901 (.0466)	.3909 (.0464)
y2	x1	−0.4487 (.0456)	−.4487 (.0455)
	x2	0.5624 (.0458)	.5620 (.0456)
	x3	−0.3998 (.0457)	−.3977 (.0457)
	x4	0.4000 (.0456)	.3961 (.0457)
	Constant	−0.5086 (.0474)	−.5079 (.0474)
y3	x1	0.3102 (.0445)	.3151 (.0446)
	x2	0.3846 (.0445)	.3875 (.0449)
	x3	−0.3188 (.0446)	−.3206 (.0447)
	x4	−0.3462 (.0446)	−.3496 (.0447)
	Constant	0.3230 (.0463)	.3210 (.0463)
y4	x1	0.4567 (.0455)	.4573 (.0457)
	x2	−0.4438 (.0455)	−.4408 (.0457)
	x3	−0.4489 (.0456)	−.4516 (.0457)
	x4	0.4555 (.0456)	.4499 (.0453)
	Constant	−0.4552 (.0472)	−.4524 (.0472)

In light of these results, use of methods like bvpmvp(…) to estimate MVP models merits consideration when computation time is an important consideration.⁹

6. Multivariate Ordered Probit Models

Analogous conceptual considerations arise in the context of multivariate ordered probit (MVOP) models in which the observed ordered outcomes are yo_j ∈ {0,…,G_j} for finite integers G_j ≥ 1 . MVOP modeling involves estimation of and inference about the parameters B and R as well as the vector of category cutpoints, C (for each outcome yo_j there are G_j cutpoints that delineate the G_j + 1 categories).¹⁰

An estimation strategy fully analogous to bvpmvp(…) is not available since the bioprobit procedure (Sajaia, 2008) does not permit postestimation prediction with the score option, as required by suest. However, an alternative, fully consistent, and computationally efficient approach is available, as follows. First, estimate M univariate ordered probit models using Stata’s oprobit procedure and store these results using estimates store. This provides consistent estimates of the B and C parameters. Second, estimate a chain of bivariate binary probit models using biprobit -- as with bvpmvp(…) -- and store these estimates using estimates store. This provides a consistent estimate of R.¹¹ Note that any thresholds used to map the ordered yo_ij to their corresponding coarsened binary outcomes should result in consistent estimates of R. biprobit uses the rule that a nonbinary outcome is treated as zero for zero values and one otherwise; this is a convenient mapping that minimizes programming burden. Third, combine all the estimates stored in these two steps using suest. The estimates from suest can then be used for inference. The do file containing the Mata code for the function bvopmvop(…) that implements this approach is available with this paper’s supplementary materials.¹² An example of bvopmvop(…) output is presented in Exhibit 2.¹³

Exhibit 2:

Sample Output from bvopmvop(...) (N=10,000, M=4, K=5)

7. Summary

This paper has presented a novel estimation strategy for consistent estimation of and inference about the parameters of MVP and MVOP models. The straightforward implementation of these approaches using available Mata programs recommends their consideration in applied work, particularly in situations involving large numbers of outcomes (M), large sample sizes (N), or in situations requiring repeated MVP estimation like bootstrapping exercises.

In closing, it should be noted that the methods suggested here may prove useful in many but not all applications of multivariate probit models. Ultimately the methods proposed here -- as well as the mvprobit method -- permit estimation of the joint conditional probability model Pr(y = k|x ) for the M-vectors of outcomes y, all possible 2^M vectors k=[k_m], k_m ∈{0,1} , and exogenous covariates x. As such, when these joint conditional probabilities are per se the estimands of interest, when they are instrumentally of interest in the estimation of other quantitites (see Mullahy, 2011, for discussion), or when reduced forms of structural models are of interest, the approach suggested here may prove useful. However in other MVN contexts with binary outcomes -- e.g. where endogenous y_m are RHS variables in the structural models for latent ${\text{y}}_{\text{j}}^{*}$ -- consistent estimation of the structural parameters will typically demand attention to the full joint probability structure, not just its bivariate marginals.¹⁴

Appendix: Additional Remarks on Combining biprobit Estimates

In general, the optimal approach to combining such multiple estimates in the overidentified case is to use a minimum-distance estimator with an optimal weight matrix (Wooldridge, 2010, section 14.5). In the present context this would amount to computing a weighted average for each point estimate, i.e. ${\widehat{\mathbf{\text{β}}}}_{\text{jkw}}={\sum }_{\begin{array}{l}\text{m}=1\\ \text{m}\ne \text{j}\end{array}}^{\text{M}}{\text{w}}_{\text{jkm}}{\widehat{\mathbf{\text{β}}}}_{\text{jkm}},\text{j}=1,\dots ,\text{M},\text{k}=1,\dots ,\text{K}.$ Implementing the minimum-distance approach can be computationally challenging, however. For example, consider the simplest case, M=3. The optimal (variance-minimizing) weights even in this instance are complicated functions of the estimates’ variances and covariances; suppressing the j,k subscripts, for {p,q,r} ∈ {1,2,3}, p ≠ q ≠ r these optimal weights are:

$${\text{w}}_{\text{r}}\text{=}\frac{{\text{σ}}_{\text{pp}}{\text{σ}}_{\text{qq}}-{\text{σ}}_{\text{pq}}^{\text{2}}-{\text{σ}}_{\text{qq}}{\text{σ}}_{\text{pr}}-{\text{σ}}_{\text{pp}}{\text{σ}}_{\text{rq}}-{\text{σ}}_{\text{pr}}{\text{σ}}_{\text{pq}}-{\text{σ}}_{\text{pq}}{\text{σ}}_{\text{rq}}}{{\text{σ}}_{\text{pp}}{\text{σ}}_{\text{rr}}{\text{+σ}}_{\text{rr}}{\text{σ}}_{\text{qq}}{\text{+σ}}_{\text{pp}}{\text{σ}}_{\text{qq}}-{\text{σ}}_{\text{pr}}^{\text{2}}-{\text{σ}}_{\text{pq}}^{\text{2}}+{\text{σ}}_{\text{rq}}^{\text{2}}+\text{2}\left({\text{σ}}_{\text{pr}}{\text{σ}}_{\text{pq}}+{\text{σ}}_{\text{pq}}{\text{σ}}_{\text{rq}}+{\text{σ}}_{\text{pr}}{\text{σ}}_{\text{rq}}-{\text{σ}}_{\text{pp}}{\text{σ}}_{\text{rq}}-{\text{σ}}_{\text{rr}}{\text{σ}}_{\text{pq}}-{\text{σ}}_{\text{qq}}{\text{σ}}_{\text{pr}}\right)}$$

where ${\sigma }_{••}$ are variances and covariances of the parameter estimates (the empirical counterpart, $\widehat{{\text{w}}_{\text{r}}},$ would use $\widehat{{\sigma }_{••}}$). The algebraic complexity of these weights increases rapidly as M increases.

The considerable additional computational complexity involved in implementing such a minimum-distance approach is unlikely to provide much benefit (in terms of precision) unless the optimal w_jkm were to diverge dramatically from 1/(M-1). The simulations undertaken here suggest this is unlikely to be the case. In general the optimal weights will diverge from the equi-weighted case of 1/(M-1) to the extent that the variances and covariances of and between the parameter point estimates differ substantively across the (M-1) estimates.¹⁵

For illustrative purposes, selecting arbitrarily the (M-1) point estimates corresponding to the parameter β₁₁ (outcome y₁, covariate x₁) for the N=10,000, M=8 and K=5 specification, the range of the seven point estimates $\widehat{{\text{β}}_{11}}$ is [.3266, .3288], the range of the corresponding seven estimated point estimate variances is [.001983, .001995], and the range of the 28 estimated point estimate covariances is [.001983, .001993]. It is thus unlikely that the optimal weights would diverge much from 1/(M-1).

The ultimately important result is that at least insofar as the simulations conducted for this paper are concerned, the differences between the mvprobit and bvpmvp(…) point estimates and estimated standard errors are inconsequentially small (see Tables 2 and 3).