BIC extensions for order-constrained model selection

Abstract

The Schwarz or Bayesian information criterion (BIC) is one of the most widely used tools for model comparison in social science research. The BIC however is not suitable for evaluating models with order constraints on the parameters of interest. This paper explores two extensions of the BIC for evaluating order constrained models, one where a truncated unit information prior is used under the order-constrained model, and the other where a truncated local unit information prior is used. The first prior is centered around the maximum likelihood estimate and the latter prior is centered around a null value. Several analyses show that the order-constrained BIC based on the local unit information prior works better as an Occam’s razor for evaluating order-constrained models and results in lower error probabilities. The methodology based on the local unit information prior is implemented in the R package ‘BFpack’ which allows researchers to easily apply the method for order-constrained model selection. The usefulness of the methodology is illustrated using data from the European Values Study.

1. Introduction

The Bayesian information criterion (BIC) is one of the most commonly used model evaluation criteria in social research, for example for categorical data (Raftery, 1986), event history analysis (Vermunt, 1997), or structural equation modeling (Raftery, 1993; Lee & Song, 2007). The BIC, originally proposed by Schwarz (1978), can be viewed as a large sample approximation of the marginal likelihood (Jeffreys, 1961) based on a so-called unit information prior. This unit information prior contains the same amount of information as would a typical single observation (Raftery, 1995).

The BIC has several useful properties. First, it can be used as a default quantification of the relative evidence in the data between two statistical models. Second, it can straightforwardly be used for evaluating multiple statistical models simultaneously. Third, it is consistent for most well-behaved problems in the sense that the evidence for the true model converges to infinity (Kass & Wasserman, 1995). Fourth it behaves as an Occam’s razor by balancing model fit (quantified by the log likelihood function at the maximum likelihood estimate (MLE)) and model complexity (quantified by the number of free parameters). Fifth, it is easy to compute using standard statistical software: only the MLEs, the maximized loglikelihood, the sample size, and the number of model parameters are needed to compute it. All these useful properties have contributed to the popularity and usefulness of the BIC in social research.

Despite the general applicability of the BIC, it is not suitable for evaluating statistical models with order constraints on certain parameters. In a regression model for instance it may be expected that the first predictor has a larger effect on the outcome variable than the second predictor, and the second predictor is expected to have a larger effect than the third predictor. This can be translated to the following order-constrained model, M₁ : β₁ > β₂ > β₃, where β_k denotes the effect of the k-th predictor on the outcome variable. This model can then be tested against conflicting models, such as a model with competing order constraints, e.g., M₂ : β₃ > β₁ > β₂, a model where the effects are expected to be equal, M₃ : β₁ = β₂ = β₃, or the complement of these models, denoted by M₄. Under the complement model M₄, the true values for the β’s do not satisfy any of the constraints under models M₁, M₂, or M₃.

The reason that the BIC is not suitable for testing models with order constraints is that the number of free parameters does not properly capture the complexity of a model. In the above model M₁, all three β parameters are free parameters but saying that M₁ is equally complex as a model with no constraints, i.e, $\left({\beta }_1,{\beta }_2,{\beta }_3\right)\in {ℝ}^3$, seems incorrect. Furthermore, the BIC is based on the Laplace approximation of the marginal likelihood. It is as yet unclear how well the approximation performs in the case of models with order constraints. The complicating factor is that the approximation assumes that the maximum value of the integrand is an interior point of the integrated region. This assumption is violated if the maximum likelihood (or posterior mode) does not lie in the integrated region.

Testing order constraints is particularly useful because effect sizes can only be interpreted relative to each other in the study, and relative to the field of research (J. Cohen, 1988). An effect size of, say, 0.3, of educational level on attitude towards immigrants might seem substantial for a sociologist, while 0.3 might not be interesting when it quantifies the effect of a medical treatment on the amount of pain of a patient. Thus, instead of interpreting the magnitudes of effects by their estimated values, it may be more informative to interpret them relative to each other, as is done using order-constrained model selection. This would allow us to assess which effects dominate other effects in the study. Further, order-constrained model selection could be useful for testing scientific expectations which can often be formulated using order constraints. Examples will be given in Section 2, but see also Klugkist et al. (2005), Hoijtink (2011), Braeken et al. (2015), Mulder & Pericchi (2018) and Mulder & Fox (2018). By testing order-constrained models we can quantify the evidence in the data for one scientific theory against others.

Order-constrained models are also naturally specified when one is interested in the effect of an ordinal categorical variable on an outcome variable of interest. Also, the inclusion of order constraints results in more statistical power. This can be explained by the smaller subspace for the parameters under an order-constrained model compared to an equivalent model without the order constraints. The order constraints make the model ‘less complex’ resulting in a smaller penalty for model complexity, and thus in more evidence for an order-constrained model that is supported by the data.

In this paper we explore how the BIC can be extended to enable order-constrained model selection. First a unit information prior is considered that is truncated in the order-constrained subspace. This results in a BIC that may not properly incorporate the relative complexity of an order-constrained model. For this reason an alternative local unit information prior is considered which is centered around a null value. This prior results in a BIC that properly incorporates the relative fit and complexity of order-constrained models. The R package ‘BFpack’ has been developed for order-constrained model selection in popular models such as generalized linear models, survival models, and ordinal regression models.

To our knowledge there have been two other proposals for the BIC for evaluating models with order (or inequality) constraints by Romeijn et al. (2012) and Morey & Wagenmakers (2014), and we will compare our proposal to theirs.

The article is organized as follows. We motivate the evaluation of statistical models with order constraints on the parameters of interest in the context of the European Values Study in Section 2. In Section 3 we discuss BIC approximations of the marginal likelihood under an order-constrained model. Section 4 provides a numerical evaluation of the methods, while Section 5 describes software to implement the methods. Section 6 explains how to apply the new method for testing social theories in the European Values Study, and Section 7 discusses the results.

2. Order-constrained model selection in social research

In this section we present two situations where order-constrained model selection is useful. First, theories often make an assumption about the relative importance of certain predictors on an outcome variable. This can be formalized by specifying order constraints on the effects of these predictor variables. We will show this in Application 1 using Ethnic Competition Theory (Scheepers et al., 2002). Second, a researcher may have an expectation about the direction of an effect of a predictor variable with an ordinal measurement level. When modeling this ordinal predictor variable using dummy variables, the expected directional effect can be translated to a set of order constraints on the effects of these dummy variables. This will be shown in Application 2 by considering Inglehart’s Generational Replacement Theory.

2. 1. Application 1: Assessing the importance of different dimensions of socioeconomic status

In most European countries, the majority of immigrants are located in the lower strata of society. For this reason lower-strata members of the European majority population who hold similar social positions as the ethnic minorities, having a relatively low social class, low educational level, or low income level, will on average compete more with ethnic minorities than will other citizens in the labour market. Therefore Ethnic Competition Theory (Scheepers et al., 2002) would predict that higher social class, educational level, or income level would result in a more positive attitude towards immigrants. Furthermore, it is likely that social class (which reflects the type of job a person has) has the largest impact because one’s social class is directly related to the labour market. The effects of education is less direct and therefore it is expected that one’s educational level has a lower impact on attitude towards immigrants than social class. Finally it would be expected that the effect of income would be the lowest, but still positive. This expectation will be formalized in model M₁ which is provided below.

Alternatively, due to the importance of education in shaping one’s identity (A. K. Cohen et al., 2013; van der Waal et al., 2015), it might be expected that education is the most important factor explaining one’s attitude towards immigrants, followed by social class and income for which no specific ordering is expected (formalized in model M₂). A third hypothesis is that all three dimensions have an equal and positive effect on attitudes towards immigrants (model M₃). Finally, it may be that none of these three hypotheses is true (model M₄).

To evaluate these expectations we first write down the linear regression model where the attitude towards immigrants is the outcome variable, and social class, educational level, and income are the predictor variables while controlling for age. The i-th observation is modeled as follows,

$$\begin{gathered} \text{attitude}\left(i\right)={\theta }_0+\text{class}\left(i\right)\times {\theta }_{\text{class}}+\text{education}\left(i\right)\times {\theta }_{\text{education}} \\ +\text{income}\left(i\right)\times {\theta }_{\text{income}}+\text{gender}\left(i\right)\times {\theta }_{\text{gender}}+\text{error}\left(i\right), \end{gathered}$$ (1)

for i = 1, …, n. The predictor variables are all standardized. In (1), θ_class, θ_education, and θ_income are the standardized coefficients for social class, educational level, and income, respectively, θ_gender is the standardized coefficient for gender, and the errors are assumed to be independent and normally distributed with unknown variance.

The four expectations given above can be formalized using competing statistical models with different order constraints on the standardized effects, namely

$$\begin{array}{l}{M}_1:{\theta }_{\text{class }}>{\theta }_{\text{education }}>{\theta }_{\text{income }}>0,\hfill \\ M_2:{\theta }_{\text{education }}>\left({\theta }_{\text{class }},{\theta }_{\text{income }}\right)>0,\hfill \\ M_3:{\theta }_{\text{class }}={\theta }_{\text{education }}={\theta }_{\text{income }}>0,\hfill \\ M_4:“\text{neither\hspace{0.17em}}{M}_1,M_2,\text{ nor }{M}_3”,\hfill \end{array}$$ (2)

where θ_class, θ_education, and θ_income denote the effects of social class, educational level, and income on attitude towards immigrants, respectively. Consequently the goal is to quantify the evidence in the data for these three models to determine which model receives the most support.

Note that nuisance parameters (e.g., effects of control variables) are omitted in the above formulation of the models of interest to simplify the notation. Further note that additional competing constrained models could be formulated in this context as well. For the current application however we restrict ourselves to these models.

2.2. Application 2: The importance of postmaterialism for young, middle and old generations

Experiences in pre-adult years are known to have a crucial impact on the development of basic values in later life. Due to the increase in welfare in recent decades, Generational Replacement Theory predicts that the values of younger generations are different from those of older generations. In particular postmaterialistic values, such as the desire for freedom, self-expression, and quality of life, are expected to have increased for younger generations as a result of improved economic standards in western countries (Inglehart & Abramson, 1999; Welzel & Inglehart, 2005).

In the European Values Study, generation was operationalized using an ordinal variable with three categories corresponding to a young, middle or old generation. Similarly, postmaterialism has been measured on an ordinal scale as well, having three categories. When setting the younger generation as the reference group and using dummy variables for the middle and older generations, Generational Replacement Theory can be translated to an order-constrained model (M₁). We contrast it with a model that assumes no generation effect on postmaterialism (M₀) and with a complementary model that assumes neither an increased effect nor a zero effect (M₂).

The models of interest can be summarized as follows:

$$\begin{array}{l}{M}_0:{\theta }_{\text{old}}={\theta }_{\text{middle}}=0,\\ M_1:{\theta }_{\text{old}}<{\theta }_{\text{middle}}<0,\\ M_2:“\text{neither\hspace{0.17em}}{M}_0,\text{nor\hspace{0.17em}}{M}_1.”\end{array}$$ (3)

Furthermore we hypothesize that the inclusion of order constraints on the generational effects of interest results in an increase of statistical power in comparison to testing the classical alternative, say, M₃ : θ_young ≠ θ_middle 6= 0 versus the null model M₂ : θ_young = θ_middle = 0. In terms of the BIC this implies we obtain more evidence against M₀ when testing it against the order-constrained model M₁ (if the constraints are supported by the data) than when testing M₀ against the unconstrained alternative M₃.

3. BIC approximations of the marginal likelihood

In this section, extensions of the BIC are derived for a model with order (or inequality) constraints on certain model parameters. Consider an order-constrained model M₁ with d unknown model parameters, denoted by θ, which are restricted by r₁ order constraints, i.e., M₁ : R₁θ > r₁ where [R₁|r₁] is an augmented r₁ ×(d+1) matrix containing the coefficients of the order constraints under M₁.

For example, the order-constrained model M₁ : θ_class > θ_education > θ_income > 0 in (2) can be translated to

$$\begin{gathered} R_1\theta >r_1\iff \left[\begin{array}{cccccc}0& 1& -1& 0& 0& 0\\ 0& 0& 1& -1& 0& 0\\ 0& 0& 0& 1& 0& 0\end{array}\right] \\ \left[\begin{array}{l}{\theta }_0\hfill \\ {\theta }_{\text{class}}\hfill \\ {\theta }_{\text{education}}\hfill \\ {\theta }_{\text{income}}\hfill \\ {\theta }_{\text{gender}}\hfill \\ {\sigma }^2\hfill \end{array}\right]>\left[\begin{array}{l}0\hfill \\ 0\hfill \\ 0\hfill \end{array}\right], \end{gathered}$$ (4)

where the first element of θ denotes the intercept, the fifth element the gender effect, and the sixth element denotes the error variance, which are nuisance parameters. The order-constrained model is nested in an unconstrained model which will be denoted by M_u.

The likelihood function under M₁ is a truncation of the likelihood under an unconstrained model, i.e., $p_1\left(D|\theta \right)=p\left(D|\theta \right)\times I_{{\Theta }_1}\left(\theta \right)$, where p(D|θ) denotes the likelihood function of the data D under the unconstrained parameter space Θ. The prior for θ under M₁ will be denoted by p₁(θ). Two different types of priors will be considered for approximating the marginal likelihoods under M₁ and M_u.

3.1. Truncated unit information prior

First we assume that the unconstrained posterior mode, denoted by ${\tilde{\theta }}_u$, falls in the inequality-constrained space of model M₁, i.e., $R_1\tilde{\theta }>r_1$. The BIC approximation of the marginal likelihood under the inequality-constrained model is then obtained using a second-order Taylor expansion of the logarithm of the integrand around the posterior mode. This approximation introduces in an error that is $\mathcal{O}\left(n^{-1}\right)$.¹ Let us define g(θ) = logp₁(D|θ) + logp₁(θ). Then, the marginal likelihood can be derived by

$$\begin{gathered} \text{log\hspace{0.17em}}{p}_1\left(D\right)=\text{log\hspace{0.17em}}{\int }_{R_1\theta >r_1}{p}_1\left(D|\theta \right)p_1\left(\theta \right)d\theta \\ =\text{log\hspace{0.17em}}{\int }_{R_1\theta >r_1}\text{exp}\left\{g\left(\theta \right)\right\}d\theta \\ =\text{log\hspace{0.17em}}{\int }_{R_1\theta >r_1}\text{exp}\left\{g\left({\tilde{\theta }}_u\right)+\frac{1}{2}\left(\theta -{\tilde{\theta }}_u\right)\prime H\left({\tilde{\theta }}_u\right)\left(\theta -{\tilde{\theta }}_u\right)\right\}d\theta +\mathcal{O}\left(n^{-1}\right) \\ =\text{log\hspace{0.17em}}{p}_1\left(D|{\tilde{\theta }}_u\right)+\text{log\hspace{0.17em}}{p}_1\left({\tilde{\theta }}_u\right)+ \\ \text{log\hspace{0.17em}}{\int }_{R_1\theta >r_1}\text{exp}\left\{\frac{1}{2}\left(\theta -{\tilde{\theta }}_u\right)\prime H\left({\tilde{\theta }}_u\right)\left(\theta -{\tilde{\theta }}_u\right)\right\}d\theta +\mathcal{O}\left(n^{-1}\right) \\ =\text{log\hspace{0.17em}}{p}_1\left(D|{\tilde{\theta }}_u\right)+\text{log\hspace{0.17em}}{p}_1\left({\tilde{\theta }}_u\right)+\frac{d}{2}\text{log\hspace{0.17em}}\left(2\pi \right)-\frac{1}{2}\text{log\hspace{0.17em}}|-H\left({\tilde{\theta }}_u\right)| \\ +\text{log\hspace{0.17em}Pr}\left(R_1\theta >r_1|D,M_u\right)+\mathcal{O}\left(n^{-1}\right), \end{gathered}$$

where $H\left({\tilde{\theta }}_u\right)$ denotes the Hessian matrix of second-order partial derivatives of g(θ) evaluated at ${\tilde{\theta }}_u$.

Hence, the only difference with the original derivation is that the resulting approximation also includes the posterior probability that the order constraints of M₁ hold under the larger unconstrained model M_u. From large sample theory, the unconstrained posterior mode can be approximated with the unconstrained maximum likelihood estimate (MLE), i.e., ${\tilde{\theta }}_u\approx {\widehat{\theta }}_u$, and $-H\left({\tilde{\theta }}_u\right)\approx n{I}_E\left({\widehat{\theta }}_u\right)$, where $I_E\left({\widehat{\theta }}_u\right)$ is the expected Fisher information matrix of one observation (which can be obtained using standard statistical software). This introduces an additional approximation error of $\mathcal{O}\left(n^{-\frac{1}{2}}\right)$. Subsequently the approximated logarithm of the marginal likelihood is given by

$$\begin{gathered} \text{log\hspace{0.17em}}{p}_1\left(D\right)=\text{log\hspace{0.17em}}{p}_1\left(D|{\widehat{\theta }}_u\right)+\text{log\hspace{0.17em}}{p}_1\left({\widehat{\theta }}_u\right)+\frac{d}{2}\text{log\hspace{0.17em}}\left(2\pi \right)-\frac{d}{2}\text{log\hspace{0.17em}}\left(n\right)-\frac{1}{2}\text{log\hspace{0.17em}}|I_E\left({\widehat{\theta }}_u\right)| \\ +\text{log\hspace{0.17em}Pr}\left(R_1\theta >r_1|D,M_u\right)+\mathcal{O}\left(n^{-\frac{1}{2}}\right). \end{gathered}$$ (5)

As was pointed out by Raftery (1995), certain terms cancel out when plugging in the so-called unit information prior (see also Kass & Wasserman, 1995). The unit information prior has a multivariate normal distribution with mean equal to the MLE and variance equal to the inverse of the expected Fisher information matrix of one observation, i.e., $p^{UI}\left(\theta \right)=N\left({\widehat{\theta }}_u,I_E{\left({\widehat{\theta }}_u\right)}^{-1}\right)$. Under the constrained model M₁ we propose using a truncated unit information prior, i.e.,

$$p_1^{UI}\left(\theta \right)=p^{UI}\left(\theta \right)\times I\left(R_1\theta >r_1\right)\times {\text{Pr}}^{UI}{\left(R_1\theta >r_1|M_u\right)}^{-1},$$ (6)

where the prior probability serves as a normalization contant so that the truncated unit information prior integrates to one, i.e.,

$${\text{Pr}}^{UI}\left(R_1\theta >r_1|M_u\right)={\int }_{R_1\theta >r_1}{p}^{UI}\left(\theta \right)d\theta .$$

Evaluating the logarithm of the unconstrained unit information prior at the unconstrained MLE yields $\text{log\hspace{0.17em}}{p}^{UI}\left({\widehat{\theta }}_u\right)=-\frac{d}{2}\text{log\hspace{0.17em}}\left(2\pi \right)+\frac{1}{2}\text{log\hspace{0.17em}}|I_E\left({\widehat{\theta }}_u\right)|$, and therefore (5) becomes

$$\begin{gathered} \text{log\hspace{0.17em}}{p}_1\left(D\right)=\text{log\hspace{0.17em}}{p}_1\left(D|{\widehat{\theta }}_u\right)+-\frac{d}{2}\text{log\hspace{0.17em}}\left(n\right)+\text{log\hspace{0.17em}Pr}\left(R_1\theta >r_1|D,M_u\right) \\ -\text{log\hspace{0.17em}}{\text{Pr}}^{UI}\left(R_1\theta >r_1|M_u\right)+\mathcal{O}\left(n^{-\frac{1}{2}}\right). \end{gathered}$$ (7)

The corresponding order-constrained BIC is then obtained by multiplying the logarithm of the approximated marginal likelihood by −2 and ignoring the error term. This yields

$$\begin{gathered} \text{OC-BIC}\left(M_1\right)=-2\text{\hspace{0.17em}log\hspace{0.17em}}{p}_1\left(D|{\widehat{\theta }}_u\right)+d\text{\hspace{0.17em}log\hspace{0.17em}}\left(n\right)-2\text{\hspace{0.17em}log\hspace{0.17em}Pr}\left(R_1\theta >r_1|D,M_u\right) \\ +2\text{\hspace{0.17em}log\hspace{0.17em}}{\text{Pr}}^{UI}\left(R_1\theta >r_1|M_u\right), \end{gathered}$$ (8)

where the first two terms form the ordinary BIC of model M₁ without the order constraints, and the additional third and fourth term are used for the evaluation of the order constraints of M₁ within M_u.

Next we consider the case where the unconstrained posterior mode does not lie in the inequality-constrained subspace of M₁, i.e., $R_1\tilde{\theta }\ngeqq r_1$. In this case the second-order Taylor expansion of g(θ) around the unconstrained posterior mode (or MLE) may not be a good approximation. The rationale is that the mode under M₁, which will have a nonzero gradient, will lie on the boundary space where R₁θ = r₁.²

Because of the exponential tails of the normal distribution, a first-order Taylor expansion of g(θ) at the posterior mode under M₁, denoted by ${\tilde{\theta }}_1$, seems more appropriate (Avramidi, 2000). For example let us consider a simple inequality-constrained model, M₁ : θ ≥ 0, and let the unconstrained mode be smaller than 0, i.e., $\tilde{\theta }<0$, so that the posterior mode under M₁ is located on the boundary, i.e., ${\tilde{\theta }}_1=0$, which has a negative gradient, g′(0) < 0. The function g(θ) for such a situation is plotted in Figure 1 (black line, solid line under θ ≥ 0, dotted line under θ ≱ 0). The second-order Taylor approximation at the unconstrained mode is also plotted (red line; solid line under θ ≥ 0, dotted line under θ ≱ 0).

Figure 1:

Plot of the log of prior times likelihood, g(θ), (black line), first-order Taylor approximation at θ = 0 (green line), and second-order Taylor approximation around the unconstrained posterior mode of $\tilde{\theta }\approx -.5$ (red line), for an inequality-constrained model M₁ : θ ≥ 0. The left panel displays the functions of the log scale and the right panel on the regular scale. The inequality-constrained region under M₁ : θ ≥ 0 has solid lines, and the complement region has dotted lines.

A first-order Taylor expansion at ${\tilde{\theta }}_1=0$ can be used to approximate the function in the region θ ≥ 0 according to

$$g\left(\theta \right)=g\left(0\right)+g\prime \left(0\right)\theta +\mathcal{O}\left({\theta }^2\right).$$

The marginal likelihood can then be approximated as follows

$$\begin{gathered} \text{log\hspace{0.17em}}{p}_1\left(D\right)=\text{log\hspace{0.17em}}{\int }_{\theta >0}{p}_1\left(D|\theta \right)p_1\left(\theta \right)d\theta =\text{log\hspace{0.17em}}{\int }_{\theta >0}\text{exp}\left\{g\left(\theta \right)\right\}d\theta \\ \approx \text{log\hspace{0.17em}}{\int }_{\theta >0}\text{exp}\left\{g\left(0\right)+g\prime \left(0\right)\theta \right\}d\theta =g\left(0\right)-\text{log\hspace{0.17em}}\left(-g\prime \left(0\right)\right). \end{gathered}$$

Hence, instead of the normal distribution which is used to compute the integral in the case of a second-order Taylor expansion, an exponential distribution is used to compute the integral using this first-order Taylor expansion. The approximated line is also plotted in Figure 1 (green line).

The figure suggests that the second-order Taylor approximation at the unconstrained posterior mode is less accurate than the first-order Taylor approximation at the boundary point. This suggests that the approximated marginal likelihood under the inequality-constrained model will generally be better using first-order approximation at the boundary point in the case that the inequality constraints are not supported by the data. In the remainder of this paper however we shall use the second-order Taylor approximation at the unconstrained posterior mode both when the posterior mode does and does not lie in the subspace of the inequality-constrained model under investigation.

When the order constraints are not supported by the data, the crudeness of the approximation is less important because the order-constrained model will not be selected because of the bad fit. Instead another, better fitting model will be selected for which the approximated marginal likelihood can be accurately estimated. Another reason for working with the second-order Taylor approximation is that it can easily be computed using (7), also for more complex systems of inequality constraints on multiple parameters, e.g., θ₁ > θ₂ > θ₃ > 0, than when using a first-order Taylor approximation at boundary point of the inequality-constrained subspace where the mode is located. Numerical experiments presented later illustrate that the approximation error is acceptable when the posterior mode does not lie in the constrained subspace.

It has been argued that data-based priors, such as the unit information prior, may result in Bayes factors that do not function as an Occam’s razor when evaluating inequality-constrained models (Mulder, 2014a, b). To see that this is also the case for the unit information prior, the approximated Bayes factor of an inequality-constrained model M₁ against an unconstrained model M_u (where the inequality constraints are omitted) is given by

$$B_{1u}^{UI}\approx \frac{\text{Pr}\left(R_1\theta >r_1|D,M_u\right)}{{\text{Pr}}^{UI}\left(R_1\theta >r_1|M_u\right)}.$$

This follows automatically from (7).

Now in the case of overwhelming evidence for M₁, i.e, $R_1{\widehat{\theta }}_u\gg r_1$, both the posterior probability and the prior probability based on the unit information prior will be approximately one, resulting in equal evidence for M₁ and M_u. This is a consequence of the fact that the unit information prior is concentrated around the MLE. Because both models fit the data equally well while the inequality-constrained model can be viewed as a less complex model (because a ‘smaller’ subspace is spanned), this property suggests that the approximated Bayes factor does not properly function as an Occam’s razor.

3.2. Truncated local unit information prior

Due to the behavior of the unit information prior when evaluating order-constrained models, we consider a ‘local’ unit information prior with a mean that is located on the boundary of the inequality-constrained space (we borrow the term ‘local’ from Johnson & Rossell, 2010). Note that the boundary space is equal to the parameter space under the null model M₀ : R₁θ = r₁. The rationale for centering the prior around the null space dates back at least to Jeffreys (1961) who argued that when the null model is false the effects are expected to be close to the null; otherwise there is no point in testing the null. This implies that the prior under the alternative model should be located around the null value. Furthermore there have been reports in the literature where the use of such local priors result in desirable selection behavior when evaluating order-constrained models (e.g., Mulder et al., 2010; Mulder, 2014a). Here we explore this class of priors for the BIC.

We set the mean of the local unit information prior equal to the MLE under the null model, denoted by ${\widehat{\theta }}_0$. Furthermore, the covariance matrix will be equal to the covariance matrix of the unit information prior. Thus, the unconstrained local unit information prior can be written as $p^{LUI}\left(\theta \right)=N\left({\widehat{\theta }}_0,I_E{\left({\widehat{\theta }}_u\right)}^{-1}\right)$. The truncated prior under M₁ : R₁θ > r₁ is then equal to

$$p_1^{LUI}\left(\theta \right)=p^{LUI}\left(\theta \right)\times I\left(R_1\theta >r_1\right)\times \frac{1}{{\text{Pr}}^{LUI}\left(R_1\theta >r_1|M_u\right)}.$$

By applying formula (14) in Kass & Raftery (1995), changing the unit information prior to the local unit information prior, results in an approximated logarithm of the marginal likelihood of

$$\begin{gathered} \text{log\hspace{0.17em}}{\widehat{p}}_1^{LUI}\left(D\right)=\text{log\hspace{0.17em}}p\left(D|{\widehat{\theta }}_u\right)-\frac{d}{2}\text{log\hspace{0.17em}}\left(n\right)-\frac{1}{2}\left({\widehat{\theta }}_u-{\widehat{\theta }}_0\right)\prime I_E\left(\widehat{\theta }\right)\left({\widehat{\theta }}_u-{\widehat{\theta }}_0\right) \\ +\text{log\hspace{0.17em}Pr}\left(R\theta >r|D,M_u\right)-\text{log\hspace{0.17em}}{\text{Pr}}^{LUI}\left(R\theta >r|M_u\right). \end{gathered}$$ (9)

Consequently, the approximated Bayes factor based on the local unit information prior of an inequality-constrained model against an unconstrained model is given by

$$B_{1u}^{LUI}\approx \frac{\text{Pr}\left(R_1\theta >r_1|D,M_u\right)}{{\text{Pr}}^{LUI}\left(R_1\theta >r_1|M_u\right)}.$$ (10)

Now in the case of overwhelming evidence for M₁, in the sense that $R_1{\widehat{\theta }}_u\gg r_1$, the Bayes factor will be equal to the reciprocal of the prior probability that the inequality constraints hold under the unconstrained local unit information prior, i.e., $B_{1u}^{LUI}\approx {\left({\text{Pr}}^L\left(R_1\theta >r_1|M_u\right)\right)}^{-1}$, which is strictly larger than one because the prior mean is located on the boundary of the constrained space where R₁θ = r₁. Note that this prior probability can be viewed as a quantification of the relative size of the inequality-constrained subspace. For example in the case of a diagonal covariance matrix, the prior probability of k one-sided constraints, θ > 0, is equal to 2^−k, and the prior probability of k order constraints, θ₁ < … < θ_k, is equal to (k!)⁻¹, similar to the Bayes factors proposed by Mulder et al. (2010) and Morey & Wagenmakers (2014).

Instead of working with (9) we consider a slightly cruder approximation where the third term, which quantifies prior fit, is omitted. This yields

$$\text{log\hspace{0.17em}}{\widehat{p}}_1^{LUI*}\left(D\right)=\text{log\hspace{0.17em}}{p}_u\left(D|{\widehat{\theta }}_u\right)-\frac{d}{2}\text{log\hspace{0.17em}}\left(n\right)+\text{log\hspace{0.17em}Pr}\left(R\theta >r|D,M_u\right)-\text{log\hspace{0.17em}}{\text{Pr}}^{LUI}\left(R\theta >r|M_u\right).$$ (11)

The rationale for omitting this term is that we are not interested in quantifying prior misfit. Another reason is that expression (11) can be combined with the ordinary BIC approximation for an unconstrained model (i.e., $‘\text{log\hspace{0.17em}}p\left(D|{\widehat{\theta }}_u\right)-\frac{d}{2}\text{log\hspace{0.17em}}\left(n\right)’$) to obtain the approximated Bayes factor in (10).

The terms on the right-hand side of (11) have the following intuitive interpretations. The first and second term can be interpreted as measures of model fit and model complexity of the unconstrained model where the inequality constraints are excluded (similar to the ordinary BIC approximation based on the unit information prior). The third term, which is the approximated posterior probability that the inequality constraints hold under the unconstrained model, can be interpreted as a measure of the relative fit of an order-constrained model M₁ relative to the unconstrained model M_u. Finally, the fourth term, which is the local prior probability that the order constraints hold under the unconstrained model, can be interpreted as a measure of the relative complexity of the order-constrained model M₁ relative to the unconstrained model M_u.

Thus (11) will behave as an Occam’s razor when evaluating order-constrained models by balancing the fit and complexity of the order-constrained model. The corresponding order-constrained BIC based on the local unit-information prior then yields

$${\text{OC-BIC}}^{*}\left(M_1\right)=-2\text{\hspace{0.17em}log\hspace{0.17em}}{p}_u\left(D|{\widehat{\theta }}_u\right)+d\text{\hspace{0.17em}log\hspace{0.17em}}\left(n\right)-2\text{\hspace{0.17em}log\hspace{0.17em}Pr}\left(R\theta >r|D,M_u\right)+2\text{\hspace{0.17em}log\hspace{0.17em}}{\text{Pr}}^{LUI}\left(R\theta >r|M_u\right).$$ (12)

3. 3. Comparison with other BIC extensions

The order-constrained BIC in (12) shows some similarities with the BIC extensions proposed by Romeijn et al. (2012) and Morey & Wagenmakers (2014). In the proposal of Romeijn et al., the prior can be chosen by users allowing a subjective quantification of the relative size of the constrained space. Although this may be useful in certain situations, the BIC is typically used in an automatic fashion, and thus it may be preferable to also let the prior probability be based on a default prior. The advantage of using the local unit information prior for this purpose is that it results in a reasonable default measure for the relative size of an order-constrained parameter space because the prior is centered on the boundary of the constrained space (unlike the (nonlocal) unit information prior). For example, when considering a univariate one-sided constraint, θ < 0, the prior probability based on the local unit information prior will be $\frac{1}{2}$, which seems reasonable because half of the unconstrained space of θ is covered by the one-sided constraint.

Furthermore Romeijn et al. set the posterior probability that the order constraints hold to 1 in the case the MLE is in agreement with the constraints, and 0 elsewhere. This additional approximation step follows directly from large sample theory: When the sample size goes to infinity the posterior probability converges to 1 if the true parameter value is an interior point of the order-constrained subspace, and 0 if it is an interior point of the complement of this subspace. Thus, for extremely large samples, the prior-adapted BIC of Romeijn et al. may perform similarly to the order-constrained BIC in (12).

For modestly sized samples however, or in the case of small effects (as is typical in social research), setting the posterior probability to either 1 or 0 may result in crude approximations of the posterior probability. As will be shown in the empirical application in Section 6.1 for example, the posterior probabilities that two competing sets of order constraints hold under an unconstrained model are equal to .50 and .18. Setting these probabilities to 1 and 0, respectively, would result in an unnecessarily crude estimate of the marginal likelihood. Instead we recommend using the actual posterior probability that the order constraints hold based on the unconstrained approximated posterior (the third term in (12)).

In the proposal of Morey & Wagenmakers (2014) the prior probability that a specific ordering of d parameters hold, e.g., θ₁ < … < θ_d, is set to 1/d!. This probability is thus based on the assumption that each ordering is equally likely a priori, similar to the priors proposed by Mulder et al. (2010) and Klugkist et al. (2005) when using Bayes factors. This probability however holds only for specific covariance structures, such as a diagonal covariance structure. The prior probability may not be invariant for reparameterizations of the model (see also Mulder, 2014a). For example, if we would define ξ_d′ = θ_d′ − θ_d′−1, for d′ = 2, …, d, and ξ₁ = θ₁, the above order constraints would be equivalent to the one-sided constraints (ξ₁, …, ξ_d−1) > 0. If one would use a prior diagonal covariance structure for ξ and zero means, the prior probability would be equal to 1/2^d−1. This may be very different from 1/d!, resulting in a serious violation of invariance to reparamaterizations. The prior probability based on the local unit information (the fourth term on the right-hand side of (12)) on the other hand would be invariant for such reparameterizations as the prior covariance structure is automatically transformed along with the reparameterization.

4. Numerical analyses

The behavior of approximated Bayes factors based on the unit information prior and the local unit information prior will be investigated in a numerical example of the linear regression model, yi = θ_{0 +} θ₁x_i1 + θ₂x_i2 + ϵ_i, with ϵ_i ~ N(0, σ²), for i = 1, …, n. Here θ₀ is the intercept, and θ₁ and θ₂ are the effects of the first and second predictor. We consider a model selection problem between an order-constrained model M₁ : θ₂ > θ₁ > 0, a null model M₀ : θ₂ = θ₁ = 0, and the complement model, M₂ : θ₂ ≱ θ₁ ≱ 0. To gain more insight into the behavior of the criterion as an Occam’s razor, we also test the order-constrained model M₁ against the unconstrained model, $M_u:\left({\theta }_1,{\theta }_2\right)\in {ℝ}^2$.

4.1. Statistical evidence for order-constrained models

To understand better how the approximated Bayes factors quantify statistical evidence for order constrained models, we computed the approximated Bayes factors for data with $\left({\widehat{\theta }}_1,{\widehat{\theta }}_2\right)=\left(a,2a\right)$, for a ∈ (−1.5, 1.5), while fixing ${\widehat{\theta }}_0=0,{\widehat{\sigma }}^2=1$, n = 20, and X′X = [n 0 0; 0 n n/2;0 n/2 n] (the exact choice of these fixed values did not qualitatively affect the results). Thus there is evidence for M₁, M₀, and M₂ when a > 0, a = 0, and a < 0, respectively.

The logarithm of the approximated Bayes factors can be found in Figure 2. Based on the approximated Bayes factors of M₁ versus M_u (left panel) we can see that the evidence based on the unit information prior (dotted line) for M₁ against M_u starts to decrease for larger effects (for approximately a = .3 and larger), which seems counterintuitive. Eventually the weight of evidence (i.e., the log Bayes factor) converges to 0. Thus, in the case of overwhelming evidence for an order constrained model, we obtain equal evidence for an order-constrained model M₁ that is fully supported by the data and the ‘larger’ unconstrained model M_u when using the unit-information prior, even though M₁ is a simpler model. This suggests that the approximated Bayes faction based on the unit information prior does not work as an Occam’s razor when evaluating order constrained models.

Figure 2:

The logarithm of the approximated Bayes factors based on the unit information prior $\text{log\hspace{0.17em}}\left({\widehat{B}}^{UI}\right)$ (dotted line), and the local unit information prior $\text{log\hspace{0.17em}}\left({\widehat{B}}^L\right)$ (solid line) of M₁ : θ₂ > θ₁ > 0 versus $M_u:\left({\theta }_1,{\theta }_2\right)\in {ℝ}^2$ (left panel), of M₁ versus the complement model M₂ (middle panel), and of M₁ versus M₀ : θ₂ = θ₁ = 0 (right panel). The criteria are plotted for n = 20 as a function of a, where $\left({\widehat{\theta }}_1,{\widehat{\theta }}_2\right)=\left(a,2a\right)$.

The evidence for M₁ against M_u based on the local unit-information prior (solid line) on the other hand increases as a function of a. Eventually the weight of evidence converges to the reciprocal of the prior probability of θ₂ > θ₁ > 0 under the unconstrained model M_u, which is strictly larger than 0. Furthermore the local unit-information prior results in more evidence for a model that is supported by the data in comparison to the unit-information prior when comparing model M₁ versus model M₂ (Figure 2, middle panel) and model M₁ versus M₀ (Figure 2, right panel). Based on these considerations we conclude that the order-constrained BIC based on the local unit-information prior better balances fit and complexity when evaluating order-constrained models than the order-constrained BIC based on the unit-information prior.

4. 2. Error probabilities

Next we investigate the probabilities of selecting the true data generating model when including order constraints in the alternative model or not. First we consider testing the null model, M₀ : θ₁ = θ₂ = 0, against an unconstrained alternative, $M_u:\theta \in {ℝ}^2$, using the ordinary BIC. Second we consider testing the null model M₀ : θ₁ = θ₂ = 0 versus M₁ : θ₂ > θ₁ > 0 against two order-constrained alternative, namely M₂ : θ₂ ≱ θ₁ ≱ 0, using the two order-constrained BICs. Note that the BIC for M₀, with no inequality constraints, is the same in both tests. Further note that because the second test contains three models instead of two, the error probabilities in the second selection problem will be slightly larger when M₀ is true, as a result of the design. The true effects will be set to (θ₁, θ₂) = (a, 2a), for a = 0, so that M₀ is true, and a = .1, .2, and .4, so that M_u (M₁) is true in the first (second) test.

Figure 3 displays the error probabilities as a function of the sample size (on a log-scale). All the criteria show consistent behavior in the sense that the error probabilities go to zero as the sample size grows. Furthermore we see that when M₀ is true (upper left panel), the error probabilities are very similar and the ordinary BIC in test 1 results in the smallest errors. In the case of a true effect in the direct of the order constraints of M₁ we see that the order-constrained BIC based on the local unit-information prior results in considerably smaller errors than the other criteria.

Figure 3:

Probability of selecting the wrong model when using the ordinary BIC for testing M₀ : θ₁ = θ₂ = 0 against $M_u:\theta \in {ℝ}^2$ (dashed line), and the two order-constrained BICs when testing M₀ : θ₁ = θ₂ = 0, M₁ : θ₂ > θ₁ > 0, and M₂ : θ₂ ≱ θ₁ ≱ 0 (dotted and solid line for the non-local and local unit-information prior, respectively) for true effects of (θ₁, θ₂) = (a, 2a), for a = 0, .1, .2, and .4. The sample size on the x-axis is on a logarithmic scale.

The error probabilities of the order-constrained BIC based on the local unit-information prior were only slightly larger in the case of a non-zero effect. This is partly a consequence of the design of the test having three instead of two models under investigation. We conclude that overall the order-constrained BIC based on the local unit-information prior performs best in terms of error probabilities.

4. 3. Approximation errors of the order-constrained BICs

Finally we investigated the relative approximation errors of the order-constrained BICs by comparing them to nonapproximated counterparts, e.g., $\frac{\text{log\hspace{0.17em}}{B}_{12}-\text{log\hspace{0.17em}}{\widehat{B}}_{12}}{\text{log\hspace{0.17em}}{B}_{12}}$ for model M₁ against M₂. The approximation errors were investigated when the order-constrained model is supported by the data, namely when testing M₁ : θ₂ > θ₁ > 0 versus M₀ : θ₁ = θ₂ = 0, with $\left({\widehat{\theta }}_1,{\widehat{\theta }}_2\right)=\left(.5,1\right)$, and when the order-constrained is not supported by the data, namely when testing M₁ : θ₂ > θ₁ > 0 against its complement M₂ : θ₁ ≱ θ₂ ≱ 0, with $\left({\widehat{\theta }}_1,{\widehat{\theta }}_2\right)=\left(-.5,-1\right)$, while increasing the sample size.

The results can be found in Figure 4. As can be seen from the left panel, the relative error goes to 0 fast when the effects are in agreement with the order constraints of model M₁. When the effect are not in agreement with the constraints (right panel), we see that the relative error does not go to zero. This is a consequence of the somewhat crude approximation we already observed in Figure 1 (red line). The approximation error however is not large enough to be a serious practical problem. Other settings resulted in qualitatively similar results.

Figure 4:

Left panel: Relative approximation error of the order-constrained BIC of M₁ : θ₂ > θ₁ > 0 versus M₀ : θ₁ = θ₂ = 0 when ($\left({\widehat{\theta }}_1,{\widehat{\theta }}_2\right)=\left(.5,1\right)$. Right panel: Relative approximation error of the order-constrained BIC of M₂ : θ₁ ≯ θ₂ ≯ 0 versus M₁ : θ₂ > θ₁ > 0 when $\left({\widehat{\theta }}_1,{\widehat{\theta }}_2\right)=\left(-.5,-1\right)$

5. Software

The R-package ‘BFpack’ was developed for evaluating order-constrained models using the order-constrained BIC based on the local unit-information prior. The R-functions can be downloaded from www.github.com/jomulder/BFpack³, or from CRAN in the near future. The order-constrained BIC based on the truncated unit-information prior was not considered because of its poorer performance that we observed in the numerical simulations. The package makes use of the mvtnorm-package (Genz et al., 2016) for computing the probabilities in (11). The key function is ‘bic_oc’, which can be used for computing the order-constrained BIC for various statistical models, including generalized linear models and survival models. As input the function needs a fitted model object (e.g., a fitted glm-object or coxph-object), a character string denoting the order constraints on certain parameters, and a Boolean argument denoting whether the order-constrained subspace or its complement is considered (the default is the order-constrained subspace).

For example, in the case of a regression model with three predictors, say, X1, X2, and X3, on an outcome variable y, and it is expected that X1 has the largest effect on the outcome variable, followed by X2, and X3 is expected to have the smallest effect, and all effects are expected to be positive, the order-constrained BIC can be computed by executing the following lines

$$\begin{array}{l}\text{fit}1<-\text{\hspace{0.17em}glm}\left(\text{y\hspace{0.17em}}~\text{\hspace{0.17em}X}1+\text{\hspace{0.17em}X}2+\text{\hspace{0.17em}X}3,\text{\hspace{0.17em}data}\right)\\ \text{bic}\_\text{oc}\left(\text{fit}1,"0<\text{\hspace{0.17em}X}1<\text{\hspace{0.17em}X}2<\text{\hspace{0.17em}X}3"\right)\end{array}$$

The use of the function will be illustrated in two empirical applications in the next section.

6. Empirical applications revisited

The models from the applications in Section 2 are evaluated using the order-constrained BIC based on the local unit information prior using the R-package BFpack. For the empirical analyses presented in this section, only the European Values Study data of Germany are considered.

6. 1. Application 1: Assessing the importance of different dimensions of socioeconomic status

Model (1) can be fitted in R using the lm function:

$$\text{lm}1<-\text{\hspace{0.17em}lm}\left(\text{atti}\_{\text{immi\hspace{0.17em}}}^{~}\text{\hspace{0.17em}class\hspace{0.17em}}+\text{\hspace{0.17em}education\hspace{0.17em}}+\text{\hspace{0.17em}income\hspace{0.17em}}+\text{\hspace{0.17em}gender},\text{\hspace{0.17em}data}=\text{EVS}\_\text{Germany}\right)$$

The estimated coefficients of interest were (β_class, β_education, β_income) = (.312, .250, .041), with standard errors .067, .075, .072, respectively.

The order-constrained BIC for model M₁ : θ_class > θ_education > θ_income > 0 in (2) can then be computed using the new bic_oc-function:

$$\text{bic}\_\text{oc}\left(\text{lm}1,"\text{class\hspace{0.17em}}>\text{\hspace{0.17em}education\hspace{0.17em}}>\text{\hspace{0.17em}income\hspace{0.17em}}>0"\right)$$

This resulted in a BIC of 3918.46. The function also provides the posterior probability that the constraints hold under the unconstrained model, which was equal to .50. Next the BIC for model M₂ : θ_education > (θ_class, θ_income) > 0 is computed using the command

$$\text{bic}\_\text{oc}\left(\text{lm}1,"\text{education\hspace{0.17em}}>\left(\text{class},\text{\hspace{0.17em}income}\right)>0"\right)$$

The resulting OC-BIC was 3921.98. For this set of constraints, the posterior probability under the unconstrained model equaled .18.

The BIC for model M₃ : θ_class = θ_education = θ_income > 0 is computed. First the model is fitted with the equality constraints on the effect but without the inequality constraint. Because the effects of social class, education, and income are equal under M₃, the regression model in (1) becomes

$$\text{atti\_immi = }{\theta }_0\text{ + }\left(\text{class + education + income}\right)\times {\theta }_{\text{class}.\text{educ}.\text{income}}\text{ + error}$$

where θ_{class.educ.income} denotes the equal effect of social class, educational level, and income on attitude towards immigrants. Thus, this model can be fitted by including the sum of the class, education, and income as a linear predictor:

$$\begin{array}{l}\text{EVS}\_\text{Germany\$class}.\text{educ}.\text{income\hspace{0.17em}}<-\text{\hspace{0.17em}EVS}\_\text{Germany\$class\hspace{0.17em}}+\text{\hspace{0.17em}EVS}\_\text{Germany\$education\hspace{0.17em}}+\text{\hspace{0.17em}EVS}\_\text{Germany\$income}\\ \text{lm}2<-\text{\hspace{0.17em}lm}\left(\text{atti}\_\text{immi\hspace{0.17em}}~\text{\hspace{0.17em}class}.\text{educ}.\text{income\hspace{0.17em}}+\text{\hspace{0.17em}gender},\text{\hspace{0.17em}data}=\text{EVS}\_\text{Germany}\right)\end{array}$$

The order-constrained BIC can then be computed based on the resulting fitted model:

$$\text{bic}\_\text{oc}\left(\text{lm}2,"\text{class}.\text{educ}.\text{income\hspace{0.17em}}>0"\right)$$

This resulted in a BIC of 3917.84.

Finally, to compute the BIC of the complement model M₄ : “neither M₁, M₂, nor M₃, “, first note that the marginal likelihood of the union of M₁, M₂, and M₃ would be the same as the marginal likelihood of the union of only M₁ and M₂ because M₃ has zero probability due to the presence of the equality constraints of M₃. Thus, we need to compute the marginal likelihood of the complement model of the joint of models M₁ and M₂. First we combine the two sets of order constraints in one vector, and then compute the order-constrained BIC using the new function:

$$\begin{array}{l}\text{constraints}\_\text{M}4<-\text{\hspace{0.17em}c}\left("\text{class\hspace{0.17em}}>\text{\hspace{0.17em}education\hspace{0.17em}}>\text{\hspace{0.17em}income\hspace{0.17em}}>0","\text{education\hspace{0.17em}}>\left(\text{class},\text{\hspace{0.17em}income}\right)>0"\right)\\ \text{bic}\_\text{oc}\left(\text{lm}1,\text{\hspace{0.17em}constraints}\_\text{M}4,\text{\hspace{0.17em}complement\hspace{0.17em}}=\text{\hspace{0.17em}TRUE}\right)\end{array}$$

This resulted in a BIC of 3926.13. The BIC values are summarized in Table 1. From these values we can conclude that model M₃ receives the most support but the evidence is negligible in comparison to the evidence for the order-constrained model M₁, given the BIC difference of .62⁴. The evidence for M₂ and M₃ is considerably lower than for M₁ and M₃.

Table 1:

Order-constrained BICs and posterior model probabilities for the competing models in Application 1.

	OC-BIC*	P(Mt|D)
M1 : θclass > θeducation > θincome > 0	3918.46	0.391
M2 : θeducation > (θclass, θincome) > 0	3921.98	0.067
M3 : θclass = θeducation = θincome > 0	3917.84	0.533
M4 : “neither M1, M2, nor M3,”	3926.13	0.008

For interpretation purposes it can be useful to translate the BICs to posterior model probabilities. A posterior model probability quantifies the probability of the data having been generated by one of the models considered, after observing the data given certain prior model probabilities. This probability is conditional on the data having been generated by one of the models considered.

In this application we assume equal prior probabilities for the models. The posterior model probabilities can be computed from the BIC values using the ‘postprob’ function in BFpack. The posterior probabilities together with the BICs can be found in Table 1. Hence the posterior probability for model M₃, which assumes equal and positive effects of social class, education, and income on attitude towards immigrants, is largest with 53.3%. The posterior probability of M₁, which assumed ordered positive effects of social class, education, and income based on the Ethnic Competition Theory, is only slightly smaller with 39.1%. There is not much evidence for either M₂ or M₂, given their posterior probabilities of 6.7% and .8%, respectively. There is thus considerable model uncertainty, and more data would be needed to choose a single best model.

6.2. Application 2: The importance of postmaterialism for young, middle and old generations

Because the outcome variable ‘postmaterialism’ has an ordinal measurement level with three categories (‘low’, ‘medium’, and ‘high’), an ordinal regression model can be fitted using the polr-function of the MASS-package. Thus, the ordinal variable ‘postmaterialism’ is regressed to the ordinal predictor ‘generation’ with categories ‘young’, ‘middle’, and ‘old’ while controlling for ‘gender’, ‘income’, and ‘education’:

$$\text{fit}3<-\text{\hspace{0.17em}polr}\left({\text{postmaterial\hspace{0.17em}}}^{~}\text{\hspace{0.17em}generation\hspace{0.17em}}+\text{\hspace{0.17em}gender\hspace{0.17em}}+\text{\hspace{0.17em}income\hspace{0.17em}}+\text{\hspace{0.17em}education},\text{\hspace{0.17em}data}=\text{EVS}\_\text{Germany},\text{\hspace{0.17em}Hess}=\text{TRUE}\right)$$

In the fitted model the ‘young’ generation is the reference group and dummy variables are created for the ‘middle’ and ‘old’ generation. These variables are called ‘generationmiddle’ and ‘generationold’ in the fitted polr-object. The estimated effects under this model were equal to $\left({\widehat{\theta }}_{\text{generationmiddle}},{\widehat{\theta }}_{\text{generationold}}\right)=\left(-.444,-.848\right)$, having standard errors of .154 and .150, respectively.

Thus, the order-constrained BIC of model M₁ : θ_{generationold} < θ_{generationmiddle} < 0, representing the Generational Replacement Theory, can be computed by the command

$$\text{bic}\_\text{oc}\left(\text{fit}3,"\text{generationold\hspace{0.17em}}<\text{\hspace{0.17em}generationmiddle\hspace{0.17em}}<0"\right)$$

The resulting BIC equaled 3154.82.

Next, the BIC of the null model M₀ : θ_{generationold} = θ_{generationmiddle} = 0 is computed with no generation effect. Because this model does not contain any order constraints, we can simply compute an ordinary BIC. This can also be done using the bic_oc-function by omitting any order constraints:

$$\begin{gathered} \text{fit}4<-\text{polr}\left({\text{postmaterial\hspace{0.17em}}}^{~}1+\text{\hspace{0.17em}gender\hspace{0.17em}}+\text{\hspace{0.17em}income\hspace{0.17em}}+\text{\hspace{0.17em}education},\text{\hspace{0.17em}data}=\text{EVS}\_\text{Germany},\text{\hspace{0.17em}Hess}=\text{TRUE}\right) \\ \text{bic}\_\text{oc}\left(\text{fit}4\right) \end{gathered}$$

The resulting BIC was equal to 3177.69.

Finally the BIC of the complement model was computed. Similarly to the previous example, this can be done as follows

$$\text{bic}\_\text{oc}\left(\text{fit}3,"\text{generationold\hspace{0.17em}}<\text{\hspace{0.17em}generationmiddle\hspace{0.17em}}<0",\text{\hspace{0.17em}complement}=\text{TRUE}\right)$$

This resulted in a BIC of 3170.15. The BICs and respective posterior model probabilities can be found in Table 2. Clearly, there is overwhelming evidence for M₁ which implies that postmaterialism has increased for younger generations.

Table 2:

Order-constrained BICs and posterior model probabilities for the competing models in Application 2.

	OC-BIC*	P(Mt|D)
M0 : θold = θmiddle = 0	3177.69	0.00
M1 : θold < θmiddle < 0	3154.82	1.00
M2 : “neither M0, nor M1,”	3170.15	0.00

Finally we show that the inclusion of order constraints in the alternative model results in more evidence against a null model if the order constraints are supported by the data. First note that the BIC for the order-constrained model, M₁, against the null model, M₀, equals BIC(M₁, M₀) = BIC(M₁) − BIC(M₀) = 3154.82 − 3177.69 = −22.87. The BIC for an unconstrained alternative model, M₃ : θ_{generationold} ≠ θ_{generationmiddle} ≠ 0, against the null model equals BIC(M₃, M₀) = BIC(M₃) − BIC(M₀) = 3158.18 − 3177.69 = −19.51.⁵ Hence, the inclusion of order constraints results in a substantial increase of the evidence against the null model in the case where the order constraints are supported by the data. We also get a more informative answer about how the effects are related to each other in the case there is evidence against the null model than when testing the null against an unconstrained alternative.

7. Discussion

We have presented two extensions of the BIC for evaluating models with order constraints on certain parameters of interest. In the first extension a truncated unit-information prior was considered under the order-constrained model and in the second extension a truncated local unit-information prior was considered. Theoretical considerations and numerical analyses revealed that the local unit-information prior resulted in better model selection behavior than the non-local unit information prior for order-constrained model selection.

The new order-constrained BIC based on the local unit-information prior can easily be computed using the new R-package ‘BFpack’. This will allow researchers to test multiple social theories that can be translated into conflicting sets of equality and order constraints on the parameters of interest. The methodology can also be used for testing directed effects of ordinal predictors, as these expectations can be translated into order-constrained models in a natural manner.