A Note on N in Bayesian Information Criterion for Item Response Models

Abstract

This brief report derives the N in the penalty term of the Schwarz’s (1978) Bayesian information criterion (BIC) for two-parameter logistic item response models. The results in this study show that the N is the number of persons for fixed item models, whereas it is the number of observations (the Number of Persons times the Number of Items) for random item models. Given these results, the authors recommend researchers to calculate the BIC or to validate the BIC value that shows in the output of software instead of accepting the output value without a further check of implicit assumptions made for the software.

It is common to use information criteria such as Schwarz’s (1978) Bayesian information criterion (BIC) in model selection for item response models, as exhibited by their automatic computation in software packages. It has been observed that there are discrepancies in the formula of the BIC for item response models in software. The discrepancies concern the sample size term $N$ in the penalty term of the BIC. As an example, for the one-parameter item response model with a random person effect and a fixed item effect using (an approximate) marginal maximum likelihood estimation, $N$ is the number of persons in flexMIRT (Cai, 2015), Mplus (Muthén & Muthén, 1998-2015), and SAS NLMIXED (e.g., Nandakumar & Hotchkiss, 2012, for item response models), whereas it is the total number of observations (i.e., the Number of Persons $\mathrm{\times }$ the Number of Items) in the glmer function of lme4 R package (Bates, Mächler, Bolker, & Walker, 2015). In the glmer function, the $N$ is the total number of observations for the one-parameter item response model with a random person effect and a random item effect. However, the model cannot be fit using flexMIRT, SAS NLMIXED, and Mplus.¹

In spite of the popularity of the BIC, to the authors’ knowledge, there is no derivation available for the $N$ to be used for the BIC of item response models. Different calculation for BIC can result in different model selections and the same calculation for BIC can be misleading if that same calculation is correct for one model but not for another. The aim of this brief report is to provide a derivation, based on the previous derivations of BIC in general (e.g., Kass & Raftery, 1995, p. 779). It focuses on two-parameter logistic (2PL) item response models for binary responses. The derivation below is applicable to the one-parameter, two-parameter, and three-parameter item response models for the binary responses.

Schwarz (1978) derived the BIC as an asymptotic approximation of the Bayesian posterior probability of a candidate model $M$:

$$\log P\left(M|\mathrm{y}\right)=\log P\left(y|\widehat{\mathbf{\vartheta }}\mathrm{,}M\right)-\frac{K}{2}\log N+O\left(N^{-1}\right)\mathrm{,}$$

where $\mathbf{y}$ is data, $\mathit{\vartheta }$ is a set of parameters, $P(\mathbf{y}|\hat{\mathit{\vartheta }},M)$ is the likelihood of data $\mathbf{y}$ evaluated at parameter estimates $\hat{\mathbf{\vartheta }}$. $K$ is the number of parameters, and $N$ is sample size. For a large sample size, the order $O\left(N^{-1}\right)$ term can be dropped in Equation 1. Below, the derivation of $N$ is presented for (a) the 2PL item response model having a random person effect and a fixed item effect and (b) the 2PL item response model having a random person effect and a random item effect. In the derivations, the notation for the candidate model $M$ as it appears in Equation 1 is omitted for reasons of simplicity.

Random Person Effect and Fixed Item Effect

The 2PL item response model can be specified as

$$\mathrm{logit}\left[P\left(y_{ji}=1|{\mathrm{\theta }}_j\right)\right]={\mathrm{\alpha }}_i\left({\mathrm{\theta }}_j-{\mathrm{\beta }}_i\right)\mathrm{,}$$

where $j$ is an index of person ($j=1,\dots ,J$), $i$ is an index of item ($i=1,\dots ,I$), $y_{JI}$ is an item response, ${\mathrm{\alpha }}_i$ is an item discrimination parameter, ${\mathrm{\beta }}_i$ is an item location parameter, and ${\mathrm{\theta }}_j$ is a latent variable. To identify the model, a standard normal distribution can be set on ${\mathrm{\theta }}_j$. For a sample of $J$ independent persons, the marginal probability of the data, $\mathbf{y}=[y_1,\dots ,y_j,\dots ,y_J]\prime$, is

$$P\left(y\right)={\displaystyle \prod _{j=1}^J}{\displaystyle \underset{\theta }{\int }\left[{\displaystyle \prod _{i=1}^I}P{\left(y_{ji}=1|{\theta }_j;{\widehat{\mathrm{\vartheta }}}_i\right)}^{y_{ji}}P{\left(y_{ji}=0|{\theta }_j;{\widehat{\mathbf{\vartheta }}}_i\right)}^{1-y_{ji}}\right]}P\left({\mathrm{\theta }}_j\right){\mathrm{d\theta }}_j\mathrm{,}$$

where ${\mathit{\vartheta }}_i=[{\mathrm{\alpha }}_i,{\mathrm{\beta }}_i]\prime$ and $P\left({\theta }_j\right)$ is a prior distribution for ${\theta }_j$.

The marginal probability of the data $\mathbf{y}$ can be approximated using Laplace approximation (Tierney & Kadane, 1986):

$$\log P\left(y\right)=\log P\left(y|\theta ;\widehat{\mathbf{\vartheta }}\right)+\log P\left(\theta \right)+\frac{K}{2}\log \left(2\mathrm{\pi }\right)-\frac{1}{2}\log \parallel A\parallel +O\left(I^{-1}\right)\mathrm{,}$$

where $\theta =[{\theta }_1,\dots ,{\theta }_J]\prime$ and $\mathit{A}$ is the expected information matrix of item parameter estimates, $\hat{\mathit{\vartheta }}=[\hat{\mathrm{\alpha }},\hat{\mathrm{\beta }}]\prime$.

For a sample of $J$ independent persons, the $\mathit{A}$ in Equation 4 approaches $J·I_j(\mathit{\vartheta })$, where $J$ is the number of persons and $I_j(\mathit{\vartheta })$ is the Fisher information (or expected information) of item parameter estimates for a person $j$ (Thissen & Wainer, 1982, p. 411):

$$A=J\cdot I_j\left(\mathbf{\vartheta }\right)\cong J\cdot E\left[-{\stackrel{··}{P}}_j\left(\widehat{\mathbf{\vartheta }}\right)\right]\mathrm{,}$$

where $-{\stackrel{··}{P}}_j(\mathit{\vartheta })$ is minus the second derivative of the loglikelihood function (called the Hessian matrix or the observed information). The matrix of ${\mathit{I}}_j(\mathit{\vartheta })$ is a ($K\times K$) matrix (e.g., $2I\times 2I$ matrix for the 2PL item response models). Thus, $\parallel A\parallel \equiv \parallel J{\mathit{I}}_j\left(\mathit{\vartheta }\right)\parallel \equiv J^K\parallel {\mathit{I}}_j\left(\mathit{\vartheta }\right)\parallel$.

Inserting $\parallel A\parallel \equiv \parallel J{\mathit{I}}_j\left(\mathit{\vartheta }\right)\parallel \equiv J^K\parallel {\mathit{I}}_j\left(\mathit{\vartheta }\right)\parallel$ into Equation 4, the approximation is written as

$$\log P\left(y\right)=\log P\left(y|\theta ;\widehat{\mathbf{\vartheta }}\right)+\log P\left(\theta \right)+\frac{K}{2}\log \left(2\mathrm{\pi }\right)-\frac{K}{2}\log \left(J\right)-\frac{1}{2}\log \parallel I_j\parallel +O\left(I^{-1}\right).$$

In Equation 6, $\log P(\theta )$, $(K/2)\log \left(2\mathrm{\pi }\right)$, and $(1/2)\log ||{\mathit{I}}_j\parallel$ have an error of order $O(1)$. When ignoring the terms having an error of order $O(1)$, Equation 6 is

$$\log P\left(y\right)=\log P\left(y|\theta ;\widehat{\mathbf{\vartheta }}\right)-\frac{K}{2}\log \left(J\right)+O\left(I^{-1}\right).$$

As presented in Equation 7, $N$ in BIC is $J$.

Random Person Effect and Random Item Effect

Item parameters in Equation 2 can be random effects in specialized measurement problems (see De Boeck, 2008, for a discussion). For a sample of $J$ independent persons and a sample of $I$ independent items, the marginal probability of the data, $\mathbf{y}=[y_{11},\dots ,y_{JI},\dots ,y_{JI}]\prime$, is

$$P\left(y\right)={\displaystyle \prod _{j=1}^J}{\displaystyle \prod _{i=1}^I}{\displaystyle \underset{\theta }{\int }{\displaystyle \underset{\xi }{\int }{[P{(y_{ji}=1|{\theta }_j\mathrm{,}{\mathrm{\xi }}_i;\widehat{\mathbf{\vartheta }})}^{y_{ji}}\left[1-P\left(y_{ji}=1|{\theta }_j\mathrm{,}{\mathrm{\xi }}_i;\widehat{\mathbf{\vartheta }}\right)\right]}^{1-y_{ji}}]}}P\left({\theta }_j\right)P({\mathrm{\xi }}_i)\mathrm{d}{\theta }_j{\mathrm{d\xi }}_i\mathrm{,}$$

where ${\mathbf{\xi }}_i=[{\alpha }_i,{\beta }_i]\prime$ and $P({\mathbf{\xi }}_i)$ is a prior distribution of item parameters. When item parameters are random, the log-normal distribution can be imposed on item discrimination parameters $({\alpha }_i~\log N(0,{\sigma }_{\alpha }))$ and the normal distribution can be imposed on item location parameters $({\beta }_i~N(0,{\sigma }_{\beta }))$. In such a case, there are three parameters $(\mathit{\vartheta }=[\mu ,{\sigma }_{\alpha },{\sigma }_{\beta }]\prime )$, where $\mu$ is a grand mean across persons and items, ${\sigma }_{\alpha }$ is a variance of a random item discrimination parameter, and ${\sigma }_{\beta }$ is a variance of a random location parameter.

Using the Laplace approximation, the marginal probability of the data $\mathbf{y}$ can be approximated as follows:

$$\log P\left(y\right)=\log P\left(y|\theta \mathrm{,}\mathrm{\xi };\widehat{\mathbf{\vartheta }}\right)+\log P\left(\theta \right)+\log P\left(\mathrm{\xi }\right)+\frac{K}{2}\log \left(2\mathrm{\pi }\right)-\frac{K}{2}\log \parallel A\parallel +O\left({\left(JI\right)}^{-1}\right)\mathrm{,}$$

where $\mathbf{\xi }=[{\xi }_1,\dots ,{\xi }_i,\dots ,{\xi }_I]\prime$ and $A$ is the expected information matrix of the parameter estimates, $\hat{\mathit{\vartheta }}=[\hat{\mathrm{\mu }},{\hat{\mathrm{\sigma }}}_{\mathrm{\alpha }},{\hat{\mathrm{\sigma }}}_{\mathrm{\beta }}]\prime$.

For a sample of independent pairs of person $j$ and item $i$, $A$ in Equation 9 approaches $\left(JI\right)·{\mathit{I}}_{ji}\left(\mathit{\vartheta }\right)$, where $J$ is the number of persons, $I$ is the number of items, and ${\mathit{I}}_{JI}\left(\mathit{\vartheta }\right)$ is the Fisher information of parameter estimates for a person $j$ and an item $i$:

$$A=\left(JI\right)\cdot I_{ji}\left(\mathbf{\vartheta }\right)\cong \left(JI\right)\cdot E\left[-{\stackrel{··}{P}}_{ji}\left(\widehat{\mathbf{\vartheta }}\right)\right]\mathrm{,}$$

where $-{\stackrel{··}{P}}_{JI}\left(\hat{\mathit{\vartheta }}\right)$ is minus the second derivative of the loglikelihood function. When ${\mathrm{IH}}_{JI}$ is a $(K\times K)$ matrix (i.e., $K=3$ for $\mathit{\vartheta }=[\mu ,{\sigma }_{\alpha },{\sigma }_{\beta }]\prime$), $\parallel A\parallel \equiv \parallel \left(JI\right)\times {\mathit{I}}_{JI}\parallel \equiv {\left(JI\right)}^K\parallel {\mathit{I}}_{JI}\parallel$.

With $\parallel A\parallel \equiv \parallel \left(JI\right)\times {\mathit{I}}_{JI}\parallel \equiv {\left(JI\right)}^K\parallel {\mathit{I}}_{JI}\parallel$, the approximation in Equation 9 can be rewritten as

$$\log P(y)=\log P\left(y|\theta \mathrm{,}\xi ;\widehat{\mathbf{\vartheta }}\right)+\log P\left(\theta \right)+\log P\left(\mathrm{\xi }\right)+\frac{K}{2}\log \left(2\mathrm{\pi }\right)-\frac{K}{2}\log \left(JI\right)-\frac{1}{2}\log \parallel I_{ji}\parallel +O\left({\left(JI\right)}^{-1}\right)\mathrm{,}$$

where $\log P(\mathrm{\theta })$, $\log P(\mathbf{\xi })$, $(K/2)\log \left(2\mathrm{\pi }\right)$, and $(1/2)\log \parallel {\mathit{I}}_{JI}\parallel$ are all approximations with an error of order $O(1)$. Ignoring the $O(1)$ terms, Equation 11 results in the following equation:

$$\log P\left(y\right)=\log P\left(y|\theta \mathrm{,}\mathrm{\xi };\widehat{\mathbf{\vartheta }}\right)-\frac{K}{2}\log \left(JI\right)+O\left({\left(JI\right)}^{-1}\right).$$

It follows from Equation 12 that $N$ is $JI$ for item response models with random item parameters.