Large Sample Confidence Intervals for Item Response Theory Reliability Coefficients

Abstract

In applications of item response theory (IRT), an estimate of the reliability of the ability estimates or sum scores is often reported. However, analytical expressions for the standard errors of the estimators of the reliability coefficients are not available in the literature and therefore the variability associated with the estimated reliability is typically not reported. In this study, the asymptotic variances of the IRT marginal and test reliability coefficient estimators are derived for dichotomous and polytomous IRT models assuming an underlying asymptotically normally distributed item parameter estimator. The results are used to construct confidence intervals for the reliability coefficients. Simulations are presented which show that the confidence intervals for the test reliability coefficient have good coverage properties in finite samples under a variety of settings with the generalized partial credit model and the three-parameter logistic model. Meanwhile, it is shown that the estimator of the marginal reliability coefficient has finite sample bias resulting in confidence intervals that do not attain the nominal level for small sample sizes but that the bias tends to zero as the sample size increases.

In classical test theory, the reliability of a test plays a central role. The reliability is a measure of the consistency of the scores from a test. Reliability, together with the concept of item and test information, also plays a role in item response theory (IRT). Two different measures of reliability in IRT are marginal reliability (Cheng, Yuan, & Liu, 2012) and test reliability (Kim & Feldt, 2010; Lord, 1977, 1980). The marginal reliability denotes the ratio of the true score variance to the total variance, expressed with respect to the estimated latent abilities. Test reliability, on the other hand, denotes the ratio of true score variance to total variance expressed with respect to the sum scores. Note that, in the terminology used in this article, both the marginal reliability and the test reliability refer to reliability with regard to a population as a whole and hence are in some sense both “marginal” measures of reliability. Having a measure of the reliability of the scores from a test in the context of IRT is useful since it provides an indication of the overall consistency of the test scores in measuring the underlying trait or the observed score generated from the IRT model. Since the object of a test in IRT is to measure the latent concept for everyone taking the test, an overall measure of the reliability of the test scores across the spectrum of the latent distribution is important to consider.

When using IRT in empirical research, a reliability coefficient is often reported. The reported reliability coefficients are estimates of the true reliability coefficient and these estimates have a certain amount of variance associated with them due to the item parameter estimation. The large sample variance for several reliability coefficient estimators in classical test theory have been derived (Yuan & Bentler, 2002). However, analytical estimates of the variance of the IRT reliability coefficients are not available in the literature and an estimated standard error or confidence interval is therefore usually not reported. In this article, the large sample variance of the estimators of the IRT marginal reliability coefficient and the IRT test reliability coefficient are derived using standard asymptotic theory. The results may be used by empirical researchers to estimate confidence intervals for the reliability coefficients.

The article is structured as follows. First, IRT is introduced and common models for either dichotomous or polytomous data are briefly described. Then, IRT marginal and test reliability coefficients are defined and the asymptotic variance of estimators of these are derived. It is then investigated how well the large sample results work with finite samples through a simulation study. Finally, concluding remarks are given.

Item Response Theory

Consider a test consisting of $J$ number of dichotomous or polytomous items. Denote the IRT item parameter vector by $\mathbf{\alpha }$ and let ${\mathbf{\alpha }}_j\in \mathbf{\alpha }$ be the item parameter vector for item $j$. Let $P_{\mathrm{jk}}(\theta ;{\mathbf{\alpha }}_j)$ be the probability of obtaining category $k\in \{1,\dots ,m_j\}$ on item $j$ conditional on the latent variable $\theta$, where $m_j$ denotes the number of possible categories for item $j$. For dichotomous items, the three-parameter logistic (3-PL) model (Lord, 1980) defines the probability as

$$P_{\mathrm{jk}}(\theta ;{\mathbf{\alpha }}_j)=u_{\mathrm{jk}}{P}_j(\theta ;{\mathbf{\alpha }}_j)+(1-u_{\mathrm{jk}})(1-P_j(\theta ;{\mathbf{\alpha }}_j)),$$

where $u_{j1}=0$ and $u_{j2}=1$ and

$$P_j(\theta ;{\mathbf{\alpha }}_j)=c_j+\frac{1-c_j}{1+\exp [-a_j(\theta -b_j)]},$$

where $a_j$ is called the discrimination parameter, $b_j$ the location parameter and $c_j$ the pseudo-guessing parameter for item $j$. For polytomous items, the generalized partial credit model (GPCM) (Muraki, 1992) can be used, with probabilities defined by

$$P_{\mathrm{jk}}(\theta ;{\mathbf{\alpha }}_j)=\frac{\exp [\sum _{v=1}^k{a}_j(\theta -b_{j,v})]}{\sum _{c=1}^{m_j}\exp [\sum _{v=1}^c{a}_j(\theta -b_{j,v})]},$$

where $a_j$ is again the discrimination parameter and each $b_{j,v}$ are called the item category parameters, with $b_{j,1}\equiv 0$. In IRT, the concept of item and test information has an important role. The test information is inversely related to the variance to the maximum likelihood estimator of the ability. Define the expected item information for a dichotomous IRT model as

$$I_j(\theta ;{\mathbf{\alpha }}_j)=\sum _{k=1}^{m_j}\frac{{\left(\frac{\partial P_{\mathrm{jk}}(\theta ;{\mathbf{\alpha }}_j)}{\partial \theta }\right)}^2}{P_{\mathrm{jk}}(\theta ;{\mathbf{\alpha }}_j)},$$

where, for the 3-PL model,

$$\frac{\partial P_{\mathrm{jk}}(\theta ;{\mathbf{\alpha }}_j)}{\partial \theta }=u_{\mathrm{jk}}\frac{\partial P_j(\theta ;{\mathbf{\alpha }}_j)}{\partial \theta }-(1-u_{\mathrm{jk}})\frac{\partial P_j(\theta ;{\mathbf{\alpha }}_j)}{\partial \theta }$$

where (Hambleton & Swaminathan, 1985)

$$\frac{\partial P_j(\theta ;{\mathbf{\alpha }}_j)}{\partial \theta }=\frac{a_j[1-P_j(\theta ;{\mathbf{\alpha }}_j)][P_j(\theta ;{\mathbf{\alpha }}_j)-c_j]}{1-c_j}.$$

For a polytomous IRT model, define the expected item information as (Magis, 2015)

$$I_j(\theta ;{\mathbf{\alpha }}_j)=\sum _{k=1}^{m_j}[\frac{{\left(\frac{\partial P_{\mathrm{jk}}(\theta ;{\mathbf{\alpha }}_j)}{\partial \theta }\right)}^2}{P_{\mathrm{jk}}(\theta ;{\mathbf{\alpha }}_j)}-\frac{{\partial }^2{P}_{\mathrm{jk}}(\theta ;{\mathbf{\alpha }}_j)}{\partial {\theta }^2}],$$

where, for the GPCM (Muraki, 1992),

$$\frac{\partial P_{\mathrm{jk}}(\theta ;{\mathbf{\alpha }}_j)}{\partial \theta }=P_{\mathrm{jk}}(\theta ;{\mathbf{\alpha }}_j)a_j(k-\sum _{c=1}^{m_j}c{P}_{\mathrm{jc}}(\theta ;{\mathbf{\alpha }}_j))$$

and

$$\frac{{\partial }^2{P}_{\mathrm{jk}}(\theta ;{\mathbf{\alpha }}_j)}{\partial {\theta }^2}=\frac{\partial P_{\mathrm{jk}}(\theta ;{\mathbf{\alpha }}_j)}{\partial \theta }{a}_j(k-\sum _{c=1}^{m_j}c{P}_{\mathrm{jc}}(\theta ;{\mathbf{\alpha }}_j))-P_{\mathrm{jk}}(\theta ;{\mathbf{\alpha }}_j)a_j\sum _{c=1}^{m_j}c\frac{\partial P_{\mathrm{jc}}(\theta ;{\mathbf{\alpha }}_j)}{\partial \theta },$$

resulting in the expected information for the GPCM having the expression

$$I_j(\theta ;{\mathbf{\alpha }}_j)=\sum _{k=1}^{m_j}{P}_{\mathrm{jk}}(\theta ;{\mathbf{\alpha }}_j)a_j\sum _{c=1}^{m_j}c\frac{\partial P_{\mathrm{jc}}(\theta ;{\mathbf{\alpha }}_j)}{\partial \theta }.$$

The test information is the sum of the information for each item, i.e.,

$$I(\theta ;\mathbf{\alpha })=\sum _{j=1}^J{I}_j(\theta ;{\mathbf{\alpha }}_j).$$

Let ${\hat{\theta }}_{\mathrm{MLE}}$ be the maximum likelihood estimator of the ability. As $J\to \infty$, the estimator ${\hat{\theta }}_{\mathrm{MLE}}$ has variance (Hambleton & Swaminathan, 1985)

$$\mathrm{Var}({\hat{\theta }}_{\mathrm{MLE}}|\theta )=\frac{1}{I(\theta ;\mathbf{\alpha })}.$$

Item Response Theory Marginal Reliability

One measure of reliability for scores on the latent variable metric which has been proposed in the literature is the marginal reliability (Green, Bock, Humphreys, Linn, & Reckase, 1984), sometimes referred to as parallel forms reliability (Kim, 2012). With the assumption of an ability distribution with density $g$ and variance 1, this reliability coefficient is defined as (Cheng et al., 2012)

$${\rho }_{\Theta }(\mathbf{\alpha })={\int }_{-\infty }^{\infty }\frac{I(\theta ;\mathbf{\alpha })}{I(\theta ;\mathbf{\alpha })+1}g(\theta )d\theta .$$

The marginal reliability coefficient can be interpreted as the reliability with regard to the maximum likelihood ability estimates. The density function $g$ is often assumed to be the standard normal distribution during estimation and this is the assumption that will be used hereafter.

Item Response Theory Test Reliability

The IRT test reliability coefficient ${\rho }_{\mathrm{XX}\prime }(\mathbf{\alpha })$ is defined as the ratio of the true score variance ${\sigma }_T^2(\mathbf{\alpha })$ and the observed score variance ${\sigma }_X^2(\mathbf{\alpha })$. It can be interpreted as the parallel forms reliability with regard to the sum score. Define $W_{\mathrm{jk}}$ as the score for obtaining category $k$ on item $j$, which we consider as a fixed constant in this article. From the definition of reliability, we can note that ${\rho }_{\mathrm{XX}\prime }(\mathbf{\alpha })$ can also be expressed as

$${\rho }_{\mathrm{XX}\prime }(\mathbf{\alpha })=\frac{{\sigma }_T^2(\mathbf{\alpha })}{{\sigma }_X^2(\mathbf{\alpha })}=\frac{{\sigma }_X^2(\mathbf{\alpha })-{\sigma }_e^2(\mathbf{\alpha })}{{\sigma }_X^2(\mathbf{\alpha })}=1-\frac{{\sigma }_e^2(\mathbf{\alpha })}{{\sigma }_X^2(\mathbf{\alpha })},$$

where, following Kim and Feldt (2010), ${\sigma }_e^2(\mathbf{\alpha })$ is the conditional error variance ${\sigma }_{e|\theta }^2(\mathbf{\alpha })$ averaged over the distribution of $\theta$, i.e.,

$${\sigma }_e^2(\mathbf{\alpha })={\int }_{-\infty }^{\infty }{\sigma }_{e|\theta }^2(\mathbf{\alpha })g(\theta )d\theta ,$$

where the conditional error variance ${\sigma }_{e|\theta }^2(\mathbf{\alpha })$ can be calculated by (Kim & Feldt, 2010)

$${\sigma }_{e|\theta }^2(\mathbf{\alpha })={\displaystyle \sum _{j=1}^J{\sigma }_{e_j|\theta }^2}({\mathbf{\alpha }}_j)={\displaystyle \sum _{j=1}^J{\sigma }_{X_j|\theta }^2}({\mathbf{\alpha }}_j)={\displaystyle \sum _{j=1}^J\left\{{\displaystyle \sum _{k=1}^{m_j}{P}_{jk}}(\theta ;{\mathbf{\alpha }}_j)W_{jk}^2-{\left[{\displaystyle \sum _{k=1}^{m_j}{P}_{jk}}(\theta ;{\mathbf{\alpha }}_j)W_{jk}\right]}^2\right\}}.$$

Consider a test that has possible observed scores from 0 to $K$. The variance of the observed score distribution, ${\sigma }_X^2(\mathbf{\alpha })$, can be calculated from the probabilities $r_i(\mathbf{\alpha })$ for each possible observed score, i.e.,

$${\sigma }_X^2(\mathbf{\alpha })=\sum _{i=0}^K{r}_i(\mathbf{\alpha })x_i^2-{(\sum _{i=0}^K{r}_i(\mathbf{\alpha })x_i)}^2,$$

where $x_i$ denotes the score value of the $i$th observed score. The probabilities $r_i(\mathbf{\alpha })$ can be calculated from the recursive formulas given for dichotomous items in Lord and Wingersky (1984) and for polytomous items in Thissen, Pommerich, Billeaud, and Williams (1995).

Asymptotic Variance of Item Response Theory Reliability Coefficient Estimators

Define $\hat{\mathbf{\alpha }}$ as the estimator of the item parameter vector and let $n$ denote the sample size. In the following, it is assumed that asymptotic normality holds for the item parameter estimator $\hat{\mathbf{\alpha }}$. Under this condition, the asymptotic variance of the reliability coefficient estimators can be derived using standard asymptotic theory (Ferguson, 1996). Let ${\Sigma }_{\hat{\mathbf{\alpha }}}$ denote the asymptotic covariance matrix of $\hat{\mathbf{\alpha }}$.

Item Response Theory Marginal Reliability

The integral in Equation (13) is approximated by a sum using Gauss-Hermite quadrature and we thus obtain the estimator of ${\rho }_{\Theta }(\mathbf{\alpha })$ as

$${\hat{\rho }}_{\Theta }(\mathbf{\alpha })=\sum _{l=1}^L\frac{\sum _{j=1}^J{I}_j({\theta }_l;\mathbf{\alpha })}{\sum _{j=1}^J{I}_j({\theta }_l;\mathbf{\alpha })+1}\frac{e^{\frac{{\theta }_l^2}{2}}}{\sqrt{2\pi }}{w}_l,$$

where $w_l$ is the weight for the $l$th Gauss-Hermite quadrature point ${\theta }_l$. The function ${\hat{\rho }}_{\Theta }(\mathbf{\alpha })$ is continuous and differentiable with regard to the item parameter vector $\mathbf{\alpha }$. We may then apply the delta method to approximate the variance of ${\hat{\rho }}_{\Theta }(\mathbf{\alpha })$ and hence we retrieve, for large $n$,

$${\sigma }_{{\hat{\rho }}_{\Theta }(\mathbf{\alpha })}^2\approx \frac{\partial {\hat{\rho }}_{\Theta }(\mathbf{\alpha })}{\partial \mathbf{\alpha }}{\Sigma }_{\hat{a}}{\left[\frac{\partial {\hat{\rho }}_{\Theta }(\mathbf{\alpha })}{\partial \mathbf{\alpha }}\right]}^{\prime },$$

where

$$\begin{array}{cc}\multicolumn{1}{c}{\frac{\partial {\hat{\rho }}_{\Theta }(\mathbf{\alpha })}{\partial \mathbf{\alpha }}}& =\frac{\partial \left[\sum _{l=1}^L\frac{\sum _{j=1}^J{I}_j({\theta }_l;{\mathbf{\alpha }}_j)}{\sum _{j=1}^J{I}_j({\theta }_l;{\mathbf{\alpha }}_j)+1}\frac{e^{\frac{{\theta }_l^2}{2}}}{\sqrt{2\pi }}{w}_l\right]}{\partial \mathbf{\alpha }}\hfill \\ \multicolumn{1}{c}{}& =\sum _{l=1}^L\{\frac{\sum _{j=1}^J\frac{\partial I_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial \mathbf{\alpha }}}{\sum _{j=1}^J{I}_j({\theta }_l;{\mathbf{\alpha }}_j)+1}-\frac{\sum _{j=1}^J{I}_j({\theta }_l;{\mathbf{\alpha }}_j)\sum _{j=1}^J\frac{\partial I_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial \mathbf{\alpha }}}{{[\sum _{j=1}^J{I}_j({\theta }_l;2a_j)+1]}^2}\}\frac{e^{\frac{{\theta }_l^2}{2}}}{\sqrt{2\pi }}{w}_l.\hfill \end{array}$$

In Equation (20), the expression of $\frac{\partial I_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial a}$ for dichotomous IRT models is

$$\frac{\partial I_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial \mathbf{\alpha }}=\sum _{k=1}^{m_j}\frac{\partial P_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l}[2\frac{{\partial }^2{P}_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial \mathbf{\alpha }}-\frac{\partial P_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l}\frac{\partial P_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial \mathbf{\alpha }}]$$

and for polytomous IRT with the GPCM is

$$\begin{array}{cc}\multicolumn{1}{c}{\frac{\partial I_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial \mathbf{\alpha }}=}& \sum _{k=1}^{m_j}\{\frac{\partial P_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial \mathbf{\alpha }}{a}_j\sum _{c=1}^{m_j}c{P}_{\mathrm{jc}}({\theta }_l;{\mathbf{\alpha }}_j)+\frac{\partial a_j}{\partial \mathbf{\alpha }}{P}_{\mathrm{jk}}({\theta }_l;\mathbf{\alpha })\sum _{c=1}^{m_j}c{P}_{\mathrm{jc}}({\theta }_l;{\mathbf{\alpha }}_j)\hfill \\ \multicolumn{1}{c}{}& +P_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)a_j\sum _{c=1}^{m_j}c\frac{{\partial }^2{P}_{\mathrm{jc}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial \mathbf{\alpha }}\}.\hfill \end{array}$$

For the 3-PL model,

$$\frac{\partial P_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial \mathbf{\alpha }}=u_{\mathrm{jk}}\frac{\partial P_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial \mathbf{\alpha }}-(1-u_{\mathrm{jk}})\frac{\partial P_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial \mathbf{\alpha }}$$

where $\frac{\partial P_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial \mathbf{\alpha }}$ is given in Hambleton and Swaminathan (1985) while for the GPCM the vector $\frac{\partial P_{\mathrm{jk}}({\theta }_l;\mathbf{\alpha })}{\partial \mathbf{\alpha }}$ is given in Muraki (1992). The vector $\frac{{\partial }^2{P}_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial \mathbf{\alpha }}$ is provided in the appendix.

Item Response Theory Test Reliability

The integral implicit in the numerator of Equation (14) is approximated by a sum using Gauss-Hermite quadrature and so are the integrals required to calculate $r_i(\mathbf{\alpha })$. We thus obtain the estimator of ${\rho }_{\mathrm{XX}\prime }(\mathbf{\alpha })$ as

$${\hat{\rho }}_{\mathrm{XX}\prime }(\mathbf{\alpha })=1-\frac{\sum _{l=1}^L\sum _{j=1}^J\{\sum _{k=1}^{m_j}{P}_{\mathrm{jk}}({\theta }_l;\mathbf{\alpha })W_{\mathrm{jk}}^2-{[\sum _{k=1}^{m_j}{P}_{\mathrm{jk}}({\theta }_l;\mathbf{\alpha })W_{\mathrm{jk}}]}^2\}\frac{e^{\frac{{\theta }_l^2}{2}}}{\sqrt{2\pi }}{w}_l}{{\sum _{i=0}^K{\hat{r}}_i(\mathbf{\alpha })x_i^2-[\sum _{i=0}^K{\hat{r}}_i(\mathbf{\alpha })x_i]}^2}.$$

Again, ${\hat{\rho }}_{\mathrm{XX}\prime }(\mathbf{\alpha })$ is a continuous and differentiable function of the item parameter vector $\mathbf{\alpha }$. Hence, the delta method can be used to approximate the variance of ${\hat{\rho }}_{\mathrm{XX}\prime }(\mathbf{\alpha })$ and we retrieve, for large $n$,

$${\sigma }_{{\hat{\rho }}_{\mathrm{XX}\prime }(\mathbf{\alpha })}^2\approx \frac{\partial {\hat{\rho }}_{\mathrm{XX}\prime }(\mathbf{\alpha })}{\partial \mathbf{\alpha }}{\Sigma }_{\hat{\mathbf{\alpha }}}\left[\frac{\partial {\hat{\rho }}_{\mathrm{XX}\prime }(\mathbf{\alpha })}{\partial \mathbf{\alpha }}\right],$$

where

$$\frac{\partial {\hat{\rho }}_{\mathrm{XX}\prime }(\mathbf{\alpha })}{\partial \mathbf{\alpha }}=\frac{1}{{\hat{\sigma }}_X^2(\mathbf{\alpha })}[\frac{{\hat{\sigma }}_e^2(\mathbf{\alpha })}{{\hat{\sigma }}_X^2(\mathbf{\alpha })}\frac{\partial {\hat{\sigma }}_X^2(\mathbf{\alpha })}{\partial \mathbf{\alpha }}-\frac{\partial {\hat{\sigma }}_e^2(\mathbf{\alpha })}{\partial \mathbf{\alpha }}].$$

Note that

$$\frac{\partial {\hat{\sigma }}_e^2(\mathbf{\alpha })}{\partial \mathbf{\alpha }}=\sum _{l=1}^L\{\sum _{j=1}^J\sum _{k=1}^{m_j}\frac{\partial P_{\mathrm{jk}}({\theta }_l;\mathbf{\alpha })}{\partial \mathbf{\alpha }}{W}_{\mathrm{jk}}^2-2\sum _{j=1}^J[\sum _{k=1}^{m_j}{P}_{\mathrm{jk}}({\theta }_l;\mathbf{\alpha })W_{\mathrm{jk}}\sum _{k=1}^{m_j}\frac{\partial P_{\mathrm{jk}}({\theta }_l;\mathbf{\alpha })}{\partial \mathbf{\alpha }}{W}_{\mathrm{jk}}]\}\frac{e^{\frac{{\theta }_l^2}{2}}}{\sqrt{2\pi }}{w}_l$$

and

$$\frac{\partial {\hat{\sigma }}_X^2(\mathbf{\alpha })}{\partial \mathbf{\alpha }}=\sum _{i=0}^K{x}_i^2\frac{\partial {\hat{r}}_i(\mathbf{\alpha })}{\partial \mathbf{\alpha }}-2\sum _{i=0}^K{x}_i{\hat{r}}_i(\mathbf{\alpha })\sum _{i=0}^K\frac{\partial {\hat{r}}_i(\mathbf{\alpha })}{\partial \mathbf{\alpha }}{x}_i,$$

where the derivatives $\frac{\partial {\hat{r}}_i(\mathbf{\alpha })}{\partial \mathbf{\alpha }}$ for the GPCM are given in Andersson (2016) and the derivatives $\frac{\partial {\hat{r}}_i(\mathbf{\alpha })}{\partial \mathbf{\alpha }}$ for the 3-PL model are given in Ogasawara (2003).

Confidence Interval Estimation

For large sample sizes, approximate confidence intervals for the reliability coefficients can be constructed using the derived variances and a normal approximation. Hence, approximate 95% confidence intervals for ${\rho }_{\Theta }(\mathbf{\alpha })$ or ${\rho }_{\mathrm{XX}\prime }(\mathbf{\alpha })$ are given by ${\hat{\rho }}_{\Theta }(\mathbf{\alpha })\pm z_{(0.975)}\sqrt{{\hat{\sigma }}_{{\hat{\rho }}_{\Theta }(\mathbf{\alpha })}^2}$ and ${\hat{\rho }}_{\mathrm{XX}\prime }(\mathbf{\alpha })\pm z_{(0.975)}\sqrt{{\hat{\sigma }}_{{\hat{\rho }}_{\mathrm{XX}\prime }(\mathbf{\alpha })}^2}$, where $z_{(0.975)}$ denotes the 0.975 quantile of the standard normal distribution. The properties of the confidence intervals in finite samples are investigated in the next section using simulations.

Simulation Study

Simulations were conducted to study the finite sample properties of the derived asymptotic variances and confidence intervals. The values of the latent variable were drawn from the $N(0,1)$ distribution and the responses to a test with nine GPCM items and the responses to a test with 40 3-PL items were simulated. The GPCM item parameters were taken from the study by Fischer, Tritt, Klapp, and Fliege (2011) and are given in Table 1. For the 3-PL model, the $a$-parameters were drawn from the $U(0.5,2)$ distribution, the $b$-parameters from the $N(0,1)$ distribution and the $c$-parameters from the $U(0.15,0.20)$ distribution. The 3-PL parameters used in the study are given in Table 2. With the selected item parameters and $N(0,1)$ as the reference distribution, for the GPCM the true marginal reliability coefficient was 0.8718 and the true test reliability coefficient was 0.8744 while for the 3-PL model the true marginal reliability coefficient was 0.8487 and the true test reliability coefficient was 0.8285. The item parameters were estimated with the R (R Development Core Team, 2016) package mirt (Chalmers, 2012) using marginal maximum likelihood with the assumption of a $N(0,1)$ latent distribution. Since maximum likelihood estimation was used, no priors were specified for the item parameters. The observed information matrix (Yuan, Cheng, & Patton, 2013) was calculated after estimation by numerically differentiating the gradient of the observed log-likelihood function at the obtained maximum likelihood estimates to provide an estimate of the asymptotic covariance matrix, using newly written R code to improve the speed of calculation. The observed information matrix is appropriate in this setting since the models are correctly specified. After estimation, the reliability coefficients and their asymptotic variances were calculated using newly written R code. The number of Gauss-Hermite quadrature points used was 49 and 2,000 replications were used. The average bias, average asymptotic standard errors (ASE), Monte Carlo standard errors (MCSE), and empirical coverage rates of 95% confidence intervals were calculated. For the GPCM, sample sizes 250, 500, 1,000, 2,000, 4,000, and 8,000 were considered and for the 3-PL model sample sizes 1,000, 2,000, 4,000, and 8,000 were considered. The reason for only considering the larger sample sizes with the 3-PL model was that reliable estimation with the smaller sample sizes was not possible due to high non-convergence rates. With the GPCM, the nonconvergence rate was 0.45% (sample size 250) and with the 3-PL model the nonconvergence rates were 9.1% (sample size 1,000) and 0.7% (sample size 2,000). For all other settings, the nonconvergence rates were 0%.

Table 1.

Generalized Partial Credit Model Item Parameters for the Nine-Item Scale Used in the Simulation Study.

Item	$a$	$b_2$	$b_3$	$b_4$
1	1.49	–1.14	0.64	0.68
2	2.73	–0.74	0.57	0.95
3	0.79	–1.52	–0.24	0.41
4	1.65	–1.62	0.20	0.40
5	0.71	–0.47	0.36	1.22
6	1.18	0.14	0.80	1.43
7	1.22	–0.70	0.77	1.01
8	0.80	0.37	1.26	1.73
9	1.40	0.89	1.92	2.26

Table 2.

Three-Parameter Logistic Item Parameters Used in the Simulation Study.

Item	$a$	$b$	$c$	Item	$a$	$b$	$c$
1	1.62	–1.85	0.19	21	1.19	0.52	0.20
2	0.81	–0.06	0.15	22	1.25	–0.85	0.18
3	1.68	2.05	0.19	23	0.80	–0.62	0.17
4	1.50	–0.52	0.15	24	1.60	–1.17	0.16
5	0.73	–0.06	0.17	25	1.79	–0.67	0.18
6	1.37	–1.16	0.20	26	0.84	–0.40	0.20
7	1.00	0.79	0.18	27	1.49	–1.94	0.19
8	1.63	–0.40	0.17	28	0.80	0.77	0.17
9	1.55	–1.37	0.18	29	1.66	–0.86	0.16
10	0.70	–1.13	0.19	30	1.13	0.07	0.19
11	1.18	1.28	0.15	31	1.69	1.20	0.18
12	1.76	–2.36	0.19	32	0.62	0.30	0.20
13	0.75	–0.47	0.20	33	0.57	1.13	0.16
14	1.38	0.74	0.17	34	1.07	–0.38	0.17
15	0.54	–1.25	0.16	35	1.55	–0.74	0.17
16	0.92	–0.36	0.19	36	0.88	1.56	0.19
17	1.73	0.50	0.17	37	1.37	–2.30	0.18
18	0.63	–0.90	0.17	38	1.65	0.98	0.16
19	0.61	–1.34	0.20	39	0.95	–0.11	0.17
20	0.75	0.27	0.15	40	1.76	1.40	0.18

Results

In Table 3 the results for the simulation with the GPCM are given. For the marginal reliability coefficient, there exists a small but statistically significant bias for sample sizes lower than 2,000. The asymptotic standard errors are accurate as an estimate of the sampling variability for all sample sizes. The empirical coverage rates of the 95% confidence intervals are slightly lower than the nominal level for sample sizes 250 and 500 but with sample sizes 1,000 and higher the empirical coverage rate is not statistically significantly different from 95%. For the test reliability coefficient, the bias is not statistically significantly different from zero for any sample size. The asymptotic standard error is accurate for all sample sizes considered. The empirical coverage rates of the 95% confidence intervals for the test reliability coefficient are not statistically significantly different to the nominal level for any sample size.

Table 3.

Mean Bias (×100), Asymptotic and Monte Carlo Standard Errors (×100), and Coverage of 95% Confidence Intervals (%) for the GPCM Reliability Coefficient Estimators, With Estimated Standard Errors in Parentheses.

Reliability	Sample size	Mean bias	ASE	MCSE	CI coverage
Marginal	250	0.14 (0.02)	0.90 (0.00)	0.90 (0.02)	92.71 (0.58)
	500	0.06 (0.01)	0.65 (0.00)	0.66 (0.01)	93.45 (0.55)
	1,000	0.03 (0.01)	0.46 (0.00)	0.46 (0.01)	94.10 (0.53)
	2,000	0.02 (0.01)	0.32 (0.00)	0.33 (0.01)	94.20 (0.52)
	4,000	0.01 (0.01)	0.23 (0.00)	0.23 (0.00)	95.70 (0.45)
	8,000	0.00 (0.00)	0.16 (0.00)	0.16 (0.00)	95.05 (0.49)
Test	250	0.02 (0.03)	1.13 (0.00)	1.13 (0.02)	94.73 (0.50)
	500	–0.01 (0.02)	0.80 (0.00)	0.81 (0.01)	94.15 (0.52)
	1,000	–0.00 (0.01)	0.57 (0.00)	0.57 (0.01)	94.55 (0.51)
	2,000	–0.00 (0.01)	0.40 (0.00)	0.40 (0.01)	95.10 (0.48)
	4,000	0.00 (0.01)	0.28 (0.00)	0.28 (0.00)	95.80 (0.45)
	8,000	–0.00 (0.00)	0.20 (0.00)	0.20 (0.00)	95.30 (0.47)

Note. GPCM = generalized partial credit model; ASE = asymptotic standard error; MCSE = Monte Carlo standard error; CI = confidence interval.

The results for the 3-PL model simulation are shown in Table 4. The estimator of the marginal reliability coefficient has a small but statistically significant bias under all settings considered. The asymptotic standard errors are accurate for all sample sizes but the empirical coverage rates of the confidence intervals are statistically significantly different from the nominal level of 95% with all sample sizes except the highest sample size of 8,000. For the test reliability coefficient, the bias is smaller and not statistically significantly different from zero with sample sizes 2,000 and higher. The asymptotic standard errors are accurate for all sample sizes and the empirical coverage rate is not statistically significantly different from the nominal level for any sample size.

Table 4.

Mean Bias (×100), Asymptotic and Monte Carlo Standard Errors (×100), and Coverage of 95% Confidence Intervals (%) for the 3-PL Reliability Coefficient Estimators, With Estimated Standard Errors in Parentheses.

Reliability	Sample size	Mean bias	ASE	MCSE	CI coverage
Marginal	1,000	0.52 (0.01)	0.60 (0.00)	0.61 (0.01)	84.21 (0.86)
	2,000	0.28 (0.01)	0.43 (0.00)	0.44 (0.01)	89.48 (0.69)
	4,000	0.15 (0.01)	0.31 (0.00)	0.32 (0.01)	91.95 (0.61)
	8,000	0.08 (0.00)	0.22 (0.00)	0.22 (0.00)	94.70 (0.50)
Test	1,000	0.05 (0.02)	0.72 (0.00)	0.74 (0.01)	94.39 (0.54)
	2,000	0.02 (0.01)	0.51 (0.00)	0.51 (0.01)	95.12 (0.48)
	4,000	0.01 (0.01)	0.36 (0.00)	0.36 (0.01)	95.25 (0.48)
	8,000	–0.00 (0.01)	0.25 (0.00)	0.26 (0.00)	95.65 (0.46)

Note. 3-PL = three-parameter logistic; ASE = asymptotic standard error; MCSE = Monte Carlo standard error; CI = confidence interval.

Concluding Remarks

With the results presented in this article, empirical researchers and practitioners have access to methods that evaluate the variability of reliability coefficient estimators in IRT and with which confidence intervals for the reliability coefficients can be estimated. The simulation study indicates that the estimated confidence intervals for the test reliability coefficient have good coverage properties with sample sizes as small as 250 with the GPCM and 1,000 with the 3-PL model. The estimator of the marginal reliability coefficient is slightly biased with small samples when using the GPCM and with all sample sizes considered in this article when using the 3-PL model. With the GPCM, the estimated confidence intervals were however still largely accurate and had correct coverage with sample size 1,000 while with the 3-PL model the confidence intervals had correct coverage only with the largest sample size considered. If sufficient computational resources are available, a nonparametric bootstrap approach (Davison & Hinkley, 1997) can be used to estimate the finite sample bias of the reliability coefficient estimators. This bias estimate can then be used together with the results in this article to generate bias-adjusted confidence intervals, which will achieve an improved coverage rate.

The differences between the properties of the marginal and test reliability estimators can be attributed to the differences with regard to the estimation of item response functions and the estimation of expected information functions. In Ogasawara (2002), it was shown that item response functions were possible to estimate accurately in spite of unstable item parameter estimates while the expected information functions were more affected by unstable item parameter estimates. Since the marginal reliability is calculated from the expected information functions while the test reliability only uses the item response functions, the difference in the accuracy of the two different reliability estimators is consistent with the results of Ogasawara (2002).

Since the marginal and test reliability coefficients measure different concepts, a direct comparison between them is not very meaningful. However, this study does indicate that the estimator of the test reliability coefficient has slightly higher sampling variability but lower bias than the estimator of the marginal reliability coefficient when using the same item parameters.

When calculating and reporting the IRT reliability coefficient estimates, it is important to note that the reliability coefficients are not invariant with regard to the latent distribution. This means that the reliability coefficients will be different for populations with different latent distributions even if the item parameters are invariant. If the distribution parameters of the population are known, the results of this paper can incorporate these without any changes to the derivations presented by suitably changing the approximation of the integrals needed. If the distribution parameters have been estimated, for example by using parametric (Mislevy, 1984), nonparametric (Bock & Aitkin, 1981), semi-parametric (Woods, 2006) or multiple group (Muthén & Lehman, 1985) methods, an extension to the methods in this article is required where the derivatives with respect to the distribution parameters are derived.

Appendix

With the GPCM, for each item $j$ and each category $k\in \{1,\dots ,m_j\}$, the required derivative vector is

$$\frac{{\partial }^2{P}_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial \mathbf{\alpha }}=\left(\begin{array}{ccccccccc}\hfill 0\hfill & \hfill \dots \hfill & \hfill \frac{{\partial }^2{P}_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial a_j}\hfill & \hfill \frac{{\partial }^2{P}_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial b_{j2}}\hfill & \hfill \dots \hfill & \hfill \frac{{\partial }^2{P}_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial b_{j{m}_j}}\hfill & \hfill 0\hfill & \hfill \dots \hfill & \hfill 0\hfill \end{array}\right),$$

where

$$\begin{array}{cc}\multicolumn{1}{c}{\frac{{\partial }^2{P}_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial a_j}=}& P_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)(k-\sum _{c=1}^{m_j}c{P}_{\mathrm{jc}}({\theta }_l;{\mathbf{\alpha }}_j))+\frac{\partial P_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial a_j}{a}_j(k-\sum _{c=1}^{m_j}c{P}_{\mathrm{jc}}({\theta }_l;{\mathbf{\alpha }}_j))\hfill \\ \multicolumn{1}{c}{}& -P_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)a_j\sum _{c=1}^{m_j}c\frac{\partial P_{\mathrm{jc}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial a_j}\hfill \end{array}$$

and, for $k^{*}=2,\dots ,m_j$,

$$\frac{{\partial }^2{P}_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial b_{j{k}^{*}}}=\frac{\partial P_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial b_{j{k}^{*}}}{a}_j(k-\sum _{c=1}^{m_j}c{P}_{\mathrm{jc}}({\theta }_l;{\mathbf{\alpha }}_j))-P_{\mathrm{jk}}({\theta }_l;{\mathbf{\alpha }}_j)a_j\sum _{c=1}^{m_j}c\frac{\partial P_{\mathrm{jc}}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial b_{j{k}^{*}}}.$$

With the 3-PL model, for each item $j$, the derivative vectors are

$$\frac{{\partial }^2{P}_{j1}({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial \mathbf{\alpha }}=\left(\begin{array}{cccccccc}\hfill 0\hfill & \hfill \dots \hfill & \hfill -\frac{{\partial }^2{P}_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial a_j}\hfill & \hfill -\frac{{\partial }^2{P}_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial b_j}\hfill & \hfill -\frac{{\partial }^2{P}_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial c_j}\hfill & \hfill 0\hfill & \hfill \dots \hfill & \hfill 0\hfill \end{array}\right)$$

and

$$\frac{{\partial }^2{P}_{j2}({\theta }_l;\mathbf{\alpha })}{\partial {\theta }_l\partial \mathbf{\alpha }}=\left(\begin{array}{cccccccc}\hfill 0\hfill & \hfill \dots \hfill & \hfill \frac{{\partial }^2{P}_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial a_j}\hfill & \hfill \frac{{\partial }^2{P}_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial b_j}\hfill & \hfill \frac{{\partial }^2{P}_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial c_j}\hfill & \hfill 0\hfill & \hfill \dots \hfill & \hfill 0\hfill \end{array}\right)$$

where

$$\begin{array}{c}\frac{{\partial }^2{P}_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial a_j}=\frac{(1-P_j({\theta }_l;{\mathbf{\alpha }}_j))(P_j({\theta }_l;{\mathbf{\alpha }}_j)-c_j)}{1-c_j}+\frac{a_j(1-P_j({\theta }_l;{\mathbf{\alpha }}_j))}{1-c_j}\frac{\partial P_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial a_j}\hfill \\ -\frac{a_j(P_j({\theta }_l;{\mathbf{\alpha }}_j)-c_j)}{1-c_j}\frac{\partial P_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial a_j},\hfill \end{array}$$

$$\frac{{\partial }^2{P}_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial b_j}=\frac{a_j(1-P_j({\theta }_l;{\mathbf{\alpha }}_j))}{1-c_j}\frac{\partial P_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial b_j}-\frac{a_j(P_j({\theta }_l;{\mathbf{\alpha }}_j)-c_j)}{1-c_j}\frac{\partial P_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial b_j}$$

and

$$\begin{array}{c}\frac{{\partial }^2{P}_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial {\theta }_l\partial c_j}=-\frac{a_j(P_j({\theta }_l;{\mathbf{\alpha }}_j)-c_j)}{1-c_j}\frac{P_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial c_j}\hfill \\ +a_j(1-P_j({\theta }_l;{\mathbf{\alpha }}_j))\frac{(\frac{\partial P_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial c_j}-1)(1-c_j)+(P_j({\theta }_l;{\mathbf{\alpha }}_j)-c_j)}{{(1-c_j)}^2}.\hfill \end{array}$$

The derivatives $\frac{\partial P_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial a_j}$, $\frac{\partial P_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial b_j}$ and $\frac{\partial P_j({\theta }_l;{\mathbf{\alpha }}_j)}{\partial c_j}$ can be found in Hambleton and Swaminathan (1985).