Information Functions of Rank-2PL Models for Forced-Choice Questionnaires

Abstract

This paper presents the item and test information functions of the Rank two-parameter logistic models (Rank-2PLM) for items with two (pair) and three (triplet) statements in forced-choice questionnaires. The Rank-2PLM model for pairs is the MUPP-2PLM (Multi-Unidimensional Pairwise Preference; Morillo et al., 2016), and for triplets, is the Triplet-2PLM (Fu et al., 2023a). Fisher’s information and directional information are described, and the test information for Maximum Likelihood (ML), Maximum A Posterior (MAP), and Expected A Posterior (EAP) trait score estimates is distinguished. Expected item/test information indexes at various levels are proposed and plotted to provide diagnostic information on items and tests. The expected test information indexes for EAP scores may be difficult to compute due to a typical test’s vast number of item response patterns. The relationships of item/test information with discrimination parameters of statements, standard error, and reliability estimates of trait score estimates are discussed and demonstrated using real data. Practical suggestions for checking the various expected item/test information indexes and plots are provided.

Introduction

In a forced-choice questionnaire (FCQ; Christiansen et al., 2005), an item is a block of statements that measures noncognitive traits, for example, attitudes, values, beliefs, or behaviors. An item includes two or more statements, which may measure the same or different traits. Respondents are asked to rank the statements based on the degree to which they endorse them, from the most to the least. The following is an example of a forced-choice item with three statements (triplet):

Order the following statements from the most to least like you:

1. I hate to lose

2. I find creative ideas come easily

3. I conceal my feeling

As an alternative to the traditional Likert scale, the forced-choice format is designed to control test takers’ impression management on noncognitive tests and thus is a popular noncognitive test format, especially in high-stakes contexts where test takers may be more inclined to respond in a socially desirable way (Morillo et al., 2019).

Some researchers have proposed item response theory (IRT) models to calibrate FCQs and generate normative scores (Brown, 2016). The notable models include the Thurstonian IRT model (TIRT; Brown & Maydeu-Olivares, 2011) and the Rank models under the ranking framework (de la Torre et al., 2012), for example, MUPP-2PLM (Multi-Unidimensional Pairwise Preference with two-Parameter Logistic IRT Model; Morillo et al., 2016) and MUPP-GGUM (MUPP with Generalized Graded Unfolding Model; Stark et al., 2005) for items with two statements (pairs). Except for MUPP-2PLM, a special case of the commonly known compensatory two-parameter logistic multidimensional IRT Model (von Davier, 2008), no item parameter estimation methods are available for either the MUPP-GGUM or the Rank models for items with more than two statements. Recently, Fu et al. (2023a) developed a maximum marginal likelihood estimation with an expectation-maximization algorithm (MML-EM; Bock & Aitkin, 1981) to estimate item parameters and their standard errors on the Rank-2PLM for items with three statements (denoted as Triplet-2PLM). In a simulation study, they showed satisfactory parameter recovery and successfully applied it to real data.

The purpose of this paper is to develop the item and test information functions (IIF and TIF) for the Rank-2PLMs for pairs (i.e., MUPP-2PLM) and triplets (i.e., Triplet-2PLM). Item and test information are critical psychometric properties of items and tests and play an important role in test assembly, especially for computer adaptive tests. They also provide diagnostic information that test developers can use to identify and adjust problematic items and tests. The current study makes several unique contributions to educational and psychological measurement literature.

First, we studied the item and test information functions for MUPP-2PLM and Triplet-2PLM. The IIF and TIF for Triplet-2PLM were developed for the first time. The current study adds to the test information literature on IRT models for FCQ, for example, TIRT (Brown & Maydeu-Olivares, 2011), MUPP-GGUM (Stark et al., 2005), and Rank-GGUM (Joo et al., 2018).

Second, we proposed various summative IIF and TIF indices and plots to provide helpful information to diagnose and adjust items and tests. These indices can also be used in an automatic test assembly program. It was a long-time issue how to use IIF and TIF in practice for IRT models for FCQs and, broadly, for multidimensional IRT models. It is the first time these indices and plots were presented systemically and demonstrated for FCQs in real data. Joo et al. (2018) also proposed two (i.e., expected overall item information and expected overall test information) of these indices presented in this paper. However, Joo et al. calculated these indices using simulated trait scores from their prior distribution rather than using their densities in a more theoretical way, as we did in this paper. In addition, the use of these indices was little discussed and not the focus of their paper.

Third, we described the inherent connections among test information, standard errors (SE) of trait score estimates, and reliability. We showed that the directional IIF was a weighted sum of all elements in an item information matrix. We discouraged its use in practice due to its disconnection with SE and reliability estimates. Using directional IIF to estimate reliability (e.g., Brown & Maydeu-Olivares, 2011) will produce an inconsistent estimate with that based on a test sample. We proposed to compare expected SE and reliability estimates based on expected overall test information on a trait to those based on a test sample as indications of the sample representativeness.

Fourth, we distinguished different TIFs for three kinds of latent score estimates: Maximum Likelihood (ML), Maximum A Posteriori (MAP), and Expected A Posteriori (EAP). These distinctions were not often emphasized in the literature. For example, in Joo et al. (2018), the TIF for ML scores was used for EAP scores. We also pointed out the difficulty in generating the proposed summative TIF indices for EAP scores.

Finally, using the real data on four FCQ forms, we obtained some interesting findings that are consistent with other studies in the literature: (a) an IIF increased with the absolute item discrimination; (b) a triplets form needed fewer statements than a pairs form to achieve a precision level of trait score estimates; and (c) the reliability estimates for ML and MAP estimates were similar although their TIF, SE, and score variance estimates were quite different.

In the following sections, we first describe the response functions of the Rank-2PLMs; second, we present three estimation methods of latent trait scores (i.e., Maximum Likelihood, Maximum A Posteriori, and Expected A Posteriori) because they are closely related to test information; third, we develop the item and test information functions: distinguish test information for the three types of trait score estimates and show the connection between the Fisher’s information and directional information; fourth, we discuss the inherent relationships between test information and standard error and reliability of trait score estimates; fifth, we propose several expected item and test information indices and plots, demonstrate their use in evaluating items and tests in real data, and provide practical recommendations; sixth, we show the relationships of item and test information with item discrimination parameters, standard errors, and reliabilities in the real data on four FCQ forms; finally we discuss the limitations of the current study and suggest future research.

Item Response Functions of Rank-2PLM

Following Luce (2005/1959), a Rank model assumes that ranking $K$ statements in a block is an iterative process of selecting the most agreeable statement among the remaining statements until all $K$ statements are ranked. For MUPP-2PLM, $K=2$ and the probability of a ranking of the two statements is

$$P_{(1,2)}(\text{θ})=\frac{P_{(1)}\left(1\mid {\theta }_1\right)P_{(2)}\left(0\mid {\theta }_2\right)}{P_{(1)}\left(1\mid {\theta }_1\right)P_{(2)}\left(0\mid {\theta }_2\right)+P_{(1)}\left(0\mid {\theta }_1\right)P_{(2)}\left(1\mid {\theta }_2\right)},$$ (1)

where $\text{θ}$ is the column vector of the latent trait scores that the statements measure; the subscripts refer to the statements with Rank 1 and 2, respectively (Rank 1 is the more agreeable statement); $P_{(1,2)}$ is the probability of a ranking order; $P_{(1)}\left(1\mid {\theta }_1\right)$ and $P_{(1)}\left(0\mid {\theta }_1\right)$ are the probabilities of endorsing and rejecting Rank 1 statement, respectively, conditional on the latent trait score ${\theta }_1$ that the statement measures; and $P_{(2)}\left(1\mid {\theta }_2\right)$ and $P_{(2)}\left(0\mid {\theta }_2\right)$ are the same probabilities for Rank 2 statement. The probability of endorsing a statement follows the item response function of the 2PLM (Birnbaum, 1968). Insert the 2PLM response function into Equation 1, and then through Algebra operations, we get the item response function of the MUPP-2PLM (Morillo et al., 2016),

$$P_{(1,2)}(\text{θ})={\left[1+\exp \left(a_2{\theta }_2+b_2-a_1{\theta }_1-b_1\right)\right]}^{-1}={\left[1+\exp \left(a_2{\theta }_2-a_1{\theta }_1-b_{12}\right)\right]}^{-1},$$ (2)

where $a_1$ and $a_2$ are the discrimination/slope parameters of Rank 1 and 2 statements, respectively, in the 2PLM, $b_1$ and $b_2$ are the intercepts, and $b_{12}=b_1-b_2$ is the estimable intercept as only one of the two intercepts is identified in Equation 2. Alternatively, one of the values can be fixed to identify the model. Note that MUPP-2PLM is just a special case of the multidimensional 2PLM. If the two statements measure the same trait, Equation 2 becomes 2PLM, and only one of the two discrimination parameters can be identified.

Based on the same rationale, we can get the item response function of Triplet-2PLM for triplets (Fu et al., 2023a),

$$\begin{gathered} P_{(1,2,3)}(\text{θ})= \\ {\left[1+\exp \left(a_2{\theta }_2+b_2-a_1{\theta }_1-b_1\right)+\exp \left(a_3{\theta }_3+b_3-a_1{\theta }_1-b_1\right)\right]}^{-1}{\left[1+\exp \left(a_3{\theta }_3+b_3-a_2{\theta }_2-b_2\right)\right]}^{-1} \end{gathered}$$ (3)

Note that for $b_1$, $b_2$ and $b_3$, only two of them can be identified in Equation 3. We can fix one of the $b$ values to identify the model. If all three statements measure a common trait, only two $a$ values can be identified, and one may be fixed to identify the model.

For a triplet with three statements, A, B, and C, there are six possible ranking patterns, ABC, ACB, BAC, BCA, CAB, and CBA, which are coded as 1 to 6, respectively, in the observed data. For pairs, there are only two possible ranking patterns: an item score is coded as 1 if the first statement is ranked the first, 0 otherwise.

Trait Score and Covariance Matrix Estimation

Item and test information in an IRT model is directly related to the error variance of a latent trait estimate. The information function of a latent trait estimate is the reciprocal of its error variance. Thus, we must first describe the different kinds of trait score estimates and their error variance estimations.

Suppose in an FCQ, there are $I$ test takers responding to $J$ items. Let $x_{ij}=s$ denote test taker $i$‘s coded score on item $j$ ($s=0,1$ for pairs; $s=1,2,\dots ,6$ for triplets), and $x_i$ is test taker $i$‘s coded response vector across $J$ items. The FCQ measures $D$ traits and $\text{θ}$ represents a random latent trait score vector sample from the test taker population with a standard multivariate normal distribution with a $D\times D$ correlation matrix ${\Sigma }_{\theta }$. $\eta$ represents the vector of item parameters in the whole FCQ, and ${\eta }_j$ is item $j$‘s parameter vector. When item parameter and intertrait correlation estimates are available, we can estimate IRT trait scores for each test taker based on Equations 2 and 3. Because the local independence assumption of item responses holds, each test taker’s trait vector can be estimated separately. Typically, there are three types of trait score estimates: Maximum Likelihood (ML), Maximum A Posterior (MAP), and Expected A Posterior (EAP) (Baker & Kim, 2004).

Maximum Likelihood (ML) Score Estimates

For ML, the log-likelihood function for a test taker $i$ is

$$l_{\text{ML}}\left({\text{θ}}_i\mid x_i,\eta ,{\Sigma }_{\text{θ}}\right)={\displaystyle \sum _{j=1}^J\log P\left(x_{ij}\mid {\text{θ}}_i,{\eta }_j\right),}$$ (4)

where $P\left(x_{ij}\mid {\text{θ}}_i,{\eta }_j\right)$ is the item response function of MUPP-2PLM/Triplet-2PLM (Equation 2/3). The symbolic first and second derivatives, $P\left(x_{ij}\mid {\text{θ}}_i,{\eta }_j\right)$, with respect to ${\text{θ}}_i$ can be obtained by the R package, Deriv (Clausen & Sokol, 2020), and the code is provided in the supplemental online materials. Then, Equation 4 can be optimized by the Newton-Raphson method (NR; Atkinson, 1989) or a quasi-Newton method, for example, the BFGS algorithm (Broyden, 1970).

The variance and covariance matrix (COV) of ML estimates (MLE) ${\widehat{\theta }}_i$ can be estimated by the negative inverse of the Hessian matrix evaluated at MLE ${\widehat{\theta }}_i$ and ${\widehat{\eta }}_j$

$$\begin{gathered} {\Omega }_{{\widehat{\theta }}_i^{\text{ML}}}^{\text{H}}=-{\left[{\displaystyle \sum _{j=1}^J\frac{{\partial }^2\log P\left(x_{ij}\mid {\widehat{\theta }}_i,{\widehat{\eta }}_j\right)}{\partial {\widehat{\theta }}_i\partial {\widehat{\theta }}_i^{\prime }}}\right]}^{-1} \\ ={\left\{{\displaystyle \sum _{j=1}^J\frac{\partial P\left(x_{ij}\mid {\widehat{\theta }}_i,{\widehat{\eta }}_j\right)}{\partial {\widehat{\theta }}_i}{\left[\frac{\partial P\left(x_{ij}\mid {\widehat{\theta }}_i,{\widehat{\eta }}_j\right)}{\partial {\widehat{\theta }}_i}\right]}^{\prime }{\left[P\left(x_{ij}\mid {\widehat{\theta }}_i,{\widehat{\eta }}_j\right)\right]}^{-2}}\right\}}^{-1} \\ -{\left[{\displaystyle \sum _{j=1}^J\frac{{\partial }^{2}P\left(x_{ij}\mid {\widehat{\theta }}_i,{\widehat{\eta }}_j\right)}{\partial {\widehat{\theta }}_i\partial {\widehat{\theta }}_i^{\prime }}}/P\left(x_{ij}\mid {\widehat{\theta }}_i,{\widehat{\eta }}_j\right)\right]}^{-1} \end{gathered}$$ (5)

Maximum A Posteriori (MAP) Score Estimates

The MAP ${\widehat{\theta }}_i$ maximizes the posterior distribution of ${\theta }_i$,

$$\log f\left({\theta }_i\mid x_i,\eta ,{\Sigma }_{\theta }\right)\propto {\displaystyle \sum _{j=1}^J\left[\log P\left(x_{ij}\mid {\theta }_i,{\eta }_j\right)+\log \varphi \left({\theta }_i\mid {\Sigma }_{\theta }\right)\right]},$$ (6)

which is just the ML log-likelihood function plus the logarithm of the multivariate normal density of ${\theta }_i$, $\varphi \left({\theta }_i\mid {\Sigma }_{\theta }\right)$. The first and second derivatives of $\varphi \left(\theta \mid {\Sigma }_{\theta }\right)$ with respect to $\text{θ}$ are

$$\frac{\partial \varphi \left(\theta \mid {\Sigma }_{\theta }\right)}{\partial \theta }=-\varphi \left(\theta \mid {\Sigma }_{\theta }\right){\Sigma }_{\theta }^{-1}\theta ,$$ (7)

$$\frac{{\partial }^2\varphi \left(\theta \mid {\Sigma }_{\theta }\right)}{\partial \theta \partial {\theta }^{\prime }}=\varphi \left(\theta \mid {\Sigma }_{\theta }\right)\left({\Sigma }_{\theta }^{-1}\theta {\theta }^{\prime }{\Sigma }_{\theta }^{-1}-{\Sigma }_{\theta }^{-1}\right)$$ (8)

The COV of MAP ${\widehat{\theta }}_i$ estimates can also be estimated by the negative inverse of the Hessian matrix of $\log f\left({\theta }_i\mid x_i,\eta ,{\Sigma }_{\theta }\right)$ evaluated at MAP ${\widehat{\theta }}_i$ and ${\widehat{\eta }}_j$.

Expected A Posteriori (EAP) Score Estimates

The EAP ${\widehat{\theta }}_i$ is the expected ${\theta }_i$ over its posterior distribution

$${\widehat{\theta }}_i^{\text{EAP}}={\displaystyle {\int }_{\text{R}}\theta f\left(\theta \mid x_i,\eta ,{\Sigma }_{\theta }\right)d\theta \approx }{\displaystyle \sum _{l=1}^L{\omega }_{l}f\left({\omega }_l\mid x_i,\eta ,{\Sigma }_{\theta }\right)},$$ (9)

$$f\left({\omega }_l\mid x_i,\eta ,{\Sigma }_{\theta }\right)=\frac{{\displaystyle \prod _{j=1}^{J}P\left(x_{ij}\mid {\eta }_j,{\omega }_l\right)\varphi \left({\omega }_l\mid {\Sigma }_{\theta }\right)}}{{\displaystyle \sum _{l^{\ast }=1}^L{\displaystyle \prod _{j=1}^{J}P\left(x_{ij}\mid {\eta }_j,{\omega }_{l^{\ast }}\right)\varphi \left({\omega }_{l^{\ast }}\mid {\Sigma }_{\theta }\right)}}},$$ (10)

where ${\omega }_l\text{s}\cdot (l=1,2,\dots ,L)$ are the quadrature points to approximate the prior distribution of $\text{θ}$, usually, the standard multivariate normal distribution with a correlation matrix ${\Sigma }_{\theta }$.

The covariance matrix of ${\widehat{\theta }}_r^{\text{EAP}}$ conditional on response pattern $r$ (Haberman & Sinharay, 2010) is

$${\Omega }_{{\widehat{\theta }}_r^{\text{EAP}}\mid x_r}\approx {\displaystyle \sum _{l=1}^{L}f\left({\omega }_l\mid x_r,\eta ,{\Sigma }_{\theta }\right)\left({\omega }_l-{\widehat{\theta }}_r^{\text{EAP}}\right){\left({\omega }_l-{\widehat{\theta }}_r^{\text{EAP}}\right)}^{\prime },}$$ (11)

where $x_r$ is response pattern $r$, and ${\widehat{\theta }}_r^{\text{EAP}}$ is the EAP score estimate for $x_r$. The expected covariance matrix of errors of EAP score estimates is the expectation of ${\Omega }_{{\widehat{\theta }}_r^{\text{EAP}}\mid x_r}$ across all possible response patterns in a test

$${\Omega }_{\left({\widehat{\text{θ}}}^{\text{EAP}}-\theta \right)}\approx {\displaystyle \sum _{r=1}^R{\Omega }_{{\widehat{\theta }}_r^{\text{EAP}}\mid x_r}P\left(x_r\mid \eta ,{\Sigma }_{\theta }\right),}$$ (12)

$$P\left(x_r\mid \eta ,{\Sigma }_{\theta }\right)={\displaystyle \sum _{l=1}^L{\displaystyle \prod _{j=1}^{J}P\left(x_{rj}\mid {\eta }_j,{\omega }_l\right)\varphi \left({\omega }_l\mid {\Sigma }_{\theta }\right),}}$$ (13)

where $R$ is the number of all possible response patterns in a test. Because the number of all possible response patterns in a test could be enormous, it may be infeasible to calculate the covariance matrix of ${\widehat{\theta }}^{\text{EAP}}$ errors based on Equations 12 and 13. Alternatively, we can estimate ${\Omega }_{\left({\widehat{\theta }}^{\text{EAP}}-\theta \right)}$ based on simulated data or a sample of test data,

$${\widehat{\Omega }}_{\left({\widehat{\theta }}^{\text{EAP}}-\theta \right)}=\frac{1}{I}{\displaystyle \sum _{i=1}^I}{\Omega }_{{\widehat{\theta }}_i^{\text{EAP}}\mid x_i}.$$ (14)

Item and Test Information

The test information of a latent trait score estimate refers to the precision of the score estimate based on a test. Item information is the precision of a latent trait score estimate that an item contributes to the test information. In other words, test information is the sum of the item information of all items in a test by assuming item local independence. For MAP and EAP latent trait score estimates, the information from the prior population distribution of latent trait scores must be added to the test information.

Item Information

The item information for an item with multiple score categories in an IRT model is the expected Fisher’s information across score categories (Muraki, 1993; Joo et al., 2018). For an FCQ pair/triplet item $j$, its item information for a trait vector $\text{θ}$ is

$$\begin{array}{l}{\beta }_j(\theta )=\sum _{s_j}^{s=1}-\frac{{\partial }^2\log P\left(x_j=s\mid \theta ,{\eta }_j\right)}{\partial \theta \partial {\theta }^{\prime }}P\left(x_j=s\mid \theta ,{\eta }_j\right)\\ ={\displaystyle \sum _{s=1}^{S_j}\frac{\partial P\left(x_j=s\mid \theta ,{\eta }_j\right)}{\partial \theta }{\left[\frac{\partial P\left(x_j=s\mid \theta ,{\eta }_j\right)}{\partial \theta }\right]}^{\prime }{\left[P\left(x_j=s\mid \theta ,{\eta }_j\right)\right]}^{-1},}\\ -{\displaystyle \sum _{s=1}^{S_j}\frac{{\partial }^{2}P\left(x_j=s\mid \theta ,{\eta }_j\right)}{\partial \theta \partial {\theta }^{\prime }}}\end{array}$$ (15)

where $S_j$ (=2 for pair; =6 for triplet) is the number of score categories of item j. Because ${\displaystyle {\displaystyle \sum _{s=1}^{S_j}}\frac{{\partial }^{2}P\left(x_j=s\mid \theta ,{\eta }_j\right)}{\partial \theta \partial {\theta }^{\prime }}=\frac{{\partial }^2{\displaystyle {\displaystyle \sum _{s=1}^{S_j}}P\left(x_j=s\mid \theta ,{\eta }_j\right)}}{\partial \theta \partial {\theta }^{\prime }}=0}$, Equation 15 is reduced to

$${\beta }_j(\theta )={\displaystyle \sum _{s=1}^{s_j}\frac{\partial P\left(x_j=s\mid \theta ,{\eta }_j\right)}{\partial \theta }{\left[\frac{\partial P\left(x_j=s\mid \theta ,{\eta }_j\right)}{\partial \theta }\right]}^{\prime }{\left[P\left(x_j=s\mid \theta ,{\eta }_j\right)\right]}^{-1}}.$$ (16)

Fisher’s information formula of Equation 15 is based on the Hessian matrix, while Equation 16 is the variance of Fisher’s score. The equality between Equations 15 and 16 is well known in an ML estimation given $\log P\left(x_j=s\mid \theta ,{\eta }_j\right)$ is twice differentiable with respect to $\text{θ}$, and under certain regularity conditions (Lehmann & Casella, 1998, p.116), which are met here. In the literature, authors used either Equation 15 (e.g., Muraki, 1993; Joo et al., 2018; Yao & Schwarz, 2006) or Equation 16 (e.g., Brown & Maydeu-Olivares, 2011; Stark et al., 2005), while the equivalence was not mentioned in these papers. Equation 16 is more computationally efficient because only the first derivatives are involved; thus, it is used here. The first derivatives in Equation 16 have been used in estimating ML $\text{θ}$ (Equation 4) and MAP $\text{θ}$ (Equation 6). The derivatives are only relevant to those ${\theta }_d$ s measured by item $j$. Thus, in the $D\times D$ information matrix, ${\beta }_j(\theta )$, only those elements with row and column traits measured by item have information, and the others are all 0s.

Some researchers used directional information that considers the direction of information in a multidimensional space (e.g., Brown & Maydeu-Olivares, 2011; Yao & Schwarz, 2006). Let $\alpha ={\left({\alpha }_1,\dots ,{\alpha }_D\right)}^{\prime }$ be a vector of angles to all $D$ axes that defines the direction from a point $\text{θ}$. The item information function for an FCQ item $j$ at the direction of $\alpha$ is defined by

$$v_j^{\alpha }(\theta )={\alpha }^{\prime }{\beta }_j(\theta )\alpha .$$ (17)

For the information along the axis of one trait, ${\theta }_d$, set the angle to ${\theta }_d$ to 0 and, thus $\cos {\alpha }_d=1$, and set the angles to the other ${\theta }_{d^{\ast }}s$. $\left(d^{\ast }\ne d\right)$ based on the correlation between ${\theta }_d$ and ${\theta }_{d^{\ast }}$ so that $\cos {\alpha }_{d^{\ast }}=cor\left({\theta }_d,{\theta }_{d^{\ast }}\right)$. This is referred to as the item information in the direction of trait ${\theta }_d$. One can see that the directional item information along a trait $d$ is just a sum of each element in the item information matrix weighted by the correlations between each trait and trait $d$. Because we emphasize the consistency of the estimates between standard errors/reliabilities of latent trait scores and item/test information, the directional information is not recommended in practice. The concept of directional information is confusing to test practitioners.

Because item information in an IRT model is defined based on Fisher’s information, and Fisher’s information is a concept defined in the general theory of ML estimation, item information is only relevant to ML $\text{θ}$.

Test Information

Based on the local independence assumption, we get the test information by summing all item information in an FCQ

$${\beta }^{\text{ML}}(\theta )={\displaystyle \sum _{j=1}^J}{\beta }_j(\theta ).$$ (18)

The test information in Equation 18 is defined for ML $\text{θ}$. For ML $\text{θ}$, Fisher’s information defines item/test information. Because Fisher’s test information matrix equals the inverse of the covariance matrix of ML $\text{θ}$ estimates, we define the test information here for MAP/EAP $\text{θ}$ as the inverse of the covariance matrix of MAP/EAP $\text{θ}$ estimates.

For MAP $\text{θ}$ test information, we need to add the prior information from $\text{θ}$ into Equation 18 and get the posterior test information (Du Toit, 2003)

$${\beta }^{\text{MAP}}(\theta )={\displaystyle \sum _{j=1}^J{\beta }_j(\theta )-\frac{{\partial }^2\log \varphi \left(\theta \mid {\Sigma }_{\theta }\right)}{\partial \theta \partial {\theta }^{\prime }},}$$ (19)

where the second derivative of $\varphi \left(\theta \mid {\Sigma }_{\theta }\right)$ is in Equation 8. The diagonal elements in the second derivative matrix of $\log \varphi \left(\theta \mid {\Sigma }_{\theta }\right)$ are equal to the diagonal elements in $-{\text{Σ}}_{\theta }^{-1}$ (Brown & Maydeu-Olivares, 2011).

For EAP $\text{θ}$ test information, we can only calculate the test information of an EAP score conditional on a response pattern. Using Equation 11 to approximate the covariance matrix of the EAP score ${\theta }_r^{\text{EAP}}$ associated with response pattern and inversing it, we get the test information matrix for ${\theta }_r^{\text{EAP}}$

$$\beta \left({\theta }_r^{\text{EAP}}\right)\approx {\left[{\displaystyle \sum _{l=1}^{L}f\left({\omega }_l\mid x_r,\eta ,{\Sigma }_{\theta }\right)\left({\omega }_l-{\theta }_r^{\text{EAP}}\right)\left({\omega }_l-{\theta }_r^{\text{EAP}}\right)}\prime \right]}^{-1}.$$ (20)

Generally, there is a one-to-one correspondence between ${\theta }_r^{\text{EAP}}$ and $x_r$(Kim, 2012). Because the number of possible response patterns could be vast, it may be practically impossible to calculate the test information for all ${\theta }_r^{\text{EAP}}$.

For a trait score, its test information is the corresponding diagonal element in the test information matrix and is equal to the squared reciprocal of the standard error estimate of the latent trait score.

Reliability of Trait Score Estimates

The test information of a trait score estimate indicates the precision of the score estimate, and the precisions are different for different score estimates in IRT. An IRT reliability coefficient is an overall index of the precision of trait scores estimated from a test analogous to the reliability in the classical test theory. For MLE $\widehat{\theta }$, the reliability of trait $d$‘s score estimates ${\widehat{\theta }}_d(d=1,\dots ,D)$. is defined as (Kim, 2012, 2013)

$${\rho }_{\text{ML}}^2=\frac{\mathrm{var}\left({\theta }_d\right)}{\mathrm{var}\left({\theta }_d\right)+E\left(e_{{\widehat{\theta }}_d\mid {\theta }_d}\right)}=\frac{\mathrm{var}\left({\widehat{\theta }}_d\right)-E\left(e_{{\widehat{\theta }}_d\mid {\theta }_d}\right)}{\mathrm{var}\left({\widehat{\theta }}_d\right)},$$ (21)

where $\mathrm{var}\left({\theta }_d\right)$ is the population variance of trait $d$‘s true scores, $E\left(e_{{\widehat{\theta }}_d\mid {\theta }_d}\right)$ is the expected value of the error variances of predicting trait $d$‘s score estimates ${\widehat{\theta }}_d$ by ${\theta }_d$, and $\text{var}\left({\widehat{\theta }}_d\right)$ is the population variance of trait ‘s score estimates and equal to $\mathrm{var}\left({\text{θ}}_d\right)+E\left(e_{{\widehat{\text{θ}}}_d\mid {\text{θ}}_d}\right)$. In Equation 21 $\mathrm{var}\left({\theta }_d\right)$ can be set to the population variance (usually 1) if known (e.g., Andersson & Xin, 2018), or can be estimated by the sample ${\widehat{\theta }}_d$ estimates of the test takers who took a test. Similarly, $E\left(e_{{\widehat{\theta }}_d\mid {\theta }_d}\right)$ can be calculated as the expected value of the reciprocal of the test information of trait $d$ scores (Equation 18) over the distribution of ${\theta }_d$ given all model parameters known or estimated by the sample mean of test takers’ error variance estimates on trait $d$. The front one is referred to as marginal reliability, and the late one is referred to as empirical reliability in the literature. The empirical reliability is estimated as

$${\widehat{\rho }}_{ML}^2=\frac{\mathrm{vâr}\left({\widehat{\text{θ}}}_d\right)-{\displaystyle \sum _{i=1}^I{\widehat{e}}_{{\widehat{\text{θ}}}_{id}}/I}}{\mathrm{vâr}\left({\widehat{\text{θ}}}_d\right)},$$ (22)

where $\text{vâr}\left({\widehat{\text{θ}}}_d\right)$ is the sample variance of ${\widehat{\text{θ}}}_d$, and ${\widehat{e}}_{{\widehat{\theta }}_{id}}$ is the error variance estimate of test taker $i$‘s estimated trait $d$‘s score ${\widehat{\theta }}_{id}$.

For MAP and EAP $\widehat{\text{θ}}$, the reliability of trait $d$‘s score estimates is defined as (Kim, 2012, 2013)

$${\rho }_{\text{M}/\text{EAP}}^2=\frac{\mathrm{var}\left({\widehat{\text{θ}}}_d\right)}{\mathrm{var}\left({\widehat{\text{θ}}}_d\right)+E\left(e_{{\text{θ}}_d\mid {\widehat{\text{θ}}}_d}\right)},$$ (23)

where $\text{vâr}\left({\widehat{\text{θ}}}_d\right)$ is the population variance of trait $d$‘s MAP/EAP score estimates ${\widehat{\theta }}_d$, and $E\left(e_{{\text{θ}}_d\mid {\widehat{\text{θ}}}_d}\right)$ is the expected value of the error variances of predicting trait ‘s true scores ${\theta }_d$ by estimates ${\widehat{\text{θ}}}_d$. $\mathrm{var}\left({\widehat{\text{θ}}}_d\right)$ is equal to the variance of the prior distribution of ${\theta }_d$, given the model holds. Usually, is estimated by the variance of the sample ${\widehat{\text{θ}}}_d$ estimates. For MAP, $E\left(e_{{\text{θ}}_d\mid {\widehat{\text{θ}}}_d}\right)$.can be calculated as the expected value of the reciprocal of the test information of trait $d$ scores (Equations 19) over the prior distribution of ${\theta }_d$, and for EAP, $E\left(e_{{\text{θ}}_d\mid {\widehat{\text{θ}}}_d}\right)$.can be calculated by Equation 12. $E\left(e_{{\text{θ}}_d\mid {\widehat{\text{θ}}}_d}\right)$ can also be estimated by the sample mean of test takers’ error variance estimates on trait $d$‘s MAP/EAP score estimates. Using a test-taker sample to estimate $\mathrm{var}\left({\widehat{\text{θ}}}_d\right)$ and $E\left(e_{{\text{θ}}_d\mid {\widehat{\text{θ}}}_d}\right)$, the empirical reliability for MAP or EAP scores is estimated as

$${\widehat{\rho }}_{\text{M}/\text{EAP}}^2=\frac{{\text{va}}^{\text{r}}\left({\widehat{\text{θ}}}_d\right)}{\mathrm{var}\left({\widehat{\text{θ}}}_d\right)+{\displaystyle \sum _{i=1}^I{\widehat{e}}_{{\theta }_{id}}}/I}$$ (24)

The $E\left(e_{{\widehat{\theta }}_d\mid {\theta }_d}\right)$ and $E\left(e_{{\text{θ}}_d\mid {\widehat{\text{θ}}}_d}\right)$ calculated based on the expected test information/error variances represent the theoretical value given the model. The expected test information is one of the expected item/test information indices we propose and demonstrate in the next section. Comparing error variance or test reliability based on the expected test information to the estimate based on a test-taker sample provides an indication of sample adequacy.

Note that Kim (2012) distinguished two types of reliabilities: parallel-forms reliability and squared-correlation reliability. The two types of reliabilities are not equal for IRT latent score estimates because all kinds of IRT score estimates are not unbiased estimators of the true scores. Equation 21 is based on the parallel-forms reliability, and Equation 23 is derived from the squared-correlations reliability under an assumption. These two reliabilities have been widely used in research and practice, partly because the statistics to calculate the two reliabilities are easily estimated from observed samples (Kim, 2013). Kim (2012) provided more formulas for IRT reliability estimations; these reliabilities can be estimated by Monte Carlo simulations with known item parameters if the statistics in the formulas are not available in the score estimations of observed samples.

Applications on Real Data

We demonstrate the estimation of item/test information based on the multidimensional MUPP-2PLM and Triplet-2PLM on four real datasets. We show how expected item/test information at different levels can be calculated and plotted. These indices and graphs provide a convenient way to show the precision distribution of latent trait score estimates at the item and test levels and provide diagnostic information for adjusting items and test assembly.

Real Data and IRT Calibrations

The real data included four FCQ forms: two pairs forms (A2 and B2) and two triplets forms (A3 and B3). Each pairs form included 130 items with 260 statements; each triplets form contained 60 items with 180 statements. These statements measured 14 interpersonal and intrapersonal skills essential to higher education and career success, such as perseverance, leadership, creativity, curiosity, responsibility, and self-discipline. All FCQ items were multidimensional items; that is, all statements in an item measured different traits. Each trait is measured by 11 to 20 statements in a pairs form and by 5 to 18 statements in a triplets form.

The four forms were assembled from 560 statements (measuring 14 traits and each trait measured by 40 statements) by an automatic assembly program using a linear programming solver. The two forms in each type were assembled to be parallel forms. The four FCQ forms were then administered to the test takers recruited from the crowdsourcing website Amazon Mechanical Turk (AMT) and the undergraduate/graduate applicants in some schools. The test orders on items within a form were random separately for each test taker. The valid sample sizes for A2, B2, A3, and B3 forms were 443, 441, 552, and 538, respectively. The samples included about 71% and 47% test takers from AMT for the pairs and triplets forms, respectively, 55% male, 75% white, and 78% with Bachelor’s or above degrees.

We conducted a model comparison study on these four forms (Fu et al., 2023b). The models compared included, among other models, MUPP/Triplet-GGUMs with fixed item parameters (from the previous administrations of the Likert scales of the statements), MUPP/Triplet-2PLMs with fixed and directly estimated item parameters, and TIRT with directly estimated item parameters. The evaluation criteria included model selection, model fit, item fit, reliabilities, and correlations of trait score estimates. The MUPP/Triplet-2PLMs with directly estimated item parameters appeared to be the best-fitted models and were recommended for the pairs/triplets forms (see Fu et al., 2023b, for details).

This paper used the directly estimated item parameters from MUPP/Triplet-2PLMs to estimate item and test information on the four forms. Note that in the MUPP/Triplet-2PLM estimations, the intercept parameter of the first statement in an item was fixed to its intercept estimate from the 2PLM calibration of the Likert forms¹ for model identification, as discussed previously in Equations 2 and 3. For latent trait scores, the MAP estimates were reported for these forms. Because all FCQ forms are multidimensional, how to use item and test information to provide useful diagnostic information on items and a test is not so straightforward. We thus proposed a series of expected information indices at different levels and demonstrated their uses in real data.

Computer Program

All four forms were estimated by the MML-EM method in the mirt 1.34 package (Chalmers, 2012) in the 64-bit R 4.1.2 (R Core Team, 2021). For pair forms, the MUPP-2PLM was estimated by the internal implementation of the multidimensional 2PLM in mirt. The Triplet-2PLM estimation was implemented for triplet forms by utilizing the function CreatItem in mirt with some modifications of mirt’s functions (see Fu et al., 2023a for details).

iteminfo function in mirt calculates item information for IRT models; however, it has limitation for our purposes: iteminfo function computes the second derivatives although only the first derivatives are needed for calculating item information, and for Triplet-2PLM iteminfo uses the numeric approximation methods to estimate the derivatives. These features of iteminfo unnecessarily lengthen computational times. Therefore, we modified the iteminfo function to use the symbolic derivatives for Triplet-2PLM and calculate only the first derivatives to speed up computations. The saving of running times was significant in our case as we calculated item information for many items and many trait score vectors. As demonstrated below, we also wrote various R functions to calculate expected item/test information at different levels and create information plots. All the R functions related to item/test information are available in the supplemental online materials.

Expected Item/Test Information Indices and Plots

We now describe and demonstrate various expected item/test information indices and plots we propose. The following description focuses on triplets; however, it applies to pairs too. For a triplet item $j$, we use a grid of latent trait score points of three traits, each with 15 equally spaced points from −3 to 3 to represent the population latent trait score distribution; ${\theta }_{jd{k}_k{h}_k}$ refers to a latent score point $h_k$ (= 1,…,15) of trait $d_k$ measured by the $k^{\text{th}}$ statement in a triplet item $j$. The weight ${\pi }_{j{h}_1{h}_2{h}_3}$ of a latent trait score vector ${\omega }_{j{h}_1{h}_2{h}_3}=\left({\theta }_{j{d}_1{h}_1},{\theta }_{j{d}_2{h}_2},{\theta }_{j{d}_3{h}_3}\right)$ is the prior probability of ${\omega }_{j{h}_1{h}_2{h}_3}$ (usually the normalized density from a standard multivariate normal distribution with the intertrait correlation matrix, as used here). Similarly, the weight ${\pi }_{j{h}_k}$ of ${\theta }_{j{d}_k{h}_k}$ is the prior probability of ${\theta }_{j{d}_k{h}_k}$(usually the normalized density of a standard normal distribution, as used here). At each latent trait score vector, ${\omega }_{j{h}_1{h}_2{h}_3}$, we calculate the item information matrix for the three traits’ scores based on Equation 15. The item information for each trait score is on the trace of the information matrix, ${\beta }_j\left({\theta }_{j{d}_1{h}_1},{\theta }_{j{d}_2{h}_2},{\theta }_{j{d}_3{h}_3}\right)$. Observing item information on multiple dimensions is difficult, especially in graphs. Thus, we work on each trait’s expected item/test information at different levels to facilitate comparisons among items and tests.

Expected Item Information of a Trait Score.

We first calculate the expected item information on a trait conditional on each latent score point of the trait, for example, for ${\theta }_{j{d}_1{h}_1}$,

$${\beta }_j\left({\theta }_{jd{d}_1{h}_1}\right)={\displaystyle \sum _{h_2=1}^{15}{\displaystyle \sum _{h_3=1}^{15}\text{dia}{\text{g}}_{d_1}\left[{\beta }_j\left({\theta }_{j{d}_1{h}_1},{\theta }_{j{d}_2{h}_2},{\theta }_{j{d}_3{h}_3}\right)\right]{\pi }_{j{h}_1{h}_2{h}_3}/{\pi }_{j{h}_1}}}$$ (25)

where $\text{dia}{\text{g}}_{d_1}\left[{\beta }_j\left({\theta }_{j{d}_1{h}_1},{\theta }_{j{d}_2{h}_2},{\theta }_{j{d}_3{h}_3}\right)\right]$ is the first diagonal element in the item information matrix representing the item information of ${\theta }_{j{d}_1{h}_1}$ in the latent trait score vector ${\omega }_{j{h}_1{h}_2{h}_3}$.

Figure 1 includes the expected item information curve (IIC) for Item 1 in Form A3 as an example. This figure contains three expected individual trait IICs, which show the average amount of information on a trait at each score point of the trait that an item contributes to the test. This plot is useful to show the distribution of the expected item information on each trait that an item measures over the range of the trait’s scores. Trait 9, measured by the third statement, appears to have the largest item information across the latent trait score range, and the item information peaks in the trait’s score range between −1 and 0. This pattern is consistent with the discrimination parameter estimates associated with the three statements in the item: −0.55, −0.61, and −0.91 for the three statements, respectively. (The discrimination and intercept parameter estimates for each statement are shown at the top of the plot.) It is well known that a higher (absolute) discrimination parameter leads to more item information (Joo et al., 2018).

Figure 1.

Expected Item Information by Traits’ Scores for Item 1 in Form A3

Note. Trait 2-Statement A = The first statement in the item measures Trait 2; Trait 4-Statement B = The second statement in the item measures Trait 4; Trait 9-Statement C = The third statement in the item measures Trait 9; a = Discrimination estimate; b = Intercept estimate.

Expected Item Information on a Trait.

We next calculate the expected item information of each trait with respect to the prior distribution of the trait. For example, for $d_1$ trait in item $j$,

$${\beta }_{j{d}_1}={\displaystyle \sum _{h_1=1}^{15}{\beta }_j\left({\theta }_{j{d}_1{h}_1}\right){\pi }_{j{h}_1}.}$$ (26)

This index shows expected overall information (a scalar) on a trait that an item contains. It is very convenient to compare overall item information on a trait across items. For a trait, we can order and plot the expected item information of the trait across items measuring the trait to compare the trait’s expected item information across items. Figure 2 is an example of Trait 1 in Form A3. Item 36 stands out as the most informative item on Trait 1. In contrast, the first three items in the plot have expected item information on Trait 1 close to 0.

Figure 2.

Expected Item Information on Trait 1 by Items in Form A3

Expected Overall Item Information.

We can also calculate the expected overall item information of an item by averaging the expected overall item information on the three traits measured by the item,

$$O_j={\displaystyle \sum _{h_1=1}^{15}{\displaystyle \sum _{h_2=1}^{15}{\displaystyle \sum _{h_3=1}^{15}\text{mean\_tr}\left[{\beta }_j\left({\theta }_{j{d}_1{h}_1},{\theta }_{j{d}_2{h}_2},{\theta }_{j{d}_3{h}_3}\right)\right]{\pi }_{j{h}_1{h}_2{h}_3},}}}$$ (27)

where mean_tr refers to the average of a matrix’s trace (diagonal elements). This scalar index provides overall item information to facilitate comparison across items. This index is the same as the overall item information proposed by Joo et al. (2018). We can also order and plot this index across items in an FCQ form. Figure 3 is an example of Form A3 where a few items back to the x-axis (e.g., Items 36, 53, and 42) stand out as the most informative items. In contrast, the first two items on the x-axis (i.e., Items 52 and 9) have little information.

Figure 3.

Expected Overall Item Information by Items in Form A3

Expected Test Information of a Trait Score.

Having the expected item information of each latent score point of a trait (Equation 25), we can calculate the expected test information of each latent score point of the trait; for latent score point $h$ of trait $d$, ${\theta }_{dh}$, its expected test information is for ML $\text{θ}$

$$T_{dh}=\sum _{j\in J_d}{\beta }_j\left({\theta }_{dh}\right),$$ (28)

and for MAP $\text{θ}$

$$T_{dh}=\sum _{j\in J_d}{\beta }_j\left({\theta }_{dh}\right)+\text{dia}{\text{g}}_d\left({\Sigma }_{\theta }^{-1}\right),$$ (29)

where $J_d$ is the set of items measuring trait. For EAP $\text{θ}$, this index should be calculated by Equation 20 through all response patterns. Then, find the response patterns with EAP score estimates at or close to ${\theta }_{dh}$, and $T_{dh}$ is the weighted average of the test information of EAP ${\theta }_{dh}$ from these response patterns with normalized weights $P\left(x_r\mid \eta ,{\Sigma }_{\theta }\right)$. Because of the vast number of response patterns for a typical real test, it is generally impractical to calculate this index for EAP scores. We can draw the expected test information curves of all traits in a test form against their latent trait scores to compare the expected test information of individual traits. Figure 4 is an example of Form A3. Due to the 14 traits measured by the form, Figure 4 looks cumbersome; however, the distributional patterns of test information on each trait within and across traits are still salient. One can easily see the shape of the distribution for each trait (e.g., the peak and low points) and compare the distributions across traits; for example, some traits (e.g., Traits 17 and 13) had much more test information than other traits (e.g., Traits 4 and 11). We may also place the expected test information curves of a trait in different test forms and compare them. Figure 5 is an example of Trait 1: B2 and A3 forms appear to have more test information on Trait 1 than A2 and B3 forms across most of the score range.

Figure 4.

Expected Test Information on Individual Traits by Trait Scores in Form A3

Note. “T” is the abbreviation of Trait so that, for example, T1 refers to Trait 1, and so on.

Figure 5.

Comparison of Expected Test Information on Trait 1 by Trait Scores Across Four Test Forms (A2, B2, A3, and B3)

Expected Test Information on a Trait.

We can calculate the expected trait test information for each trait

$$T_d={\displaystyle \sum _{h=1}^{15}{T}_{dh}{\pi }_{jh}.}$$ (30)

This scalar index represents overall test information on a trait. It can be used to compare trait test information across traits in a test form or across test forms. During assembling the four test forms, this index based on MUPP/Triplet-GGUM was used to control the variance of trait test information within a form and across forms.

Expected Overall Test Information.

Within a test form, the expected test information on each trait can be averaged to obtain the expected overall test information of a form

$$T={\displaystyle \sum _{d=1}^D{T}_d/D.}$$ (31)

For EAP scores, the expected overall test information can also be obtained by

$$T=\text{mean\_tr}\left[{\displaystyle \sum _{r=1}^R\beta \left({\text{θ}}_r^{EAP}\right)P\left(x_r\mid \eta ,{\Sigma }_{\text{θ}}\right)}\right],$$ (32)

where $\beta \left({\theta }_r^{EAP}\right)$ is the test information matrix of the EAP score for response pattern $r$ (Equation 20). This scalar index on the overall test information of a test form was also proposed by Joo et al. (2018). It can be used to compare overall test information across test forms. This index was the statistic that the test assembly program tried to maximize in assembling the four forms.

We can order and plot the expected test information of all the traits measured in a form. Figure 6 shows the 14 traits in Form A3 as an example: Trait 11 had the lowest, and Trait 7 had the highest expected test information. The expected overall test information (5.61) appears as a reference line across the plot. We can also draw the expected test information of all the traits across test forms in one plot to facilitate across-form comparisons (see Figure 7). Figure 7 shows that although there are some variants in the expected test information on some traits within each type of form, the expected overall test information is basically the same between the two forms in each type. Note that the two forms in each type were designed to be parallel. The two triplets forms have slightly more test information than the two pairs forms (5.6 vs. 5.1).

Figure 6.

Comparison of Expected Test Information on Individual Traits in Form A3

Note. Overall TIF = Expected overall Test Information.

Figure 7.

Comparison of Expected Test Information on Individual Traits Across Four Test Forms (A2, B2, A3, and B3)

Note. Overall TIF = Expected overall Test Information.

Practical Recommendations

The item/test information indices and plots presented in the paper could be very useful in practice. We provide suggestions below for reviewing the plots from the top down to identify/remedy information issues in a test form for practitioners.

First, users should check Figure 7 if comparing multiple forms or Figure 6 if checking only one form. Identify the traits with inadequate expected test information that need remedy (trait level). For one form, users may start with expected overall item information by item in Figure 3 to identify the items with very low overall item information (item level) and then jump to the third step (statement level).

Second, users should check the plots of expected item information of individual traits by items like Figure 2. Identify the items with low information on a trait (item level).

Third, users should look at the plots of these items’ expected item information on individual traits by latent trait scores (Figure 1) to identify the problematic statements (statement level). Usually, the low item information issue is caused by statements’ low absolute discrimination parameters. Replacing the statements with ones having larger absolute discrimination parameters will resolve the issue.

There are no one-fit-all criteria regarding the threshold values on the various indices to judge whether an item/test contains satisfactory overall information or information on a particular trait. Rather, they depend on individual test programs’ objectives, for example, the ideal precision distribution of score estimates. Each test program will develop its standards on item and test information, among other requirements on classic item statistics, IRT item parameters, and test statistics (e.g., reliability). These criteria may be determined based on empirical data, simulation studies, or both. Then, these criteria can be implemented as constraints in an automatic test assembly program to meet these criteria. As demonstrated here, they can also be used in the post-administration review of items and test forms to adjust items accordingly.

Standard Error of Trait Score Estimates, Reliability, and Item Discrimination

As discussed previously, test information is directly related to the standard errors (SEs) of trait score estimates, and comparing expected SEs of a trait’s score estimates and reliabilities based on expected test information to those based on sample estimates of SEs can check the representativeness of a test sample. Figure 8 compares the expected SE estimates of MAP trait score estimates from the two approaches in Form A3, and Figure 9 compares the reliabilities. The gaps in the two statistics from the two approaches are consistent with small variations across traits: the (more theoretical) TIF approach is consistently better than the sample approach with smaller SEs and higher reliabilities. Figures 8 and 9 also compare the two statistics for ML trait score estimates, and the same pattern appears². Therefore, the test sample for this triplets form does not seem to be adequate. Note that MAP test information is ML test information plus the prior information (Equation 19). Thus, the SE estimates of ML score estimates based on TIF and the test sample are much higher than those of MAP score estimates. In fact, in the expected MAP test information on individual traits in this triplets form, the prior information contributes 29% to 60%, varied by traits. At the same time, based on Equation 6, it is well known that MAP score estimates shrink to their population mean compared to ML estimates. Thus, the variance of MAP estimates is smaller than that of ML estimates (average 0.55 vs. 1.90 for the test sample on this triplets form). Then, based on Equations 21 and 23, the reliability of MAP estimates is not necessarily higher than that of ML estimates, which is shown in Figure 9 — the reliabilities for both types of trait score estimates are quite similar, and for some traits, the reliabilities for ML estimates are higher than those for MAP estimates. However, as it is well known, MLE produces much more inestimable scores on extreme/aberrant responses than MAP and EAP estimations. This issue of MLE seems more severe on high dimensional FCQs: for example, in this triplets form, 52 test takers’ ML score estimations are not convergent versus none for MAP estimations. Thus, MAP scores are used for reporting in these FCQ forms.

Figure 8.

Comparison of Expected Standard Errors (SEs) of Trait Score Estimates in Form A3

Note. MAP_TIF = Square root of the reciprocal of the expected test information on a trait for MAP scores; MAP_Sample = Mean of the SE estimates of the sample MAP trait score estimates; ML_TIF = Squared root of the reciprocal of the expected test information on a trait for ML scores; ML_Sample = Mean of the SE estimates of the sample ML trait score estimates.

Figure 9.

Comparison of Reliabilities of Trait Score Estimates in Form A3

Note. MAP_TIF = Reliability based on TIF of MAP trait scores; MAP_Sample = Reliability based on the sample mean of the standard error estimates of the MAP trait score estimates; ML_TIF = Reliability based on TIFs of ML trait scores; ML_Sample = Reliability based on the sample mean of the standard error estimates of the ML trait score estimates.

A trait’s item information is closely related to the absolute discrimination parameter of the statement measuring the trait. Table 1 shows the correlations between absolute discrimination parameters and expected item information on trait for the four forms, respectively. All correlations are 0.97 or 0.98, indicating larger absolute discrimination will almost certainly lead to high item information. This finding is consistent with the simulation result in Joo et al. (2018). Table 1 also lists the mean and SD of absolute discrimination parameters and expected overall ML test information for each form (as ML TIF directly relates to discrimination parameters). One can see that within pairs/triplets form, the two forms are almost identical in these three statistics, which supports the parallelism of the two forms. The triplets forms have slightly higher means on absolute discrimination parameters and expected overall ML test information than the pairs forms. Note that a pairs form includes 80 more statements than a triplets form. Thus, a triplets form can measure traits at a level of precision with fewer statements than a pairs form. This finding is also consistent with the simulation result in Joo et al. (2018).

Table 1.

Discrimination Parameters and Test Information

Form	Correlation	Absolute discrimination parameter		ML test information
		M	SD
A2	0.97	0.78	0.49	2.77
B2	0.97	0.78	0.49	2.78
A3	0.97	0.85	0.39	3.29
B3	0.98	0.83	0.46	3.29

Note. Correlation = Correlation between absolute discrimination parameters and expected item information on a trait; ML test information = Expected overall test information for ML trait scores.

Discussion

In this paper, we provided deep and comprehensive discussions on the information functions of Rank-2PLMs in particular and IRT models in general from theoretical and practical perspectives. This paper has valuable contributions to the IRT literature, especially the IRT models for FCQs, as listed in the Introduction section. Below, we discussed the current study’s limitations and suggested future studies.

Limitations and Future Research

This study did not look deeply into how information functions behave with respect to item parameters in MUPP-2PLM and Triplet-2PLM. A future study may investigate it analytically or use simulations/examples, as done in Muraki (1993) and Joo et al. (2018) for their models.

The TIRT model is closely related to Rank-2PLMs. For pairs, it is easy to show that TIRT is equivalent to MUPP-2PLM (as the equivalence between the normal ogive and logistic forms of an IRT model)³. For a block with three or more statements, the two models model the ranking patterns of a block differently. The TIRT model recodes ranking patterns into binary response patterns of pairs. For example, a triplet with statements A, B, and C can form into three pairs: AB, AC, and BC. Currently, in the item estimation, the dependence among pairs is considered; however, in the score estimation, the local independence among pairs is assumed to simplify the score estimation (to avoid multidimensional integrals), which may have a negative impact on the score estimation. Triplet-2PLM avoids this problem by modeling a triplet directly. Thus, it is an interesting future study to compare Triplet-2PLM and TIRT estimations, including their information functions.