Scale Linking for the Testlet Item Response Theory Model

Abstract

In their 2005 paper, Li and her colleagues proposed a test response function (TRF) linking method for a two-parameter testlet model and used a genetic algorithm to find minimization solutions for the linking coefficients. In the present paper the linking task for a three-parameter testlet model is formulated from the perspective of bi-factor modeling, and three linking methods for the model are presented: the TRF, mean/least squares (MLS), and item response function (IRF) methods. Simulations are conducted to compare the TRF method using a genetic algorithm with the TRF and IRF methods using a quasi-Newton algorithm and the MLS method. The results indicate that the IRF, MLS, and TRF methods perform very well, well, and poorly, respectively, in estimating the linking coefficients associated with testlet effects, that the use of genetic algorithms offers little improvement to the TRF method, and that the minimization function for the TRF method is not as well-structured as that for the IRF method.

Educational test forms are often constructed using clusters of items based on a common stimulus or content area. For example, test items may be grouped around a reading passage, scenario, chart, or section associated with particular content. Wainer and Kiely (1987) called such a group of items a testlet and adopted it as a construction unit in computerized adaptive testing. From the perspective of item response theory (IRT; Lord, 1980; Yen & Fitzpatrick, 2006), the assumption of local independence among items nested within a testlet, given the primary latent trait, would be violated to some extent because the responses of examinees to the items might be affected by the testlet effect as well as the primary factor. An efficient way to deal with local dependence is to use the testlet model (Wainer et al., 2007), in which a secondary, random-effect factor is added to the primary factor. Researchers (e.g., DeMars, 2006; Li et al., 2006; Rijmen, 2010) have shown that the testlet model is a constrained version of the bi-factor model (Gibbons & Hedeker, 1992).

Like other IRT models, the testlet model has a model identification problem, specifically a scale indeterminacy problem, because the item parameters and person parameters are invariant within a linear transformation of the latent trait scale. In practice, scale indeterminacy typically is solved by choosing a scale such that the mean and standard deviation (SD) of the person parameters are arbitrarily set to certain values (e.g., 0 and 1) for the examinee group being analyzed (Rijmen, 2010). According to that convention, the latent scales obtained from separate calibrations of sample data from different populations are not likely to be equivalent, but they are assumed to be linearly related. This non-equivalency creates the need for a common scale, which can be developed through scale linking (or scale transformation), in which one scale is linked to another (base) scale with a linear function.

This paper is primarily concerned with the methods used to estimate the linking parameters for the testlet model under the common-item nonequivalent groups (CING) design (Kolen & Brennan, 2014). Many linking methods have been presented for use with traditional dichotomous IRT models such as the two-parameter logistic and three-parameter logistic (3PL) models (e.g., Divgi, 1985; Haebara, 1980; Loyd & Hoover, 1980; Marco, 1977; Stocking & Lord, 1983), and they have been extended to polytomous models (Kim & Lee, 2006). Most relevant to the present paper, Kim (2019) presented three linking methods for the 3PL bi-factor model, the direct least squares (DLS), item response function (IRF), and test response function (TRF) methods, which are bi-factor extensions of Divgi’s (1985), Haebara’s (1980), and Stocking and Lord’s (1983) approaches, respectively. Kim (2019) showed through simulations that the IRF, DLS, and TRF methods differed little in estimating the slope (dilation) linking coefficients, but they exhibited substantial differences in estimating the intercept (translation) linking coefficients, with the IRF method being the most accurate and the TRF method being the least accurate. However, in the IRT literature, only the TRF method has been formally extended for use with the testlet model. That extension is found in Li et al. (2005), who presented the TRF method for a two-parameter normal ogive (2PNO) testlet model. In this paper, Li et al.’s TRF method is presented under the 3PL testlet model since this general model is more widely used than the 2PNO testlet model in practice.

Questions and Purposes

As described in detail later, Li et al. (2005) formulated the linking task under the 2PNO testlet model such that, given $k$ common testlets, the linking parameters should include the means (denoted by ${\mu }_{{\gamma }_s}$ ) of the testlet effect factors ${\gamma }_s$ , $s=$ 1,…, $k$ , with the constraint ${\displaystyle \sum _{s=1}^k{\mu }_{{\gamma }_s}=0}$ , in addition to the linking coefficients $A$ and $B$ for the primary factor $\theta$ . The criterion function (also known as the loss function) for the TRF method is nonlinear with respect to the linking parameters, and thus a multivariate search technique such as the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm, one of the quasi-Newton methods (Dennis & Schnabel, 1996), should be implemented to estimate the linking parameters. Li et al. (2005) combined the GENOUD genetic algorithm (Sekhon & Mebane, 1998; see also Mebane & Sekhon, 2011) with the BFGS algorithm. Li and her colleagues used the genetic algorithm because they were concerned that if there were three or more linking parameters to be estimated, the criterion function might have multiple minimum or saddle points and the BFGS method might be unable to find or fail to converge to the global minimum. However, the GENOUD genetic algorithm is very computationally intensive and time-consuming (taking 25 or more minutes for a linking task, as reported by Li et al.).

The present study was motivated by some related questions regarding Li et al.’s (2005) approach to the linking solutions for the TRF method. The first question is “Is it necessary to use genetic algorithms to find the linking solutions for the TRF method?” This question is important, because previous studies into multidimensional IRT linking (e.g., Davey et al., 1996; Oshima et al., 2000) that considered six or more linking parameters in a rotation matrix and translation vector have not reported any problem in finding the linking solutions using a modified version of the Newton method. If the genetic method is not substantially superior to the BFGS method, there would be no compelling reason to use it in practice. The second and third questions, which are closely related to each other, are “Are linking methods for the testlet model other than the TRF method available?” and “How do the different linking methods for the testlet model compare in their performance?” These questions are also important, because the availability of different methods for scale linking allows practitioners to choose the appropriate method depending on the situation. The choice of a linking method can be made more wisely if more information about the relative performance of different methods is given to the practitioners. Even if one method is operationally used, other methods should still be implemented for diagnostic purposes (Kolen & Brennan, 2014).

The primary purposes of this paper are two-fold. One is to answer the first question posed above regarding Li et al.’s (2005) TRF method. The other is to present the mean/least squares (MLS) and IRF methods for the testlet model and investigate their performance in linking accuracy relative to the TRF method. To achieve these purposes, we first present the 3PL testlet model (instead of the 2PNO model) in the next section for generality and reformulate the linking task formulated by Li et al. (2005) into a special case under the bi-factor modeling. Next we use the reformulated linking framework to present the MLS, TRF, and IRF methods and conduct a simulation study to compare the accuracy of these methods.

Linking Methods for the 3PL Testlet Model

The IRT literature contains several versions of the testlet model for dichotomous items that differ slightly in parameterization (e.g., Bradlow et al., 1999; Glas et al., 2000; Wainer & Wang, 2000). For the purposes of this paper, we use the parameterization of Glas et al. (2000) to write a 3PL testlet model that defines the probability that an examinee $j$ will answer item $i$ correctly as

$$P_i\left({\theta }_j,{\gamma }_{js\left(i\right)}\right)=P\left({\theta }_j,{\gamma }_{js\left(i\right)};a_i,b_i,c_i\right)=c_i+\frac{1-c_i}{1+\exp \left[-D{a}_i\left({\theta }_j+{\gamma }_{js\left(i\right)}-b_i\right)\right]},$$ (1)

where $a_i$ , $b_i$ , and $c_i$ are the discrimination, difficulty, and lower asymptote parameters for item $i$ , respectively, $D$ is a scaling constant (usually set to 1 or 1.7); ${\theta }_j$ is the primary trait (ability) parameter of examinee $j$ ; and ${\gamma }_{js\left(i\right)}$ is a random-effect parameter (assumed to be independent of ${\theta }_j$ ) for examinee $j$ of testlet $s$ , the testlet to which item $i$ belongs. Equation (1), the IRF for the 3PL testlet model, can be viewed as a special case of the 3PL bi-factor model, written as

$$P_i\left({\theta }_{j0},{\theta }_{js}\right)=P\left({\theta }_{j0},{\theta }_{js};a_i,C_{s\left(i\right)},d_i,c_i\right)=c_i+\frac{1-c_i}{1+\exp \left[-D\left(a_i{\theta }_{j0}+a_i{C}_{s\left(i\right)}{\theta }_{js}+d_i\right)\right]},$$ (2)

where $a_i$ , $c_i$ , and $D$ are the same as in Equation (1); $d_i=-a_i{b}_i$ is the intercept parameter; ${\theta }_{j0}$ is the parameter for examinee $j$ of the primary factor (i.e., ${\theta }_0$ = $\theta$ ); ${\theta }_{js}$ is the parameter of the specific factor of testlet $s$ with the relationship ${\gamma }_{s\left(i\right)}$ = $C_{s\left(i\right)}{\theta }_s$ ; and $C_{s\left(i\right)}$ is a proportionality constant across all items nested within testlet $s$ .

Whether expressed as Equations (1) or (2), the 3PL testlet model cannot be identified unless some restrictions are imposed on the parameters. For Equation (1), the mean and SD of $\theta$ are typically fixed to 0 and 1, respectively, and the means of the ${\gamma }_{s\left(i\right)}$ s are fixed to 0, with each of their SDs, ${\sigma }_{{\gamma }_s}$ , being free parameters. For Equation (2), the mean and SD of ${\theta }_0$ and ${\theta }_s$ are fixed to 0 and 1, respectively, and $C_{s\left(i\right)}$ = ${\sigma }_{{\gamma }_s}$ are considered the free parameters to be estimated. In other words, a standardized scale (0–1 scale) is independently used for each $\theta$ dimension to remove model indeterminacy. Throughout this paper, we assume that the 3PL testlet model is identified by 0–1 scaling, and its parameters are estimated using sample data.

The Linking Parameters Estimated

Consider two examinee groups, a base group and a new group, that can differ in each $\theta$ dimension. Assume that an identical test, consisting of $k$ testlets, has been administered to both groups and that for each group, separate calibration has been conducted using 0–1 scaling to estimate all item parameters, including $C_s$ (dropping the nested subscript $i$ for simplicity). Furthermore, define the 0–1 scales from the base and new groups as ${\mathbf{\theta }}_B$ and ${\mathbf{\theta }}_N$ , respectively, where ${\mathbf{\theta }}_B=$ ( ${\theta }_{0B}$ , ${\theta }_{1B}$ ,…, ${\theta }_{kB}$ ) and ${\mathbf{\theta }}_N=$ ( ${\theta }_{0N}$ , ${\theta }_{1N}$ ,…, ${\theta }_{kN}$ ). Use $a_{iB}$ , $b_{iB}$ , $c_{iB}$ , $d_{iB}$ , and $C_{sB}$ to denote the item/testlet parameters estimated on the ${\mathbf{\theta }}_B$ scale, and use $a_{iN}$ , $b_{iN}$ , $c_{iN}$ , $d_{iN}$ , and $C_{sN}$ to denote the counterparts on the ${\mathbf{\theta }}_N$ scale.

By the “within a linear transformation” invariance property of IRT, the ${\mathbf{\theta }}_B$ and ${\mathbf{\theta }}_N$ scales are linearly related as follows (Kim, 2019):

$${\theta }_{0B}=A{\theta }_{0N}+B,$$ (3)

$${\theta }_{sB}={\lambda }_s{\theta }_{sN}+{\beta }_s,s=1,\dots ,k,$$ (4)

where $A$ and $B$ are the linking coefficients for the ${\theta }_0$ dimension and ${\lambda }_s$ and ${\beta }_s$ are the linking coefficients for the ${\theta }_s$ dimension. The slopes, $A$ and ${\lambda }_s$ , adjust for unit differences between the new and base scales, and the translation intercepts, $B$ and ${\beta }_s$ , adjust for location differences. If scale linking is perfect, the two sets of item/testlet parameter estimates from separate calibrations should be related as follows:

$$a_{iB}=a_{iN}/A,$$ (5)

$$\frac{C_{sB}}{A}=\frac{C_{sN}}{{\lambda }_s},$$ (6)

$$d_{iB}=d_{iN}-a_{iN}\left(\frac{1}{A}\right)B-a_{iN}\left(\frac{C_{sB}}{A}\right){\beta }_s$$ (7a)

$$=d_{iN}-\frac{1}{A}\left[\begin{array}{cc}{a}_{iN}& a_{iN}\end{array}\right]\left[\begin{array}{cc}1& 0\\ 0& C_{sB}\end{array}\right]\left[\begin{array}{c}B\\ {\beta }_s\end{array}\right],$$ (7b)

$$b_{iB}=A\left[b_{iN}+\left(\frac{C_{sB}}{A}\right){\beta }_s\right]+B,$$ (8)

$$c_{iB}=c_{iN}.$$ (9)

However, Equations (5) through (9) do not perfectly hold among estimated item/testlet parameters because of sampling errors and possible model-data misfit. In general, linking errors are unavoidable with sample data, and the linking coefficients should be properly estimated so as to minimize the errors (Kim & Lee, 2006; Kolen & Brennan, 2014).

The descriptions above might be read as if the linking methods for the 3PL testlet model should estimate $2(k+1)$ linking coefficients ( $A$ , ${\lambda }_1$ to ${\lambda }_k$ and $B$ , ${\beta }_1$ to ${\beta }_k$ ), but that is not the case. As can be seen from Equation (6), each ${\lambda }_s$ ( $s=$ 1,…, $k$ ) is a function of $A$ , $C_{sB}$ , and $C_{sN}$ , so if $A$ is estimated and the two constants are given, its value is determined. Thus, $k$ lambda coefficients are not considered as linking parameters to be estimated. For the beta coefficients, ${\beta }_s$ , $k-1$ can be uniquely estimated due to the linear dependence among them, such as ${\displaystyle \sum _{s=1}^k{\beta }_s=0}$ . Note that the linear dependence ${\displaystyle \sum {\beta }_s}=0$ agrees with the constraint ${\displaystyle \sum {\mu }_{{\gamma }_s}}=0$ used in Li et al. (2005), where ${\mu }_{{\gamma }_s}=C_{sB}{\beta }_s/A$ given $A$ . Therefore, the three linking methods for the testlet model described below estimate $k+1$ “free” parameters, the $A$ , $B$ , and $k-1$ ${\beta }_s$ coefficients. Because the meaning of the linear dependence among the ${\beta }_s$ values can be clearly revealed in presenting the MLS method, and the TRF and IRF methods are akin to each other, we present the MLS method first and then present the two response function methods. For the following presentation, it is assumed that a common testlet $s$ contains $n_s$ items and the total number of items in the $k$ common testlets for linking is $n$ = ${\displaystyle \sum n_s}$ .

MLS Method

The MLS method presented here is a hybrid in that it uses part of the mean/mean method (Loyd & Hoover, 1980) to estimate the slope $A$ and then uses the linear least squares approach to estimate the intercepts, $B$ and ${\beta }_s$ . Unlike the TRF and IRF methods, the MLS method can estimate the linking coefficients without an iterative search for the solutions.

Taking the mean over a-parameters based on Equation (5) and solving the resulting equation for $A$ leads to a legitimate statistical solution for $A$ (Loyd & Hoover, 1980):

$$A=\frac{\mathrm{Mean}\left({\mathbf{a}}_N\right)}{\mathrm{Mean}\left({\mathbf{a}}_B\right)}=\frac{{\displaystyle \sum _i{a}_{iN}}/n}{{\displaystyle \sum _i{a}_{iB}}/n},$$ (10)

where ${\mathbf{a}}_N$ and ${\mathbf{a}}_B$ represent all the discrimination parameters estimated on the new and base scales, respectively. Once the $A$ coefficient is estimated by Equation (10), the $B$ and beta coefficients need to be simultaneously estimated because as seen from by Equation (7) or (8) they are related to each other in an equation.

Based on Equation (7a), let $d_{iN}^{\ast }=d_{iN}-a_{iN}B/A-a_{iN}{C}_{sB}{\beta }_s/A$ and $e_i=d_{iN}^{\ast }-d_{iB}$ . According to the statistical approaches used in Divgi (1985) and Oshima et al. (2000), the $B$ and beta coefficients can be estimated as the values that minimize the sum of squared differences ( $e_i^2$ ) between $d_{iN}^{\ast }$ and $d_{iB}$ for all $i$ . To obtain the solutions for the intercept coefficients using the least squares method, we first write an error model, based on Equation (7b), as

$${\mathbf{d}}_N-{\mathbf{d}}_B=\frac{1}{A}\mathbf{P}\mathbf{D}\mathbf{\beta }+\mathbf{e},$$ (11)

where ${\mathbf{d}}_N$ = $(d_{1N},\mathrm{...},d_{nN}{)}^{\prime }$ and ${\mathbf{d}}_B$ = $(d_{1B},\mathrm{...},d_{nB}{)}^{\prime }$ are $n\times 1$ vectors of d-parameter estimates; $\mathbf{e}$ is an $n\times 1$ error vector; $\mathbf{P}$ is an $n\times (k+1)$ matrix whose row elements are “factor loadings,” $a_{iN}$ s, associated with the $(k+1)$ -dimensional space $\mathbf{\theta }=$ ( ${\theta }_0$ , ${\theta }_1$ ,…, ${\theta }_k$ ); $\mathbf{D}$ is a diagonal matrix whose $k+1$ diagonal elements are 1, $C_{1B}$ ,…, $C_{kB}$ ; and $\mathbf{\beta }$ =( $B,{\beta }_1,\mathrm{...},{\beta }_k{)}^{\prime }$ is a vector of size $k+1$ . For instance, if there are two testlets and two items within each testlet, $\mathbf{P}$ , $\mathbf{D}$ , and $\mathbf{\beta }$ are expressed as

$$\mathbf{P}=\left[\begin{array}{c}{a}_{1N}\\ a_{2N}\\ a_{3N}\\ a_{4N}\end{array}\begin{array}{c}{a}_{1N}\\ a_{2N}\\ 0\\ 0\end{array}\begin{array}{c}0\\ 0\\ a_{3N}\\ a_{4N}\end{array}\right],\mathbf{D}=\left[\begin{array}{ccc}1& 0& 0\\ 0& C_{1B}& 0\\ 0& 0& C_{2B}\end{array}\right],\text{and}\mathbf{\beta }=\left[\begin{array}{c}B\\ {\beta }_1\\ {\beta }_2\end{array}\right].$$ (12)

Although the error model in Equation (11) resembles a regression model, where the dependent variable is ${\mathbf{d}}_N-{\mathbf{d}}_B$ and the coefficient vector is $\mathbf{\beta }$ , the solutions of $\mathbf{\beta }$ cannot be computed using the ordinary least squares approach because the factor pattern matrix $\mathbf{P}$ is not of full column rank (as shown by the example matrix in Equation (12)). With the condition $n\ge k+1$ , usually met in practice, the rank of $\mathbf{P}$ is $k$ , not $k+1$ . Such rank deficiency implies that except for the $B$ coefficient, the $k$ ${\beta }_s$ coefficients in $\mathbf{\beta }$ are linearly dependent, and only $k-1$ ones need to be estimated. Although the linear dependence among ${\beta }_s$ can be formulated in many ways, we here choose the constraint ${\displaystyle \sum {\beta }_s=0}$ , corresponding to the constraint ${\displaystyle \sum {\mu }_{{\gamma }_s}}=0$ used in Li et al. (2005). In addition, we introduce a transformation matrix $\mathbf{T}$ , which relates $\mathbf{\beta }$ to an estimation vector $\mathbf{\alpha }$ = $(B,{\alpha }_1,\mathrm{...},{\alpha }_{k-1}{)}^{\prime }$ such that

$$\mathbf{\beta }=\mathbf{T}\mathbf{\alpha },$$ (13)

where $\mathbf{T}=\left[\begin{array}{cc}1& 0\\ 0& {\mathbf{T}}_D\end{array}\right]$ is a $(k+1)\times k$ matrix, and ${\mathbf{T}}_D=\left[\begin{array}{cccc}1/k& 1/k& \mathrm{...}& 1/k\\ (1-k)/k& 1/k& \mathrm{...}& 1/k\\ 1/k& (1-k)/k& \mathrm{...}& 1/k\\ .& .& & .\\ .& .& & .\\ .& .& & .\\ 1/k& 1/k& \mathrm{...}& (1-k)/k\end{array}\right]$ . Now the free coefficients in $\mathbf{\alpha }$ can be estimated using the least squares method, and the solution formula can be derived as

$$\mathbf{\alpha }=A{\left[(\mathbf{P}\mathbf{D}\mathbf{T}{)}^{\prime }(\mathbf{P}\mathbf{D}\mathbf{T})\right]}^{-1}(\mathbf{P}\mathbf{D}\mathbf{T}{)}^{\prime }({\mathbf{d}}_N-{\mathbf{d}}_B).$$ (14)

Finally, the MLS solutions for the $B$ and ${\beta }_s$ coefficients in $\mathbf{\beta }$ are obtained by plugging the resulting $\mathbf{\alpha }$ into Equation (13). Note that the ${\beta }_s$ estimates surely satisfy the zero-sum constraint due to the use of the $\mathbf{T}$ matrix.

TRF Method

For the traditional 3PL model, the TRF at a given $\theta$ is defined as the sum of IRFs over all the items on the test, written as $T\left(\theta \right)={\displaystyle \sum P_i\left(\theta \right)}$ . $T\left(\theta \right)$ is the true score for an examinee with ability $\theta$ . Conceptually, the analog of $T\left(\theta \right)$ for the 3PL testlet model can be defined as the sum of the marginalized IRFs, each of which is computed by integrating the nuisance dimension ${\gamma }_s$ (or ${\theta }_s$ ) out from the IRF in Equation (1) (or 2). In accordance with this conception, let $P_i\left({\theta }_B\right)$ and $T\left({\theta }_B\right)$ denote the marginalized IRF and TRF computed with the item/testlet parameters estimated on the ${\theta }_B$ scale, respectively, and let $P_i^{\ast }\left({\theta }_B\right)$ and $T^{\ast }\left({\theta }_B\right)$ denote the marginalized IRF and TRF computed with the parameter estimates transformed to the ${\theta }_B$ scale. The TRF method finds the solutions of $A$ and $\mathbf{\beta }={\left(B,{\beta }_1,\mathrm{...},{\beta }_k\right)}^{\prime }$ that minimize the criterion function, $f_T\left(A,\mathbf{\beta }\right)$ ,

$$f_T\left(A,\mathbf{\beta }\right)=\frac{1}{N}{\displaystyle \sum _{q=1}^N{\left[T\left({\theta }_{qB}\right)-T^{\ast }\left({\theta }_{qB}\right)\right]}^2}=\frac{1}{N}{\displaystyle \sum _{q=1}^N{\left[{\displaystyle \sum _{i=1}^n{P}_i\left({\theta }_{qB}\right)}-{\displaystyle \sum _{i=1}^n{P}_i^{\ast }\left({\theta }_{qB}\right)}\right]}^2},$$ (15)

where $q=$ 1, 2,…, $N$ indexes $N$ arbitrary points over the ${\theta }_B$ scale.

Although the marginal IRF and TRF can be straightforwardly computed for each pair of $\theta$ and ${\gamma }_s$ , Li et al. (2005) used a new composite variable ${\xi }_s=\theta +{\gamma }_s$ for linking purposes. They noted that, assuming $\theta$ and ${\gamma }_s$ have independent normal distributions with zero means and variances equal to 1 and ${\sigma }_{{\gamma }_s}^2$ , respectively, ${\xi }_s$ given $\theta$ is distributed as $N(\theta ,{\sigma }_{{\gamma }_s}^2)$ . With ${\xi }_s$ , more explicitly ${\xi }_{s\left(i\right)}$ , the 3PL testlet model in Equation (1) can be expressed as

$$P_i\left({\xi }_s\right)=P\left({\xi }_s;a_i,b_i,c_i\right)=c_i+\frac{1-c_i}{1+\exp \left[-D{a}_i\left({\xi }_s-b_i\right)\right]}.$$ (16)

Then, the probability of answering item $i$ within testlet $s$ correctly, conditional on $\theta$ and ${\sigma }_{{\xi }_s}(={\sigma }_{{\gamma }_s})$ , that is, the marginalized $P_i\left(\theta \right)$ , is expressed as

$$P_i\left(\theta \right)\equiv P_i\left(\theta ;{\sigma }_{{\xi }_s}\right)={\displaystyle \int P_i\left({\xi }_s\right)h\left({\xi }_s|\theta ;{\sigma }_{{\xi }_s}\right)d{\xi }_s},$$ (17)

where $h$ is the probability density function of ${\xi }_s$ given $\theta$ . The integral in Equation (17) can be approximated to any desired degree of accuracy by using Gauss–Hermite quadrature.

Let ${\xi }_{sB}$ and ${\xi }_{sN}$ denote the ${\xi }_s$ scales defined with the base and new groups, respectively. If the two scales are related as ${\xi }_{sB}$ = $A{\xi }_{sN}+B$ , the item parameters on the ${\xi }_{sN}$ scale can be transformed into those on the ${\xi }_{sB}$ scale as follows (Li et al., 2005):

$$a_{iN}^{\ast }=a_{iN}/A,$$ (18)

$$b_{iN}^{\ast }=A{b}_{iN}+B.$$ (19)

Although both transformations are legitimate in a technical sense, the transformation of $b_{iN}$ by Equation (19) is insufficient for linking purposes because ${\xi }_{sB}$ = $A{\xi }_{sN}+B$ takes into account possible mean and SD differences in $\theta$ between the base and new groups but not possible mean differences in ${\gamma }_s$ between the two groups. Li et al. (2005) pointed out that if separate calibration results were obtained using the model in Equation (16), possible differences in the mean of ${\gamma }_s$ between the base and new groups would be absorbed into $b_{iN}$ , which would lead to a shift in $b_{iN}$ . Therefore, they used the following transformation to account for that possible shift

$$b_{iN}^{\ast }=A\left(b_{iN}+{\mu }_{{\gamma }_s}\right)+B.$$ (20)

They further indicated that the zero-sum constraint ${\displaystyle \sum {\mu }_{{\gamma }_s}}=0$ ( $s=1,\mathrm{...},k$ ) should be imposed for model identification, although they did not detail why the model needed that constraint. Note that based on Equation (8), the $b_{iN}^{\ast }$ in Equation (20) can also be written as

$$b_{iN}^{\ast }=A\left[b_{iN}+\left(\frac{C_{sB}}{A}\right){\beta }_s\right]+B,$$ (21)

where $C_{sB}{\beta }_s/A={\mu }_{{\gamma }_s}$ . Of course, the constraint ${\displaystyle \sum {\beta }_s}=0$ is necessary for the reason revealed when the MLS method was addressed above.

The criterion function $f_T(A,\mathbf{\beta })$ in Equation (15), defined with the two sets, { $a_{iB},b_{iB},c_{iB},C_{sB}$ } and { $a_{iN}^{\ast },b_{iN}^{\ast },c_{iN}^{\ast }$ }, for the $n$ common items associated with $k$ testlets, is nonlinear with respect to the linking coefficients, $A$ , $B$ , and ${\beta }_s$ , where the zero-sum constraint can be dealt with in practice by setting ${\beta }_k=-{\displaystyle {\sum }_{s=1}^{k-1}{\beta }_s}$ . Thus a multivariate search technique is required to find the linking solutions for the TRF method. Previous linking studies (e.g., Kim & Lee, 2006; Oshima et al., 2000) suggest that the minimization solutions can be obtained by using a modified Newton or quasi-Newton approach such as the BFGS method. However, Li et al. (2005) combined the GENOUD algorithm (Sekhon & Mebane, 1998) with the BFGS method to ensure that the global, not the local, minimum solutions are obtained. All of the search techniques are based on the vector of partial derivatives (i.e., gradient) of the criterion function with respect to the parameters. The analytic formulas for the gradient of $f_T(A,\mathbf{\beta })$ with respect to the linking coefficients are presented in the Appendix.

IRF Method

Given the marginalized IRFs $P_i\left({\theta }_B\right)$ and $P_i^{\ast }\left({\theta }_B\right)$ for all common items, the IRF linking method (Haebara, 1980) for the traditional 3PL model can be straightforwardly extended to the 3PL testlet model. Similarly to the TRF method, the IRF method finds the solutions of $A$ and $\mathbf{\beta }={\left(B,{\beta }_1,\mathrm{...},{\beta }_k\right)}^{\prime }$ that minimize the criterion function, $f_I\left(A,\mathbf{\beta }\right)$ ,

$$f_I\left(A,\mathbf{\beta }\right)=\frac{1}{Nn}{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{i=1}^n{\left[P_i\left({\theta }_{qB}\right)-P_i^{\ast }\left({\theta }_{qB}\right)\right]}^2}},$$ (22)

where, as denoted earlier, $n$ is the number of common items and $q=$ 1, 2,…, $N$ indexes $N$ arbitrary points over the ${\theta }_B$ scale. Although the marginalized IRFs $P_i\left(\theta \right)$ or $P_i\left({\theta }_0\right)$ , as generally denoted, can be evaluated by Equation (17), they can also be computed using the bi-factor model in Equation (2) as follows:

$$P_i\left(\theta \right)\equiv P_i\left({\theta }_0\right)={\displaystyle \int P_i\left({\theta }_0,{\theta }_s\right)g\left({\theta }_s\right)d{\theta }_s},$$ (23)

where $g$ is the probability density function of ${\theta }_s$ . Of course, in that case, the $P_i\left({\theta }_{qB}\right)$ and $P_i^{\ast }\left({\theta }_{qB}\right)$ in Equation (22) are the probabilities evaluated at ${\theta }_0={\theta }_{qB}$ with the parameter sets { $a_{iB}$ , $c_{iB}$ , $d_{iB}$ , $C_{sB}$ } and { $a_{iN}^{\ast }$ , $c_{iN}^{\ast }$ , $d_{iN}^{\ast }$ }, respectively.

The criterion function $f_I\left(A,\mathbf{\beta }\right)$ is nonlinear, as is $f_T\left(A,\mathbf{\beta }\right)$ , with respect to the linking coefficients, and thus a search technique is required to find the linking solutions. In this paper, we use the BFGS algorithm to find the linking solutions for the IRF method. The analytic formulas for the gradient of $f_I\left(A,\mathbf{\beta }\right)$ are presented in the Appendix.

Simulation Study

A simulation study was conducted to compare the performance of the TRF, MLS, and IRF methods. Two versions of the TRF method were conducted: one using the GENOUD algorithm and the other using the BFGS algorithm. The IRF method was implemented using only the BFGS algorithm. The design and methodology of this simulation study were closely matched to those used by Li et al. (2005) so that the comparison might be made under nearly the same conditions as those used in the previous study.

Design and Data

The CING design was used to evaluate the linking parameter recovery of the four methods for the 3PL testlet model: (a) the GENOUD-TRF method, (b) the BFGS-TRF method, (c) the MLS method, and (d) the IRF method based on the BFGS algorithm. As in Li et al. (2005), simulated tests and data sets were generated using different sets of item/testlet parameters and linking parameters. Each simulated test form consisted of six testlets, each of which contained 5 items, giving 30 items in total. The number $k$ of common testlets between the two (“base” and “new”) test forms to be linked was considered as the simulation factor. Two levels of $k$ were used: $k$ =2 and $k$ =4, resulting in the two common testlets condition (Condition 1) and the four common testlets condition (Condition 2), respectively.

For each simulation condition, 10 pairs of simulated tests with 5000 examinees per form were generated, as in Li et al. (2005). For each test form, with $D=1.7$ , the $a_i$ parameters were generated from $LN(0,{0.5}^2)$ , the log-normal distribution with log-mean=0 and log-SD = .5; the $b_i$ parameters were generated from $N(0,1)$ under the restriction that $-3\le b_i\le 3$ ; and the $c_i$ parameters were generated from a uniform distribution ranging from .05 to .35. For each test form, the variances of ${\gamma }_s$ (that is, $C_{s\left(i\right)}^2$ ) were set to three levels, .1 (small testlet effect), .5 (medium testlet effect), and 1 (large testlet effect), and they were assigned to three testlet pairs that were randomly matched. Note that 10 or 20 common items had the same parameters between the base and new forms to be linked.

Because the linking coefficients $A$ and $B$ reflect, respectively, the differences in the SD and mean of the primary factor ${\theta }_0$ between the base and new populations, and the ${\beta }_s$ coefficients reflect differences in the mean of the testlet effect factors ${\theta }_s$ , the generation of linking parameters began by fixing the distributions of all factors for the base population to $N(0,1)$ . Then the slope coefficients $A$ (the SDs of ${\theta }_0$ for the new population) were generated from LN (0, 0.2²), and the intercept coefficients $B$ (the means of ${\theta }_0$ for the new population) were generated from N (0, 0.3²). Ten combinations of $A$ and $B$ values were generated, and they were applied to both Conditions 1 and 2. Note that for the first combination, the values of $A$ and $B$ were set at 1 and 0, respectively, so that it could serve as the baseline combination. For each simulation condition, the ${\beta }_s$ coefficients (the means of ${\theta }_s$ for the new population) were generated from N (0, 0.3²), subject to the constraint $\sum _{s=1}^k{\beta }_s=0$ , where the first $k-1$ beta coefficients were randomly sampled from the distribution and the last beta coefficient was set as ${\beta }_k=-{\displaystyle \sum _{s=1}^{k=1}{\beta }_s}$ . Associated with the first combination of $A$ =1 and $B$ =0, all beta coefficients were set at 0. The true linking parameters, $A$ , $B$ , and ${\beta }_s$ , used to generate 20 data sets (10 data sets per condition) are presented in Table 1.

Table 1.

True Linking Coefficients for Simulation Conditions 1 and 2.

		Condition 1			Condition 2
$A$	$B$	Data Set	${\beta }_1$	${\beta }_2$	Data Set	${\beta }_1$	${\beta }_2$	${\beta }_3$	${\beta }_4$
1.000	.000	1	.000	.000	11	.000	.000	.000	.000
.850	−.244	2	−.052	.052	12	−.202	.006	−.541	.737
1.250	.370	3	−.362	.362	13	−.164	.507	.063	−.406
1.013	.051	4	.224	−.224	14	−.141	.512	−.112	−.259
.900	−.360	5	.145	−.145	15	−.024	.238	−.143	−.071
1.093	.202	6	−.017	.017	16	−.138	−.008	.658	−.512
1.234	−.180	7	.132	−.132	17	.142	−.172	−.021	.051
.986	.026	8	−.321	.321	18	.087	−.188	.053	.048
.961	.150	9	.239	−.239	19	−.475	−.247	.155	.567
.857	−.108	10	.020	−.020	20	−.082	.013	.364	−.295

Estimation and Evaluation

For each data set, the item and testlet parameters for the 3PL testlet model were estimated using the computer program flexMIRT (Cai, 2017). By default, flexMIRT uses 0–1 scaling for each factor to estimate item parameters, and we applied that scaling approach to the separate calibrations of base and new sample data. With the separate calibration results, the linking parameters were estimated using the statistical programming language R (R Development Core Team, 2018). Specifically, the linking solutions for the MLS method were computed using the built-in linear algebra functions. The solutions for the GENOUD-TRF method were found using the “genoud” function included in the R package genoud (Mebane & Sekhon, 2011). The solutions for the BFGS-TRF and IRF methods were found using the “optim” function included in the R package stats. For the TRF and IRF methods, 41 ${\theta }_B$ points, equally spaced from −4 to 4, were used to define their criterion functions (see Equations (15) and (22)).

For each data set in each condition, differences between the estimated and true linking parameters (i.e., estimation errors) were computed to evaluate the performance of each linking method. In addition, the means of the absolute differences across the 10 data sets in each condition were computed to summarize the estimation errors for each of the linking parameters.

Results

Results of Condition 1

The linking parameter recovery results of the GENOUD- and BFGS-TRF methods for Condition 1 (data sets 1–10), in which two common testlets were used, are presented in Table 2. The two TRF methods performed nearly equally in estimating the true linking parameters. For the $A$ , $B$ , and ${\beta }_s$ coefficients, in most cases, the estimates produced by the GENOUD-TRF method were equal to those by the BFGS-TRF method up to three decimal places. These results suggest that the use of a genetic algorithm offers little improvement to the TRF method. The recovery of the linking parameters differed by the type of linking coefficients. For most data sets, the estimation errors for $\widehat{A}$ and $\widehat{B}$ were close to zero, and the mean absolute errors of $\widehat{A}$ and $\widehat{B}$ were .021 and .029, respectively, indicating that the two TRF methods perform well in estimating the linking coefficients for the primary factor ${\theta }_0$ . By contrast, the estimation errors for ${\widehat{\beta }}_1$ and ${\widehat{\beta }}_2$ were greater (by more than .4 for data sets 3 and 9), and their mean absolute errors were .182 and .182, respectively (the zero-sum constraint causes the two values to be the same). It is noteworthy that for the first baseline data set, the estimation errors for the beta coefficients (−.174 and .174) are much larger than the error for the $B$ coefficient (.013). This finding suggests that the TRF method can be poor at estimating the mean differences in testlet factors between the examinee groups being analyzed for linking.

Table 2.

Estimation Errors of the Linking Parameters from the Two TRF Methods for Condition 1.

Data Set	$\widehat{A}-A$	$\widehat{B}-B$	${\widehat{\beta }}_1-{\beta }_1$	${\widehat{\beta }}_2-{\beta }_2$
GENOUD-TRF method
1	−.019	.013	−.174	.174
2	.007	−.016	.053	−.053
3	−.018	.056	.445	−.445
4	−.045	.042	−.162	.162
5	−.007	−.023	−.069	.069
6	−.038	−.001	.063	−.063
7	.029	−.054	.154	−.154
8	−.020	−.056	.121	−.121
9	−.030	.025	−.530	.530
10	.002	.006	.045	−.045
Mean absolute error	.021	.029	.182	.182
BFGS-TRF method
1	−.019	.013	−.174	.174
2	.007	−.016	.053	−.053
3	−.017	.056	.444	−.444
4	−.045	.042	−.163	.163
5	−.007	−.023	−.069	.069
6	−.038	−.001	.063	−.063
7	.029	−.054	.154	−.154
8	−.020	−.056	.121	−.121
9	−.030	.025	−.530	.530
10	.002	.006	.045	−.045
Mean absolute error	.021	.029	.182	.182

The recovery results from the MLS and IRF methods for Condition 1 (data sets 1–10) are presented in Table 3. For both methods, the estimation errors of $\widehat{A}$ and $\widehat{B}$ were close to zero in most cases, as was found with the two TRF methods. The mean absolute errors for $\widehat{A}$ and $\widehat{B}$ with the MLS method were .035 and .037, respectively, and those with the IRF method were .021 and .033. For the recovery of the beta coefficients, the estimation errors for ${\widehat{\beta }}_1$ and ${\widehat{\beta }}_2$ with the MLS and IRF methods were closer to zero than those with the two TRF methods. The mean absolute error for either beta coefficient with the MLS method was .068, and that with the IRF method was .036. This finding suggests that the IRF, MLS, and TRF methods perform best, second best, and worst, respectively, in estimating the intercept linking coefficients ( ${\beta }_s$ ).

Table 3.

Estimation Errors of the Linking Parameters from the MLS and IRF Methods for Condition 1.

Data Set	$\widehat{A}-A$	$\widehat{B}-B$	${\widehat{\beta }}_1-{\beta }_1$	${\widehat{\beta }}_2-{\beta }_2$
MLS method
1	.015	−.019	.013	−.013
2	.108	−.025	−.155	.155
3	.013	−.150	−.182	.182
4	.025	.021	.002	−.002
5	−.032	−.006	.084	−.084
6	.056	−.067	−.190	.190
7	.017	−.020	.025	−.025
8	−.030	−.039	−.018	.018
9	−.044	.016	.006	−.006
10	.009	.006	.007	−.007
Mean absolute error	.035	.037	.068	.068
IRF method
1	.007	.003	−.005	.005
2	−.003	−.013	.026	−.026
3	.027	−.075	−.091	.091
4	−.033	.043	.012	−.012
5	−.036	−.050	.102	−.102
6	−.032	.013	.037	−.037
7	.025	−.036	.007	−.007
8	−.010	−.069	−.024	.024
9	.019	−.031	.046	−.046
10	.014	−.001	.013	−.013
Mean absolute error	.021	.033	.036	.036

Results of Condition 2

The recovery results of the GENOUD- and BFGS-TRF methods in Condition 2 (data sets 11–20) are presented in Table 4, and the results of the MLS and IRF methods are presented in Table 5. As was found in the results in Condition 1, all methods produced estimation errors for $\widehat{A}$ and $\widehat{B}$ that were close to zero in most cases. The mean absolute errors for $\widehat{A}$ and $\widehat{B}$ were .022 and .046, respectively, with the GENOUD-TRF method, .023 and .045 with the BFGS-TRF method, .026 and .053 with the MLS method, and .014 and .036 with the IRF method.

Table 4.

Estimation Errors of the Linking Parameters from the Two TRF Methods for Condition 2.

Data Set	$\widehat{A}-A$	$\widehat{B}-B$	${\widehat{\beta }}_1-{\beta }_1$	${\widehat{\beta }}_2-{\beta }_2$	${\widehat{\beta }}_3-{\beta }_3$	${\widehat{\beta }}_4-{\beta }_4$
GENOUD-TRF method
11	−.067	.006	−.504	.063	−.189	.630
12	−.013	−.088	−.317	−.621	.563	.376
13	−.055	−.021	−.062	−.072	−.189	.324
14	.013	−.055	.180	−.029	−.245	.094
15	−.009	.061	−.304	−.110	.091	.323
16	−.027	.040	−.335	−.196	.361	.170
17	−.001	−.013	.067	.054	−.194	.074
18	.001	.044	−.022	.119	−.298	.201
19	−.017	.115	−.193	−.097	−.063	.352
20	−.018	.020	.102	−.202	−.240	.340
Mean absolute error	.022	.046	.209	.156	.243	.288
BFGS-TRF method
11	−.067	.006	−.500	.057	−.189	.632
12	−.010	−.085	−.310	−.578	.555	.332
13	−.055	−.024	−.088	−.059	−.182	.330
14	.014	−.048	.066	−.038	−.160	.132
15	−.026	.060	.053	−.114	−.101	.162
16	−.028	.040	−.317	−.158	.296	.179
17	.001	−.012	.054	.047	−.169	.068
18	.001	.044	−.027	.119	−.295	.202
19	−.017	.115	−.193	−.097	−.063	.353
20	−.014	.019	.114	−.176	−.244	.306
Mean absolute error	.023	.045	.172	.144	.225	.270

Table 5.

Estimation Errors of the Linking Parameters from the MLS and IRF Methods for Condition 2.

Data Set	$\widehat{A}-A$	$\widehat{B}-B$	${\widehat{\beta }}_1-{\beta }_1$	${\widehat{\beta }}_2-{\beta }_2$	${\widehat{\beta }}_3-{\beta }_3$	${\widehat{\beta }}_4-{\beta }_4$
MLS method
11	.082	−.093	−.014	.078	−.010	−.054
12	.032	−.036	−.012	.170	−.167	.010
13	−.029	−.063	−.300	.126	.120	.054
14	.013	−.052	.005	.094	−.034	−.064
15	−.003	.039	.036	.267	−.085	−.218
16	−.038	.071	.008	−.226	.256	−.038
17	.003	−.004	.012	−.068	.012	.043
18	.005	−.021	.096	.063	−.017	−.141
19	−.020	.139	−.100	−.220	−.078	.398
20	−.035	−.012	−.142	.029	.209	−.097
Mean absolute error	.026	.053	.073	.134	.099	.112
IRF method
11	.047	−.006	.003	−.009	.044	−.037
12	.007	−.051	.086	−.036	−.081	.031
13	−.016	.005	−.038	−.001	.010	.029
14	−.004	−.042	.004	.066	−.018	−.053
15	.005	.063	−.066	.167	−.041	−.060
16	−.013	.053	−.124	−.167	.345	−.054
17	.006	.001	−.025	−.041	−.009	.074
18	.014	.019	.011	−.034	−.027	.050
19	−.011	.113	−.076	−.249	−.028	.353
20	.017	−.006	−.103	.041	.137	−.075
Mean absolute error	.014	.036	.053	.081	.074	.082

For the recovery of the beta coefficients, the two TRF methods produced estimation errors for ${\widehat{\beta }}_1$ to ${\widehat{\beta }}_4$ that deviated from zero by more than .3 in many cases. The mean absolute errors for ${\widehat{\beta }}_1$ to ${\widehat{\beta }}_4$ were .209, .156, .243, and .288, respectively, with the GENOUD-TRF method and .172, .144, .225, and .270 with the BFGS-TRF method. Thus, using a genetic algorithm for scale linking can lead to worse solutions for the beta coefficients than using a quasi-Newton algorithm. In contrast, the estimation errors for ${\widehat{\beta }}_1$ to ${\widehat{\beta }}_4$ with the MLS and IRF methods were closer to zero than those with the two TRF methods. The mean absolute errors of ${\widehat{\beta }}_1$ to ${\widehat{\beta }}_4$ were .073, .134, .099, and .112, respectively, with the MLS method and .053, .081, .074, and .082 with the IRF method. It is noteworthy that with the baseline data set 11, the two TRF methods resulted in much larger estimation errors for the beta coefficients than for the $B$ coefficient. In sum, the IRF, MLS, and TRF methods performed best, second best, and worst, respectively, in estimating the ${\beta }_s$ coefficients.

Discussion and Conclusions

Li et al. (2005) proposed a TRF linking method for the 2PNO testlet model and used the GENOUD genetic algorithm to find minimization solutions for the linking parameters by updating a population of solutions from generation to generation. In the present paper, we used the 3PL testlet model for generality to formulate the linking task from the perspective of bi-factor modeling and presented two alternatives (MLS and IRF) to the TRF linking method for the model. One of the purposes of the simulation study was to examine whether there is a compelling reason to use the genetic algorithm instead of the BFGS algorithm, which is one of the quasi-Newton methods widely applied, when using the TRF method to find linking solutions. The other purpose was to investigate the performance of the TRF method (based on either the GENOUD or BFGS algorithm) against the other linking methods, MLS and IRF.

The following main results were found from the stimulation study. For the simulated linking data sets using two common testlets (Condition 1), the performance of the GENOUD-TRF method was nearly the same as that of the BFGS-TRF method in recovering the true linking parameters, $A$ (slope) and $B$ (intercept) for the primary dimension factor and ${\beta }_s$ (intercepts) for the testlet factors, subject to the zero-sum constraint ${\displaystyle \sum {\beta }_s}=0$ . In Condition 2 involving four common testlets, the GENOUD-TRF method performed nearly as well as the BFGS-TRF method in estimating the $A$ and $B$ coefficients, but it tended to estimate the ${\beta }_s$ coefficients less accurately than the BFGS-TRF method. This finding suggests that using a genetic algorithm does not lead to better solutions for the linking coefficients, particularly the beta coefficients, than using a quasi-Newton algorithm. In both simulation conditions, there was a small difference in linking accuracy among the linking methods for the $A$ and $B$ coefficients, whereas for the ${\beta }_s$ coefficients, the methods differed substantially. In recovering the true ${\beta }_s$ coefficients, on average, the IRF method showed the least estimation error, and the TRF methods produced the largest errors, more than double the average error of the IRF method. Taken together, these results suggest that the IRF, MLS, and TRF methods perform best, second best, and worst, respectively, in estimating the linking parameters associated with testlet effects.

The poor performance of the TRF method against the IRF method in estimating the ${\beta }_s$ coefficients may be regarded as a bit unusual, but is not a new finding, as shown by Kim (2019). To understand why the TRF method estimated the beta coefficients more poorly than the IRF method, we examined the contour plots (i.e., level curves) of the negative criterion functions, $-f_T\left(A,\mathbf{\beta }\right)$ and $-f_I\left(A,\mathbf{\beta }\right)$ , for the two methods. With ${\beta }_k$ = $-{\displaystyle \sum _{s=1}^{k-1}{\beta }_s}$ and $A$ and/or $B$ fixed to their true values, we drew the contour plots with the axes of $B$ and ${\beta }_1$ for the data sets in Condition 1 and the axes of each of the three pairs ( ${\beta }_1$ and ${\beta }_2$ , ${\beta }_1$ and ${\beta }_3$ , and ${\beta }_2$ and ${\beta }_3$ ) for the data sets in Condition 2. As illustrated in Figure 1 (where the contour plots for data sets 1 and 11 are presented as examples), for all data sets in Condition 1, the contour plots of $-f_T\left(A,\mathbf{\beta }\right)$ had top level curves shaped like elongated rings, narrow along the $B$ -axis but wide along the ${\beta }_1$ -axis, whereas those of $-f_I\left(A,\mathbf{\beta }\right)$ had top level curves shaped like small ellipses, narrow along both axes. Compared with the top-level curves for the IRF method, the shape of the top-level curves for the TRF method indicates that the $B$ coefficient can be more accurately estimated than the ${\beta }_1$ coefficient. For all data sets in Condition 2, the contour plots of $-f_T\left(A,\mathbf{\beta }\right)$ had top level curves shaped like distorted ellipses, big and wide, whereas those of $-f_I\left(A,\mathbf{\beta }\right)$ had top level curves shaped like small ellipses, tilted diagonally.

Figure 1.

Contour Plots of $-f_T\left(A,\mathbf{\beta }\right)$ and $-f_I\left(A,\mathbf{\beta }\right)$ with Data Set 1 in Condition 1 and Data Set 11 in Condition 2.

The difference in shape between the top-level curves suggests that the TRF method produces less stable estimates of the beta coefficients than the IRF method and that the GENOUD and BFGS algorithms can converge to different neighborhoods and reach a global minimum. In other words, the criterion function of the IRF method is well-structured for the linking solutions but that of the TRF method is not. The criterion function of the IRF method is based on the sum of the squared differences in category response functions at the item level, and so the possible location differences between separate calibrations for each testlet are preserved and not mixed with those for other testlets. But the criterion function of the TRF method is based on the squared differences in true test scores, so that the possible location differences for each testlet are likely confounded at the test level. Such confounding likely leads to unstable estimation of the beta coefficients.

From the simulation results, we find no compelling reason to use genetic algorithms instead of quasi-Newton algorithms to find minimization solutions for the TRF method. Furthermore, in a numerical sense, we find that the criterion function of the IRF method produces better-structured solutions than that of the TRF method. From a practical point of view, the BFGS algorithm should be preferred to the GENOUD algorithm because the latter takes much more time than the former to find the minimization solutions. In this simulation study, the GENOUD-TRF method often took more than 5 and 10 minutes for the data sets in conditions 1 and 2, respectively, whereas the BFGS-TRF and IRF methods took less than 5 and 15 seconds for the corresponding data sets.

As pointed out by Li et al. (2005), the zero-sum constraint for the beta coefficients indicates that the average (across testlets) of testlet factor means should be the same between the examinee groups being analyzed for scale linking. Although the zero-sum constraint seems to be reasonable and flexible, the linear dependence among the beta coefficients can be solved in other ways. One feasible approach is choosing a common testlet whose factor mean is expected to change little across examinee groups and fixing the beta coefficient for the common testlet to zero. Then the rest of the beta coefficients are considered as the free parameters. An analog of this approach is found in differential item functioning (DIF) analyses. If we use the constraint that the overall mean difference across examinee groups in item difficulties is zero, any performance differences are absorbed into differences in ability and treated as impact. If we suspect that the DIF may not cancel out across items, we can designate a set of anchor items to have zero DIF. Of course, to apply that “fixed-to-zero” constraint to linking tasks, the transformation matrix in Equation (13) should be modified appropriately for the MLS method, and the partial derivatives of the criterion functions with respect to the linking parameters should be properly computed for the IRF and TRF methods.

Some studies need to be conducted to improve understanding of the three linking methods presented in this paper and enable a wise choice among them in practice. First, the simulation study in this paper did not address the effects of linking errors in transformed item parameter estimates on the estimation of the ability ( ${\theta }_0$ ) parameters for the new group examinees. The accuracy of ability estimation would be affected most by the estimates of the $b_{iN}^{\ast }$ or $d_{iN}^{\ast }$ parameters, which are functions of the beta coefficients, given $A$ , $B$ , and $C_{sB}$ (see Equation (21)). It should be examined whether the ability parameters can be estimated better when the linking coefficients from the IRF method are used than when those from the MLS or TRF method are used. Second, although the separate calibration and linking approach is a basic and dependable method for developing a common IRT scale, a common scale can also be developed by using multiple-group concurrent calibration (Bock & Zimowski, 1997) or fixed parameter calibration (Kim, 2006). A comparative study on the performance of the three calibration types will offer practitioners in the areas of test equating and vertical scaling useful information about the advantages and disadvantages of the various linking methods. Third, the three linking methods presented for the 3PL testlet model need to be extended to polytomous testlet models such as the graded response testlet model and the generalized partial credit testlet model. It would be meaningful to investigate the performance of the linking methods using a variety of real and simulated data. Finally, it would be very useful to derive analytic formulas for the standard errors of linking coefficient estimates obtained from the different linking methods because those standard errors of estimates can be used as indices of linking precision in practice.

Appendix: Partial Derivatives.

Based on $c_{iN}^{\ast }=c_{iN}$ and Equations (18) and (21), let us write $P_i^{\ast }\left({\xi }_{sB}\right)$ and $P_i^{\ast }\left({\theta }_B\right)$ as

$$P_i^{\ast }\left({\xi }_{sB}\right)=c_{iN}^{\ast }+\left(1-c_{iN}^{\ast }\right)/\left\{1+\exp \left[-D{a}_{iN}^{\ast }\left({\xi }_{sB}-b_{iN}^{\ast }\right)\right]\right\},$$ (A1)

$$P_i^{\ast }\left({\theta }_B\right)={\displaystyle \int P_i^{\ast }\left({\xi }_{sB}\right)h\left({\xi }_{sB}|{\theta }_B;C_{sB}\right)d{\xi }_{sB}}.$$ (A2)

The partial derivatives of $P_i^{\ast }\left({\theta }_B\right)$ with respect to $A$ , $B$ , and ${\beta }_s$ (where $i\in s$ ) are computed as

$$\frac{\partial P_i^{\ast }\left({\theta }_B\right)}{\partial A}={\displaystyle \int -D{a}_{iN}^{\ast }\left(\frac{{\xi }_{sB}-C_{sB}{\beta }_s-B}{A}\right)\left[\frac{P_i^{\ast }\left({\xi }_{sB}\right)-c_{iN}^{\ast }}{1-c_{iN}^{\ast }}\right]\left[1-P_i^{\ast }\left({\xi }_{sB}\right)\right]h\left({\xi }_{sB}|{\theta }_B;C_{sB}\right)d{\xi }_{sB}},$$ (A3)

$$\frac{\partial P_i^{\ast }\left({\theta }_B\right)}{\partial B}={\displaystyle \int -D{a}_{iN}^{\ast }\left[\frac{P_i^{\ast }\left({\xi }_{sB}\right)-c_{iN}^{\ast }}{1-c_{iN}^{\ast }}\right]\left[1-P_i^{\ast }\left({\xi }_{sB}\right)\right]h\left({\xi }_{sB}|{\theta }_B;C_{sB}\right)d{\xi }_{sB}},$$ (A4)

$$\frac{\partial P_i^{\ast }\left({\theta }_B\right)}{\partial {\beta }_s}={\displaystyle \int -D{a}_{iN}^{\ast }{C}_{sB}\left[\frac{P_i^{\ast }\left({\xi }_{sB}\right)-c_{iN}^{\ast }}{1-c_{iN}^{\ast }}\right]\left[1-P_i^{\ast }\left({\xi }_{sB}\right)\right]h\left({\xi }_{sB}|{\theta }_B;C_{sB}\right)d{\xi }_{sB}}.$$ (A5)

Then, the partial derivatives of $f_I(A,\mathbf{\beta })$ with respect to $A$ and $B$ are given by

$$\frac{\partial f_I}{\partial A}=-\frac{2}{Nn}{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{i=1}^n\left[P_i\left({\theta }_{qB}\right)-P_i^{\ast }\left({\theta }_{qB}\right)\right]\frac{\partial P_i^{\ast }\left({\theta }_{qB}\right)}{\partial A}}},$$ (A6)

$$\frac{\partial f_I}{\partial B}=-\frac{2}{Nn}{\displaystyle \sum _{q=1}^N{\displaystyle \sum _{i=1}^n\left[P_i\left({\theta }_{qB}\right)-P_i^{\ast }\left({\theta }_{qB}\right)\right]\frac{\partial P_i^{\ast }\left({\theta }_{qB}\right)}{\partial B}}}.$$ (A7)

When ${\beta }_k=-{\displaystyle {\sum }_{s=1}^{k-1}{\beta }_s}$ , the partial derivatives of $f_I(A,\mathbf{\beta })$ with respect to ${\beta }_s$ ( $s=1,\mathrm{...},k-1$ ) are given by

$$\frac{\partial f_I}{\partial {\beta }_s}=-\frac{2}{Nn}{\displaystyle \sum _{q=1}^N\left\{{\displaystyle \sum _{i\in s}\left[P_i\left({\theta }_{qB}\right)-P_i^{\ast }\left({\theta }_{qB}\right)\right]\frac{\partial P_i^{\ast }\left({\theta }_{qB}\right)}{\partial {\beta }_s}}-{\displaystyle \sum _{i\in k}\left[P_i\left({\theta }_{qB}\right)-P_i^{\ast }\left({\theta }_{qB}\right)\right]\frac{\partial P_i^{\ast }\left({\theta }_{qB}\right)}{\partial {\beta }_k}}\right\}}.$$ (A8)

With Equations (A1) to A5, the partial derivatives of $f_T(A,\mathbf{\beta })$ with respect to $A$ and $B$ are given by

$$\frac{\partial f_T}{\partial A}=-\frac{2}{N}{\displaystyle \sum _{q=1}^N\left[{\displaystyle \sum _{i=1}^n{P}_i\left({\theta }_{qB}\right)}-{\displaystyle \sum _{i=1}^n{P}_i^{\ast }\left({\theta }_{qB}\right)}\right]{\displaystyle \sum _{i=1}^n\frac{\partial P_i^{\ast }\left({\theta }_{qB}\right)}{\partial A}}},$$ (A9)

$$\frac{\partial f_T}{\partial B}=-\frac{2}{N}{\displaystyle \sum _{q=1}^N\left[{\displaystyle \sum _{i=1}^n{P}_i\left({\theta }_{qB}\right)}-{\displaystyle \sum _{i=1}^n{P}_i^{\ast }\left({\theta }_{qB}\right)}\right]{\displaystyle \sum _{i=1}^n\frac{\partial P_i^{\ast }\left({\theta }_{qB}\right)}{\partial B}}}.$$ (A10)

And if ${\beta }_k=-{\displaystyle \sum _{s=1}^{k-1}{\beta }_s}$ , the partial derivatives of $f_T\left(A,\mathbf{\beta }\right)$ with respect to ${\beta }_s$ ( $s=1,\mathrm{...},k-1$ ) are given by

$$\frac{\partial f_T}{\partial {\beta }_s}=-\frac{2}{N}{\displaystyle \sum _{q=1}^N\left[{\displaystyle \sum _{i=1}^n{P}_i\left({\theta }_{qB}\right)}-{\displaystyle \sum _{i=1}^n{P}_i^{\ast }\left({\theta }_{qB}\right)}\right]\left\{{\displaystyle \sum _{i\in s}\frac{\partial P_i^{\ast }\left({\theta }_{qB}\right)}{\partial {\beta }_s}}-{\displaystyle \sum _{i\in k}\frac{\partial P_i^{\ast }\left({\theta }_{qB}\right)}{\partial {\beta }_k}}\right\}}.$$ (A11)

ORCID iD

Seonghoon Kim https://orcid.org/0000-0002-0357-8639